Digital flip synthesis device and program of the same

Abstract

To provide a digital flip synthesis device capable of switching and synthesizing digital materials on a flip-board shown in a camera image.SOLUTION: A digital flip synthesis device 1 comprises: a material adjustment unit 110 for setting, for a plurality of digital materials, relative positions and sizes of the digital materials with respect to an AR marker on marker coordinates, and converting the digital materials into digital materials on the marker coordinates; a marker detection unit 100 for detecting an AR marker on camera coordinates from a camera image; a matrix calculation unit 101 for calculating a projection conversion matrix that converts the AR marker on the marker coordinates into the AR marker on the camera coordinates; a material switching unit 112 for switching and selecting the digital materials; and an image synthesis unit 12 for converting the digital materials selected by the material switching unit 112 with the projection conversion matrix, and synthesizing thereof into the camera image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a digital flip synthesis device and its program.

Background Art

[0002] In recent years, AR (Augmented Reality) technology (see Patent Documents 1, 2, etc.), which recognizes a specific pattern from camera images and synthesizes CG (Computer Graphics) in real time at a position corresponding to the pattern, has been widely used. This pattern is called an AR marker. For example, a method of shooting an AR marker with a smartphone incorporating a specific application and displaying a still image, a moving image, a 3D character, etc. corresponding to the position and inclination of the AR marker on the screen is being utilized at events, tourist spots, etc.

[0003] On the other hand, in broadcast information programs and the like, an effect of explaining information content using a flip board on which characters, figures, etc. are described is widely performed. This flip board is usually made of a plate-shaped cardboard or the like and is created according to the information content.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Even in the flip board used in broadcast programs and the like, an effect of synthesizing digital materials such as still images, moving images, and 3D characters on the flip board using AR technology has been desired. However, in the conventional AR technology, since one digital material corresponds to the position of one AR marker, there is a problem that it is difficult to present a plurality of contents by changing them, like a flip board for an information program or the like. For example, when performing an effect of presenting information content by sticking dedicated sticky papers for flipping at multiple locations on a flip board and appropriately flipping them, a plurality of AR markers must be prepared on the flip board, and a great deal of labor is required for its creation.

[0006] Therefore, an object of the present invention is to provide a digital flip synthesis device and a program thereof that can switch and synthesize digital materials on a flip board reflected in a camera image by simply preparing one AR marker on the flip board.

Means for Solving the Problems

[0007] To solve the above problems, a digital flip synthesis device according to the present invention is a digital flip synthesis device that generates a composite image in which a digital material is synthesized with an AR marker on a flip board reflected in a camera image, and includes a material adjustment unit, a marker detection unit, a matrix calculation unit, a material switching unit, and an image synthesis unit.

[0008] In such a configuration, the digital flip synthesis device sets, by the material adjustment unit, the relative position and size of the digital material with respect to the AR marker on the predetermined marker coordinates for a plurality of digital materials, converts the digital materials into digital materials on the marker coordinates, and stores them in the storage unit. Then, the digital flip synthesis device detects an AR marker on the camera coordinates from the camera image by the marker detection unit. Furthermore, the digital flip synthesis device calculates a projective transformation matrix (homography matrix) that converts the AR marker on the marker coordinates into the AR marker on the camera coordinates by the matrix calculation unit.

[0009] Then, the digital flip synthesis device switches and selects the digital materials stored in the storage unit according to the operator's instructions by the material switching unit. Then, the digital flip synthesis device converts the digital materials selected by the material switching unit with the projection transformation matrix calculated by the matrix calculation unit by the video synthesis unit, and synthesizes them with the camera video.

[0010] As a result, the digital materials selected by the operator are synthesized on the AR marker on the flip board shown in the camera video. In addition, since this digital material can be switched by the material switching unit, even with one AR marker, different digital materials can be switched and synthesized. Note that the digital flip synthesis device can be operated by a program for causing a computer to function as each of the above-described units.

Effect of the Invention

[0011] According to the present invention, by simply preparing one AR marker on the flip board, digital materials can be switched and synthesized on the flip board shown in the camera video.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Outline of Processing of Digital Flip Synthesis Device] The digital flip synthesis device 1 (Figure 3) according to an embodiment of the present invention generates a synthesized video in which digital materials are synthesized onto an AR marker on a flip board shown in a camera video. First, with reference to FIGS. 1 and 2, the processing outline of the digital flip synthesis device 1 will be described.

[0014] The camera video I in FIG. 1(a) IN shows the video input to the digital flip synthesis device 1 taken by a camera (not shown). The camera video I IN is the video in which the flip board FB is shown. On the flip board FB, an AR marker MK is drawn, pasted, etc. The AR marker MK may be any pattern as long as it can detect at least four points. Here, the AR marker MK is a pattern in which four horizontal × four vertical black dots d are regularly arranged at equal intervals with white as the background on the flip board FB. The dot d is a black perfect circle having a certain area. For example, each dot d has a diameter of 30 mm and a center distance of 64 mm. Note that the white of the background of the AR marker MK and the black of the dots may be light gray, dark gray, etc. within a range distinguishable by binarization processing.

[0015] The synthesized video I in FIG. 1(b) OUT shows the video generated by the digital flip synthesis device 1. The synthesized video I OUT is a video in which one digital material D such as a still image, a moving image, a 3D character, etc. is synthesized at a position corresponding to the flip board FB of the camera video I IN .

[0016] As shown in FIG. 2(a), the digital flip synthesis device 1 associates the AR marker MK on the flip board FB and the digital material D in the same coordinate system (marker coordinate system xy) in advance. Then, as shown in FIG. 2(b), the digital flip synthesis device 1 detects the AR marker MK in the camera video I IN . Then, the digital flip synthesis device 1 determines the coordinates of the center of gravity g of each dot of the AR marker MK preset in the marker coordinate system xy of FIG. 2(a) in the camera video I respectively.IN Calculate a homography matrix (projection matrix) that converts the coordinates of the center of gravity of each dot of the AR marker MK into the coordinates of the camera coordinate system (camera coordinate system XY). Then, as shown in Fig. 2(c), the digital flip composite device 1 projects and transforms the digital material D on the marker coordinate system xy by the homography matrix to obtain the camera image I on the camera coordinate system XY IN and superimposes and synthesizes it on the composite image I OUT to generate it.

[0017] Also, the digital flip composite device 1 switches and synthesizes the digital material D according to the operator's instruction for one type of AR marker MK reflected in the camera image I IN Hereinafter, the configuration and operation of the digital flip composite device 1 will be described in detail. Hereinafter, the configuration and operation of the digital flip composite device 1 will be described in detail.

[0018] [Configuration of Digital Flip Composite Device] With reference to Fig. 3, the configuration of the digital flip composite device 1 will be described. The digital flip composite device 1 inputs a camera image and a digital material, generates and outputs a composite image in which the digital material is synthesized on the flip board reflected in the camera image. The digital flip composite device 1 includes a conversion matrix generation unit 10, a material operation unit 11, an image synthesis unit 12, and a storage unit 13. The digital flip composite device 1 is externally connected to a display device 2.

[0019] The conversion matrix generation unit 10 generates a conversion matrix for projecting and transforming the digital material onto the flip board of the camera image using the AR marker on the flip board reflected in the camera image. Here, the conversion matrix generation unit 10 includes a marker detection unit 100 and a matrix calculation unit 101.

[0020] The marker detection unit 100 detects the AR marker on the camera coordinates for each frame from the camera image. Here, the marker detection unit 100 recognizes the AR marker for each frame and performs tracking. The marker detection unit 100 detects an AR marker and calculates, as its position information, the centroid coordinates of each dot of the AR marker in camera coordinates.

[0021] Here, with reference to FIG. 4, the marker recognition method of the marker detection unit 100 will be described. FIG. 4 is a diagram schematically showing the processing performed by the marker detection unit 100 on the camera image. FIG. 4(a) shows a certain frame of the camera image (hereinafter referred to as a camera image). As shown in FIG. 4(b), the marker detection unit 100 binarizes the camera image in FIG. 4(a) to generate a binarized image. At this time, the marker detection unit 100 converts the RGB values of the camera image into grayscale and performs binarization by threshold processing. Note that, in order to speed up the subsequent processing, the marker detection unit 100 may reduce the grayscale camera image, smooth it, and then perform binarization.

[0022] Then, as shown in FIG. 4(c), the marker detection unit 100 performs black-and-white inversion on the binarized image in FIG. 4(b). Then, as shown in FIG. 4(d), the marker detection unit 100 performs labeling as shown by round numbers on the white regions of the binarized image in FIG. 4(c). Finally, as shown in FIG. 4(e), the marker detection unit 100 detects, as the AR marker MK, the locations where the labeled white regions in FIG. 4(d) are regularly arranged, and calculates the coordinates of the centroid G of each dot.

[0023] Here, with reference to FIGS. 5 and 6, the processing in FIGS. 4(d) and 4(e) will be described in more detail. FIG. 5 shows a part of the binarized camera image and shows the labeled white regions in FIG. 4(d). Note that the round numbers labeled in FIG. 4(d) are attached for the purpose of explaining the concept and do not correspond to the numbers labeled in FIG. 5. The numbers in FIG. 5 indicate the actually labeled numbers (label Nos). As shown in FIG. 5, the marker detection unit 100 leaves only the regions within a predetermined range of size (horizontal pixel number, vertical pixel number) among the labeled white regions as candidates for the AR marker MK. Here, the regions with label No1 (number 1) and label No16 (number 16) are excluded because they are smaller than the predetermined size. Of course, although not shown in the figure, regions larger than the predetermined size are also excluded. In FIG. 5, the ○ marks indicate the centroids of the respective white regions. Then, the marker detection unit 100 recognizes the AR marker MK from the regions remaining based on the size criterion.

[0024] The marker detection unit 100 first excludes, from the candidates for each region that is a candidate for a dot of the AR marker MK, the regions where the angle formed by two vectors directed from the centroid of the region to the centroids of two neighboring regions searched by neighborhood search is 0° or more and less than 90°, or 120° or more. For the neighborhood search, a general method such as neighborhood search using a kd-tree may be used.

[0025] A specific example will be described with reference to FIG. 6(a). FIG. 6(a) is a view focusing only on the regions with label No2, No3, and No9 (regions B2, B3, B9) shown in FIG. 5. Here, let the regions searched by neighborhood search for region B2 be regions B3 and B9. When the angle θ formed by the vector directed from the centroid G2 of region B2, which is the determination target of the candidate for the dot of the AR marker MK, to the centroid G3 and the vector directed from the centroid G2 to the centroid G9 is 0° or more and less than 90°, or 120° or more, the marker detection unit 100 excludes region B2 from the candidates.

[0026] Then, for the regions not excluded from the candidates, the marker detection unit 100 synthesizes two vectors directed from the centroid of the region to the centroids of two neighboring regions. If the end point of the synthesized vector belongs to a predetermined range from the centroid of another region, the marker detection unit 100 sets the four points including the centroid of the region as the centroid of the dots of the AR marker MK.

[0027] A specific example will be described with reference to FIG. 6(b). FIG. 6(b) is a view focusing only on the regions (regions B3, B4, B10, B11) of label Nos. 3, 4, 10, and 11 shown in FIG. 5. Here, the regions obtained by performing a neighborhood search on region B3 are regions B4 and B10. Since the end point of the composite vector of the vector from the centroid G3 of region B3 to centroid G4 and the vector from centroid G3 to centroid G10 belongs to region B11, the marker detection unit 100 sets centroids G3, G4, and G10, including the centroid G11 of region B11, as part of the centroids of the dots of the AR marker MK. Thereby, the marker detection unit 100 can obtain the coordinates of the centroid G indicating the positions of the dots of the AR marker MK shown in FIG. 4(e).

[0028] In addition, in FIG. 4(b), when the camera image is reduced, the marker detection unit 100 enlarges the centroid coordinates according to the reduction ratio and converts them into the coordinates in the coordinate system of the original camera image. Returning to FIG. 3, the description of the configuration of the digital flip synthesis device 1 will be continued.

[0029] Also, the marker detection unit 100 performs tracking of the AR marker in the camera video using the homography matrix calculated in the immediately previous (one frame before) frame by the matrix calculation unit 101. Specifically, the marker detection unit 100 projects the coordinates of the centroid g of each dot of the AR marker MK preset in the marker coordinate system xy in FIG. 2(a) to the coordinates in the camera coordinate system in FIG. 2(b) by the homography matrix calculated in the immediately previous camera video by the matrix calculation unit 101. The coordinates after this projective transformation are highly likely to be the centroid coordinates of each dot of the AR marker MK in the immediately previous frame. Therefore, the marker detection unit 100 sets the centroid of the region closest to the centroid of each dot of the AR marker MK in the immediately previous frame among the white regions of the binarized image in FIG. 4(c) as the centroid of each dot of the AR marker MK in the new frame.

[0030] That is, as shown in FIG. 7, the marker detection unit 100 projects and transforms the center of gravity g of each dot of the AR marker MK preset in the marker coordinate system xy by the homography matrix calculated in the immediately preceding frame, and sets the center of gravity g OLD as such. Then, the marker detection unit 100 sets the center of gravity of the region closest to the center of gravity g OLD in the current frame as the center of gravity g of each dot of the AR marker MK in the current frame NEW as such.

[0031] In addition, when a part of the AR marker MK is out of the frame, that is, when there is no white area in the vicinity of the center of gravity g OLD or the distance to the white area is longer than a predetermined distance, the marker detection unit 100 estimates the dot position of the AR marker MK outside the frame from the arrangement positions of the other dots constituting the AR marker MK.

[0032] The marker detection unit 100 outputs the center-of-gravity coordinates of each dot constituting the detected AR marker (including the center-of-gravity coordinates of the tracked AR marker) to the matrix calculation unit 101. In addition, the marker detection unit 100 notifies the material operation unit 11 that the AR marker has been detected.

[0033] The matrix calculation unit 101 calculates a homography matrix (projection transformation matrix) for converting the AR marker on the marker coordinates into the AR marker on the camera coordinates. Here, the matrix calculation unit 101 uses the center-of-gravity coordinates of each dot of the AR marker detected (recognized and tracked) by the marker detection unit 100 in the camera coordinate system and the center-of-gravity coordinates of each dot of the preset AR marker in the marker coordinate system to calculate a homography matrix for projecting and transforming a plane from the marker coordinate system to the camera coordinate system. The matrix calculation unit 101 calculates a homography matrix for each of the four adjacent dots in the vicinity of the AR marker detected by the marker detection unit 100. Since the method of calculating the homography matrix using four points is common, the description is omitted here.

[0034] The matrix calculation unit 101 calculates the barycentric coordinates in the camera coordinate system from the barycentric coordinates in the marker coordinate system using a plurality of homography matrices calculated for each dot of the four adjacent neighboring points. Then, the matrix calculation unit 101 selects a homography matrix that minimizes the error between the calculated barycentric coordinates and the actual barycentric coordinates detected by the marker detection unit 100.

[0035] For example, let it be assumed that the barycenters g1 to g16 of each dot constituting the AR marker in the marker coordinate system xy in Fig. 8(a) correspond to the barycenters G1 to G16 of each dot constituting the AR marker in the camera coordinate system XY shown in Fig. 8(b). In this case, the matrix calculation unit 101 calculates a homography matrix for projective transformation of the four adjacent neighboring points of the barycenters g1 to g16 shown in Fig. 8(a), for example, g1, g2, g5, g6, to the points G1, G2, G5, G6 shown in Fig. 8(b). Here, in addition to g1, g2, g5, g6, there are a total of nine sets of four neighboring dots such as g2, g3, g6, g7. Therefore, the matrix calculation unit 101 calculates a homography matrix for each set.

[0036] Then, for each dot of the four adjacent neighboring points in the marker coordinate system xy, projective transformation is performed using each calculated homography matrix, and a homography matrix that minimizes the error from the coordinates of the four detected points in the corresponding camera coordinate system XY is selected. The matrix calculation unit 101 outputs the selected one homography matrix to the video composition unit 12.

[0037] The material operation unit 11 performs various operations for synthesizing digital materials such as still images, moving images, and 3D characters on a flip board. Here, the material operation unit 11 includes a material adjustment unit 110, a switching order setting unit 111, and a material switching unit 112.

[0038] The material adjustment unit 110 sets the relative position and size of the digital material with respect to the AR marker on the predetermined marker coordinates for a plurality of digital materials, and converts the digital material into a digital material on the marker coordinates. First, the material adjustment unit 110 sequentially captures digital materials from the outside and sets the arrangement positions (position, size, etc.) of the digital materials with respect to the AR markers on the flip board. The material adjustment unit 110 associates the position, size, etc. of the digital material with the marker coordinate system of the AR marker MK on the flip board FB described in FIG. 2(a).

[0039] Specifically, the material adjustment unit 110 displays the flip board FB on the screen of the display device 2 in a predetermined size, and arranges the digital material to be displayed on the flip board FB. Then, the material adjustment unit 110 sets the position, size, etc. of the digital material on the screen by an input device such as a mouse or a keyboard (not shown).

[0040] By inputting the position, rotation angle, magnification ratio, etc. in the marker coordinate system, the material adjustment unit 110 deforms the digital material at the specified position, rotation angle, and magnification ratio, and stores it in the storage unit 13. Also, the deformed digital material is displayed on the screen of the display device 2.

[0041] Here, with reference to FIG. 9, a specific example of a screen (material capture setting screen SC1) for the material adjustment unit 110 to capture and set the arrangement position of the digital material will be described. The material capture setting screen SC1 has at least a material arrangement area PA and a setting value input area AD.

[0042] The material arrangement area PA is an area for arranging the digital material D captured from the outside on the flip board FB. The material adjustment unit 110 sets the coordinate system inside the material arrangement area PA as the marker coordinate system, and displays the flip board FB with a predetermined size at a predetermined position. In this state, the center of gravity g of each dot of the AR marker MK on the flip board FB shown in FIG. 2(a) is specified in the marker coordinate system. The material adjustment unit 110 arranges the digital material D on the flip board FB within the material arrangement area PA.

[0043] The setting value input area AD is an area where the operator inputs the position, rotation angle, magnification ratio, etc. of the digital material D arranged in the material placement area PA. "Position" indicates the position coordinates in the xyz directions in the marker coordinate system of the material placement area PA. Note that z is the coordinate indicating the front or back with respect to the plane of the flip board FB. "Rotation" indicates the rotation angles in the xyz directions in the marker coordinate system. "Magnification" indicates the magnification ratios in the xyz directions in the marker coordinate system.

[0044] By inputting each value into the setting value input area AD using a mouse, keyboard, etc., the material adjustment unit 110 deforms the digital material D at the specified position, rotation angle, and magnification ratio and displays it in the material placement area PA. Thereby, the operator can visually confirm the state where the digital material D is arranged on the flip board FB.

[0045] After the operator completes the setting of the placement position, by pressing the GO button (decision button) BT1 with the mouse, the material adjustment unit 110 completes the setting of one digital material D and takes in the next digital material. Here, every time the setting of the placement position of one digital material D is completed, the material adjustment unit 110 displays the thumbnail image of the digital material D in the material list area DL, and stores the digital material D after setting (deformation) and the thumbnail image in the storage unit 13 in association with each other. Returning to FIG. 3, the description of the configuration of the digital flip composition device 1 will be continued.

[0046] The switching order setting unit 111 is for setting the order in which the operator switches between a plurality of digital materials. The switching order setting unit 111 selects digital materials to be synthesized into the camera image from the digital materials whose arrangement positions are set and stored in the storage unit 13. Here, the switching order setting unit 111 selects digital materials according to the operator's instructions and sets the order of synthesis into the camera image. Further, the switching order setting unit 111 can also select, as digital materials, materials having images that reflect external inputs in real time.

[0047] Specifically, the switching order setting unit 111 displays information (identification information) for identifying the digital material D stored in the storage unit 13 on the screen of the display device 2, and selects the digital material to be synthesized by having the identification information selected. Further, the switching order setting unit 111 specifies the order of synthesis into the camera image by arranging the selected identification information in order on the screen of the display device 2.

[0048] Here, with reference to FIGS. 10 and 11, a specific example of a screen (material selection screen SC2) for the switching order setting unit 111 to select digital materials and set the switching order will be described. The material selection screen SC2 has at least a material list area DL and a switching order arrangement area OD.

[0049] The material list area DL is an area for displaying identification information of selectable digital materials. Here, the switching order setting unit 111 arranges the thumbnail images of the digital materials stored in the storage unit 13 in the material list area DL with numbers as identification information. FIG. 11 shows an enlarged view of the thumbnail images arranged in numbers 1 to 4 in the material list area DL. In this way, by arranging the thumbnail images, the operator can visually confirm the digital materials.

[0050] The switching order arrangement area OD is an area for arranging the identification information of the digital material selected from the material list area DL in the order of synthesis on the flip board. Here, the switching order setting unit 111 selects identification information (thumbnail image) from the material list area DL by a mouse or the like, and is arranged between the first right arrow R1 and the second right arrow R2 indicating the switching order, thereby associating the digital material with its switching order. As a result, the operator can visually confirm the order of the digital materials to be synthesized on the flip board FB. After the operator determines the switching order, by pressing the GO button (decision button) BT2 with the mouse, the switching order setting unit 111 stores the switching order of the digital material D in the storage unit 13. Returning to FIG. 3, the description of the configuration of the digital flip synthesis apparatus 1 will be continued.

[0051] The material switching unit 112 is for the operator to switch and select a plurality of digital materials. The material switching unit 112 displays the identification information of the next digital material to be synthesized on the screen of the display device 2, and switches the digital material according to the operator's instruction. The material switching unit 112 manages the currently synthesized digital material in the order selected by the switching order setting unit 111 stored in the storage unit 13, and instructs the video synthesis unit 12 to synthesize the digital material into the camera video at the timing when the instruction to switch to the next digital material is received.

[0052] Note that at the timing when the marker detection unit 100 of the conversion matrix generation unit 10 detects the AR marker, the material switching unit 112 instructs the video synthesis unit 12 to synthesize the first digital material selected by the switching order setting unit 111.

[0053] Here, with reference to FIG. 12, a specific example of a screen (switching execution screen SC3) for the material switching unit 112 to switch digital materials will be described. The switching execution screen SC3 has a next material display area NX that displays at least the identification information of the digital material to be switched next. Here, the switching execution screen SC3 has a preview area PV, a current material display area NW, and a next material display area NX.

[0054] The preview area PV is an area for visualizing the video that the composite video will be broadcast as the broadcast video. The material switching unit 112 draws the current composite video in which the digital material is synthesized with the camera video in the video synthesizing unit 12 in the preview area PV.

[0055] The current material display area NW is an area for displaying the digital material currently synthesized with the camera video. The material switching unit 112 reads out the digital material currently synthesized from the storage unit 13 and draws it in the current material display area NW.

[0056] The next material display area NX is an area for displaying the digital material to be next synthesized with the camera video. The material switching unit 112 reads out the digital material to be next synthesized from the storage unit 13 and draws it in the next material display area NX. Here, when the material switching unit 112 presses the keyboard key "Enter" indicating the switching to the next digital material, it instructs the video synthesizing unit 12 to switch the digital material. Note that the material switching unit 112 can also return the digital material to the previous one by pressing the keyboard key "B" indicating the switching back to the previous digital material to the video synthesizing unit 12.

[0057] Also, here, when the material switching unit 112 presses the keyboard key "ESC" indicating the end of the operation, it ends the operation. As a result, the operator can switch the digital material at an arbitrary timing while confirming the synthesis of the digital material. Returning to FIG. 3, the description of the configuration of the digital flip composite device 1 will be continued.

[0058] The video synthesizing unit 12 converts the digital material by a projective transformation matrix and synthesizes it with the camera video. The video synthesizing unit 12 reads out the digital material specified by the material operation unit 11 from the storage unit 13, performs projective transformation using the transformation matrix (homography matrix) generated by the transformation matrix generation unit 10, and synthesizes the digital material with the camera video for each frame.

[0059] Since the transformation matrix generated by the transformation matrix generation unit 10 transforms the plane of the marker coordinate system based on the flip board into the plane of the camera coordinate system of the camera image, digital materials will be synthesized on the flip board of the camera image. Note that the video composition unit 12 will fill the conversion target area of the digital material with a predetermined color (for example, white which is the background color of the flip board) in the camera image and then perform projective transformation on the digital material. The video composition unit 12 outputs the synthesized video after synthesis to the outside.

[0060] The storage unit 13 stores the digital materials whose positions, sizes, etc. are set by the material operation unit 11. The digital materials stored in the storage unit 13 are associated with the marker coordinate system of the AR marker with their positions, sizes, etc. set by the material adjustment unit 110 of the material operation unit 11. In addition, the digital materials stored in the storage unit 13 are associated with thumbnail images together with numbers by the material adjustment unit 110. In addition, for the digital materials stored in the storage unit 13, the order of synthesis is set by the switching order setting unit 111 of the material operation unit 11 according to the numbers. This storage unit 13 can be configured with a general storage medium such as a semiconductor memory.

[0061] With the configuration described above, the digital flip synthesis device 1 can generate an image with the digital material changed on the flip board by simply preparing one AR marker on the flip board.

[0062] In addition, since the digital flip synthesis device 1 uses dots arranged regularly continuously in the horizontal and vertical directions as the AR marker, even if a part of the AR marker is out of the camera image, it can display the digital material following the AR marker. In addition, the digital flip synthesis device 1 allows an operator to switch digital materials at any timing, and can perform various effects such as the contents at multiple locations on the flip board being switched. Note that the digital flip synthesis device 1 can be operated by a program for causing a computer (not shown) to function as each of the above-described units.

[0063] [Operation of Digital Flip Synthesis Device] Next, with reference to FIG. 13 (refer to FIG. 3 as appropriate for the configuration), the operation of the digital flip synthesis device 1 will be described. As a pre-stage for generating a composite video, the digital flip synthesis device 1 performs the processes of the following steps S1 and S2.

[0064] In step S1, the material adjustment unit 110 of the material operation unit 11 takes in a digital material from the outside and sets the position, size, etc. of the digital material with respect to the AR marker on the flip board. Here, the material adjustment unit 110 displays the material capture setting screen SC1 shown in FIG. 9 on the display device 2, and sets the position, size, etc. of the digital material in the marker coordinate system according to the operator's instructions. The material adjustment unit 110 stores the set digital material in the storage unit 13 (not shown as a step).

[0065] In step S2, the switching order setting unit 111 of the material operation unit 11 selects the digital material to be synthesized with the camera video and sets the switching order. Here, the switching order setting unit 111 displays the material selection screen SC2 shown in FIG. 10 on the display device 2, and sets the selection of the digital material to be synthesized with the camera video and the switching order according to the operator's instructions. The switching order setting unit 111 stores the switching order of the digital materials in the storage unit 13 (not shown as a step). The digital flip synthesis device 1 synthesizes the digital material with the camera video by the operations after step S3.

[0066] In step S3, the marker detection unit 100 of the conversion matrix generation unit 10 detects (recognizes and tracks) the AR marker for each frame from the camera video. Here, the marker detection unit 100 recognizes the AR marker by detecting the barycentric coordinates of the dots constituting the AR marker in the frame (camera image) according to the procedure described in FIG. 4. Further, the marker detection unit 100 tracks the AR marker in the sequentially input frames.

[0067] In step S4, the matrix calculation unit 101 of the conversion matrix generation unit 10 calculates a homography matrix for projecting a plane from the marker coordinate system to the camera coordinate system using the AR marker detected in step S3. Here, the matrix calculation unit 101 calculates a homography matrix for each of the four adjacent neighboring dots of the AR marker and selects the homography matrix with the minimum error. In step S5, the video composition unit 12 reads out the digital material stored in the storage unit 13 and performs projective transformation using the homography matrix calculated in step S4, thereby compositing the digital material into the camera video for each frame.

[0068] In step S6, the material switching unit 112 of the material operation unit 11 determines whether there is an instruction to switch the digital material to be composited into the camera video. Here, the material switching unit 112 displays the switching execution screen SC3 shown in FIG. 12 on the display device 2 and determines whether an instruction to switch to the next digital material (pressing "Enter") has been given.

[0069] Here, when an instruction to switch to the next digital material is given (Yes in step S6), in step S7, the video composition unit 12 switches the digital material to be composited into the camera video. That is, the video composition unit 12 reads out the next digital material stored in the storage unit 13 and performs projective transformation using the homography matrix calculated in step S4, thereby compositing the digital material into the camera video for each frame. Then, the digital flip composition device 1 returns to the operation in step S3.

[0070] On the other hand, when not instructed to switch to the next digital material (No in step S6), in step S8, the material switching unit 112 determines whether there is an instruction to end. Here, the material switching unit 112 determines whether an instruction to end the operation (pressing "ESC") has been given on the switching execution screen SC3 shown in FIG. 12 of the display device 2. Here, when no end instruction is given (No in step S8), the digital flip compositing device 1 returns the operation to step S3.

[0071] On the other hand, when an end instruction is given (Yes in step S8), the digital flip compositing device 1 ends the operation. In this case, the video compositing unit 12 outputs the input camera image as it is. Through the above operations, the digital flip compositing device 1 can generate a video with multiple contents changed on the flip board according to the operator's instructions.

[0072] Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and design changes and the like within the scope not departing from the gist of the present invention are also included.

[0073] For example, here, in the matrix calculation unit 101, a homography matrix is calculated for every four adjacent neighboring points of each dot constituting the AR marker, and the homography matrix with the minimum error is selected. However, the matrix calculation unit 101 may calculate a homography matrix from corresponding points of four or more points (for example, the center of gravity of all dots of the AR marker) using, for example, the DLT (Direct Linear Transform) method.

[0074] Also, here, the switching order setting unit 111 is configured to set in advance the order in which digital materials are switched. However, when there are few digital materials to be switched, etc., the material switching unit 112 may display thumbnail images of all digital materials on the screen without using the switching execution screen SC3 in FIG. 12, and the operator can select a thumbnail image to switch the digital material to be synthesized. In that case, the switching order setting unit 111 may be omitted from the configuration. Of course, for the convenience of the operator, it is preferable to set in advance the order in which the digital materials are switched by the switching order setting unit 111.

Explanation of Signs

[0075] 1 Digital flip synthesis device 10 Conversion matrix generation unit 100 Marker detection unit 101 Matrix calculation unit 11 Material operation unit 110 Material adjustment unit 111 Switching order setting unit 112 Material switching unit 12 Video synthesis unit 13 Memory unit FB Flip board MK AR marker D Digital material

Claims

1. A digital flip synthesis device that generates a synthesized video by synthesizing digital materials onto an AR marker on a flip board shown in a camera video, comprising: a material adjustment unit that sets the relative position and size of the digital material with respect to the AR marker on predetermined marker coordinates for a plurality of the digital materials, converts the digital material into a digital material on the marker coordinates, and stores it in a storage unit; a marker detection unit that detects the AR marker on the camera coordinates from the camera video; a matrix calculation unit that calculates a projective transformation matrix for converting the AR marker on the marker coordinates into the AR marker on the camera coordinates; a material switching unit that switches and selects a plurality of digital materials stored in the storage unit; a video synthesis unit that converts the digital material selected by the material switching unit with the projective transformation matrix and synthesizes it onto the camera video; A digital flip synthesis device characterized by comprising the above.

2. The AR marker has a structure in which dots having a predetermined size are regularly arranged at equal intervals in the horizontal and vertical directions, The digital flip synthesis device according to claim 1, wherein the marker detection unit detects the positions of the dots from the camera video based on the regularity of the arrangement of the dots.

3. The digital flip synthesis device according to claim 2, wherein the marker detection unit tracks the dots for each frame of the camera video to obtain a new AR marker.

4. The digital flip synthesis device according to claim 2 or claim 3, wherein the matrix calculation unit calculates a projective transformation matrix for each of the adjacent four neighboring dots detected by the marker detection unit, and selects a projective transformation matrix that minimizes the conversion error from the AR marker on the marker coordinates to the AR marker on the camera coordinates.

5. Further comprising a switching order setting unit that sets the order of switching a plurality of digital materials stored in the storage unit, The digital flip synthesis device according to claim 1, wherein the material switching unit selects the digital material from the storage unit in the order set by the switching order setting unit according to an external switching instruction.

6. A program for causing a computer to function as the digital flip synthesis device according to claim 1.

Images