Verification method, computer program and projection system

Abstract

To facilitate determining validity of geographic correction in automatic geographic correction during projection of a projection device.SOLUTION: In a verification method for geometric correction of a projection system that projects an image onto a projection surface using a projection device, the method being executed by an arithmetic device that can access a storage device, the storage device stores a pattern image that includes a plurality of characteristic points that represent coordinates on the image projected onto the projection surface by the projection device. The arithmetic device causes the projection device to project the pattern image onto the projection surface, acquires first captured image data that includes the pattern image projected onto the projection surface from an imaging device, compares characteristic points included in the first captured image data with the plurality of characteristic points included in the pattern image to extract a plurality of non-projection characteristic points that are not projected onto the projection surface, and generates non-projection characteristic point data in which the extracted plurality of non-projection characteristic points are arranged at corresponding positions within the image projected by the projection device.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a projection system that projects an image onto a projection surface using a projection device, a method for verifying geometric correction in the projection system, and a computer program. [Background technology]

[0002] In recent years, projection devices have been used in a variety of situations. Projection devices often perform geometric correction to display a desired image. Furthermore, the variety of projection surfaces onto which images are projected by projection devices is increasing, and they are no longer limited to rectangular, flat projection surfaces; for example, they can also be projected onto curved surfaces. When projecting onto such curved projection surfaces, geometric correction becomes more complex.

[0003] For geometric correction of a projection device, there is also a method of using a photographing device to photograph an image projected by the projection device and then automatically performing geometric correction using the photographed image. For example, Patent Document 1 also describes a technology that provides correction information that can accurately correct distortion in a projected image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-57876 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional automatic geometric correction, it is not possible to check the process of the automatic geometric correction, and it is not possible to identify the cause of failure of the geometric correction. For example, geometric correction requires that the coordinates of the original image to be projected by a projection device, the coordinates of the projected image output from the projection device, the coordinates on the projection surface projected from the projection device, and the coordinates of the captured image obtained by capturing the image on the projection surface correspond to each other. However, with automatic geometric correction, it is not possible to check information regarding the correspondence of these coordinates as intermediate data.

[0006] The present disclosure provides a verification method, a computer program, and a projection system that facilitate determining the validity of geometric correction during automatic geometric correction performed by a projection device during projection. [Means for solving the problem]

[0007] The verification method disclosed herein is a method for verifying geometric correction in a projection system that is executed by a computing device that can access a storage device and that projects an image onto a projection surface using a projection device, wherein the storage device stores a pattern image that includes a plurality of feature points that indicate coordinates on the image to be projected onto the projection surface by the projection device, the computing device projects the pattern image from the projection device onto the projection surface, the computing device acquires first captured image data from a photographing device that includes the pattern image projected onto the projection surface, compares the feature points included in the first captured image data with each feature point included in the pattern image to extract non-projected feature points that are not projected onto the projection surface, and generates non-projected feature point data in which the extracted non-projected feature points are arranged at corresponding positions within the range of the projection image projected by the projection device.

[0008] These general and specific aspects may be realized by a system, a method, and a computer program, as well as combinations thereof. [Effects of the Invention]

[0009] The verification method, computer program, and projection system disclosed herein can facilitate determining the validity of geometric correction when the projection device automatically performs geometric correction during projection. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a projection system according to a first embodiment. [Figure 2] 2 illustrates various components of the projection system of FIG. 1 arranged in space. [Figure 3] 2 is a schematic diagram showing images used by each configuration of the projection system of FIG. 1. [Figure 4] 10 is an explanatory diagram of the correspondence between coordinates of an original image to be projected and a projected image projected from a projection device. FIG. [Figure 5A] 10 shows an example of projected feature point data. [Figure 5B] 10 shows an example of unprojected feature point data. [Figure 6A] 1 shows an example of a pattern image including feature points. [Figure 6B] 6B shows an example in which the pattern image of FIG. 6A is projected. [Figure 6C] 6B shows an example of the relationship between the pattern image and the placement reference markers in FIG. 6A. [Figure 6D] 10 shows an example of a projected reference marker. [Figure 7] 1 shows an example of visualized image data. [Figure 8] An example of result data is shown below. [Figure 9A] 4 is a flowchart illustrating a verification method according to the first embodiment. [Figure 9B] 9B is a flowchart illustrating the verification method according to the first embodiment, following FIG. 9A. [Figure 9C] 9B is a flowchart illustrating the verification method according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing an image processing apparatus according to a second embodiment. [Figure 11] 11 is a schematic diagram showing images used by each component of the virtual space generated by the image processing device of FIG. 10. [Figure 12A] 10 is a flowchart illustrating a verification method according to the second embodiment. [Figure 12B] 12B is a flowchart illustrating the verification method according to the second embodiment, following FIG. 12A. [Figure 12C] 12B is a flowchart illustrating the verification method according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present disclosure provides a verification method, a computer program, and a projection system that facilitate determining the validity of geometric correction in automatic geometric correction during projection by a projection device. Specifically, the verification method, the computer program, and the projection system of the present disclosure enable a user to understand the process of geometric correction in automatic geometric correction by a projection device, thereby enabling the user to recognize whether the geometric correction is appropriate.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings as appropriate. However, in the detailed description, unnecessary parts of the description of the prior art and substantially identical configurations may be omitted. This is for the sake of simplicity. Furthermore, the following description and the accompanying drawings are disclosed to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0013] The following defines various terms used in this specification. The "projection surface" is a surface onto which an image is projected by a projection device such as a projector. The projection surface is not limited to a flat screen, but may be curved or have some unevenness.

[0014] "Projected image" refers to an image output by a projection device for projection onto a projection surface.

[0015] The "projection surface image" refers to the image on the projection surface when the projection image is projected onto the projection surface by the projection device.

[0016] "Photographed image" refers to an image photographed by a photographing device.

[0017] "Geometric correction" refers to the correction of geometric distortions that occur when an image is projected from a projection device onto a projection surface. For example, geometric correction is used to display an image as a square when an image that should be square is projected onto the projection surface as a trapezoid (keystone distortion), when the image appears to bulge in the center (barrel distortion), or when the image appears to contract in the center (pincushion distortion). Geometric correction is achieved by adjusting parameter values ​​in the projection device.

[0018] "Virtual space" refers to a space within a computer that represents an environment equivalent to a real space using spatial information that represents the surrounding environment of the real space in which the projection device and projection surface are placed.

[0019] "Spatial information" refers to information regarding the size of a space, including its size and shape, the building materials that form the space, the lighting devices used in the space, the shape and size of objects present in the space, the placement location of those objects, the material and color of those objects, etc. For example, spatial information may include the floor area and wall height of the space, the material and color of the floor and walls, etc. Furthermore, for example, if components such as pillars or beams exist in the space, spatial information may include the shape, size, placement location, material, and color of these parts. Note that objects present in the space are preferably those that are constantly present in a specific position in the space, such as air conditioning equipment. Spatial information may include coordinates indicating the placement location of a display device in the space. Spatial information may also include the size of a projection surface such as a screen, the position of the projection surface (coordinates and the angle formed between the projection direction of the projection device and the projection surface, etc.), the material, color, etc.

[0020] [First embodiment] In the projection system according to the first embodiment, the image projection and automatic geometric correction by the projection device can be verified using an image processing device.

[0021] Projection System 1, a projection system 1 according to the first embodiment of the present disclosure includes an image processing device 10, a projection device 20, and an image capturing device 30. A display device 14 and an input device 15 can be connected to the image processing device 10. FIG. 2 is a diagram showing a space having the projection system 1.

[0022] In the projection system 1, the image processing device 10 operates the projection device 20 to project an image onto the projection surface 40. Note that the "image" does not necessarily have to be a still image, and may be a moving image. In this specification, a still image will be exemplified as the "image" in the description.

[0023] The image capturing device 30 captures, as a captured image, an image including at least the projection surface image Im2 projected onto the projection surface 40 by the projection device 20. In the example shown in Fig. 2, the image projected from the projection device 20 does not fit entirely within the projection surface 40, and part of the image is projected outside the projection surface 40. In this case, the image processing device 10 geometrically corrects the projection surface image Im2 so that the entire projection surface image Im2 fits within the projection surface 40.

[0024] <<Image Processing Device>> 1, an image processing device 10 according to the present disclosure is an information processing device including a calculation device 11, a communication device 12, and a storage device 13. The image processing device 10 can be connected to a display device 14 and an input device 15.

[0025] The arithmetic unit 11 is a controller that controls the entire image processing device 10. For example, the arithmetic unit 11 performs various processes for executing verification by reading and executing a computer program P stored in the storage device 13. Furthermore, the arithmetic unit 11 is not limited to a device that performs predetermined functions through the cooperation of hardware and software, but may be a hardware circuit designed specifically to perform predetermined functions. In other words, the arithmetic unit 11 can be realized by various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, an ASIC, etc.

[0026] The communication device 12 is a communication means for enabling data communication with an external device (e.g., the projection device 20, the image capture device 30, etc.). The above-mentioned data communication may be wired and / or wireless, and may be performed in accordance with known communication standards. For example, wired data communication is performed by using, as the communication device 12, a communication controller of a semiconductor integrated circuit that operates in accordance with the Ethernet (registered trademark) standard and / or the USB (registered trademark) standard. Wireless data communication is performed by using, as the communication device 12, a communication controller of a semiconductor integrated circuit that operates in accordance with the IEEE 802.11 standard for LANs (Local Area Networks) and / or the fourth-generation / fifth-generation mobile communication systems, so-called 4G / 5G, for mobile communications.

[0027] The storage device 13 is a recording medium for recording various information. The storage device 13 is realized, for example, by a RAM, a ROM, a flash memory, an SSD (Solid State Drive), a hard disk drive, or other storage devices, or by an appropriate combination thereof. The storage device 13 stores a computer program P, which is a computer program executed by the arithmetic unit 11, and various data used for executing verification of automatic geometric correction related to projection. For example, the storage device 13 stores image information 131, first captured image data 132, feature point coordinate data 133, second captured image data 134, a coordinate conversion table 135, visualized image data 136, projected feature point data 137, non-projected feature point data 138, etc.

[0028] The image processing device 10 may be realized by a plurality of information processing devices connected to each other so as to be able to communicate with each other. Also, a part of the data stored in the storage device 13 may be stored in an external storage device, and the image processing device 10 may read and use the data from the external storage device.

[0029] The display device 14 is a display means such as a display that displays data obtained during the verification process, verification results, etc. The input device 15 is an input means such as an operation button, keyboard, mouse, touch panel, microphone, etc. that is used for operations and data input.

[0030] The image information 131 includes various image data that is projected onto a projection surface from the projection device 20 by operating the image processing device 10. For example, the image information 131 includes image data such as a pattern image, a marker image, and a content image.

[0031] The pattern image includes a plurality of feature points indicating coordinates on the image projected from the projection device 20 onto the projection surface 40. Specifically, the pattern image includes, for example, approximately 1,000 feature points within the area of ​​the projection image projected by the projection device 20. The pattern image can be used to identify the correspondence between the coordinates of the optical elements (DMD, liquid crystal element, etc.) used to display the image on the projection device 20 and the coordinates of the optical elements (image sensor, etc.) used to acquire the image on the image capture device 30. Specifically, the pattern image has a plurality of feature points arranged in a regular pattern. As a result, each feature point is displayed in a uniquely identifiable state by projecting the pattern image, and therefore can be used to identify the correspondence between the coordinates of the projected image output by the projection device 20 and the captured image acquired by the image capture device 30.

[0032] The marker image is a marker image displayed by the projection device 20. This marker image includes multiple projection reference markers that indicate specific coordinates defined as marks on the image projected onto the projection surface 40 by the projection device 20. For example, the marker image is an image that includes the projection reference markers within the image display area projected by the projection device 20. The marker image can be used to identify the correspondence between the coordinates of the optical elements for image display in the projection device 20 and the coordinates of the projection surface 40. For example, the projection reference markers are placed at four corners of the projection surface 40. Furthermore, if the projection surface 40 includes a curved surface, the number of projection reference markers placed on the projection surface 40 is increased according to the curvature. Specifically, the projection reference markers can be placed horizontally and vertically on the projection surface at equal intervals, such as dividing the distance into halves or thirds, or at any point desired by the user.

[0033] The arithmetic unit 11 executes various processes such as projection processing, image acquisition processing, coordinate acquisition processing, feature point extraction processing, coordinate correspondence processing, generation processing, adjustment processing, reception processing, determination processing, and output processing.

[0034] In the projection process, the arithmetic unit 11 reads out a pattern image from the image information 131 stored in the storage device 13, and causes the projection device 20 to project the pattern image onto the projection surface 40. As described above, as shown in FIG. 3, the image output from the projection device 20 onto the projection surface 40 in the projection process is a "projected image Im1," and the image projected onto the projection surface 40 is a "projection surface image Im2." Furthermore, the image captured by the imaging device 30 thereafter is a "captured image Im3."

[0035] In the image acquisition process, the arithmetic device 11 acquires, from the photographing device 30, first photographed image data 132 including a pattern image projected onto the projection surface 40, which is an image photographed by the photographing device 30. The arithmetic device 11 also stores the acquired first photographed image data 132 in the storage device 13.

[0036] In the coordinate acquisition process, the arithmetic device 11 acquires the coordinates of feature points on the projection surface 40 included in the first photographed image data 132, associates the acquired coordinates of each feature point with the first photographed image data 132, and stores them in the storage device 13 as feature point coordinate data 133. Specifically, the arithmetic device 11 acquires coordinates that match the conditions indicating the feature points, and generates data including the acquired multiple coordinates as the feature point coordinate data 133.

[0037] In the feature point extraction process, the arithmetic device 11 compares feature points included in the first captured image data 132 with each feature point included in the pattern image to extract "non-projected feature points," which are feature points that are not projected onto the projection surface 40. In addition, in the feature point extraction process, the arithmetic device 11 compares feature points included in the first captured image data with each feature point included in the pattern image to extract "projected feature points," which are feature points that are projected onto the projection surface 40. Specifically, the arithmetic device 11 compares the coordinates of each feature point included in the feature point coordinate data 133 acquired in the coordinate acquisition process with the coordinates of each feature point in the pattern image to determine whether a feature point corresponding to each feature point in the pattern image is included on the projection surface 40 of the first captured image data 132. When a feature point corresponding to a feature point in the pattern image is included on the projection surface 40 of the first captured image data 132, the arithmetic device 11 designates the feature point as a "projected feature point." On the other hand, when a feature point corresponding to a feature point of the pattern image is not included on the projection surface 40 of the first captured image data 132, the calculation device 11 determines that feature point as a "non-projected feature point." In other words, of the feature points included in the pattern image, the feature points projected onto the projection surface 40 are determined as "projected feature points." On the other hand, of the feature points included in the pattern image, the feature points not projected onto the projection surface 40 are determined as "non-projected feature points." The calculation device 11 adds the result of the feature point extraction process to the feature point coordinate data 133 stored in the storage device 13.

[0038] In the coordinate correspondence process, the arithmetic unit 11 associates the coordinates of each feature point of the pattern image projected by the projection device 20, the coordinates of each feature point of the pattern image projected onto the projection surface 40, and the coordinates of each feature point photographed by the photographing device 30. For example, the arithmetic unit 11 stores, in the feature point coordinate data 133 of the storage device 13, the correspondence relationships between the coordinates of the projected feature points on the projection image Im1, the coordinates of the projected feature points on the projection surface image Im2, and the coordinates of the projected feature points on the photographed image Im3. Using these correspondence relationships, it is possible to indicate how the image output from the projection device 20 will be displayed on the projection surface 40.

[0039] An example of coordinate correspondence processing will be described using FIG. 4. FIG. 4 shows an example in which 41 × 26 feature points are provided in a pattern image that is the source of generation of the projection image Im1, and the coordinate number of the projection image Im1 projected from the projection device 20 is 201 × 126. Note that in FIG. 4, for simplicity of the drawing, not all coordinates and feature points are shown. For example, the relationship between the coordinates (xa, ya) of an arbitrary feature point a in the pattern image and the coordinates (ua, va) corresponding to feature point a on the projection image Im1 when the pattern image is projected onto the projection surface by the projection device 20 is shown in the following equation (1). The relationship between the coordinates of these pattern images and the coordinates of the projection image Im1 is uniquely defined for each test pattern image and each projection image.

number

[0040] For example, the relationship between the coordinates (0,0) of the top left corner of the pattern image and the corresponding coordinates on the projection image Im1 is Q 0_p , the relationship between the coordinates (41,0) of the top right edge of the pattern image and the corresponding coordinates on the projection image Im1, Q 41_p , the relationship between the coordinates (0,26) of the bottom left edge of the pattern image and the corresponding coordinates on the projection image Im1, Q 1026_p The relationship between the coordinates (4,26) of the bottom right edge of the pattern image and the corresponding coordinates on the projection image Im1 is Q. 1066_p can be defined by the following formulas (1-1) to (1-4), respectively.

number

[0041] Furthermore, the relationship between the coordinates (xa, ya) of an arbitrary feature point a in the pattern image and the coordinates (sa, ta) corresponding to the coordinates (xa, ya) of the feature point a on the captured image Im3 obtained by capturing the projection surface image Im2 projected onto the projection surface 40 by the projection device 20 with the capture device 30 is shown in the following formula (2). The relationship between these coordinates of the pattern image and the coordinates of the projection surface image Im2 on the projection surface 40 is determined according to the shape of the projection surface image Im2 on the projection surface 40 and the capture angle of view formed by the projection surface 40 and the capture device 30 that captures the projection surface 40.

number

[0042] Furthermore, the relationship between the coordinates (ua, va) on the projection image Im1 corresponding to the coordinates (xa, ya) of the feature point a in the pattern image and the coordinates (sa, ta) corresponding to the coordinates (xa, ya) of the feature point a on the captured image Im3 obtained by capturing the projection surface image Im2 on the projection surface 40 can be expressed by the following equation (3).

number

[0043] Equation (3) is a coordinate transformation matrix from the coordinates of the projected image Im1 to the coordinates of the captured image Im3. Therefore, this coordinate transformation matrix T a_pc By using the above formula, it is possible to obtain the coordinates of any feature point in the captured image Im3 from the coordinates in the projected image Im1, or vice versa, the coordinates in the projected image Im1 from the coordinates in the captured image Im3.

[0044] In the generation process, the arithmetic device 11 generates projected feature point data 137 in which the multiple projected feature points extracted in the extraction process are arranged at corresponding positions on the projection surface 40 captured by the image capturing device 30. Specifically, the arithmetic device 11 uses the feature point coordinate data 133 to generate projected feature point data 137 in which marks indicating feature points are arranged at positions corresponding to the projected feature points on the projection surface 40 of image data representing the state captured by the image capturing device 30. FIG. 5A shows an example of the projected feature point data 137. For example, by referring to the projected feature point data 137, it is possible to understand what proportion of the image data projected from the projection device 20 is actually projected by the projection device 20. Specifically, if the number of projected feature points confirmed in the projected feature point data 137 is significantly smaller than the number of feature points included in the actual pattern image, it is considered that the image data projected from the projection device 20 is too large for the projection surface 40. In this case, for example, the number of feature points, the number of projected feature points, the ratio of the number of projected feature points to the number of feature points, etc. may be displayed together. Furthermore, for example, by referring to the projection feature point data 137, it is possible to grasp how the image data projected from the projection device 20 will be distorted when projected. Specifically, in the projection feature point data 137, it is preferable that the feature points are uniformly dispersed and arranged on the projection surface 40, but if the feature points are densely concentrated in one area, it is possible to grasp that the distortion is occurring in that area.

[0045] In the generation process, the arithmetic device 11 generates non-projected feature point data 138 by arranging multiple non-projected feature points extracted in the extraction process at corresponding positions within the range of the projection image projected by the projection device 20. Specifically, the arithmetic device 11 uses the feature point coordinate data 133 to generate non-projected feature point data 138 by arranging marks indicating feature points at positions corresponding to the non-projected feature points within the region of image data indicating the state of projection by the projection device 20. FIG. 5B shows an example of the non-projected feature point data 138. For example, by referring to the non-projected feature point data 138, it is possible to understand what proportion of the image data projected from the projection device 20 is projected and what proportion is not projected, and how distorted the projection is. Specifically, the wider the range in which non-projected feature points are present that can be confirmed in the non-projected feature point data 138, the more likely it is that the image data projected from the projection device 20 is too large for the projection surface 40. In this case, for example, the number of feature points and the number of non-projected feature points, the ratio of the number of non-projected feature points to the number of feature points, etc. may be displayed together.

[0046] In addition, in the projection process, the arithmetic unit 11 reads out a marker image from the image information 131 stored in the storage unit 13, and causes the projection device 20 to project the marker image onto the projection surface 40.

[0047] Furthermore, in the image acquisition process, the computing device 11 acquires, from the imaging device 30, second captured image data 134 including marker images projected onto the projection surface 40 on which a plurality of placement reference markers corresponding to the plurality of projection reference markers are arranged. The placement reference markers are marks that the user can arrange at desired positions on the projection surface 40.

[0048] Furthermore, in the generation process, the arithmetic device 11 generates a coordinate conversion table 135 for projecting non-projection feature points onto the projection surface 40, using a comparison result between the positions of a plurality of placement reference markers included in the second captured image data 134 and the positions of a plurality of projection reference markers. In other words, by using the coordinate conversion table 135, it is possible to convert each coordinate of the image data into each coordinate of the projection surface and perform geometric correction in order to project the image data from the projection device 20 onto the projection surface 40. Specifically, the arithmetic device 11 generates the coordinate conversion table 135 that associates the coordinate system of the projection device 20 with the coordinate system of the projection surface 40. The arithmetic device 11 stores the generated coordinate conversion table 135 in the storage device 13.

[0049] An example of generating the coordinate conversion table 135 will be described using FIGS. 6A to 6D. FIG. 6A shows an example of a pattern image including 41 feature points horizontally and 26 feature points vertically, for a total of 41 × 26 feature points. FIG. 6B is an external view showing the state in which placement reference markers are arranged on the projection surface 40 onto which the pattern image shown in FIG. 6A is projected. The user places placement reference markers at at least four points on the projection surface 40 to indicate the projection range of the projection device 20 and the expected range. In the example shown in FIG. 6B, placement reference markers are arranged at four points (P11 to P14) on the projection surface 40, respectively: the upper left, upper right, lower left, and lower right. On the pattern image, the points (P11 to P14) of the placement reference markers shown in FIG. 6B correspond to the four points (P21 to P24) shown in FIG. 6C. Therefore, the position of each projection reference marker projected from the projection device 20 is uniquely determined. 6B, when the projection surface 40 is photographed by the photographing device 30 and each feature point of the pattern image in the photographed image Im3 is detected, the feature points of the projection image Im1 can be calculated, and the positions of the placement reference markers in the photographed image Im3 can be uniquely determined. Because the range in which the user expects projection is the range surrounded by the placement reference markers, the coordinate conversion table 135 is generated by interpolating the positions of the feature points included in the photographed image Im3 based on the feature points included and not included in the photographed image Im3.

[0050] Furthermore, in the generation process, the calculation device 11 generates visualized image data 136 by visualizing the projected feature points and the non-projected feature points using the coordinate conversion table 135. FIG. 7 shows an example of the visualized image data 136. The visualized image data 136 shown in FIG. 7 has an outer frame that indicates the range that can be projected by the projection device 20. For example, assume that a predetermined grid is set for the image data to be projected. In this case, the visualized image data 136 shown in FIG. 7 has the grid lines of the image data to be projected represented by gray curves in the projected image. Since the range that can be projected by the projection device 20 is wider than the area of ​​the projection surface 40, as shown in FIG. 7, the coordinate conversion table 135 is used to deform the projected image so that the image data to be projected fits within a portion of the image. Note that the projection system 1 may project an image that extends beyond the projection surface 40, and therefore the outer frame of the gray portion is not necessarily aligned with the projection surface 40.

[0051] In the output process, the calculation device 11 may generate result data including projected feature point data 137, non-projected feature point data 138, and visualized image data 136, as shown in FIG.

[0052] Furthermore, in the adjustment process, the arithmetic unit 11 may use the coordinate conversion table 135 to adjust the image data so that the entire image data is projected onto the projection surface 40. Furthermore, in the projection process, the arithmetic unit 11 may cause the projection device 20 to project the adjusted image data onto the projection surface 40. Note that the image data generated in the adjustment process may be a pattern image in addition to a content image to be projected in the projection system 1. The pattern image is displayed again, making it easier to determine whether the generated coordinate conversion table 135 is appropriate.

[0053] In the reception process, the arithmetic unit 11 receives a designation of a predetermined display range. For example, the arithmetic unit 11 receives a pixel count indicating a predetermined range from the periphery based on the pixel count of the projection image projected by the projection device 20, and can determine the range designated by the received pixel count as the display range.

[0054] "Determining the display range" is a determination of whether the entire image or a desired range of the image is projected onto the projection surface 40. For example, in the projection system 1, the projection device 20 may cause a predetermined pixel shift in the vertical and horizontal directions of the image due to the characteristics of the device. In contrast, the image processing device 10 determines a display range that takes into account possible amounts of shift due to the characteristics of the device in advance, and determines whether the image will be displayed within that range, so that the entire range of the image can be set to be displayed on the projection surface 40 even if a shift occurs.

[0055] Furthermore, the arithmetic device 11 accepts the specification of the transformation conditions in the reception process. For example, the arithmetic device 11 can determine the degree of transformation of each coordinate relative to the original image as the transformation condition. Specifically, the arithmetic device 11 can accept an allowable angle (e.g., 10°) for each of the x and y directions, and can determine movement within that angle as the transformation condition for the coordinate. Also, for example, the arithmetic device 11 can determine the maximum distance between adjacent pixels and can determine that maximum distance as the transformation condition.

[0056] The "determination of transformation conditions" is a determination of whether the image projected onto the projection surface 40 can be changed. For example, in the projection system 1, when the projection device 20 deforms and projects an image onto a curved or other non-flat projection surface 40, the projection device 20 may not be able to deform the image to fit the projection surface 40 due to the characteristics of the device, depending on the degree of curvature. On the other hand, it is physically possible to determine in advance whether the image can be appropriately deformed. Therefore, the image processing device 10 can set the image to be displayed on the projection surface 40 by determining whether the image can be deformed. For example, by changing the position of the projection device 20, the angle between the projection device 20 and the projection surface 40 can be changed, and the image can be displayed on the projection surface 40. Alternatively, by changing the projection angle of the lens of the projection device 20, the image can be displayed on the projection surface 40.

[0057] In the determination process, the arithmetic device 11 determines whether the display using the coordinate conversion table is within a predetermined display range that is allowable for coordinate conversion. Furthermore, in the determination process, the arithmetic device 11 determines whether the display using the coordinate conversion table satisfies predetermined transformation conditions that allow image transformation. Furthermore, in the output process, the arithmetic device 11 can output the determination result. For example, if the determination result indicates that the transformation conditions are not satisfied, the shape of the projection surface, the arrangement of the projection device, and / or the projection lens can be changed as described above to adjust the transformation conditions so that they are satisfied.

[0058] <<Verification Method>> The verification method according to the present disclosure will be described using the flowcharts shown in FIGS. 9A to 9C. First, the user sets up the projection device 20, the projection surface 40, and the image capturing device 30 in a space (S001).

[0059] Further, the position of the projection device 20 is adjusted (S002). Specifically, the position of the projection device 20 is adjusted so that an image is projected from the projection device 20 onto the entire projection surface 40, taking into consideration the projection direction as well as the placement position in the three-dimensional space that takes into consideration the relationship with the space and the projection surface 40, etc.

[0060] Next, the position of the image capturing device 30 is adjusted (S003). Specifically, the position of the image capturing device 30 is adjusted so that the entire projection surface 40 can be captured, taking into consideration the image capturing direction as well as the placement position in three-dimensional space that takes into consideration the relationship with the space, the projection device 20, the projection surface 40, etc. The processing of steps S001 to S003 is a physical adjustment.

[0061] When the physical adjustment is completed, the calculation device 11 operates the projection device 20 to project a pattern image onto the projection surface 40 (S004). The pattern image includes a plurality of feature points.

[0062] Furthermore, the arithmetic device 11 acquires first captured image data from the image capturing device 30 (S005). The first captured image data includes a pattern image projected onto the projection surface 40 by the projection device 20.

[0063] Furthermore, the arithmetic unit 11 acquires the coordinates of a plurality of feature points projected onto the projection surface 40 from the first captured image data acquired in step S005 (S006).

[0064] Next, the calculation device 11 operates the projection device 20 to project a marker image onto the projection surface 40 (S007). The marker image includes a projection reference marker for transforming the projection image into a desired shape.

[0065] Furthermore, the arithmetic device 11 acquires second captured image data from the image capturing device 30 (S008). The second captured image data includes the marker image projected onto the projection surface 40 by the projection device 20.

[0066] Furthermore, the arithmetic unit 11 acquires the coordinates of the plurality of projection reference markers projected onto the projection surface 40 from the second captured image data acquired in step S008 (S009).

[0067] Furthermore, the arithmetic unit 11 acquires the coordinates of a plurality of projection reference markers arranged on the projection surface 40 from the second captured image data acquired in step S008 (S010).

[0068] The calculation device 11 extracts non-projected feature points and projected feature points using the pattern image and the result of acquisition in step S006 (S011). Specifically, the coordinates of each feature point included in the pattern image are compared with each feature point acquired in step S006 to find corresponding feature points, and the feature points extracted in step S006 are set as projected feature points, and the feature points not extracted are set as non-projected feature points.

[0069] The calculation device 11 uses the coordinates of the placement reference markers in step S010 and the extraction result in step S011 to specify a projection range in which the projection image is projected from the projection device 20 (S012).

[0070] Furthermore, the arithmetic unit 11 generates a coordinate conversion table 135 for converting the projection image projected from the projection device 20 to match the shape of the projection surface 40 based on the projection range specified in step S012 (S013).

[0071] The arithmetic unit 11 receives the display range and the transformation conditions (S014).

[0072] The arithmetic unit 11 determines whether or not the image to be transformed using the coordinate conversion table 135 generated in step S013 satisfies the display range and transformation conditions accepted in step S014 (S015). If the image is not displayed within the display range and / or does not satisfy the transformation conditions (NO in S016), the arithmetic unit 11 returns to step S002 and repeats the processes of steps S003 to S015.

[0073] Thereafter, the arithmetic unit 11 generates the projected feature point data 137 in which the projected feature points are arranged, and displays it on the display unit 14 (S017).

[0074] Furthermore, the arithmetic unit 11 generates non-projected feature point data 138 in which the non-projected feature points are arranged, and displays the data on the display unit 14 (S018).

[0075] Furthermore, the arithmetic unit 11 generates visualized image data 136 using the coordinate conversion table, and displays it on the display unit 14 (S019).

[0076] Furthermore, the arithmetic unit 11 transforms the image using the coordinate conversion table 135 generated in step S013 (S020).

[0077] The arithmetic unit 11 operates the projection device 20 to project the image transformed in step S020 onto the projection surface 40 (S021).

[0078] If the image projected onto the projection surface 40 is as expected, the end is operated and the series of processes related to verification in the projection system 1 ends (YES in S022).

[0079] On the other hand, if the image projected onto the projection screen 40 is not as expected (NO in S022), readjustment is required, so the process returns to step S002 and the processes of steps S002 to S022 are repeated. Specifically, after physical adjustment of the projection device 20 and the image capturing device 30 is performed, the verification process is performed again.

[0080] As described above, the projection system 1 according to the first embodiment makes it possible to check information generated during automatic geometric correction, thereby facilitating the determination of the validity of the geometric correction performed by the projection device.

[0081] [Second embodiment] According to the image processing device of the second embodiment, even when no projection system actually exists, such as when a projection system is not yet constructed, virtual space information and parameter information including the positions of a projection device, a projection surface, an image capture device, etc. to be installed can be used to imagine a space in which the projection system will be installed and verify the projection of an image by the projection device and automatic geometric correction. For example, in a situation where a projection device, a projection surface, an image capture device, etc. are already installed in a space, the virtual space information and parameter information can be used to adjust the position of the projection device, etc. in the space. Furthermore, in a situation where a projection device, etc. is not installed in the space, the virtual space information and parameter information can be used to determine the position of the projection device, etc. in the space. Note that the following describes an example of verification in a situation where a projection system is not yet constructed and a projection device, a projection surface, and an image capture device are not installed in the space. However, the same applies to a case where a virtual space is generated and verification is performed when a projection device, etc., does not exist in the real space.

[0082] <<Image Processing Device>> 10, an image processing device 10A according to the second embodiment is an information processing device including a calculation device 11, a communication device 12, a storage device 13, etc. The image processing device 10A can be connected to a display device 14 and an input device 15. The configurations of the calculation device 11, the communication device 12, the storage device 13, the display device 14, and the input device 15 are the same as those described above with reference to FIG.

[0083] As shown in FIG. 10, the memory device 13 of the image processing device 10A differs from the memory device 13 of the image processing device 10 described above using FIG. 1 in that it does not store first captured image data 132, feature point coordinate data 133, and second captured image data 134, but stores space information 141, parameter information 142, virtual space information 143, first virtual captured data 144, virtual feature point coordinate data 145, and second virtual captured data 146.

[0084] The space information 141 is information about a real space in which a projection device and the like are planned to be placed. For example, the space information 141 can include at least any of the following information: the placement positions of the projection device and the projection surface in the space, material information and size of the projection surface, the placement position of the image capture device in the space, information about the size of the space, information about the building materials that make up the space, information about the lighting used in the space, and information about objects to be placed in the space. For example, this space information is coordinate information that indicates the space, coordinate information including the placement positions of objects to be placed in the space, and information about the specifications of the objects.

[0085] The parameter information 142 may include parameter values ​​to be set in a projection device to be connected to the image processing device 10. For example, the parameter information 142 may include at least one of the resolution, brightness, chromaticity, lens zoom, shift amount, and throw ratio to be set in the projection device. The parameter information 142 may also include parameter values ​​to be set in an imaging device. For example, the parameter information 142 may include at least one of the focal length, exposure, and angle of view to be set in the imaging device to be connected to the image processing device 10.

[0086] The virtual space information 143 is information indicating a virtual space generated in the image processing device 10A based on the space information 141. In the virtual space information 143, virtual parameter values ​​based on the parameter information 142 are set for the virtual display device and the virtual image capturing device arranged based on the space information 141. This virtual space information 143 is information including coordinate information and the like represented by the space information 141 and virtual parameter values ​​and the like.

[0087] The arithmetic unit 11 executes various processes such as virtual space generation process, virtual projection process, virtual image acquisition process, virtual coordinate acquisition process, feature point extraction process, coordinate correspondence process, generation process, adjustment process, reception process, determination process, and output process.

[0088] In the virtual space generation process, the calculation device 11 reads out the space information 141 stored in the storage device 13 and generates virtual space information 143 indicating the virtual space in which the virtual projection device, the virtual projection surface, and the virtual shooting device are arranged.

[0089] In the virtual projection process, the arithmetic device 11 uses the pattern image included in the image information 131, the parameter information 142, and the virtual space information 143 to project the pattern image onto a virtual projection surface from a virtual projection device placed in the virtual space indicated by the virtual space information 143. For example, as shown in FIG. 11 , in the virtual projection process, the image output from the virtual projection device 20′ onto the virtual projection surface 40′ is a “projected image Im1,” and the image projected onto the virtual projection surface 40′ is a “virtual projection surface image Im2′.” Furthermore, an image subsequently captured in the virtual space by the virtual imaging device 30′ is a “virtual captured image Im3′.”

[0090] In the virtual image acquisition process, the arithmetic device 11 uses the parameter information 142 and the virtual space information 143 to acquire first virtual shooting data 144 including a pattern image projected onto the virtual projection surface 40′ from the virtual shooting device 30′ in the virtual space indicated by the virtual space information 143. The arithmetic device 11 also stores the acquired first virtual shooting data 144 in the storage device 13.

[0091] In the virtual coordinate acquisition process, the calculation device 11 acquires the coordinates of feature points on the virtual projection surface 40' included in the first virtual shooting data 144, associates the acquired coordinates of each feature point with the first virtual shooting data 144 as virtual feature point coordinate data 145, and stores the data in the storage device 13.

[0092] In the feature point extraction process, the arithmetic device 11 compares the feature points included in the first virtual shooting data 144 with each feature point included in the pattern image to extract non-projected feature points that are not projected onto the virtual projection plane 40'. In addition, in the feature point extraction process, the arithmetic device 11 compares the feature points included in the first virtual shooting data 144 with each feature point included in the pattern image to extract projected feature points that are projected onto the virtual projection plane 40'. Specifically, the arithmetic device 11 compares the coordinates of each feature point included in the virtual feature point coordinate data 145 acquired in the virtual coordinate acquisition process with the coordinates of each feature point in the pattern image to determine whether each feature point of the pattern image is included on the virtual projection plane 40' of the first virtual shooting data 144. When a virtual feature point is included on the virtual projection plane 40' of the first virtual shooting data 144, the arithmetic device 11 designates the virtual feature point as a "projected feature point." On the other hand, when a feature point is not included on the virtual projection surface 40' of the first virtual shooting data 144, the calculation device 11 designates the feature point as a "non-projected feature point." In other words, of the feature points included in the pattern image, the feature points projected onto the virtual projection surface 40' are designated as "projected feature points." On the other hand, of the feature points included in the pattern image, the feature points not projected onto the virtual projection surface 40' are designated as "non-projected feature points." The calculation device 11 adds the result of the feature point extraction process to the feature point coordinate data 133 stored in the storage device 13.

[0093] In the coordinate correspondence processing, the arithmetic device 11 associates the coordinates of each feature point of the pattern image projected by the virtual projection device 20', the coordinates of each feature point of the pattern image projected onto the virtual projection plane 40', and the coordinates of each feature point photographed by the photographing device 30. For example, the arithmetic device 11 stores, in virtual feature point coordinate data 145 in the storage device 13, the correspondence between the coordinates of the projected and non-projected feature points on the projection image Im1, the coordinates of the projected feature points on the virtual projection plane image Im2', and the coordinates of the projected feature points on the virtual photographed image Im3'. The arithmetic device 11 executes the coordinate correspondence processing as described above with reference to FIG. 4.

[0094] In the generation process, the calculation device 11 generates projection feature point data 137 in which the multiple projection feature points extracted in the extraction process are arranged at corresponding positions on a virtual projection surface 40' captured by a virtual imaging device 30'.

[0095] In the generation process, the calculation device 11 generates non-projected feature point data 138 in which the multiple non-projected feature points extracted in the extraction process are arranged at corresponding positions within the range of the virtual projection image projected by the virtual projection device 20′.

[0096] In addition, in the virtual projection process, the calculation device 11 uses the marker image included in the image information 131, the parameter information 142, and the virtual space information 143 to project the marker image from the virtual projection device 20' indicated by the virtual space information 143 onto the virtual projection surface 40'.

[0097] Furthermore, in the virtual image acquisition process, the arithmetic device 11 uses the parameter information 142 and the virtual space information 143 to acquire second virtual shooting data 146 including marker images projected from the virtual shooting device 30′ onto a virtual projection surface 40′ on which a plurality of placement reference markers corresponding to the plurality of projection reference markers are arranged, in the virtual space indicated by the virtual space information 143. Furthermore, the arithmetic device 11 stores the acquired second virtual shooting data 146 in the storage device 13.

[0098] Furthermore, in the generation process, the calculation device 11 generates a coordinate conversion table 135 for projecting non-projected feature points onto the virtual projection surface 40′ using a comparison result between the positions of the plurality of placement reference markers and the positions of the plurality of projection reference markers included in the second virtual shooting data 146. Because the virtual space is an imitation of the real space, by using the coordinate conversion table 135 generated here, it is possible to project image data from the projection device 20 onto the projection surface 40 in the real space corresponding to the virtual space.

[0099] In addition, in the generation process, the calculation device 11 uses the coordinate conversion table 135 to generate visualized image data 136 that visualizes the projected feature points and the non-projected feature points.

[0100] Furthermore, in the adjustment process, the calculation device 11 may adjust the image data using the coordinate conversion table 135 so that the entire image data is projected onto the virtual projection surface 40'. As described above, the image data adjusted by the coordinate conversion table 135 can be projected onto the virtual projection surface 40', and can also be projected onto the projection surface 40.

[0101] Furthermore, in the virtual projection process, the arithmetic device 11 may project adjusted image data from the virtual projection device 20' onto the virtual projection surface 40'. Specifically, the arithmetic device 11 can generate a virtual space in which desired image data is projected onto the virtual projection surface 40' by projecting the generated adjusted image data. Furthermore, the arithmetic device 11 displays the virtual captured image on the display device 14, allowing the user to confirm it.

[0102] In the reception process, the arithmetic unit 11 receives a designation of a predetermined display range. In addition, in the reception process, the arithmetic unit 11 receives a designation of a transformation condition.

[0103] In the determination process, the arithmetic unit 11 determines whether the display using the coordinate conversion table 135 is within a predetermined display range that is permitted for coordinate conversion. Also, in the determination process, the arithmetic unit 11 determines whether the display using the coordinate conversion table 135 satisfies predetermined transformation conditions that permit image transformation.

[0104] In the output process, the arithmetic unit 11 outputs the determination result.

[0105] <<Verification Method>> The verification method according to the present disclosure will be described using the flowcharts shown in FIGS. 12A to 12C. First, the arithmetic device 11 generates virtual space information indicating a virtual space in which the virtual projection device 20', the virtual projection surface 40', and the virtual image capturing device 30' are installed (S101).

[0106] Furthermore, the position of the virtual projection device 20' in the virtual space is adjusted (S102).

[0107] Next, the position of the virtual camera 30' in the virtual space is adjusted (S103).

[0108] Thereafter, the arithmetic unit 11 causes the virtual projection device 20' to project a pattern image onto the virtual projection surface 40' (S104).

[0109] The calculation device 11 also acquires first virtual shooting data 144 from the virtual image capturing device 30' (S105). The first virtual shooting data 144 includes a pattern image projected onto the virtual projection surface 40' from the virtual projection device 20'.

[0110] Furthermore, the arithmetic unit 11 acquires the coordinates of a plurality of feature points projected onto the virtual projection surface 40' from the first virtual photographed data 144 acquired in step S105 (S106).

[0111] Next, the arithmetic unit 11 operates the virtual projection device 20' to project the marker image onto the virtual projection surface 40' (S107).

[0112] The computing device 11 also acquires second virtual shooting data 146 virtually captured by the virtual imaging device 30 (S108). The second virtual shooting data 146 includes a marker image projected onto the virtual projection surface 40′ from the virtual projection device 20′.

[0113] Furthermore, the calculation device 11 acquires the coordinates of a plurality of projection reference markers projected onto the virtual projection surface 40' from the second virtual imaging data 146 acquired in step S108 (S109).

[0114] Furthermore, the calculation device 11 acquires the coordinates of a plurality of placement reference markers arranged on the virtual projection plane 40' from the second virtual shooting data 146 acquired in step S108 (S110).

[0115] The calculation device 11 extracts non-projected feature points and projected feature points using the pattern image and the results obtained in step S106 (S011).

[0116] The calculation device 11 uses the coordinates of the placement reference markers in step S110 and the extraction result in step S111 to specify a projection range into which the projection image is projected from the virtual projection device 20' (S112).

[0117] The arithmetic unit 11 generates a coordinate conversion table 135 for converting the projection image projected from the virtual projection device 20' to fit the shape of the virtual projection surface 40' based on the projection range specified in step S112 (S113).

[0118] The arithmetic unit 11 receives the display range and the transformation conditions (S114).

[0119] The arithmetic unit 11 determines whether or not the image to be transformed using the coordinate conversion table 135 generated in step S113 satisfies the display range and transformation conditions accepted in step S114 (S115). If the image is not displayed within the display range and / or does not satisfy the transformation conditions (NO in S116), the arithmetic unit 11 returns to step S102 and repeats the processes of steps S003 to S015.

[0120] Thereafter, the arithmetic unit 11 generates the projected feature point data 137 in which the projected feature points are arranged, and displays it on the display unit 14 (S117).

[0121] Furthermore, the arithmetic unit 11 generates non-projected feature point data 138 in which the non-projected feature points are arranged, and displays it on the display unit 14 (S118).

[0122] Furthermore, the arithmetic unit 11 generates visualized image data 136 using the coordinate conversion table 135, and displays it on the display unit 14 (S119).

[0123] Furthermore, the arithmetic unit 11 transforms the image using the coordinate conversion table 135 generated in step S112 (S120).

[0124] The arithmetic unit 11 operates the projection device 20 to project the image transformed in step S119 onto the projection surface 40 (S121).

[0125] When the image projected onto the projection surface 40 is as expected, an end operation is performed, and the verification process in the projection system 1 ends (YES in S122).

[0126] On the other hand, if the image projected onto the projection surface 40' is not as expected (NO in S122), readjustment is necessary, so the process returns to step S102 and the processes of steps S102 to S122 are repeated. Specifically, after the virtual space is adjusted regarding the physical positions of the virtual projection device 20' and the virtual image capturing device 30', the verification process is executed again.

[0127] As described above, the image processing device 10A according to the second embodiment can generate a virtual space in which a virtual projection device is placed, even in the absence of a projection system, and can confirm information generated during the automatic geometric correction process, thereby facilitating the determination of the validity of the geometric correction performed by the projection device.

[0128] Overview of the embodiment (1) The verification method disclosed herein is a method for verifying geometric correction in a projection system that is executed by a computing device that can access a storage device and that projects an image onto a projection surface using a projection device, wherein the storage device stores a pattern image that includes a plurality of feature points that indicate coordinates on the image to be projected onto the projection surface by the projection device, the computing device projects the pattern image from the projection device onto the projection surface, the computing device acquires first captured image data from a capturing device that includes the pattern image projected onto the projection surface, compares the feature points included in the first captured image data with each feature point included in the pattern image to extract non-projected feature points that are not projected onto the projection surface, and generates non-projected feature point data in which the extracted non-projected feature points are positioned at corresponding positions within the range of the projection image projected by the projection device.

[0129] This makes it easier to determine the validity of the geometric correction performed by the projection device.

[0130] (2) In (1), the feature points included in the first captured image data may be compared with the feature points included in the pattern image to extract the projected feature points projected onto the projection surface, and projected feature point data may be generated in which the extracted multiple projected feature points are arranged at corresponding positions on the projection surface captured by the photographing device.

[0131] This makes it easier to determine the validity of the geometric correction performed by the projection device.

[0132] (3) In (2), the storage device may store a marker image including a plurality of projection reference markers indicating coordinates on an image to be projected onto the projection surface by the projection device, the calculation device may project the marker image onto the projection surface from the projection device, the second captured image data may be obtained from the photographing device, the second captured image data may include the marker image projected onto the projection surface on which a plurality of placement reference markers corresponding to the plurality of projection reference markers are arranged, and a coordinate conversion table may be generated for projecting the non-projected feature points onto the projection surface using a comparison result between the positions of the plurality of placement reference markers included in the second captured image data and the positions of the plurality of projection reference markers.

[0133] This makes it possible to generate a coordinate conversion table that allows the projection device to appropriately adjust and project onto the projection surface.

[0134] (4) In (3), visualized image data may be generated by visualizing the projected feature points and the non-projected feature points using a coordinate conversion table.

[0135] This allows the user to visually grasp the range of the image data that is projected onto the projection surface and the range that is not projected onto the projection surface.

[0136] (5) In (3) or (4), a coordinate conversion table may be used to generate adjusted image data adjusted so that the entire image data is projected onto the projection surface, and the adjusted image data may be projected onto the projection surface from the projection device.

[0137] This allows the user to check how the image data will be displayed using the coordinate conversion table.

[0138] (6) In (3) to (5), it may be determined whether the display using the coordinate conversion table is within a predetermined display range that is allowed for coordinate conversion, and the determination result may be output.

[0139] This makes it possible to check whether the image will be converted into a predetermined display range.

[0140] (7) In (6), the predetermined display range may be designated.

[0141] This allows the user to specify the display range.

[0142] (8) In (3) to (7), it may be determined whether the display using the coordinate conversion table satisfies a predetermined transformation condition under which transformation of the image is permitted, and the result of the determination may be output.

[0143] This makes it possible to check whether the change is within a range that satisfies the transformation conditions.

[0144] (9) In (8), specification of transformation conditions may be accepted.

[0145] This allows the user to specify the transformation conditions.

[0146] (10) A verification method of the present disclosure is a method for verifying geometric correction in a projection system that is executed by a computing device that can access a storage device and that projects an image onto a projection surface using a projection device, wherein the storage device stores spatial information of a real space in which the projection device, the projection surface, and an imaging device that can image the projection surface are arranged, parameter information set for the projection device and the imaging device, and a pattern image including a plurality of feature points that indicate coordinates on an image to be projected onto the projection surface by the projection device, and the computing device generates a virtual projection device in a virtual space that is virtually constructed based on the spatial information, in which virtual parameter values ​​based on the parameter information are set. and generate virtual space information including a state in which a virtual imaging device is placed, project a pattern image from the virtual projection device onto a virtual projection surface in the virtual space information, obtain virtual first captured image data including the pattern image captured by the virtual imaging device and projected onto the virtual projection surface, compare feature points included in the virtual first captured image data with each feature point included in the pattern image to extract non-projected feature points that are not projected onto the virtual projection surface, and generate non-projected feature point data in which the extracted multiple non-projected feature points are placed at corresponding positions within the range of the virtual projection image projected by the virtual projection device.

[0147] This makes it possible to verify the validity of the geometric correction performed by the projection device through simulation, even if the virtual system does not actually exist.

[0148] (11) In (10), the feature points included in the virtual first captured image data may be compared with each feature point included in the pattern image to extract projected feature points projected onto a virtual projection surface, and projected feature point data may be generated in which the extracted multiple projected feature points are arranged at corresponding positions on the virtual projection surface captured by a virtual imaging device.

[0149] This makes it possible to easily determine the validity of the geometric correction performed by the projection device through simulation.

[0150] (12) In (11), the storage device may store a marker image including a plurality of projection reference markers indicating coordinates on an image to be projected onto a projection surface by a projection device, project the marker image onto the virtual projection surface from a virtual projection device using a calculation device, acquire virtual second captured image data from a virtual capture device including the marker image projected onto a virtual projection surface on which a plurality of virtual placement reference markers corresponding to the plurality of projection reference markers are placed, and generate a coordinate conversion table for projecting feature points that are not projected onto the virtual projection surface onto the virtual projection surface using a comparison result between the positions of the plurality of virtual placement reference markers included in the virtual second captured image data and the positions of the plurality of projection reference markers.

[0151] This makes it possible to generate a coordinate conversion table by simulation that allows the projection device to appropriately adjust and project onto the projection surface.

[0152] (13) In (12), visualized image data may be generated by visualizing the projected feature points and the non-projected feature points using the coordinate conversion table.

[0153] This allows the simulation to visually grasp the range of the image data that is projected onto the projection surface and the range that is not projected.

[0154] (14) In (12) or (13), a coordinate conversion table may be used to generate adjusted image data adjusted so that the entire image data is projected onto a virtual projection surface, and the adjusted image data may be projected onto the virtual projection surface from a virtual projection device.

[0155] This allows the user to confirm, through simulation, how image data will be displayed using the coordinate conversion table.

[0156] (15) A computer program according to the present disclosure causes a computing device to execute the verification method according to any one of (1) to (14).

[0157] This makes it possible to verify the validity of the geometric correction performed by the projection device.

[0158] (16) The projection system of the present disclosure includes a computing device that can access a storage device, a projection device controlled by the computing device and that projects an image onto a projection surface, and a photographing device that photographs an image including the projection surface, wherein the storage device stores a pattern image including a plurality of feature points that indicate coordinates on the image to be projected onto the projection surface by the projection device, and the computing device, during verification of the projection system, projects the pattern image onto the projection surface from the projection device, obtains first photographed image data from the photographing device that includes the pattern image projected onto the projection surface, compares the feature points included in the first photographed image data with each feature point included in the pattern image, extracts non-projected feature points that are not projected onto the projection surface, and generates non-projected feature point data in which the extracted non-projected feature points are positioned at corresponding positions within the range of the projection image projected by the projection device.

[0159] This makes it possible to verify the validity of the geometric correction performed by the projection device.

[0160] The simulation device and simulation method described in all claims of the present disclosure are realized by hardware resources, such as a processor, a memory, and cooperation with a computer program. [Industrial Applicability]

[0161] The verification method, computer program, and projection system of the present disclosure are useful for realizing automatic geometric correction during projection by a projection device. [Explanation of symbols]

[0162] 1. Projection System 10 Image processing device 11 Arithmetic unit 12. Communications equipment 13 Storage device 20 Projection device 30 Imaging equipment 40 Projection plane

Claims

1. A method for verifying geometric correction in a projection system that projects an image onto a projection surface using a projection device, the method being executed by a computing device that can access a storage device, the method comprising: The storage device includes: storing a pattern image including a plurality of feature points indicating coordinates on the image to be projected onto the projection surface by the projection device; The computing device projecting the pattern image from the projection device onto the projection surface; acquiring first captured image data including the pattern image projected onto the projection surface from an imaging device; comparing feature points included in the first captured image data with feature points included in the pattern image to extract non-projection feature points that are not projected onto the projection surface; Generate non-projected feature point data in which the extracted non-projected feature points are arranged at corresponding positions within the range of the projection image projected by the projection device. Verification method.

2. comparing feature points included in the first captured image data with feature points included in the pattern image to extract projected feature points projected onto the projection surface; Generate projection feature point data in which the extracted projection feature points are arranged at corresponding positions on the projection surface photographed by the photographing device The verification method of claim 1 .

3. The storage device includes: storing a marker image including a plurality of projection reference markers indicating coordinates on the image to be projected onto the projection surface by the projection device; The computing device projecting the marker image from the projection device onto the projection surface; acquiring second photographed image data from the photographing device, the second photographed image data including the marker images projected onto the projection surface on which a plurality of placement reference markers respectively corresponding to the plurality of projection reference markers are arranged; Using a comparison result between the positions of the plurality of placement reference markers included in the second captured image data and the positions of the plurality of projection reference markers, a coordinate conversion table is generated for projecting the non-projection feature points onto the projection surface. The verification method of claim 2 .

4. Using the coordinate conversion table, visualized image data is generated by visualizing the projected feature points and the non-projected feature points. The verification method according to claim 3 .

5. Using the coordinate conversion table, adjusted image data is generated so that the entire image data is projected onto the projection surface; The adjusted image data is projected from the projection device onto the projection surface. The verification method according to claim 3 or 4.

6. determining whether the display using the coordinate conversion table is within a predetermined display range that is allowable for coordinate conversion; Output the judgment result The verification method according to any one of claims 3 to 5.

7. Accept the designation of the predetermined display range The verification method according to claim 6.

8. determining whether the display using the coordinate conversion table satisfies a predetermined transformation condition under which transformation of the image is permitted; Output the judgment result The verification method according to any one of claims 3 to 7.

9. Accept the specification of the transformation conditions The verification method of claim 8.

10. A method for verifying geometric correction in a projection system that projects an image onto a projection surface using a projection device, the method being executed by a computing device that can access a storage device, the method comprising: The storage device includes: spatial information of a real space in which the projection device, the projection surface, and an image capturing device capable of capturing an image of the projection surface are arranged; parameter information set in the projection device and the imaging device; a pattern image including a plurality of feature points indicating coordinates on an image to be projected onto the projection surface by the projection device; Remember, The computing device generating virtual space information including a state in which a virtual projection device and a virtual image capture device, to which virtual parameter values ​​based on the parameter information are set, are arranged in a virtual space virtually constructed based on the spatial information; projecting the pattern image from the virtual projection device onto a virtual projection surface in the virtual space information; acquiring virtual first captured image data including the pattern image captured by the virtual imaging device and projected onto the virtual projection surface; comparing feature points included in the virtual first captured image data with feature points included in the pattern image to extract non-projection feature points that are not projected onto the virtual projection surface; generating non-projected feature point data in which the extracted non-projected feature points are arranged at corresponding positions within a range of a virtual projection image projected by the virtual projection device; Verification method.

11. comparing feature points included in the virtual first captured image data with feature points included in the pattern image to extract projected feature points projected onto the virtual projection surface; generating projection feature point data in which the extracted projection feature points are arranged at corresponding positions on the virtual projection surface photographed by the virtual photographing device; The verification method of claim 10.

12. The storage device includes: storing a marker image including a plurality of projection reference markers indicating coordinates on the image to be projected onto the projection surface by the projection device; The computing device projecting the marker image from the virtual projection device onto the virtual projection surface; acquiring, from the virtual imaging device, virtual second captured image data including the marker images projected onto the virtual projection plane on which a plurality of virtual placement reference markers respectively corresponding to the plurality of projection reference markers are arranged; generating a coordinate conversion table for projecting the non-projection feature points onto the virtual projection plane using a comparison result between the positions of the plurality of virtual placement reference markers included in the virtual second captured image data and the positions of the plurality of projection reference markers; The verification method of claim 11.

13. Using the coordinate conversion table, visualized image data is generated by visualizing the projected feature points and the non-projected feature points. The verification method of claim 12.

14. Using the coordinate conversion table, adjusted image data is generated so that the entire image data is projected onto the virtual projection surface; The adjusted image data is projected from the virtual projection device onto the virtual projection surface.

14. The verification method according to claim 12 or 13.

15. A computer program causing the computing device to execute the verification method according to any one of claims 1 to 14.

16. A projection system including: a computing device that can access a storage device; a projection device that is controlled by the computing device and projects an image onto a projection surface; and an image capture device that captures an image including the projection surface, The storage device storing a pattern image including a plurality of feature points indicating coordinates on the image to be projected onto the projection surface by the projection device; The computing device, when verifying the projection system, projecting the pattern image from the projection device onto the projection surface; acquiring first captured image data including the pattern image projected onto the projection surface from an imaging device; comparing feature points included in the first captured image data with feature points included in the pattern image to extract non-projection feature points that are not projected onto the projection surface; Generate non-projected feature point data in which the extracted non-projected feature points are arranged at corresponding positions within the range of the projection image projected by the projection device. Projection system.

Images