Image processing method, information processing device, and program

Abstract

To correct an image with a simple device configuration.SOLUTION: An image processing method comprises the steps of: acquiring a first photographed image including a first point obtained by imaging an object by using an imaging element at first time; acquiring a second photographed image including a second point corresponding to the first point obtained by imaging the object by using the imaging element at the second time different from the first time; and outputting first output information being information for image processing on the basis of at least one of the first photographed image and the second photographed image when a first index value generated on the basis of coordinates of the first point in the first photographed image and coordinates of the second point in the second photographed image is equal to or less than a threshold.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an image processing method, an information processing apparatus, and a program. Background Art

[0002] Technologies relating to a method for correcting the shape, display position, and the like of an image projected from a projector have been developed. For example, Patent Literature 1 discloses a projector including a sensor that detects motion and a camera that captures an image of an imaging object. The projector determines whether to correct the projected image based on a difference between a captured image stored in a storage means and a captured image captured after the sensor detects motion of the projector. Prior Art Literature Patent Literature

[0003] Patent Literature 1 Japanese Unexamined Patent Publication No. 2009-253575 Summary of the Invention Problems to be Solved by the Invention

[0004] The projector described in Patent Literature 1 instructs the camera to perform imaging based on a signal output from a motion detection sensor. For this reason, the projector is required to include the sensor, which gives rise to the problem that the device configuration becomes complicated. Means for Solving the Problems

[0005] One aspect of the image processing method according to the present invention includes: acquiring a first image image including a first point obtained by imaging an object using an image sensor at a first time; acquiring a second image image including a second point corresponding to the first point obtained by imaging an object using an image sensor at a second time different from the first time; and outputting first output information, which is information for image processing, based on at least one of the first and second image images, when a first index value generated based on the coordinates of the first point in the first image image and the coordinates of the second point in the second image image is less than or equal to a threshold.

[0006] One embodiment of the information processing device according to the present invention comprises a processing device which performs the following: acquiring a first image containing a first point obtained by imaging an object using an image sensor at a first time; acquiring a second image containing a second point corresponding to the first point obtained by imaging an object using an image sensor at a second time different from the first time; and outputting first output information, which is information for image processing, based on at least one of the first image and the second image, when a first index value generated based on the coordinates of the first point in the first image and the coordinates of the second point in the second image is less than or equal to a threshold.

[0007] One aspect of the program according to the present invention causes a processing unit to perform the following actions: acquire a first image containing a first point obtained by imaging an object using an image sensor at a first time; acquire a second image containing a second point corresponding to the first point obtained by imaging an object using an image sensor at a second time different from the first time; and output first output information, which is information for image processing, based on at least one of the first and second image images, if a first index value generated based on the coordinates of the first point in the first image and the coordinates of the second point in the second image is less than or equal to a threshold value. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram illustrating the image display system Sys according to the embodiment. [Figure 2] This is a block diagram showing the configuration of the image display system Sys according to the embodiment. [Figure 3] This is a block diagram showing the configuration of the storage device 10 according to the embodiment. [Figure 4] This is a schematic diagram to explain image GF. [Figure 5] This is a schematic diagram illustrating the captured image GS1. [Figure 6] This is a schematic diagram illustrating the captured image GS2. [Figure 7] This is a schematic diagram to explain statistical coordinates. [Figure 8] This is a schematic diagram illustrating the captured image GS3. [Figure 9] This is a schematic diagram illustrating the captured image GS4. [Figure 10] This is a flowchart illustrating the operation of computer 1 according to the embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings. Note that the dimensions and scale of each part in the drawings may differ from those of the actual parts, and some parts are shown schematically for ease of understanding. Furthermore, the scope of the present invention is not limited to these forms unless otherwise stated in the following description. In addition, in this specification and the claims, when a numerical range is expressed using "Φ~Ψ" (where Φ and Ψ are both numerical values), the range includes the numerical values ​​of the upper limit (Ψ) and the lower limit (Φ). Furthermore, the units of the upper limit (Ψ) and the lower limit (Φ) are the same.

[0010] 1. Embodiment In this embodiment, an image display system comprising a projector that projects an image onto a wall surface of an object, a camera that captures images of the wall surface on which the projected image is displayed, and a computer that outputs information for correcting the projected image based on the captured images acquired from the camera will be used as an example to explain the image processing method, information processing device, and program according to the present invention.

[0011] 1.1. Overview of the Image Display System Figure 1 is a schematic diagram illustrating an image display system Sys according to an embodiment. The image display system Sys includes a projector 5 that projects a projected image GP onto a wall surface W of an object Ob by projecting light, a camera 3 that captures images of the wall surface W on which the projected image GP projected from the projector 5 is displayed, and a computer 1 that outputs information for correcting the shape and display position of the image projected from the projector 5 based on the captured image GS acquired from the camera 3. The computer 1 is communicated with the camera 3 and the projector 5.

[0012] The projected image GP has a pattern in which multiple black squares and multiple white squares are arranged alternately. The projected image GP includes multiple points DP. In Figure 1, points DP are points located at the positions where the vertices of four squares coincide, i.e., grid points. The multiple points DP include point DP1. Note that the pattern of the projected image GP is not limited to white and black, but may be a pattern consisting of two other colors. The pattern of the projected image GP is not limited to a pattern in which squares are arranged alternately, but any known pattern in which points DP can be detected can be used. It is preferable that the pattern of the projected image GP uses two colors with high contrast.

[0013] The camera 3 is disposed facing the wall surface W of the object Ob. The camera 3 acquires a captured image GS by capturing an image of the wall surface W on which the projection image GP is displayed. The captured image GS includes an image GVP. The image GVP is an image representing the projection image GP. Further, the image GVP includes a plurality of points DS. Each of the plurality of points DS has a one-to-one correspondence with each of the plurality of points DP included in the projection image GP. The camera 3 repeatedly captures images of the wall surface W on which the projection image GP is displayed, and sequentially outputs the acquired captured images GS to the computer 1. In the present embodiment, the camera 3 is not fixed and is disposed on a flat surface such as a desk. That is, the camera 3 may slightly move due to unintended vibration caused by, for example, a user contacting the desk.

[0014] The computer 1 acquires the captured image GS from the camera 3. Further, the computer 1 detects the points DS by performing image processing on the captured image GS. Further, the computer 1 causes the display device 16 included in the computer 1 to display the captured image GS. Further, the computer 1 causes the display device 16 to display the captured image GS in a state where an image GDS for indicating the positions of the detected points DS is superimposed on the captured image GS. The computer 1 repeatedly acquires captured images GS from the camera 3, and detects the points DS from each of the plurality of acquired captured images GS. Further, the computer 1 determines whether the degree of difference between the plurality of captured images GS is sufficiently small based on the coordinates of the plurality of detected points DS. When it is determined that the degree of difference between the plurality of captured images GS is small, the computer 1 outputs information for correcting the shape, display position, and the like of the image projected from the projector 5.

[0015] 1.2. Configuration and Functions of the Image Display System Hereinafter, the configuration and functions of the image display system Sys according to the embodiment will be described with reference to FIGS. 2 to 9.

[0016] FIG. 2 is a block diagram showing a configuration of an image display system Sys according to an embodiment. A computer 1 includes a storage device 10 that stores various types of information, a processing device 12 that controls the operation of the computer 1, an operation device 14 that receives input operations from a user of the computer 1, a display device 16 that displays images, and a communication device 18 that executes communication with a camera 3, a projector 5, an external storage device, an external server, or the like. The processing device 12 has functions as an acquisition unit 120, a detection unit 121, a determination unit 122, a display control unit 123, a data management unit 124, a statistics unit 125, and a correction unit 126. The camera 3 includes an imaging lens 30 and an image sensor 32.

[0017] The storage device 10 is configured to include, for example, volatile memory such as RAM and non-volatile memory such as ROM. Here, RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory.

[0018] Figure 3 is a block diagram showing the configuration of the storage device 10 according to the embodiment. The non-volatile memory of the storage device 10 stores a program 100 that defines the operation of the computer 1, projection image information 101 for forming the projection light projected when the projector 5 displays the projection image GP, projection grid point information 102 representing the arrangement and number of multiple points DF included in the image GF described later, imaging image information 103 representing the captured image GS, detection grid point information 108 representing the coordinates and number of multiple points DS detected from the captured image GS, statistical coordinate information 113 generated by statistical processing of the coordinates of points DS detected from the multiple captured images GS, and index value information 116 representing an index value used to evaluate the degree of difference between captured images GS. The captured image information 103 includes first captured image information 104, second captured image information 105, third captured image information 106, and fourth captured image information 107. The detected grid point information 108 includes the first detected grid point information 109, the second detected grid point information 110, the third detected grid point information 111, and the fourth detected grid point information 112. The statistical coordinate information 113 includes the first statistical coordinate information 114 and the second statistical coordinate information 115. The index value information 116 includes the first index value information 117 and the second index value information 118.

[0019] Furthermore, the volatile memory of the storage device 10 is used by the processing unit 12 as a work area when executing the program 100.

[0020] Furthermore, part or all of the storage device 10 may be located on an external storage device or an external server. Also, part or all of the various information stored in the storage device 10 may be stored in the storage device 10 in advance, or it may be obtained from an external storage device or an external server.

[0021] Figure 4 is a schematic diagram illustrating image GF. Image GF is the image shown by projection image information 101. Image GF has a pattern in which multiple black squares and multiple white squares are arranged alternately. Image GF contains multiple points DF. In Figure 4, points DF are points located at the positions where the vertices of four squares coincide, i.e., grid points. The multiple points DF include point DF1.

[0022] Projector 5 acquires projection image information 101 from computer 1 and projects projection light formed based on the projection image information 101 to display the projected image GP on the wall surface W. That is, image GF and projected image GP have similar patterns. Therefore, each of the multiple points DF corresponds one-to-one with each of the multiple points DP included in the projected image GP. For example, point DF1 and point DP1 are corresponding points. Also, the number of points DF included in image GF, the number of points DP included in projected image GP, and the number of points DS included in captured image GS are the same.

[0023] Returning to Figure 2, the processing unit 12 is composed of one or more CPUs. However, the processing unit 12 may also include a programmable logic device such as an FPGA instead of a CPU, or in addition to a CPU. Here, CPU is an abbreviation for Central Processing Unit, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0024] The processing unit 12 functions as the acquisition unit 120, detection unit 121, determination unit 122, display control unit 123, data management unit 124, statistics unit 125, and correction unit 126 shown in Figure 2, by having the CPU and other components of the processing unit 12 execute the program 100 and operating in accordance with the program 100.

[0025] The acquisition unit 120 controls the communication device 18 to acquire various information from external terminals such as the camera 3, projector 5, external storage device, or external server, which are connected to the computer 1 in a communicative manner. The acquisition unit 120 also stores the acquired information in the storage device 10. In this embodiment, the acquisition unit 120 acquires captured image information 103 from the camera 3. The acquisition unit 120 then stores the acquired captured image information 103 in the storage device 10.

[0026] The detection unit 121 performs image processing on the captured image GS to detect points DS contained in the captured image GS. In the function related to point DS detection, known image processing techniques such as pattern matching may be used. Detailed technical explanations regarding point DS detection are omitted in this specification.

[0027] The determination unit 122 performs various determinations. For example, the determination unit 122 determines whether all points DS included in the captured image GS have been detected based on the number of detected points DS, the coordinates of the detected points DS, and the projection grid point information 102.

[0028] The display control unit 123 controls the display device 16 to display the captured image GS. Furthermore, if all points DS contained in the captured image GS are detected, the display control unit 123 displays the image GDS superimposed on the captured image GS on the display device 16. As mentioned above, the image GDS is an image that indicates the positions of the detected points DS. Specifically, the display control unit 123 displays the image GDS superimposed on the positions of the detected points DS on the display device 16.

[0029] The data management unit 124 manages various types of information. For example, the data management unit 124 controls the storage device 10 to store detection grid point information 108, which represents the coordinates and number of multiple points DS detected from the captured image GS. The data management unit 124 also controls the storage device 10 to erase the stored detection grid point information 108.

[0030] The statistics unit 125 performs statistical processing on the coordinates of points DS detected from multiple captured images GS.

[0031] The statistical processing in the image display system Sys according to this embodiment will be described below with reference to Figures 5 to 7.

[0032] In the following, in the captured image GS, the upper left vertex of the image is defined as the origin, the direction parallel to the longitudinal direction of the captured image GS and extending to the right from the origin toward the image is defined as the X direction, and the direction parallel to the short direction of the captured image GS and extending downward from the origin is defined as the Y direction. Furthermore, the axis extending from the origin in the X direction is defined as the X-axis, and the axis extending from the origin in the Y direction is defined as the Y-axis. The coordinates of a point in the captured image GS are represented as (x,y). Here, the value x represents the X coordinate of the point in the captured image GS, and the value y represents the Y coordinate of the point in the captured image GS.

[0033] In this embodiment, the statistical processing performed by the statistical unit 125 is to calculate the average value of the X coordinate and the average value of the Y coordinate of point DS included in multiple captured images GS that corresponds to point DP included in the projection image GP. This statistical processing is sometimes referred to as "averaging processing". For example, the coordinates of point DS that corresponds to point DP1 in the nth captured image GS out of N captured images GS are (x1 n ,y1 n If we assume that point DP1 corresponds to point DS, then the average value of the X coordinates a of point DS included in N captured images GS is... x1 This is expressed by equation (1) shown below.

number

number

[0034] Figure 5 is a schematic diagram illustrating the captured image GS1. The captured image GS1 is the image indicated by the first captured image information 104. At time T1, the camera 3 acquires the captured image GS1 by imaging the wall surface W on which the projected image GP is displayed using the imaging lens 30 and the image sensor 32. The captured image GS1 includes image GVW1. Image GVW1 is an image representing the wall surface W. Image GVW1 also includes image GVP1. Image GVP1 is an image representing the projected image GP. Image GVP1 also includes multiple points DS1. Each of the multiple points DS1 corresponds one-to-one with each of the multiple points DP included in the projected image GP. The coordinates and number of the detected multiple points DS1 are stored in the storage device 10 as the first detected grid point information 109. The multiple points DS1 also include point DS11. Hereinafter, the coordinates of point DS11 will be represented as (x11, y11). Point DS11 corresponds to point DP1.

[0035] Figure 6 is a schematic diagram illustrating the captured image GS2. The captured image GS2 is the image indicated by the second captured image information 105. At time T2, the camera 3 acquires the captured image GS2 by imaging the wall surface W on which the projected image GP is displayed using the imaging lens 30 and the image sensor 32. The captured image GS2 includes image GVW2. Image GVW2 is an image representing the wall surface W. Image GVW2 also includes image GVP2. Image GVP2 is an image representing the projected image GP. Image GVP2 also includes multiple points DS2. Each of the multiple points DS2 corresponds one-to-one with each of the multiple points DP included in the projected image GP. The coordinates and number of the detected multiple points DS2 are stored in the storage device 10 as the second detected grid point information 110. The multiple points DS2 also include point DS21. Hereinafter, the coordinates of point DS21 will be represented as (x12, y12). Point DS21 corresponds to point DP1. That is, point DS21 corresponds to point DS11.

[0036] Figure 7 is a schematic diagram illustrating statistical coordinates. Specifically, Figure 7 shows the image GS2 superimposed on the image GS1. For the sake of explanation, images GVW1 and GVW2 are omitted from Figure 7.

[0037] In this embodiment, the coordinates indicated by the average X coordinates and the average Y coordinates obtained by statistical processing using the coordinates of point DS included in a plurality of captured images GS that correspond to point DP are referred to as "statistical coordinates." Point DR1 is the point indicated by the statistical coordinates obtained by statistical processing using the coordinates of point DS11 and point DS21 that correspond to point DP1. That is, the coordinates of point DR1 are obtained by substituting N=2 into equations (1) and (2) above, resulting in ((x11+x12) / 2,(y11+y12) / 2).

[0038] Statistical coordinate information 113 is information representing statistical coordinates. In other words, statistical processing is the process of generating statistical coordinate information 113 based on the coordinates of points DS included in multiple captured images GS that correspond to point DP. For example, the statistical unit 125 generates first statistical coordinate information 114 based on the coordinates of point DS11 and point DS21 that correspond to point DP1. First statistical coordinate information 114 is information representing the coordinates of point DR1.

[0039] The statistical unit 125 performs statistical processing for each point DP included in the projected image GP. That is, the statistical unit 125 performs statistical processing using the coordinates of each of the multiple points DP included in the projected image GP and the multiple points DS that correspond to them.

[0040] Furthermore, the statistics unit 125 calculates the distance between the statistical coordinates and the coordinates of point DS used in the statistical processing to determine the statistical coordinates. In this embodiment, this distance is referred to as the "index value." The index value information 116 is information representing the index value. In other words, the statistics unit 125 generates the index value information 116 based on the statistical coordinates and the coordinates of point DS used in the statistical processing to determine the statistical coordinates.

[0041] For example, when calculating an index value based on statistical coordinates obtained by statistical processing using the coordinates of point DS included in N captured images GS that correspond to point DP1, and the coordinates of point DS used in the statistical processing to obtain those statistical coordinates, a total of N index values ​​are obtained. The index value d1 is calculated based on those statistical coordinates and the coordinates of point DS included in the nth captured image GS out of the N captured images GS that correspond to point DP1. n This is expressed by equation (3) shown below.

number

[0042] When N=2, the point indicated by the statistical coordinates is the midpoint of the line segment whose endpoints are the two points used in the statistical processing, and therefore the two index values ​​obtained based on these statistical coordinates are equal. That is, the index value d11 calculated based on the coordinates of point DS11 and point DR1, and the index value d12 calculated based on the coordinates of point DS21 and point DR1 are equal. The first index value information 117 is information representing the index values ​​d12 and d11.

[0043] The processing unit 12 determines, based on the index value, whether the degree of difference between multiple captured image GSs is sufficiently small. If it is determined that the degree of difference between multiple captured image GSs is large, the processing unit 12 erases the detection grid point information 108 stored in the storage device 10 and acquires the captured image information 103 from the camera 3 again.

[0044] Figure 8 is a schematic diagram illustrating the captured image GS3. The captured image GS3 is the image indicated by the third captured image information 106. At time T3, the camera 3 acquires the captured image GS3 by imaging the wall surface W on which the projected image GP is displayed using the imaging lens 30 and the image sensor 32. The captured image GS3 includes image GVW3. Image GVW3 is an image representing the wall surface W. Image GVW3 also includes image GVP3. Image GVP3 is an image representing the projected image GP. Image GVP3 also includes multiple points DS3. Each of the multiple points DS3 corresponds one-to-one with each of the multiple points DP included in the projected image GP. The coordinates and number of the detected multiple points DS3 are stored in the storage device 10 as the third detected grid point information 111. The multiple points DS3 also include point DS31. Hereinafter, the coordinates of point DS31 will be represented as (x13, y13). Point DS31 corresponds to point DP1. That is, point DS31 corresponds to points DS11 and DS21.

[0045] Figure 9 is a schematic diagram illustrating the captured image GS4. The captured image GS4 is the image indicated by the fourth captured image information 107. At time T4, the camera 3 acquires the captured image GS4 by imaging the wall surface W on which the projected image GP is displayed using the imaging lens 30 and the image sensor 32. The captured image GS4 includes image GVW4. Image GVW4 is an image representing the wall surface W. Image GVW4 also includes image GVP4. Image GVP4 is an image representing the projected image GP. Image GVP4 also includes multiple points DS4. Each of the multiple points DS4 corresponds one-to-one with each of the multiple points DP included in the projected image GP. The coordinates and number of the detected multiple points DS4 are stored in the storage device 10 as the fourth detected grid point information 112. The multiple points DS4 also include point DS41. Hereinafter, the coordinates of point DS41 will be represented as (x14, y14). Point DS41 corresponds to point DP1. That is, point DS41 corresponds to points DS11, DS21, and DS31.

[0046] If the processor determines that there is a large difference between multiple captured images GS, the processor 12 detects points DS from captured images GS3 and GS4. The processor 12 also generates statistical coordinate information 113 based on the detected points DS. The processor 12 also generates index value information 116 based on the statistical coordinates indicated in the statistical coordinate information 113 and the coordinates of the points DS used in the statistical processing to obtain those statistical coordinates. For example, the processor 12 generates second statistical coordinate information 115 based on the coordinates of point DS31 and point DS41. The second statistical coordinate information 115 represents the statistical coordinates obtained by statistical processing using the coordinates of point DS31 and point DS41. The processor 12 also generates second index value information 118 based on the statistical coordinates indicated in the second statistical coordinate information 115 and the coordinates of point DS41. The second index value information 118 represents the distance between the statistical coordinates indicated in the second statistical coordinate information 115 and the coordinates of point DS41.

[0047] If the degree of difference between multiple captured images GS is determined to be small, the processing unit 12 outputs information representing the coordinates of points included in the captured images GS. If the output information is based on at least one of captured images GS1 and GS2, it may be referred to as "first output information." If the output information is based on at least one of captured images GS3 and GS4, it may be referred to as "second output information." The first output information and the second output information are information for correcting the shape and display position of the projected image GP.

[0048] In this embodiment, statistical coordinate information 113 is output as information representing the coordinates of points included in the captured image GS. Note that the output information is not limited to statistical coordinate information 113, but may also be, for example, detection grid point information 108. In this embodiment, the output information is stored in the storage device 10.

[0049] Returning to Figure 2, the correction unit 126 deforms the shape of image GF by editing the projected image information 101. Subsequently, the projector 5 acquires the projected image information 101 edited by the correction unit 126 from the computer 1 and projects the light formed based on the edited projected image information 101, thereby deforming the shape of the projected image GP displayed on the wall surface W. In other words, the correction unit 126 corrects the shape of the projected image GP displayed on the wall surface W. Note that the correction unit 126 can correct the shape of any image projected from the projector 5 onto the wall surface W, not just the projected image GP. Furthermore, in this embodiment, the correction unit 126 corrects the shape of the projected image GP based on the statistical coordinate information 113.

[0050] The operating device 14 is an input interface that receives input operations from the user of computer 1 to computer 1. The operating device 14 is, for example, a keyboard or pointing device provided on computer 1. For example, if the operating device 14 includes a keyboard, the operating device 14 outputs data identifying the pressed button to the processing device 12. In this way, the content of the input operation to computer 1 is transmitted to the processing device 12.

[0051] The display device 16 includes a display panel for displaying images. The display device 16 displays images by controlling multiple pixel circuits provided by the display control unit 123. For example, if the display device 16 includes a liquid crystal panel, each of the multiple pixel circuits is equipped with a liquid crystal element, and the image is displayed by adjusting the transmittance of light emitted from the backlight using the liquid crystal element. Alternatively, if the display device 16 includes an organic EL panel, each of the multiple pixel circuits is equipped with a light-emitting element, and the image is displayed by adjusting the intensity of light emitted from the light-emitting element. Here, EL is an abbreviation for Electro-Luminescence.

[0052] The communication device 18 includes, for example, an interface board having a connector and an interface circuit, and has the function of receiving various information from an external terminal, an external storage device or an external server, and the function of transmitting various information to an external terminal, an external storage device or an external server, etc. In this embodiment, the communication device 18 is communicably connected to the camera 3 and the projector 5, and transmits and receives various information between the camera 3 and the projector 5.

[0053] Camera 3 includes an imaging lens 30 for focusing light and an image sensor 32 that converts the focused light into an electrical signal. The image sensor 32 is an image sensor, such as a CCD or CMOS. Here, CCD is an abbreviation for Charge Coupled Device, and CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. Camera 3 acquires an image GS by imaging the wall surface W on which the projected image GP is displayed. Camera 3 also outputs image information 103 representing the acquired image GS to computer 1.

[0054] Projector 5 comprises a light source device including a halogen lamp, xenon lamp, ultra-high pressure mercury lamp, LED or laser light source, an optical modulator that generates image light, and a projection optical system that projects the image light. The optical modulator includes a DMD or liquid crystal panel. The projection optical system also includes a projection lens. Here, LED is an abbreviation for Light Emitting Diode, and DMD is an abbreviation for Digital Mirror Device. Projector 5 displays a projected image GP on the wall surface W by projecting projection light formed based on the projected image information 101.

[0055] 1.3. Computer Operation Figure 10 is a flowchart illustrating the operation of computer 1 according to this embodiment. The series of operations shown in the flowchart of Figure 10 are initiated, for example, when computer 1 is powered on and receives an input operation from the user to start the operation. It is assumed that during the execution of the series of operations shown in the flowchart of Figure 10, projector 5 is displaying a projected image GP on the wall surface W. That is, the projected image information 101 is output from computer 1 to projector 5 in advance before the execution of the series of operations shown in the flowchart of Figure 10.

[0056] In step S101, the acquisition unit 120 acquires captured image information 103 from the camera 3 by controlling the communication device 18. In other words, the acquisition unit 120 acquires the captured image GS from the camera 3 by controlling the communication device 18. The acquisition unit 120 then stores the acquired captured image information 103 in the storage device 10.

[0057] In step S102, the detection unit 121 performs image processing on the captured image GS indicated by the captured image information 103 and detects points DS, which are grid points included in the captured image GS.

[0058] In step S103, the determination unit 122 determines whether the detection of grid points was successful. Specifically, the determination unit 122 determines whether all points DS included in the captured image GS have been detected based on the number of detected points DS, the coordinates of the detected points DS, and the projection grid point information 102. If all points DS included in the captured image GS have been detected, that is, if the result in step S103 is YES, the determination unit 122 proceeds to step S104. If not all points DS included in the captured image GS have been detected, that is, if the result in step S103 is NO, the determination unit 122 proceeds to step S105.

[0059] In step S104, the display control unit 123 controls the display device 16 to display the captured image GS. The display control unit 123 also displays an image GDS, which indicates the position of the detected point DS, on the display device 16, superimposed on the position of the detected point DS.

[0060] In step S105, the display control unit 123 controls the display device 16 to display the captured image GS.

[0061] If not all points DS included in the captured image GS are detected, the display control unit 123 displays the captured image GS on the display device 16 without superimposing the image GDS.

[0062] In step S106, the data management unit 124 controls the storage device 10 to store detection grid point information 108, which represents the coordinates and number of multiple points DS detected from the captured image GS.

[0063] In step S107, the determination unit 122 determines whether the number of detection grid point information 108 stored in the storage device 10 is N or more. If the number of detection grid point information 108 stored in the storage device 10 is N or more, that is, if the result in step S107 is YES, the determination unit 122 proceeds to step S108. If the number of detection grid point information 108 stored in the storage device 10 is less than N, that is, if the result in step S107 is NO, the determination unit 122 proceeds to step S101.

[0064] Until the required number of captured images GS for obtaining statistical coordinates are available, the processing unit 12 acquires captured image information 103 from the camera 3 and detects points DS from the captured images GS indicated by the captured image information 103. For example, when performing statistical processing using the coordinates of multiple points DS contained in two captured images GS, N=2.

[0065] In step S108, the statistics unit 125 performs statistical processing on the coordinates of point DS detected from N captured images GS. That is, the statistics unit 125 obtains statistical coordinate information 113 based on the coordinates of point DS included in the N captured images GS that correspond to point DP.

[0066] In step S109, the statistics unit 125 generates index value information 116 based on the statistical coordinates indicated by the statistical coordinate information 113 and the coordinates of point DS used in the statistical processing to determine the said statistical coordinates.

[0067] In step S110, the determination unit 122 determines whether the index value indicated by the index value information 116 is below a threshold. If the index value indicated by the index value information 116 is below a threshold, that is, if the result in step S110 is YES, the determination unit 122 proceeds to step S112. If the index value indicated by the index value information 116 is greater than the threshold, that is, if the result in step S110 is NO, the determination unit 122 proceeds to step S111.

[0068] If the index value indicated by index value information 116 is below the threshold, the degree of difference between multiple captured images GS is considered to be sufficiently small. Conversely, if the index value indicated by index value information 116 is greater than the threshold, the degree of difference between multiple captured images GS is considered to be large. Furthermore, if N=2 and point DS is detected from captured image GS1 and captured image GS2, the determination unit 122 determines whether the index value indicated by first index value information 117 is below the threshold. Furthermore, if N=2 and point DS is detected from captured image GS3 and captured image GS4, the determination unit 122 determines whether the index value indicated by second index value information 118 is below the threshold. Note that a value such as 3 pixels is set as the threshold in the determination in step S110.

[0069] In step S111, the data management unit 124 erases the stored detection grid point information 108 by controlling the storage device 10.

[0070] If the result of the determination in step S103 is NO, for example, the captured image GS may be blurred because the camera 3 moved significantly when capturing the projected image GP. Also, if the result of the determination in step S110 is NO, the degree of difference between the multiple captured images GS is large, which may indicate that the camera 3 moved or the direction in which the projector 5 projects light changed when acquiring the captured image GS. In such cases, the computer 1 resets the detection grid point information 108 stored in the storage device 10 and acquires the captured image information 103 from the camera 3 again.

[0071] In step S112, the data management unit 124 outputs statistical coordinate information 113. The data management unit 124 also controls the storage device 10 to store the statistical coordinate information 113.

[0072] If the statistical coordinate information 113 output in step S112 includes first statistical coordinate information 114, then the first statistical coordinate information 114 is the first output information. Also, if the statistical coordinate information 113 output in step S112 includes second statistical coordinate information 115, then the second statistical coordinate information 115 is the second output information.

[0073] After the processing in step S112 is completed, the processing unit 12 completes the series of operations shown in the flowchart of Figure 10.

[0074] When the series of operations shown in the flowchart of Figure 10 are completed, the correction unit 126 corrects the shape of the projected image GP displayed on the wall surface W by editing the projected image information 101 based on the statistical coordinate information 113. Specifically, when the first output information is output, the correction unit 126 corrects the shape of the projected image GP displayed on the wall surface W by editing the projected image information 101 based on the first output information. Also, when the second output information is output, the correction unit 126 corrects the shape of the projected image GP displayed on the wall surface W by editing the projected image information 101 based on the second output information.

[0075] Based on the above, according to the embodiment, the computer 1 can determine whether the degree of difference between multiple captured images GS is sufficiently small based on the coordinates of points DS detected from the multiple captured images GS. In other words, if the degree of difference between multiple captured images GS is sufficiently small, the computer 1 can correct the shape of the projected image GP with high precision based on the coordinates of points DS detected when high-precision imaging is performed.

[0076] Furthermore, according to the embodiment, the image display system Sys can correct the projected image GP even when the computer 1, camera 3, and projector 5 are separated. Moreover, the image display system Sys can correct the projected image GP without using sensors to detect the movement of the projector 5 or camera 3. In other words, since the image display system Sys can omit sensors, the configuration of the device can be simplified, and the manufacturing cost of the product can be reduced. In addition, since the image display system Sys can correct the projected image GP based on the captured image GS without using sensors, even if a difference occurs between multiple captured images GS due to the movement of the object Ob being captured, for example, the shape of the projected image GP can be appropriately corrected.

[0077] As described above, the image processing method according to the embodiment includes: acquiring an image GS1 containing point DS11 obtained by imaging object Ob using the image sensor 32 at time T1; acquiring an image GS2 containing point DS21 corresponding to point DS11 obtained by imaging object Ob using the image sensor 32 at a time T2 different from time T1; and outputting first output information, which is information for image processing, based on at least one of the image GS of the image GS1 and the image GS2, when the index value d12 generated based on the coordinates of point DS11 in the image GS1 and the coordinates of point DS21 in the image GS2 is less than or equal to a threshold.

[0078] Furthermore, the computer 1 according to this embodiment includes a processing unit 12, which performs the following: acquiring an image GS1 containing point DS11 obtained by imaging object Ob using the image sensor 32 at time T1; acquiring an image GS2 containing point DS21 corresponding to point DS11 obtained by imaging object Ob using the image sensor 32 at a time T2 different from time T1; and outputting first output information, which is information for image processing, based on at least one of the image GS1 and GS2, if the index value d12 generated based on the coordinates of point DS11 in the image GS1 and the coordinates of point DS21 in the image GS2 is less than or equal to a threshold.

[0079] Furthermore, the program 100 according to the embodiment causes the processing unit 12 to perform the following actions: acquire an image GS1 containing point DS11 obtained by imaging object Ob using the image sensor 32 at time T1; acquire an image GS2 containing point DS21 corresponding to point DS11 obtained by imaging object Ob using the image sensor 32 at a time T2 different from time T1; and output first output information, which is information for image processing, based on at least one of the image GS1 and GS2, if the index value d12 generated based on the coordinates of point DS11 in the image GS1 and the coordinates of point DS21 in the image GS2 is less than or equal to a threshold.

[0080] In other words, the image processing method, computer 1, and program 100 according to this embodiment selectively use the image GS obtained when the degree of difference between multiple captured images GS is small, or in other words, when high-precision imaging is performed, to acquire and output information for image correction. As a result, the image display system Sys can output information for image processing without using special devices such as motion detection sensors, and can perform image correction with a simple device configuration.

[0081] In this embodiment, time T1 is an example of a "first time," image sensor 32 is an example of an "image sensor," object Ob is an example of an "object," point DS11 is an example of a "first point," captured image GS1 is an example of a "first captured image," time T2 is an example of a "second time," point DS21 is an example of a "second point," captured image GS2 is an example of a "second captured image," index value d12 is an example of a "first index value," captured image GS is an example of an "captured image," computer 1 is an example of an "information processing device," program 100 is an example of a "program," and processing device 12 is an example of a "processing device." Furthermore, "first output information" is exemplified by first statistical coordinate information 114.

[0082] Furthermore, in the image processing method according to the embodiment, object Ob includes a wall surface W on which the projected image GP projected from projector 5 is displayed, the projected image GP includes point DP1 corresponding to points DS11 and DS21, and the first output information indicates the coordinates of one or more points included in the captured image GS.

[0083] In other words, because the projected image GP includes point DP1 corresponding to points DS11 and DS21, computer 1 can understand the correspondence between point DS in the captured image GS and point DP in the projected image GP. This allows computer 1 to accurately determine whether or not point DS has been detected. Furthermore, computer 1 uses the captured image GS, which is an image of the wall W on which the projected image GP projected from projector 5 is displayed, to determine the degree of difference between multiple captured images GS. This makes it possible for computer 1 to detect changes in the captured image GS due to the movement of camera 3, changes in the captured image GS due to the movement of projector 5, and changes in the captured image GS due to the movement of object Ob, which has the wall W. Therefore, computer 1 can also detect changes in the captured image GS caused by factors other than the movement of projector 5, thereby preventing a decrease in accuracy when acquiring the captured image GS.

[0084] In this embodiment, projector 5 is an example of a "projector," projected image GP is an example of a "projected image," wall surface W is an example of a "display surface," and point DP1 is an example of a "third point." Also, "one or more points" refers to point DR1 as an example.

[0085] Furthermore, in the image processing method according to the embodiment, the index value d12 is generated based on the coordinates of point DR1 obtained by statistical processing using the coordinates of point DS11 and point DS21, and the coordinates of point DS21.

[0086] In other words, computer 1 determines the criteria for evaluating the degree of difference between multiple captured image GSs based on statistical processing using the coordinates of points included in the captured image GSs being evaluated. This allows computer 1 to appropriately set the criteria for evaluating the degree of difference between multiple captured image GSs, enabling accurate evaluation.

[0087] In this embodiment, the coordinates of point DR1 are an example of "statistical coordinates".

[0088] Furthermore, in the image processing method according to this embodiment, the statistical processing is an averaging process using the coordinates of point DS11 and point DS21.

[0089] This allows computer 1 to appropriately set evaluation criteria even when there are two captured images GS to be evaluated for the degree of difference.

[0090] Furthermore, the image processing method according to the embodiment further includes: deleting the first detection grid point information 109 and the second detection grid point information 110 when the index value d12 is greater than a threshold; acquiring an imaged image GS3 including point DS31 obtained by imaging object Ob using the image sensor 32 at a time T3 different from times T1 and T2; acquiring an imaged image GS4 including point DS41 corresponding to point DS31 obtained by imaging object Ob using the image sensor 32 at a time T4 different from times T1, T2 and T3; and outputting second output information, which is information for image processing, based on at least one of the imaged image GS3 and the imaged image GS4, when the index value indicated by the second index value information 118 generated based on the coordinates of point DS31 in the imaged image GS3 and the coordinates of point DS41 in the imaged image GS4 is less than or equal to a threshold.

[0091] In other words, if the degree of difference between multiple captured image GSs is not small, or in other words, if high-precision imaging could not be performed, computer 1 erases the information indicating the coordinates of the detected points and acquires the captured image again. As a result, computer 1 can always acquire and output information for performing image correction using the captured image GSs obtained when high-precision imaging was performed.

[0092] In this embodiment, the first detected grid point information 109 is an example of "information representing the coordinates of the first point," the second detected grid point information 110 is an example of "information representing the coordinates of the second point," time T3 is an example of "the third time," point DS31 is an example of "the fourth point," captured image GS3 is an example of "the third captured image," time T4 is an example of "the fourth time," point DS41 is an example of "the fifth point," captured image GS4 is an example of "the fourth captured image," and the index value indicated by the second index value information 118 is an example of "the second index value." Furthermore, "second output information" is exemplified by the second statistical coordinate information 115.

[0093] Furthermore, in the image processing method according to the embodiment, object Ob includes a wall surface W on which the projected image GP projected from projector 5 is displayed, and further includes correcting the shape of the projected image GP based on first output information or second output information.

[0094] This allows computer 1 to accurately correct the image projected from projector 5.

[0095] 2. Variations The above embodiments can be modified in various ways. Specific examples of modifications are given below. Furthermore, two or more embodiments arbitrarily selected from the following examples can be combined as appropriate, within the bounds of mutual consistency. In the modified examples given below, for elements whose operation and function are equivalent to those in the above embodiments, the same reference numerals used in the above description will be reused, and detailed explanations of each will be omitted as appropriate.

[0096] 2.1. Variation 1 In the embodiments described above, an example was given in which averaging was performed as a statistical process, but the present invention is not limited to this embodiment. For example, in the case of N≧3, instead of averaging, a process may be performed to obtain the median X coordinate and the median Y coordinate of a point DS included in a plurality of captured images GS that corresponds to a point DP included in the projected image GP. Alternatively, the coordinates indicated by the obtained median X coordinate and median Y coordinate may be used as statistical coordinates. That is, by outputting statistical coordinate information 113 representing these statistical coordinates, the shape of the projected image displayed on the wall surface W by the projected light projected from the projector 5 may be corrected.

[0097] Furthermore, if N ≥ 3, outliers may be excluded before averaging. Outlier detection may be performed, for example, based on the mean value of the coordinates of multiple points DS that are subject to statistical processing and the standard deviation of the coordinates of those points DS.

[0098] Regarding statistical processing methods, various methods can be applied, not limited to those described above. Furthermore, information representing statistical coordinates obtained through statistical processing methods other than those described above can be output and used for image correction. 2.2. Variation Example 2 In the embodiments and modified examples described above, the image display system Sys is exemplified as comprising a computer 1, a camera 3, and a projector 5. However, the present invention is not limited to such embodiments. For example, by using a projector having similar functions to the computer 1 and projector 5 instead of the computer 1 and projector 5, the configuration of the image display system can be made simpler. [Explanation of Symbols]

[0099] 1...Computer, 3...Camera, 5...Projector, 10...Storage device, 12...Processing device, 14...Operating device, 16...Display device, 18...Communication device, 30...Imaging lens, 32...Image sensor, 100...Program, 101...Projection image information, 102...Projection grid point information, 103...Imagine image information, 104...First image image information, 105...Second image image information, 106...Third image image information, 107...Fourth image image information, 108...Detection grid point information, 109...First detection grid point information, 110...Second detection Grid point information, 111...3rd detection grid point information, 112...4th detection grid point information, 113...Statistical coordinate information, 114...1st statistical coordinate information, 115...2nd statistical coordinate information, 116...Index value information, 117...1st index value information, 118...2nd index value information, 120...Acquisition unit, 121...Detection unit, 122...Determination unit, 123...Display control unit, 124...Data management unit, 125...Statistical unit, 126...Correction unit, GF...Image, GP...Projected image, GS...Acquired image, DF·DP·DS...Point, Sys...Image display system.

Claims

1. A camera that captures images of a display surface on which a projected image having a known pattern projected from a projector is displayed, and an image processing method for outputting information for correcting at least one of the shape of the image projected from the projector and the display position based on the captured images acquired from the camera, To obtain a first image image containing a first point detectable by the pattern, obtained by imaging the display surface using an image sensor at a first time point, Obtaining a second image image, which includes a second point corresponding to the first point, obtained by imaging the display surface using the image sensor at a second time different from the first time; If a first index value indicating the degree of difference between the first and second captured images, generated based on the coordinates of the first point in the first captured image and the coordinates of the second point in the second captured image, is less than or equal to a threshold, then first output information, which is information for correction, is output based on at least one of the first and second captured images. including, Image processing methods.

2. The projected image includes a third point corresponding to the first point and the second point, The first output information indicates the coordinates of one or more points included in the captured image. The image processing method according to claim 1.

3. The first index value is generated based on statistical coordinates obtained by statistical processing using the coordinates of the first point and the coordinates of the second point, and the coordinates of the second point. The image processing method according to claim 1 or 2.

4. The aforementioned statistical processing is an averaging process using the coordinates of the first point and the coordinates of the second point. The image processing method according to claim 3.

5. If the first index value is greater than the threshold, Deleting the information representing the coordinates of the first point and the information representing the coordinates of the second point, To obtain a third image image including a fourth point detectable by the pattern, obtained by imaging the display surface using the image sensor at a third time different from the first and second times, To obtain a fourth image image including a fifth point corresponding to the fourth point, obtained by imaging the display surface using the image sensor at a fourth time different from the first, second, and third times, If a second index value indicating the degree of difference between the third and fourth captured images, generated based on the coordinates of the fourth point in the third captured image and the coordinates of the fifth point in the fourth captured image, is less than or equal to the threshold, then a second output information, which is information for the correction, is output based on at least one of the third and fourth captured images. This also includes, The image processing method according to claim 1 or 2.

6. Correcting at least one of the shape and display position of the image projected from the projector based on the first output information or the second output information, This also includes, The image processing method according to claim 5.

7. Equipped with a processing device, The aforementioned processing apparatus is Based on an image captured by a camera that captures the display surface on which a projected image having a known pattern is displayed, information is output to correct at least one of the shape and display position of the image projected from the projector. To obtain a first image image containing a first point detectable by the pattern, obtained by imaging the display surface using an image sensor at a first time point, Obtaining a second image image, which includes a second point corresponding to the first point, obtained by imaging the display surface using the image sensor at a second time different from the first time; If a first index value indicating the degree of difference between the first and second captured images, generated based on the coordinates of the first point in the first captured image and the coordinates of the second point in the second captured image, is less than or equal to a threshold, then first output information, which is information for correction, is output based on at least one of the first and second captured images. Execute Information processing device.

8. In the processing unit, Based on an image captured by a camera that captures the display surface on which a projected image having a known pattern is displayed, information is output to correct at least one of the shape and display position of the image projected from the projector. To obtain a first image image containing a first point detectable by the pattern, obtained by imaging the display surface using an image sensor at a first time point, Obtaining a second image image, which includes a second point corresponding to the first point, obtained by imaging the display surface using the image sensor at a second time different from the first time; If a first index value indicating the degree of difference between the first and second captured images, generated based on the coordinates of the first point in the first captured image and the coordinates of the second point in the second captured image, is less than or equal to a threshold, then first output information, which is information for correction, is output based on at least one of the first and second captured images. To execute program.

Images