Processing method, information processing device, and processing program

Abstract

When calculating a correction value to correct the shape of an image based on multiple captured images obtained by capturing a pattern image projected onto a screen with a camera, the processing load for determining whether or not the pattern image is contained within the imaging range is reduced, while calculating an accurate correction value. [Solution] The information processing device 10 includes a first processing unit 140a that performs processing related to the first image capture when acquiring multiple captured images by capturing a pattern image projected onto a screen with a camera, and a second processing unit 140b that performs processing related to the second image capture. The first processing unit 140a determines whether or not the pattern image is contained within the imaging range by edge detection. The second processing unit 140b uses the processing result of the first processing unit 140a to determine whether or not the pattern image is contained within the imaging range.

Description

[Technical Field]

[0001] This disclosure relates to a processing method, an information processing device, and a processing program. [Background technology]

[0002] Patent Document 1 discloses an invention for a portable terminal device that detects four groups of edge pixels connected in a linear fashion from an image obtained by imaging a rectangular object with an imaging unit such as a camera, and determines whether or not the vertices of the rectangle enclosed by the straight lines corresponding to each of the four line segments indicated by the four groups of edge pixels are located within the imaging range of the imaging unit. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2013-168119 [Overview of the project] [Problems that the invention aims to solve]

[0004] Some projection devices, such as projectors, have a correction function that corrects the shape of the projected image so that the image projected onto the screen is rectangular. In projectors with a correction function, a pattern image such as a chessboard pattern is projected onto the screen, and a correction value for correcting the shape of the image is calculated based on multiple captured images obtained by capturing the projected pattern image from various angles with a camera. In this case, it is necessary to determine whether the pattern image is contained within the imaging range for each of the multiple captured images, but if the process of detecting edge pixel groups one by one is performed as in the technology disclosed in Patent Document 1, the speed at which the determination result is output decreases. [Means for solving the problem]

[0005] One aspect of the processing method of the present disclosure involves, using one or more processors, acquiring a first image captured by a camera of a screen onto which a drawn image including a structured light pattern drawn on an optical modulation element is projected via the projection lens of a projector during a first period; identifying a first correspondence relationship that associates the coordinate system of the first image with the coordinate system of the optical modulation element based on the first image; extracting a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first image based on the results of analyzing the screen and the drawn image contained in the first image; and determining that each of the plurality of first coordinates corresponds to the coordinate of the optical modulation element based on the first correspondence relationship. The method is characterized by including: identifying a plurality of second coordinates converted to coordinates in a system; acquiring a second image captured by the camera of the screen onto which a drawn image including a structured light pattern drawn on the optical modulation element is projected via the projection lens during a second period following the first period; identifying a second correspondence relationship that associates the coordinate system of the second image with the coordinate system of the optical modulation element based on the second image; identifying a plurality of third coordinates obtained by converting each of the plurality of second coordinates to coordinates in the coordinate system of the second image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second image.

[0006] Furthermore, one embodiment of the information processing apparatus of the present disclosure includes one or more processors and a camera, the one or more processors, in a first period, acquire a first image obtained by imaging a screen on which a drawn image including a structured light pattern drawn on an optical modulation element is projected via the projection lens of a projector using the camera; identify a first correspondence relationship that associates the coordinate system of the first image with the coordinate system of the optical modulation element based on the first image; extract a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first image based on the results of analyzing the screen and the drawn image included in the first image; and based on the first correspondence relationship, the plurality of first coordinates The method is characterized by: identifying a plurality of second coordinates, each converted to coordinates in the coordinate system of the optical modulation element; acquiring a second captured image obtained by the camera, in a second period following the first period, on which a drawn image including a structured light pattern drawn on the optical modulation element is projected through the projection lens onto the screen; identifying a second correspondence relationship that associates the coordinate system of the second captured image with the coordinate system of the optical modulation element based on the second captured image; identifying a plurality of third coordinates, each of the plurality of second coordinates converted to coordinates in the coordinate system of the second captured image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second captured image.

[0007] Furthermore, one aspect of the processing program of the present disclosure involves a computer acquiring a first captured image, which is captured by a camera on a screen onto which a drawn image, including a structured light pattern drawn on an optical modulation element, is projected via the projection lens of a projector during a first period; identifying a first correspondence relationship that associates the coordinate system of the first captured image with the coordinate system of the optical modulation element based on the first captured image; extracting a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first captured image, based on the results of analyzing the screen and the drawn image included in the first captured image; and determining that each of the plurality of first coordinates corresponds one-to-one with the coordinate system of the optical modulation element based on the first correspondence relationship. The method is characterized by: identifying a plurality of second coordinates converted to coordinates; acquiring a second image captured by the camera of the screen onto which a drawn image including a structured light pattern drawn on the light modulation element is projected via the projection lens during a second period following the first period; identifying a second correspondence relationship that associates the coordinate system of the second image with the coordinate system of the light modulation element based on the second image; identifying a plurality of third coordinates obtained by converting each of the plurality of second coordinates to coordinates in the coordinate system of the second image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second image. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of a projection system 1 including an information processing device 10 according to one embodiment of the present disclosure. [Figure 2] This figure shows an example configuration of the information processing device 10. [Figure 3] This flowchart shows the processing flow in the first process executed by the processing unit 140 of the information processing device 10 according to the program PR1. [Figure 4] This figure shows an example of the first captured image. [Figure 5] This figure shows an example of an intersection point extracted from the first image. [Figure 6] It is a flowchart showing a processing flow in a second process executed by the processing device 140 of the information processing device 10 in accordance with the program PR1. [Figure 7] It is a diagram showing an example of a second captured image. [Figure 8] It is a diagram showing an example of intersection points extracted from the second captured image. [Figure 9] It is a diagram showing an example of intersection points extracted from the second captured image. [Figure 10] It is a flowchart showing a processing flow in a first process executed by the processing device 140 of the information processing device 10 in accordance with the program PR1. [Figure 11] It is a flowchart showing a processing flow in a fourth process executed by the processing device 140 of the information processing device 10 in accordance with the program PR1. [Figure 12] It is a diagram for explaining the content of interpolation processing. [Figure 13] It is a diagram for explaining the content of interpolation processing. [Figure 14] It is a flowchart showing a processing flow in a second correction value calculation process. [Figure 15] It is a diagram for explaining processing for calculating a coordinate system transformation matrix. [Figure 16] It is a diagram for explaining processing for calculating coordinates of four corners after correction. [Figure 17] It is a diagram for explaining processing for calculating coordinates of four corners after correction. [Figure 18] It is a diagram for explaining processing for calculating coordinates of four corners after correction. [Figure 19] It is a diagram for explaining processing for calculating coordinates of four corners after correction.

Mode for Carrying Out the Invention

[0009] Various technically preferable limitations are attached to the embodiments described below. However, the embodiments of the present disclosure are not limited to the embodiments described below. A. Embodiment Figure 1 shows an example configuration of a projection system 1 including an information processing device 10 according to one embodiment of the present disclosure. As shown in Figure 1, the projection system 1 includes a projection device 20, which is, for example, a projector, in addition to the information processing device 10. The projection device 20 includes a display panel on which an image is drawn and an optical system that guides the image light emitted from the display panel to a screen SC. The display panel is an example of an optical modulation element, and in this embodiment, it is a liquid crystal panel. The optical modulation element may also be a digital mirror device or the like. In Figure 1, the display panel and optical system are not shown. The projection device 20 displays an image on the screen SC by projecting image light onto the screen SC under the control of the information processing device 10. In this embodiment, the screen SC is provided on the wall surface of the room in which the projection device 20 is installed.

[0010] The projection device 20 has a correction function that corrects the shape of the projected image so that the image projected onto the screen SC is rectangular. In this embodiment, the projection device 20 projects a pattern image, such as a chessboard pattern, onto the screen under the control of the information processing device 10. The information processing device 10 is equipped with a camera and calculates a correction value for correcting the shape of the image based on multiple captured images obtained by capturing the pattern image projected onto the screen SC from various angles with the camera, and sets this correction value in the projection device 20.

[0011] The information processing device 10 is a smartphone that communicates with the projection device 20 via, for example, a wireless LAN (Local Area Network). Figure 2 shows an example configuration of the information processing device 10. As shown in Figure 2, the information processing device 10 includes a communication device 100, a UI (User Interface) device 110, a camera 120, a storage device 130, a processing device 140, and a sensor 150. Each of the communication device 100, UI device 110, camera 120, storage device 130, and sensor 150 is connected to the processing device 140 via a bus (not shown in Figure 2). Note that the camera 120 may be a separate device from the information processing device 10 (for example, a camera connected to the information processing device 10 via a wireless LAN).

[0012] The communication device 100 includes an antenna for transmitting and receiving signals related to wireless communication and a communication interface circuit for encoding and decoding such signals. The communication device 100 passes data received from the projection device 20 via a wireless LAN or the like to the processing device 140, while transmitting data provided by the processing device 140 to the projection device 20.

[0013] The UI device 110 includes an input device equipped with multiple controls, such as a numeric keypad, and a display device for displaying various images under the control of the processing device 140. A liquid crystal display is a specific example of the display device. In Figure 2, the input device and display device are not shown. When an operation is performed on the input device included in the UI device 110, such as pressing any of the controls, the UI device 110 passes data corresponding to the user's operation to the processing device 140. In this way, the content of the user's operation is transmitted to the processing device 140.

[0014] The processing unit 140 includes one or more processors. The processors include, for example, a CPU (Central Processing Unit). The processing unit 140 is an example of a computer in this disclosure. As will be described in detail later, the processing unit 140 functions as the control center of the information processing unit 10 by operating according to program PR1 which is pre-stored in the storage device 130. Program PR1 is an example of a processing program in this disclosure.

[0015] The camera 120 includes a CMOS (Complementary Metal-Oxide-Semiconductor) or CCD (Charge Coupled Device) image sensor. In this embodiment, the camera 120 captures an image of the shooting area under the control of the processing unit 140. The camera 120 outputs image data representing the captured image to the processing unit 140. The sensor 150 is, for example, a 3-axis accelerometer. In this embodiment, the direction of gravitational acceleration is determined from the output value of the sensor 150.

[0016] The storage device 130 includes non-volatile memory such as flash ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory). The non-volatile memory of the storage device 130 stores program PR1, which causes the processing unit 140 to function as the control center of the information processing unit 10, and pattern image data DD, which represents a chessboard pattern. The volatile memory of the storage device 130 is used by the processing unit 140 as a work area when executing program PR1. When an operation is made to instruct the UI device 110 to execute program PR1, the processing unit 140 reads program PR1 from the non-volatile memory to the volatile memory and starts executing program PR1 read into the volatile memory. The processing unit 140, operating according to program PR1, functions as a first processing unit 140a, a second processing unit 140b, a third processing unit 140c, and a fourth processing unit 140d.

[0017] The first processing unit 140a performs processing related to the first image when the screen SC on which the pattern image is projected is imaged multiple times from various angles. The second processing unit 140b performs processing related to the second image when the screen SC on which the pattern image is projected is imaged multiple times from various angles. The third processing unit 140c performs processing related to the third and subsequent images when the screen SC on which the pattern image is projected is imaged multiple times from various angles. The fourth processing unit 140d calculates a correction value based on the processing results of the second processing unit 140b and the processing results of the third processing unit 140c. The processing method in this disclosure includes processing performed by the first processing unit 140a, processing performed by the second processing unit 140b, processing performed by the third processing unit 140c, and processing performed by the fourth processing unit 140d. The period during which processing by the first processing unit 140a is performed is an example of the first period in this disclosure. The period during which processing by the second processing unit 140b is performed is an example of the second period in this disclosure.

[0018] Figure 3 is a flowchart showing the processing flow in the first process executed by the first processing unit 140a. As shown in Figure 3, the first processing unit 140a first projects a drawing image including the original image such as structured light as a pattern image onto the projection device 20 (step SA110), and then has the camera 120 image the screen SC onto which the drawing image has been projected via the projection lens of the projection device 20 (step SA120). Then, the first processing unit 140a acquires image data representing the image captured by the camera 120 from the camera 120. Hereinafter, in the first process, the pattern image projected onto the screen SC will be referred to as the first pattern image. Also, in the first process, the image captured by the camera 120 will be referred to as the first captured image, and the image data representing the first captured image will be referred to as the first image data. Figure 4 is a diagram showing an example of the first captured image G1. In Figure 4, the rectangle T1 drawn with a solid line corresponds to the frame of the screen SC. In Figure 4, the rectangle T2 drawn with a dotted line corresponds to the maximum drawing area on the display panel of the projection device 20, and the rectangle T3 drawn with a dashed line corresponds to the first pattern image. As shown in Figure 4, the first pattern image is a small pattern image distributed only near the center of the maximum drawing area on the display panel of the projection device 20. Since the first pattern image is a small pattern image distributed only near the center of the maximum drawing area on the display panel of the projection device 20, in step SA110, the first processing unit 140a draws a white image in the blank areas (areas where the first pattern image is not drawn) on the display panel of the projection device 20. That is, the drawing image projected from the projection device 20 in step SC110 includes the first pattern image and a white image that surrounds it without any gaps.

[0019] Furthermore, the first processing unit 140a does not have to execute step SA110. That is, the pattern image may be projected not in response to a command from the information processing device 10, but in response to an operation by the user to the projection device 20, or an operation to a remote controller that remotely operates the projection device 20.

[0020] Next, the first processing unit 140a identifies a correspondence between the coordinate system that defines the position on the image represented by the first captured image data (hereinafter referred to as the camera coordinate system) and the coordinate system that defines the position on the display panel (hereinafter referred to as the panel coordinate system) based on the first captured image data acquired from the camera 120 (Figure 3: Step SA130). In order for the first processing unit 140a to identify the correspondence, a checkerboard pattern or a phase shift pattern is used as structured light. Existing technologies may be used as appropriate for identifying the correspondence based on the captured image data. The correspondence identified based on the first captured image data is referred to as the first correspondence. The camera coordinate system that defines the position on the image represented by the first captured image data is referred to as the first camera coordinate system.

[0021] Next, the first processing unit 140a analyzes the screen SC and the drawing image included in the first captured image G1 and extracts a plurality of first coordinates (coordinates in the first camera coordinate system) that correspond one-to-one with a plurality of first pixels that define the projection area in the first captured image G1. In this embodiment, the first pixel is a pixel that corresponds to at least one intersection point where a first line segment indicating the edge of the drawing image included in the first captured image G1 and a second line segment indicating the edge of the screen SC included in the first captured image G1 intersect each other, and is also called an intersection pixel.

[0022] More specifically, the first processing unit 140a first identifies a plurality of first edge pixels that indicate the edges of the drawing image contained in the first captured image G1 using an existing edge detection algorithm, and identifies a first line segment based on the plurality of first edge pixels (Figure 3: step SA140). An example of an existing edge detection algorithm is Canny edge detection. Next, the first processing unit 140a identifies a plurality of second edge pixels that indicate the edges of the screen SC contained in the first captured image G1, and identifies a second line segment based on the plurality of second edge pixels. In step SA150 following step SA140, the first processing unit 140a identifies an intersection pixel corresponding to the intersection of the first line segment and the second line segment, and extracts the coordinates of the intersection pixel in the first camera coordinate system as the first coordinate. Figure 5 shows an example of intersections P1 to P6 extracted from the first captured image G1 shown in Figure 4.

[0023] In step SA160, which follows step SA150, the first processing unit 140a identifies multiple second coordinates by converting each of the multiple first coordinates into coordinates in the panel coordinate system based on the first correspondence relationship. There is a one-to-one correspondence between the multiple second coordinates and the multiple first coordinates. By converting the first coordinates into coordinates in the panel coordinate system based on the first correspondence relationship, the second coordinates corresponding to those first coordinates are obtained. This is the flow of the first processing performed by the first processing unit 140a. Once the execution of the first processing is complete, the second processing unit 140b executes the second processing.

[0024] Figure 6 is a flowchart showing the processing flow in the second process executed by the second processing unit 140b. As shown in Figure 6, the second processing unit 140b projects a drawing image including a pattern image such as structured light onto the projection device 20, similar to the processing in step SA110 of the first process (step SB110). Next, the second processing unit 140b causes the camera 120 to image the screen SC onto which the drawing image has been projected via the projection lens of the projection device 20 (step SB120), and acquires image data representing the image captured by the camera 120 from the camera 120. Hereinafter, in the second process, the pattern image projected onto the screen SC will be referred to as the second pattern image. Also, in the second process, the image captured by the camera 120 will be referred to as the second image, and the image data representing the second image will be referred to as the second image data. The second processing unit 140b writes the acquired second image data to volatile memory.

[0025] Figure 7 shows an example of the second captured image G2. In Figure 7, as in Figure 4, the rectangle T1 drawn with a solid line corresponds to the frame of the screen SC, the rectangle T2 drawn with a dotted line corresponds to the maximum drawing area on the display panel of the projection device 20, and the rectangle T4 drawn with a dashed line represents the second pattern image. As is clear from comparing Figure 4 and Figure 7, the second pattern image is a larger image than the first pattern image. In this embodiment, the type of the second pattern image is the same as the type of the first pattern image. For example, if the type of the first pattern image is a checkerboard pattern, then the type of the second pattern image is also a checkerboard pattern. The type of the second pattern image may be different from the type of the first pattern image. In step SB110, the second processing unit 140b draws a white image in the blank areas (areas where the second pattern image is not drawn) on the display panel of the projection device 20. In other words, the drawing image projected from the projection device 20 in step SC110 includes the first pattern image and a white image that completely surrounds it without any gaps.

[0026] In step SB130, which follows step SB120, the second processing unit 140b identifies a correspondence between the coordinate system that defines the position on the image represented by the second image data (hereinafter referred to as the second camera coordinate system) and the panel coordinate system, based on the second image data acquired from the camera 120. The correspondence identified based on the second image data is referred to as the second correspondence.

[0027] In step SB140, which follows step SB130, the second processing unit 140b identifies multiple third coordinates by converting each of the multiple second coordinates into coordinates in the second camera coordinate system based on the second correspondence relationship. There is a one-to-one correspondence between the multiple third coordinates and the multiple second coordinates. By converting the second coordinates into coordinates in the second camera coordinate system based on the second correspondence relationship, the third coordinates corresponding to those second coordinates are obtained. In other words, in step SB140, the second processing unit 140b identifies multiple third coordinates based on the second correspondence relationship, rather than by an edge detection algorithm.

[0028] In step SB150, which follows step SB140, the second processing unit 140b stores the multiple third coordinate data, which represent the multiple third coordinates identified in step SB140, by writing them to volatile memory in association with the aforementioned second image data. Then, in step SB160, which follows step SB150, the second processing unit 140b determines whether each of the multiple third coordinates falls within the range of the second image G2. For example, the second processing unit 140b has prior knowledge of the maximum horizontal pixel coordinates and the maximum vertical pixel coordinates in the coordinate system of the second image G2, and determines whether each of the multiple third coordinates falls within the range defined by the maximum horizontal pixel coordinates and the maximum vertical pixel coordinates. As shown in Figure 8, if all of the intersection points P1 to P6 corresponding to each of the multiple third coordinates fall within the range of the second image G2, the result of the determination in step SB150 is "Yes". As shown in Figure 9, if at least one of the intersection points P1 to P6 is not within the range of the second image G2, the result of step SB150 will be "No".

[0029] If the result of step SB160 is "Yes", the second processing unit 140b maintains the state in which the second image data is stored in the volatile memory, erases the multiple third coordinate data associated with the second image data from the volatile memory (step SB170), and executes the processing from step SB190 onwards. Conversely, if the result of step SB160 is "No", the second processing unit 140b erases the second image data and the third coordinate data from the volatile memory (step SB180), and executes the processing from step SB110 onwards again. As a result, when the processing from step SB190 onwards is executed, only the appropriate second image data that includes all of the multiple intersection points extracted from the first image G1 is stored in the volatile memory. The second processing unit 140b determines at least once whether the second image G2 contains the second pattern image using the processing result by the first processing unit 140a, rather than using an edge detection algorithm, thus reducing the processing load compared to an approach that performs edge detection one by one to make the determination.

[0030] In steps SB190 and SB200, the second processing unit 140b analyzes the second captured image G2 and stores a plurality of fourth coordinates (coordinates in the second camera coordinate system) that correspond one-to-one with a plurality of second pixels that define the projection area in the second captured image G2. In this embodiment, the second pixels are pixels corresponding to the edges corresponding to the frame of the screen SC and images corresponding to the edges of the drawn image. The edge detection algorithm used in steps SB190 and SB200 is the same as the edge detection algorithm used in step SA140, for example. In this embodiment, steps SB190 and SB200 are performed after the determination result of step SB160 is determined to be "Yes". Therefore, in this embodiment, it is possible to suppress the second processing unit 140b from unnecessarily executing the edge detection algorithm.

[0031] In step SB210, which follows step SB200, the second processing unit 140b stores the coordinates of the feature points in the second pattern image. This is the flow of the second processing performed by the second processing unit 140b. Once the execution of the second processing is complete, the third processing is performed by the third processing unit 140c.

[0032] Figure 10 is a flowchart showing the processing flow in the third process executed by the third processing unit 140c. In the third process, the third and subsequent imaging of the screen SC onto which a drawing image including a pattern image such as structured light is projected is performed. In the third process, the aforementioned second pattern image is used as the pattern image to be projected onto the screen SC. As shown in Figure 10, the third processing unit 140c causes the camera 120 to image the screen SC onto which the drawing image including the second pattern image is projected via the projection lens of the projection device 20, and acquires image data representing the captured image from the camera 120 (step SC110).

[0033] In step SC120, which follows step SC110, the third processing unit 140c determines whether the second pattern image fits within the field of view of the camera 120 by analyzing the image data acquired in step SC110. In this determination in step SC120, intersection coordinates are not identified; instead, the determination of whether the second pattern image fits within the field of view of the camera 120 is made based on the image data acquired in step SC110. If the second pattern image fits within the field of view of the camera 120, the determination result in step SC110 is "Yes". Conversely, if at least a part of the second pattern image extends beyond the field of view of the camera 120, the determination result in step SC110 is "No". If the determination result in step SC120 is "No", the third processing unit 140c outputs a message to the UI device 110, for example, "Please switch the display mode of the pattern image so that the entire pattern image fits within the camera's field of view", and then executes the processing from step SC110 onward again.

[0034] If the result of step SC120 is "Yes", the third processing unit 140c executes the processing from step SC130 onward. In step SC130, the third processing unit 140c stores the coordinates of the feature points in the second pattern image, similar to step SB210 in the second processing.

[0035] In step SC140, which follows step SC130, the third processing unit 140c determines whether N images (where N is an integer greater than or equal to 2, in this embodiment N=9) have been captured and whether N sets of feature point coordinates have been stored. If the result of step SC140 is "No", the third processing unit 140c executes the processing from step SC110 onward. If the result of step SC140 is "Yes", the third processing unit 140c completes the execution of the third process, and thereafter the fourth processing unit 140d executes the fourth process. In this embodiment, a drawing image including the second pattern image is projected onto the projection device 20, and the screen SC onto which the drawing image is projected via the projection lens of the projection device 20 is imaged by the camera 120. In the captured image, one set of feature point coordinates is stored in the second process, and nine sets in the third process, for a total of 10 sets, in the volatile memory. The fourth process is then executed using these 10 sets of feature point coordinates. In this embodiment, N=9, but N may be a value between 2 and 8, or a value of 10 or more.

[0036] Figure 11 is a flowchart showing the processing flow in the fourth process executed by the fourth processing unit 140d. The fourth process is the process of calculating a correction value to correct the shape of the image projected from the projection device 20 onto the screen SC into a rectangular shape. As shown in Figure 11, the fourth process includes the processes of steps SD110 to SD160. In step SD110, the fourth processing unit 140d refers to a plurality of fourth coordinates written to volatile memory and determines whether four sides corresponding to each side of the frame of the screen SC have been detected. If the number of detected sides is 4, the result of step SD110 is "Yes". If the number of detected sides is less than 4, the result of step SD110 is "No". If the result of step SD110 is "Yes", the fourth processing unit 140d executes the first correction value calculation process (step SD160). In the first correction value calculation process, the fourth processing unit 140d determines the coordinates of the four corners of the corrected projected image so that the image projected from the projection device 20 becomes a rectangle on the screen SC, and converts these coordinates into coordinates in the panel coordinate system based on the second correspondence relationship. Then, the fourth processing unit 140d calculates geometric correction values ​​for distortion correction from the coordinates of the four corners and the coordinates of the four corners of the projected image, and sets them in the projection device 20.

[0037] If the determination result of step SD110 is "No", it means that there is a shortage of multiple fourth coordinates that correspond one-to-one with multiple second pixels that define the projection area in the second captured image G2, and the aforementioned first correction value calculation process cannot be executed as is. In step SD120, which is executed when the determination result of step SD110 is "No", the fourth processing unit 140d calculates the normal vector of the screen SC. More specifically, the fourth processing unit 140d calculates a transformation matrix that projects the image on the display panel of the projection device 20 onto the captured image of the camera 120, based on each of the coordinates of the aforementioned 10 sets of feature points. That is, in this embodiment, 10 transformation matrices are calculated. Existing technology may be used as appropriate for calculating these transformation matrices. Next, the fourth processing unit 140d calculates the normal vector of the screen SC using the optimal path method based on the 10 transformation matrices. For details of the optimal path method, please refer to Japanese Patent Application No. 2024-029280 by the present applicant.

[0038] In step SD130, which follows step SD120, the fourth processing unit 140d determines whether the number of detected edges is 1 or more. If the result of step SD130 is "Yes", the fourth processing unit 140d performs interpolation processing (step SD140) to interpolate multiple fourth coordinates that correspond one-to-one with multiple second pixels that define the projection region in the second captured image G2, and then performs first correction value calculation processing (step SD160). If the result of step SD130 is "No", the fourth processing unit 140d performs second correction value calculation processing (step SD150). Details of the second correction value calculation processing will be revealed later.

[0039] In the interpolation process (step SD140), the fourth processing unit 140d interpolates multiple fourth coordinates by interpolating the undetected edges of the four edges (top edge, bottom edge, right edge, and left edge) corresponding to the frame of the screen SC, based on the actually detected edges and the normal vectors of the screen SC. For example, if only the bottom edge is detected, the fourth processing unit 140d interpolates the top edge, right edge, and left edge based on the normal vectors of the bottom edge and the screen SC, in the manner of STEPA1 to STEPA4 shown in Figure 12, thereby interpolating the missing fourth coordinates.

[0040] In STEPA1, the fourth processing unit 140d identifies the normal vector N=(Nx,Ny,Nz) of the screen SC in the normalized panel coordinate system, and the normal vector L=(a,b,c) of the line equation ax+by+c=0 that represents the bottom edge B1. Since only the bottom edge B1 of the screen SC has been found, the fourth processing unit 140d identifies the line segment C1 that corresponds to the top edge of the display panel in the normalized panel coordinate system, instead of the top edge of the frame of the screen SC.

[0041] In STEPA2, the fourth processing unit 140d determines the coordinates (vector VH) of the horizontal vanishing point HI from the cross product of the normal vector N and the normal vector LL identified in STEPA1. Next, the fourth processing unit 140d determines the innermost of the two lines, DL1 which passes through one end of line segment C1 and the horizontal vanishing point HI, and DL2 which passes through the other end of line segment C1 and the horizontal vanishing point HI, as the upper edge for screen fitting. In the example shown in Figure 12, the line DL1 becomes the upper edge for screen fitting.

[0042] In STEPA3, the fourth processing unit 140d calculates the coordinates of the vertical vanishing point VI (vector VV) from the cross product of the normal vector N and vector VH. Then, the fourth processing unit 140d selects the innermost line from among the lines DL3 passing through the left end of line segment C1 and the vertical vanishing point VI, and the line DL4 passing through the left end of the lower edge B1 of the screen SC and the vertical vanishing point VI, as the left side for screen fitting. In the example shown in Figure 12, line DL4 is the left side for screen fitting. Similarly, the fourth processing unit 140d selects the innermost line from among the lines DL5 passing through the right end of line segment C1 and the vertical vanishing point VI, and the line DL6 passing through the right end of the lower edge B1 of the screen SC and the vertical vanishing point VI, as the right side for screen fitting. In the example shown in Figure 12, line DL6 is the left side for screen fitting.

[0043] In STEPA4, the fourth processing unit 140d clips the coordinates of each vertex so that any vertices corresponding to the four corners of the screen SC frame extend beyond the projection area, so that any vertices that extend beyond the projection area are contained within the display panel. Specifically, the fourth processing unit 140d uses the intersection of the line corresponding to the left edge of the screen fit and the line corresponding to the top edge of the screen fit, the intersection of the line corresponding to the right edge of the screen fit and the line corresponding to the top edge of the screen fit, the intersection of the line corresponding to the left edge of the screen fit and the bottom edge B1, and the intersection of the line corresponding to the right edge of the screen fit and the bottom edge B1 as the vertices of the interpolated frame. In Figure 12, these intersections are shown as white circles.

[0044] Furthermore, if only the top and bottom edges of the screen SC are detected, the fourth processing unit 140d completes the missing fourth coordinates by completing the right and left edges based on the normal vectors of the top and bottom edges and the screen SC, in the manner of STEP B1 to STEP B4 shown in Figure 13.

[0045] In STEP B1, the fourth processing unit 140d identifies the normal vector N in the normalized panel coordinate system, the normal vector L1=(a1,b1,c1) of the line equation a1x+b1y+c1=0 representing the upper side, and the normal vector L2=(a2,b2,c2) of the line equation a2x+b2y+c2=0 representing the lower side.

[0046] In STEP B2, the fourth processing unit 140d uses the property that in a normalized panel coordinate system, the coordinates of a horizontal vanishing point can be found by taking the cross product of the normal vectors of straight lines, to determine the coordinates of the horizontal vanishing point HI. Specifically, the fourth processing unit 140d obtains the coordinates of the horizontal vanishing point HI (vector VH) from the cross product of the normal vectors L1 and L2 identified in STEP B1.

[0047] In STEP B3, the fourth processing unit 140d calculates the coordinates of the vertical vanishing point VI (vector VV) from the cross product of the normal vector N and vector VH. Then, the fourth processing unit 140d selects the innermost line among the lines DL7 passing through the left end of the top edge of the screen SC and the vertical vanishing point VI, and the line DL8 passing through the left end of the bottom edge of the screen SC and the vertical vanishing point VI, as the left side for screen fitting. In the example shown in Figure 13, line DL8 is the left side for screen fitting. Similarly, the fourth processing unit 140d selects the innermost line among the lines passing through the right end of the top edge of the screen SC and the vertical vanishing point VI, and the line passing through the right end of the bottom edge of the screen SC and the vertical vanishing point VI, as the right side for screen fitting. In STEP B4, the fourth processing unit 140d recalculates the positions of the four vertices of the screen SC, similar to STEP B4 described above, but STEP B4 may be omitted. Furthermore, if only the left and right sides of the screen SC are detected, the fourth processing unit 140d completes the top and bottom sides based on the left and right sides and the normal vector of the screen SC, in the manner of STEP B1 to STEP B4. In this case, vector VV is calculated first, and vector VH is calculated from the cross product of vector VV and the normal vector N.

[0048] Furthermore, if only the top and left edges of the screen SC are detected, the fourth processing unit 140d will fill in the missing fourth coordinates by filling in the right and bottom edges based on the normal vectors of the top and left edges and the screen SC, in the manner of STEPC1 to STEPC5 below.

[0049] In STEPC1, the fourth processing unit 140d identifies the normal vector N in the normalized panel coordinate system, the normal vector L1=(a1,b1,c1) of the line equation a1x+b1y+c1=0 representing the top edge, and the normal vector L3=(a3,b3,c3) of the line equation a3x+b3y+c3=0 representing the left edge. In STEPC2, the fourth processing unit 140d finds the coordinates of the vertical vanishing point (vector VV) from the cross product of the normal vector L3 and the normal vector N. In STEPC3, the fourth processing unit 140d identifies the innermost line among the group of lines that pass through the vertical vanishing point and intersect the top edge of the screen SC frame as the right edge. In STEPC4, the fourth processing unit 140d finds the coordinates of the horizontal vanishing point (vector VH) from the cross product of the normal vector L1 and the normal vector N. In STEPC5, the fourth processing unit 140d determines the bottom edge of the screen SC by selecting the innermost line from the group of lines that pass through the horizontal vanishing point and intersect the left edge of the screen SC's frame. The fourth processing unit 140d may also recalculate the positions of the four vertices of the screen SC, similar to STEPA4 described above. Furthermore, if only the bottom and right edges of the screen SC are detected, the fourth processing unit 140d may fill in the top and left edges based on the normal vectors of the screen SC and the bottom and right edges, in the same manner as in STEPC1 to STEPC5.

[0050] Next, we will explain the process for calculating the second correction value. In the second correction value calculation process, the fourth processing unit 140d calculates a geometric correction value for distortion correction based on the output value of the sensor 150 and the normal vector of the screen SC, and sets it in the projection device 20. Figure 14 is a diagram illustrating the processing flow in the second correction value calculation process. In step SE110 of the second correction value calculation process, the fourth processing unit 140d calculates a coordinate system transformation matrix H for converting the normalized panel coordinate system to the screen coordinate system.PS This is calculated based on the output value of sensor 150 and the normal vector of screen SC, in the manner shown in Figure 15. The screen coordinate system is the coordinate system as viewed from the front of screen SC.

[0051] To explain in more detail, the fourth processing unit 140d first uses the property that the cross product of the vector G indicating the direction of gravity pointed to by the output value of the sensor 150 and the normal vector N of the screen SC is the X-axis direction of the roll-compensated screen coordinate system to determine the horizontal vanishing point (vector H c Next, the fourth processing unit 140d calculates the vector H. c From the cross product of vectors G and vector V, the perpendicular vanishing point (vector V c The fourth processing unit 140d calculates the vector H. c and vector V c Based on this, the coordinate system transformation matrix H PS Calculate.

[0052] In step SE120, which follows step SE110, the fourth processing unit 140d generates a projection transformation matrix for projecting the normalized panel coordinate system to the screen coordinate system, along with a transformation matrix for projecting the image on the display panel of the projection device 20 to the image captured by the camera 120 and a coordinate system transformation matrix H PS It is calculated based on the following.

[0053] In step SE130 subsequent to step SE120, the fourth processing unit 140d calculates the corrected coordinates of the four corners on the basis of the coordinates of the four corners of the display panel in the normalized panel coordinate system (vector PC0, vector PC1, vector PC2, and vector PC3), the resolution of the display panel of the projection device 20, and the projection transformation matrix calculated in step SE120. In the present embodiment, the vector PC0 corresponds to the coordinates of the top-left corner of the display panel in the normalized panel coordinate system. The vector PC1 corresponds to the coordinates of the top-right corner of the display panel in the normalized panel coordinate system. The vector PC2 corresponds to the coordinates of the bottom-right corner of the display panel in the normalized panel coordinate system. The vector PC3 corresponds to the coordinates of the bottom-left corner of the display panel in the normalized panel coordinate system. The coordinates of the four corners of the display panel in the normalized panel coordinate system are calculated from the aforementioned fourth coordinates. As shown in FIG. 16, the processing of step SE130 includes the respective processing of steps SE131 to SE135.

[0054] In step SE131, the fourth processing unit 140d processes the vector PC0, vector PC1, vector PC2, and vector PC3, and the coordinate system transformation matrix H PS and a coefficient α determined according to the resolution of the display panel of the projection device 20, and calculates the coordinates of the four corners of the display panel in the screen coordinate system (vector PS0, vector PS1, vector PS2, and vector PS3) using the following formula (1). In the present embodiment, the vector PS0 corresponds to the coordinates of the top-left corner of the display panel in the screen coordinate system. The vector PS1 corresponds to the coordinates of the top-right corner of the display panel in the screen coordinate system. The vector PS2 corresponds to the coordinates of the bottom-right corner of the display panel in the screen coordinate system. The vector PS3 corresponds to the coordinates of the bottom-left corner of the display panel in the screen coordinate system. Note that n is 0 to 3 in formula (1). H PS PC n =αPS n ···(1)

[0055] In step SE132, the fourth processing unit 140d identifies the central intersection point PD, which is the intersection point of the diagonal line D1 passing through the upper left and lower right corners of the display panel in the screen coordinate system and the diagonal line D2 passing through the lower left and upper right corners of the display panel in the screen coordinate system, as shown in Figure 17.

[0056] In step SE133, the fourth processing unit 140d determines a rectangle R1 with a panel aspect ratio centered at the central intersection PD, as shown in Figure 18. More specifically, the fourth processing unit 140d calculates the intersections of the lines extending to the four corners of a rectangle with a desired aspect ratio centered at the central intersection PD and the four outer sides of the rectangle, and searches for the closest intersection (hereinafter referred to as the reference intersection) from the central intersection PD. Then, by enlarging the rectangle with a desired aspect ratio centered at the central intersection PD, a rectangle with a desired aspect ratio where the reference intersection is one of the four corners is determined as rectangle R1. In the example shown in Figure 18, a rectangle with a desired aspect ratio centered at the central intersection PD and where the lower left corner corresponds to the reference intersection is determined as rectangle R1.

[0057] In step SE134, the fourth processing unit 140d determines rectangle R2 by expanding rectangle R1 with the reference intersection as the origin. In the example shown in Figure 19, the fourth processing unit 140d expands rectangle R1 in three directions: up, right, and upper right, calculates the intersections with the four outer sides, determines rectangle R2 as the rectangle formed by the intersection with the smallest expansion ratio and the reference intersection, and calculates the four corners of rectangle R2 (vectors QS0, QS1, QS2, and QS3).

[0058] In step SE135, the fourth processing unit 140d processes vectors QS0, QS1, QS2, and QS3, and the coordinate system transformation matrix H PS Based on the coefficient α mentioned above, the coordinates of the four corners of the corrected display panel in the normalized panel coordinate system (vectors QS0, QS1, QS2, and QS3) are calculated using the following equation (2). Note that H in equation (2) PS -1 H is the coordinate system transformation matrix. PS This is the inverse matrix of [the given matrix]. H PS-1 QS n =αQC n ...(2) The above details the processing performed by the fourth processing unit 140d in step SE130.

[0059] In step SE140, which follows step SE130, the fourth processing unit 140d calculates geometric correction values ​​for distortion correction from vectors PS0, SS1, PS2 and PS3, and vectors QS0, QS1, QS2 and QS3, and sets them in the projection device 20.

[0060] According to this embodiment, a correction value is calculated to correct at least one of the shape or position of the image projected from the projection device 20, based on a plurality of fourth coordinates and a second correspondence relationship. According to this embodiment, a highly accurate correction value can be calculated using the recalculated plurality of fourth coordinates. Furthermore, since recalculation is performed only when the image fits within the field of view, unnecessary recalculations can be prevented, and a decrease in the speed until the judgment result is output can be avoided compared to an embodiment that performs a process to detect edge pixel groups one by one.

[0061] B. Transformation The above embodiment can be modified as follows. (1) Prior to acquiring the first image G1, the processing device 140 may output a message to the UI device 110 prompting it to capture the first image G1 within the field of view.

[0062] (2) In the above embodiment, the processing unit 140 of the information processing device 10, which communicates with the projection device 20 via a wireless LAN, was made to execute the determination method of this disclosure. However, the program PR1 may be installed in the projection device 20, and the computer of the projection device 20 may be made to execute the determination method of this disclosure by operating the computer in accordance with the program PR1. In this case, the camera 120 may be included in the projection device 20, or it may be connected to the projection device 20 via a wireless LAN or the like.

[0063] (3) In the above embodiment, the first processing unit 140a, the second processing unit 140b, the third processing unit 140c, and the fourth processing unit 140d were software modules. However, at least one of the first processing unit 140a, the second processing unit 140b, the third processing unit 140c, and the fourth processing unit 140d may be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if at least one of the first processing unit 140a, the second processing unit 140b, the third processing unit 140c, and the fourth processing unit 140d is a hardware module, the same effects as in the above embodiment will be achieved.

[0064] (4) Program PR1 may be manufactured as a standalone product and may be provided for a fee or free of charge. Specific ways in which Program PR1 may be provided include writing Program PR1 to a computer-readable recording medium such as flash ROM, or providing Program PR1 via download over a telecommunications line such as the Internet.

[0065] C. Summary of this disclosure This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms. The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate. A summary of this disclosure is provided below.

[0066] (Note 1) One aspect of the processing method of the present disclosure involves, using one or more processors, acquiring a first image captured by a camera of a screen onto which a drawn image including a structured light pattern drawn on an optical modulation element is projected via the projection lens of a projector during a first period; identifying a first correspondence relationship that associates the coordinate system of the first image with the coordinate system of the optical modulation element based on the first image; extracting a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first image based on the results of analyzing the screen and the drawn image contained in the first image; and determining that each of the plurality of first coordinates corresponds to the coordinate of the optical modulation element based on the first correspondence relationship. The method is characterized by including: identifying a plurality of second coordinates converted to coordinates in a system; acquiring a second image captured by the camera of the screen onto which a drawn image including a structured light pattern drawn on the optical modulation element is projected via the projection lens during a second period following the first period; identifying a second correspondence relationship that associates the coordinate system of the second image with the coordinate system of the optical modulation element based on the second image; identifying a plurality of third coordinates obtained by converting each of the plurality of second coordinates to coordinates in the coordinate system of the second image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second image.

[0067] According to the processing method of this embodiment, it is determined whether each of the multiple third coordinates falls within the range of the second captured image. In the processing method of this embodiment, instead of extracting multiple third coordinates that define the area of ​​the drawing image projected onto the screen in the second captured image by analyzing the second captured image, multiple third coordinates obtained by converting each of the multiple second coordinates to the coordinate system of the second captured image based on the second correspondence are used. Therefore, according to the processing method of this embodiment, the time required to determine whether each of the multiple third coordinates falls within the range of the second captured image can be reduced compared to when analyzing the second captured image acquired in the second period.

[0068] (Note 2) A more preferred embodiment of the processing method is the processing method described in (Appendix 1), which includes: acquiring the second image using one or more processors, storing the data of the second image in memory; maintaining the state in which the data of the second image is stored in memory when it is determined that each of the plurality of third coordinates falls within the range of the second image; and deleting the data of the second image from memory when it is determined that at least one of the plurality of third coordinates does not fall within the range of the second image. According to this embodiment, only appropriate second images are stored in memory.

[0069] (Note 3) A more preferred embodiment of the processing method is the processing method described in (Note 1) or (Note 2), which includes: identifying the plurality of third coordinates using one or more processors, storing third coordinate data indicating the plurality of third coordinates in memory;, when it is determined that each of the plurality of third coordinates is within the range of the second image, extracting a plurality of fourth coordinates that correspond one-to-one with a plurality of second pixels defining the projection area in the second image based on the results of analyzing the screen and the drawing image included in the second image; calculating a correction value for correcting at least one of the shape or position of the image projected from the projector based on the plurality of fourth coordinates and the second correspondence; and, when it is determined that at least one of the plurality of third coordinates is not within the range of the second image, deleting the third coordinate data from memory. According to this embodiment, a highly accurate correction value can be calculated using the recalculated plurality of fourth coordinates. Furthermore, since recalculation is performed only when the coordinates are within the field of view, unnecessary recalculations can be prevented.

[0070] (Note 4) A more preferred embodiment of the processing method involves extracting the plurality of first coordinates, which includes: using one or more processors to identify a plurality of first edge pixels that indicate the edges of the drawing image included in the first captured image; identifying a first line segment that indicates the edges of the drawing image based on the plurality of first edge pixels; identifying a plurality of second edge pixels that indicate the edges of the screen included in the first captured image; identifying a second line segment that indicates the edges of the screen based on the plurality of second edge pixels; identifying at least one intersection pixel that corresponds to at least one intersection point where the first line segment and the second line segment intersect each other; and setting the coordinates of the at least one intersection pixel as the plurality of first coordinates, as described in (Note 1), (Note 2), or (Note 3).

[0071] (Note 5) Furthermore, one embodiment of the information processing apparatus of the present disclosure includes one or more processors and a camera, the one or more processors, in a first period, acquire a first image obtained by imaging a screen on which a drawn image including a structured light pattern drawn on an optical modulation element is projected via the projection lens of a projector using the camera; identify a first correspondence relationship that associates the coordinate system of the first image with the coordinate system of the optical modulation element based on the first image; extract a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first image based on the results of analyzing the screen and the drawn image included in the first image; and based on the first correspondence relationship, the plurality of first coordinates The present invention is characterized by: identifying a plurality of second coordinates, each converted to coordinates in the coordinate system of the optical modulation element; acquiring a second captured image obtained by the camera, in a second period following the first period, on the screen onto which a drawn image including a structured light pattern drawn on the optical modulation element is projected via the projection lens; identifying a second correspondence relationship that associates the coordinate system of the second captured image with the coordinate system of the optical modulation element based on the second captured image; identifying a plurality of third coordinates, each of the plurality of second coordinates converted to coordinates in the coordinate system of the second captured image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second captured image. The present invention provides an information processing device that, compared to the method described in (Appendix 1) which extracts a plurality of third coordinates by analyzing the second captured image, reduces the time required to determine whether each of the plurality of third coordinates falls within the range of the second captured image.

[0072] (Note 6) Furthermore, one aspect of the processing program of the present disclosure involves a computer acquiring a first captured image, which is captured by a camera on a screen onto which a drawn image, including a structured light pattern drawn on an optical modulation element, is projected via the projection lens of a projector during a first period; identifying a first correspondence relationship that associates the coordinate system of the first captured image with the coordinate system of the optical modulation element based on the first captured image; extracting a plurality of first coordinates that correspond one-to-one with a plurality of first pixels defining a projection area in the first captured image, based on the results of analyzing the screen and the drawn image included in the first captured image; and determining that each of the plurality of first coordinates corresponds one-to-one with the coordinate system of the optical modulation element based on the first correspondence relationship. The processing program of this embodiment is characterized by: identifying a plurality of second coordinates converted to coordinates; acquiring a second image captured by the camera of the screen onto which a drawn image including a structured light pattern drawn on the optical modulation element is projected via the projection lens during a second period following the first period; identifying a second correspondence relationship that associates the coordinate system of the second image with the coordinate system of the optical modulation element based on the second image; identifying a plurality of third coordinates into which each of the plurality of second coordinates is converted to coordinates in the coordinate system of the second image based on the second correspondence relationship; and determining whether each of the plurality of third coordinates falls within the range of the second image. According to the processing program of this embodiment, compared to the embodiment in which a plurality of third coordinates are extracted by analyzing the second image, similar to the processing method described in (Appendix 1), the time required to determine whether each of the plurality of third coordinates falls within the range of the second image can be reduced. [Explanation of symbols]

[0073] 10... Information processing device, 20... Projection device, 100... Communication device, 110... UI device, 120... Camera, 130... Memory device, 140... Processing device, 150... Sensor, 140a... First processing unit, 140b... Second processing unit, 140c... Third processing unit, 140d... Fourth processing unit, PR1... Program.

Claims

1. One or more processors, In the first period, a first image is acquired by a camera, which captures a screen onto which a drawn image including a structured light pattern drawn on a light modulation element is projected via the projection lens of a projector, Based on the first captured image, a first correspondence is identified that associates the coordinate system of the first captured image with the coordinate system of the optical modulation element. Based on the results of analyzing the screen and the drawing image contained in the first captured image, a plurality of first coordinates that correspond one-to-one with a plurality of first pixels that define the projection area in the first captured image are extracted, Based on the first correspondence, each of the plurality of first coordinates is transformed into a plurality of second coordinates in the coordinate system of the optical modulation element, In a second period following the first period, a second image is obtained by the camera, on which the screen onto which the drawn image including the structured light pattern drawn on the light modulation element is projected via the projection lens, Based on the second captured image, a second correspondence is identified that associates the coordinate system of the second captured image with the coordinate system of the optical modulation element. Based on the second correspondence, each of the plurality of second coordinates is transformed into a plurality of third coordinates in the coordinate system of the second captured image, To determine whether each of the plurality of third coordinates falls within the range of the second captured image, A processing method that includes this.

2. The one or more processors described above, After acquiring the second image, the data of the second image is saved to memory. When it is determined that each of the plurality of third coordinates falls within the range of the second captured image, the memory maintains a state in which the data of the second captured image is stored. If it is determined that at least one of the plurality of third coordinates is not within the range of the second captured image, the data of the second captured image is deleted from the memory. The processing method according to claim 1, including the method described in claim 1.

3. The one or more processors described above, After identifying the plurality of third coordinates, the third coordinate data representing the plurality of third coordinates is stored in memory. If it is determined that each of the plurality of third coordinates falls within the range of the second captured image, the following steps are taken: extract a plurality of fourth coordinates that correspond one-to-one with a plurality of second pixels defining the projection area in the second captured image, based on the results of analyzing the screen and the drawing image included in the second captured image; calculate a correction value to correct at least one of the shape or position of the image projected from the projector based on the plurality of fourth coordinates and the second correspondence; and delete the third coordinate data from the memory if it is determined that at least one of the plurality of third coordinates does not fall within the range of the second captured image. The processing method according to claim 1, including the method described in claim 1.

4. Extracting the plurality of first coordinates by the one or more processors is: Identifying a plurality of first edge pixels that indicate the edges of the drawing image included in the first captured image, Based on the plurality of first edge pixels, a first line segment indicating the edge of the drawn image is identified, Identifying a plurality of second edge pixels that indicate the edges of the screen included in the first captured image, Based on the plurality of second edge pixels, a second line segment indicating the edge of the screen is identified, Identifying at least one intersection pixel corresponding to at least one intersection point where the first line segment and the second line segment intersect each other, The coordinates of at least one intersection pixel are set as the plurality of first coordinates, The processing method according to claim 1, including the method described in claim 1.

5. It includes one or more processors and a camera, The one or more processors described above are: In the first period, a first captured image is obtained by capturing a screen on which a drawn image including a structured light pattern drawn on a light modulation element is projected via the projection lens of a projector, using the camera. Based on the first captured image, a first correspondence is identified that associates the coordinate system of the first captured image with the coordinate system of the optical modulation element. Based on the results of analyzing the screen and the drawing image contained in the first captured image, a plurality of first coordinates that correspond one-to-one with a plurality of first pixels that define the projection area in the first captured image are extracted, Based on the first correspondence, each of the plurality of first coordinates is transformed into a plurality of second coordinates in the coordinate system of the optical modulation element, In a second period following the first period, a second image is obtained by having the camera project a drawn image including a structured light pattern drawn on the light modulation element onto the screen via the projection lens, Based on the second captured image, a second correspondence is identified that associates the coordinate system of the second captured image with the coordinate system of the optical modulation element. Based on the second correspondence, each of the plurality of second coordinates is transformed into a plurality of third coordinates in the coordinate system of the second captured image, The process involves determining whether each of the aforementioned plurality of third coordinates falls within the range of the second captured image, and performing the following: Information processing device.

6. On the computer, In the first period, a first image is acquired by a camera, which captures a screen onto which a drawn image including a structured light pattern drawn on a light modulation element is projected via the projection lens of a projector, Based on the first captured image, a first correspondence is identified that associates the coordinate system of the first captured image with the coordinate system of the optical modulation element. Based on the results of analyzing the screen and the drawing image contained in the first captured image, a plurality of first coordinates that correspond one-to-one with a plurality of first pixels that define the projection area in the first captured image are extracted, Based on the first correspondence, each of the plurality of first coordinates is transformed into a plurality of second coordinates in the coordinate system of the optical modulation element, In a second period following the first period, a second image is obtained by the camera, on which the screen onto which the drawn image including the structured light pattern drawn on the light modulation element is projected via the projection lens, Based on the second captured image, a second correspondence is identified that associates the coordinate system of the second captured image with the coordinate system of the optical modulation element. Based on the second correspondence, each of the plurality of second coordinates is transformed into a plurality of third coordinates in the coordinate system of the second captured image, To determine whether each of the plurality of third coordinates falls within the range of the second captured image, A processing program that executes [something].

Images