Feature point extraction method, feature point extraction system, and program product

Abstract

This invention provides a feature point extraction method, a feature point extraction system, and a program product to alleviate the inconvenience of geometric alignment timing for projectors. The feature point extraction method includes the following steps: inputting first input data into a first learning model (213), which outputs output data when input data containing captured images is input, the captured images being obtained by capturing an area including the projected object and its four corners, the output data representing four feature points in the captured image corresponding one-to-one with the four corners, the first input data including a first captured image (GS1), the first captured image (GS1) being obtained by capturing a first area (R1) including the screen (SC) and its four corners (CN1); and obtaining first output data from the first learning model (213) representing the four feature points (FP1).

Description

Technical Field

[0001] This invention relates to a feature point extraction method, a feature point extraction system, and a program product. Background Technology

[0002] Programs with settings-maintaining functions that maintain the position and shape of a projected image from a projector onto a projected object over a long period are known. For example, Patent Document 1 discloses a program that extracts several candidate feature points required to generate a projection transformation matrix for correcting the projected image, and allows a user to select feature points from the extracted candidate feature points.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2024-99941

[0004] However, in the technology of Patent Document 1, in order to select one feature point corresponding to the corner of the screen from the extracted candidate feature points, the user needs to study the feature point, which is troublesome for the user. Summary of the Invention

[0005] To address the aforementioned issues, one aspect of the feature point extraction method disclosed herein includes the following steps: inputting first input data into a learned model, wherein the learned model, upon receiving input data including a captured image, outputs output data, the captured image being obtained by capturing an area including a projection object from a projection device and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner, the first input data including a first captured image, the first captured image being obtained by capturing a first region including a first projection object and at least one first corner of the first projection object; and obtaining first output data output from the learned model, wherein the first output data representing at least one first feature point corresponding to the at least one first corner.

[0006] To address the aforementioned issues, one embodiment of the feature point extraction system disclosed herein includes: an external device having a learned model that, upon inputting input data including a captured image, outputs output data, the captured image being obtained by an imaging device capturing a region including a projection object from which an image is projected and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner; a first projection device that projects a first projected image onto a first projection object; and a first imaging device that captures a first region including the first projected image, the first projection object, and at least one first corner of the first projection object, the first imaging device sending first input data including the first captured image obtained by capturing the first region to the external device, the external device inputting the first input data to the learned model, obtaining first output data output from the learned model representing at least one first feature point corresponding to the at least one first corner, and sending the first output data to the first projection device.

[0007] To address the aforementioned issues, one method of the program disclosed herein causes a computer to perform the following processing: inputting first input data representing a first captured image to a learned model, wherein the learned model, upon being input with input data containing capturing information, outputs output data, the first capturing information representing a captured image obtained by capturing an area including a projection object from a projection device and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner, the first captured image including a first projection object and at least one first corner of the first projection object; and obtaining first output data output from the learned model, wherein the first output data represents at least one first feature point corresponding to the at least one first corner. Attached Figure Description

[0008] Figure 1 This is a structural diagram of the image display system according to the first embodiment.

[0009] Figure 2 This is a schematic diagram illustrating the display of the first projected image during the first period.

[0010] Figure 3 It is shown Figure 1 A block diagram of the structure of a projector.

[0011] Figure 4 It is shown Figure 1 A block diagram of the structure of the storage device of the projector.

[0012] Figure 5 This is a schematic diagram illustrating the display of a third projected image.

[0013] Figure 6 This is a schematic diagram illustrating the display of the fourth projected image.

[0014] Figure 7 This is a schematic diagram illustrating the display of the fifth projected image.

[0015] Figure 8 This is a schematic diagram illustrating the display of the sixth projected image.

[0016] Figure 9 This is a schematic diagram illustrating the display of the second projected image.

[0017] Figure 10 It is shown Figure 1 A block diagram of the server's structure.

[0018] Figure 11 It is shown Figure 3 Processing device and Figure 10 A flowchart of the operation of the processing device.

[0019] Figure 12 It is shown Figure 11 A flowchart of an example of the actions of a subroutine.

[0020] Figure 13 This is a flowchart illustrating an example of the action of variation 8.

[0021] Label Explanation

[0022] 1…Image display system, 10…Projector, 12…Processing device, 13…Shooting device, 14…Projection device, 20…Server, 22…Processing device, 213…First learning model, CN1, CN1-1, CN1-2, CN1-3, CN1-4…Angle, FP1, FP1-1, FP1-2, FP1-3, FP1-4, FP2, FP2-1, FP2-2, FP2-3, FP2-4…Feature point, GP1…First projected image, GP2…Second projected image, GS1…First captured image, GS2…Second captured image, R1…First region, SC…Screen. Detailed Implementation

[0023] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, the dimensions and scales of the parts in the drawings may differ from actual dimensions, and some parts are shown schematically for ease of understanding. Moreover, unless otherwise specified in the following description, the scope of the present invention is not limited to these embodiments.

[0024] 1. First Implementation Method

[0025] 1.1. Overview of Image Display Systems

[0026] The following is for reference Figures 1-12 The outline of the image display system 1 according to the first embodiment will be described. The image display system is an example of a feature point extraction system.

[0027] Figure 1 This is a structural diagram of the image display system 1 according to the first embodiment. The image display system 1 includes a projector 10, a server 20, and a network NET.

[0028] A network (NET) includes one or both of a wired network and a wireless network. A network (NET) is an electrical communication line that includes the Internet, intranets, etc. In the image display system 1, the projector 10 and the server 20 are connected via the network (NET) in a manner that enables them to communicate with each other.

[0029] The projector 10 displays a projected image on the projection surface by projecting light onto the projection surface. The projector 10 has the function of correcting the shape, brightness, hue, etc. of the projected image using a captured image obtained by capturing the displayed projected image.

[0030] Server 20 has the following functions: acquiring a captured image from projector 10, and extracting feature points for correcting the shape of the projected image based on the acquired image. Additionally, server 20 has the function of sending feature point information related to the extracted feature points to projector 10.

[0031] Therefore, the image display system 1 is a system that extracts feature points from the server 20 to correct the shape of the projected image of the projector 10, and corrects the shape of the projected image based on the extracted feature points. In this specification, the feature point extraction process, which is the initial stage of correcting the shape of the projected image, will be described.

[0032] 1.2. Overview of Projectors

[0033] Figure 2This is a schematic diagram illustrating the display of the first projected image GP1 during a first period. The projector 10 displays the first projected image GP1 by projecting light onto a first region R1 of the screen SC, which serves as the projection surface. The first projected image GP1 is a pure white image. A pure white image is an example of a white image. The first region R1 is larger than the area enclosed by the frame FR of the screen SC, covering the entire screen SC. The first period includes the time during which the user activates the setting function of the projector 10 and begins extracting feature points corresponding to the four corners CN1-1 to CN1-4 of the frame FR of the screen SC. The grayscale of each color (R, G, B) of the pure white image is set to maximum. However, the grayscale of the pure white image does not necessarily have to be maximum. Furthermore, the first projected image GP1 is not limited to a pure white image and may also be an image of other colors. Furthermore, the first projected image GP1 is not limited to a monochrome image and may also be an image containing multiple colors.

[0034] The projector 10 includes: an imaging device 13 for capturing images of a projection surface covering a predetermined area; and a projection device 14 for projecting light onto the projection surface. The imaging device 13 includes an imaging lens 132 for focusing light and an imaging element 131 for generating an image by converting the light focused by the imaging lens 132 into an electrical signal. The imaging element 131 has multiple pixels. The projection device 14 includes: a light source (not shown); a light modulator 141 for modulating light emitted from the light source into projection light for displaying a projected image on the projection surface; and a projection lens 142 for projecting the projection light modulated by the light modulator 141 onto the projection surface. The light modulator 141 has multiple pixels. The projector 10 displays a projected image on the projection surface by controlling the projection device 14. In this embodiment, the projector 10 displays a projected image on a screen SC by controlling the projection device 14.

[0035] 1.3. Structure of the Projector

[0036] Figure 3 It is shown Figure 1 A block diagram illustrating the structure of a projector 10. The projector 10 includes: a storage device 11 for storing various information; a processing device 12 for controlling the operation of the projector 10; an imaging device 13 for imaging a range encompassing a predetermined area of ​​the projection surface; a projection device 14 for projecting light onto the projection surface; an operation device 15 for receiving input operations from a user; and a communication device 16 for communicating with other devices. The various elements of the projector 10 are interconnected via one or more buses for information communication. The imaging device 13 is an example of a first imaging device, and the projection device 14 is an example of a first projection device.

[0037] The processing device 12 functions as a projection control unit 121, an image capture control unit 122, a first determination unit 123, a communication control unit 124, a second determination unit 125, an extraction unit 126, and a third determination unit 127. As described above, the image capture device 13 includes an image capture element 131 and an image capture lens 132. Furthermore, as described above, the projection device 14 includes a light source (not shown), a light modulator 141, and a projection lens 142.

[0038] The storage device 11 is configured, for example, to include volatile memory such as RAM and non-volatile memory such as ROM. Here, RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory.

[0039] Figure 4 It is shown Figure 1 A block diagram of the structure of the storage device 11 of the projector 10. The storage device 11 has non-volatile memory storing: a control program 100 that specifies the operation of the projector 10; projected image information 101 that represents the image projected onto the projection surface; captured image information 108 that represents the result of capturing a region on the projection surface containing the projected image; and coordinate information 111 that represents the coordinates of points contained in various images.

[0040] The projected image information 101 includes: first projected image information 102, which represents the image projected when displaying the first projected image GP1; second projected image information 103, which represents the image projected when displaying the second projected image GP2; third projected image information 104, which represents the image projected when displaying the third projected image GP3; fourth projected image information 105, which represents the image projected when displaying the fourth projected image GP4; fifth projected image information 106, which represents the image projected when displaying the fifth projected image GP5; and sixth projected image information 107, which represents the image projected when displaying the sixth projected image GP6.

[0041] The captured image information 108 includes first captured image information 109 representing a first captured image GS1 and second captured image information 110 representing a second captured image GS2. The first captured image GS1 is an image captured when the first projected image GP1 is projected. In other words, the first projected image GP1 is an image projected when the first captured image GS1 is captured.

[0042] Coordinate information 111 represents the coordinate information of the extracted feature points.

[0043] The volatile memory of the storage device 11 is used by the processing device 12 as a working area when executing the control program 100.

[0044] Furthermore, part or all of the storage device 11 may be located on an external storage device or an external server, etc. Additionally, part or all of the various information stored in the storage device 11 may be pre-stored in the storage device 11 or retrieved from an external storage device or an external server, etc. (See again...) Figure 3 The processing device 12 is configured to include one or more CPUs. However, the processing device 12 may replace a CPU or, in addition to a CPU, include programmable logic devices such as FPGAs or ASICs. Here, CPU is an abbreviation for Central Processing Unit, FPGA is an abbreviation for Field-Programmable Gate Array, and ASIC is an abbreviation for Application-Specific Integrated Circuit. The processing device 12 reads the control program 100 from the storage device 11. The processing device 12 executes the read control program 100 as... Figure 3 The projection control unit 121, shooting control unit 122, first determination unit 123, communication control unit 124, second determination unit 125, extraction unit 126, and third determination unit 127 shown in the diagram perform their functions.

[0045] The projection control unit 121 controls the projection device 14 to project projection light for displaying images onto the projection surface. Specifically, the projection control unit 121 displays a projected image on the projection surface by causing the projection device to project projection light based on the projected image information 101. In other words, the projection control unit 121 displays a projected image on the projection surface by projecting the image shown in the projected image information 101 from the projection device. Additionally, the projection control unit 121 controls the projection device 14 to display images on the projection surface to assist user operation.

[0046] In this embodiment, the projection control unit 121 controls the projection device 14 to project projection light for displaying images onto the screen SC, which serves as the projection surface.

[0047] Specifically, the projection control unit 121 displays a projected image on the screen SC by causing the projection device 14 to project projection light based on the projected image information 101. More specifically, the projection control unit 121 displays a first projected image GP1 on the screen SC by causing the projection device 14 to project projection light based on the first projected image information 102. Furthermore, the projection control unit 121 displays a second projected image GP2 on the screen SC by causing the projection device 14 to project projection light based on the second projected image information 103.

[0048] Furthermore, the projection control unit 121 displays a third projected image GP3 on the screen SC by projecting projection light based on the third projected image information 104 onto the projection device 14. Additionally, the projection control unit 121 displays a fourth projected image GP4 on the screen SC by projecting projection light based on the fourth projected image information 105 onto the projection device 14. Furthermore, the projection control unit 121 displays a fifth projected image GP5 on the screen SC by projecting projection light based on the fifth projected image information 106 onto the projection device 14. Finally, the projection control unit 121 displays a sixth projected image GP6 on the screen SC by projecting projection light based on the sixth projected image information 107 onto the projection device 14.

[0049] The imaging control unit 122 controls the imaging device 13 to capture images of a region including the area on the projection surface where the projected image is displayed. Furthermore, the imaging control unit 122 acquires an image representing the result of the imaging from the imaging device 13. Additionally, the imaging control unit 122 stores image information 108 representing the acquired image in the storage device 11.

[0050] In this embodiment, the shooting control unit 122 controls the shooting device 13 to capture an image of a region including the first area R1 on the screen SC displaying the first projected image GP1. Furthermore, the shooting control unit 122 acquires a captured image representing the result of the shooting from the shooting device 13. Additionally, the shooting control unit 122 stores first captured image information 109 representing the acquired captured image in the storage device 11.

[0051] The first determination unit 123 determines whether the projector 10 is connected to the network NET. More specifically, the first determination unit 123 determines that the projector 10 is connected to the network NET (“Yes”) if a response is received from a specific address on the Internet via the ping (Packet Internet Groper) command. On the other hand, the processing device 12 determines that the projector 10 is not connected to the network NET if no response is received from the specific address on the Internet via the ping command within a specified time (“No”).

[0052] When the projector 10 is connected to the network NET, the communication control unit 124 sends the first captured image GS1 to the server 20 via the communication device 16. Furthermore, the communication control unit 124 sends the second captured image GS2 to the server 20 via the communication device 16.

[0053] Figure 5 This is a schematic diagram illustrating the display of the third projected image GP3. (For example...) Figure 5As shown, the third projected image GP3 is an image displaying a dialog box DB1 and a selection button SB1 on a completely white image. Therefore, the third projected image GP3 is an image of the first projected image GP1 with the dialog box DB1 and the selection button SB1 added on top.

[0054] In dialog box DB1, the message "Connecting to the network will speed up setup preparation. Connect to the network?" and selection button SB1 are displayed. Selection button SB1 includes a "Yes" button SB1-1 and a "No" button SB1-2. The user can select either the "Yes" button SB1-1 or the "No" button SB1-2 by operating the operating device 15.

[0055] Refer again Figure 3 If the projector 10 is not connected to the network NET, the second determination unit 125 determines whether the user has connected the projector 10 to the network NET. More specifically, the second determination unit 125 determines that the user has connected the projector 10 to the network NET ("Yes") based on whether the user has selected the "Yes" button SB1-1 and whether wired or wireless LAN communication has been established. On the other hand, the processing device 12 determines that the user has not connected the projector 10 to the network NET ("No") based on whether the user has selected the "No" button SB1-2.

[0056] When the user selects not to connect the projector 10 to the network NET, the extraction unit 126 extracts feature points FP2-1 to FP2-4 that correspond one-to-one with the four corners CN1-1 to CN1-4 of the screen SC. In the functions related to the extraction of feature points FP2-1 to FP2-4, known image processing techniques can also be used. Examples of known image processing techniques related to feature point detection include template matching, centroid detection, and an algorithm called "AKAZE (Accelerated KAZE)". In this embodiment, "AKAZE" is used as the image processing technique. Detailed technical descriptions of the "AKAZE" algorithm are omitted in this specification.

[0057] In this embodiment, the extraction unit 126 performs first image processing on the image represented by the first captured image information 109 or the image represented by the second captured image information 110 to obtain coordinate information 111 representing the coordinates of four sets of feature points FP2-1 to FP2-4, which correspond one-to-one with the four corners, contained in the first captured image GS1 represented by the first captured image information 109 or the second captured image GS2 represented by the second captured image information 110. Furthermore, the extraction unit 126 stores the obtained coordinate information 111 in the storage device 11. The first image processing is image processing for detecting feature points.

[0058] Figure 6 This is a schematic diagram illustrating the display of the fourth projected image GP4. (As shown) Figure 6 As shown, according to the extraction unit 126, multiple feature points are extracted from the four groups of feature points FP2-1 to FP2-4 respectively. In this case, the user needs to select one feature point from the multiple feature points.

[0059] Figure 7 This is a schematic diagram illustrating the display of the fifth projected image GP5. (For example...) Figure 7 As shown, the fifth projected image GP5 is an image displaying four feature points FP1-1 to FP1-4 on a completely white image.

[0060] Figure 8 This is a schematic diagram illustrating the display of the sixth projected image GP6. (As shown) Figure 8 As shown, the sixth projected image GP6 is an image displaying four feature points FP1-1 to FP1-4, a dialog box DB2, and a selection button SB2 on a completely white image. Therefore, the sixth projected image GP6 is an image of the fifth projected image GP5 with the dialog box DB2 and the selection button SB2 added.

[0061] The dialog box DB2 displays the message "Are the positions of the displayed feature points correct?" and a selection button SB2. Selection button SB2 includes a "Yes" button SB2-1 and a "No" button SB2-2. The user can select either "Yes" button SB2-1 or "No" button SB2-2 by operating the operating device 15. In other words, the dialog box DB2 prompts the user to check whether the positions of the four projected feature points FP1-1 to FP1-4 correspond to the four corresponding corners CN1-1 to CN1-4.

[0062] Figure 9 This is a schematic diagram illustrating the display of the second projected image GP2. (As shown) Figure 9 As shown, the second projected image GP2 is a completely white image with a lower brightness than the first projected image GP1. That is, the second projected image GP2 is an image different from the first projected image GP1. Furthermore, the second projected image GP2 is not limited to a completely white image with a lower brightness than the first projected image GP1. Moreover, the second projected image GP2 is not limited to a completely white image; it can also be an image of other colors. Furthermore, the second projected image GP2 is not limited to a monochrome image; it can also be an image containing multiple colors.

[0063] Refer again Figure 3The third determination unit 127 responds to the message from the sixth projected image GP6 to determine whether the user's judgment is "correct". More specifically, the processing device 12 determines that the user's judgment is "correct" ("Yes") based on the user selecting the "Yes" button SB2-1. On the other hand, the processing device 12 determines that the user's judgment is not "correct" ("No") based on the user selecting the "No" button SB2-2.

[0064] The imaging element 131 is, for example, an image sensor such as a CCD or CMOS. Here, CCD is short for Charge Coupled Device, and CMOS is short for Complementary Metal Oxide Semiconductor.

[0065] The imaging device 13, under the control of the imaging control unit 122, captures images of a region on the projection surface displaying the projected image. Furthermore, the imaging device 13 outputs image information 108, representing the result of capturing images of the region on the projection surface displaying the projected image, to the processing device 12. In other words, the imaging device 13 outputs the captured image shown in the image information 108 to the processing device 12.

[0066] The light modulator 141 may be configured to include one or more liquid crystal panels. Alternatively, the light modulator 141 may be configured to include a DMD instead of a liquid crystal panel. Based on a signal input from the processing device 12, the light modulator 141 modulates light emitted from a light source into projected light for displaying a projected image on the projection surface. The light source may include, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED, or a laser light source. Here, LED is an abbreviation for Light Emitting Diode, and DMD is an abbreviation for Digital Mirror Device.

[0067] Under the control of the projection control unit 121, the projection device 14 projects projection light for displaying a projected image on the projection surface. In other words, the projection device 14 projects the image input from the processing device 12 onto the projection surface.

[0068] The operating device 15 receives input operations from the user of the projector 10. The operating device 15 may be configured to include, for example, a touch panel or operation buttons disposed on the housing of the projector 10. When the operating device 15 includes a touch panel, it outputs data indicating the detected touch location to the processing device 12. Alternatively, when the operating device 15 includes operation buttons, it outputs data identifying the pressed button to the processing device 12. The operating device 15 may also include a receiving device for receiving operation signals output from a remote control based on the user's operation. When the operating device 15 includes a receiving device, it outputs data representing the operation signals received from the remote control to the processing device 12. Thus, the content of the input operation to the projector 10 is transmitted to the processing device 12.

[0069] The communication device 16 is hardware used as a transceiver for communicating with other devices. The communication device 16 is also referred to as a network device, network controller, network interface card (NIC), communication module, etc. The communication device 16 may also have a connector for wired connection and an interface circuit corresponding to that connector. Additionally, the communication device 16 may also have a wireless communication interface. Examples of wired connectors and interface circuits include those conforming to HDMI (High-Definition Multimedia Interface; registered trademark), DisplayPort (registered trademark), wired LAN, IEEE 1394, USB, etc. Examples of wireless communication interfaces include those conforming to wireless LAN and Bluetooth (registered trademark).

[0070] 1.4. Server Structure

[0071] Figure 10 It is shown Figure 1 A block diagram of the structure of server 20. Server 20 includes a storage device 21 for storing various information, a processing device 22 for controlling the operation of server 20, and a communication device 23 for communicating with other devices. The various elements of server 20 are interconnected through one or more buses for communicating information. Server 20 is an example of an external device.

[0072] The processing device 22 functions as a communication control unit 221, a generation unit 222, an extraction unit 223, and a decision unit 224.

[0073] Storage device 21 is configured to include, for example, volatile memory such as RAM and non-volatile memory such as ROM. Storage device 21 stores control program 211, coordinate information 212, first learning model 213, second learning model 214, first training data 215, second training data 216, image information 217, etc. The volatile memory of storage device 21 is used by processing device 22 as the working area of ​​processing device 22. Control program 211 is a program that controls the entire server 20.

[0074] Furthermore, part or all of the storage device 21 may be located on an external storage device, an external server, or the like. In addition, part or all of the various information stored in the storage device 21 may be pre-stored in the storage device 21 or obtained from an external storage device, an external server, or the like.

[0075] The processing device 22 is configured to include one or more CPUs. However, the processing device 12 may also replace the CPU or include programmable logic devices such as FPGAs or ASICs in addition to CPUs.

[0076] Processing device 22 reads control program 211 from storage device 21. Processing device 22 executes the read control program 211 as... Figure 10 The communication control unit 221, generation unit 222, extraction unit 223, and decision unit 224 shown in the diagram perform their functions.

[0077] The communication control unit 221 receives various information, including the captured image output from the projector 10, via the communication device 23. The communication control unit 221 transmits various information, including coordinate information and image information, to the projector 10 via the communication device 23.

[0078] The generation unit 222 generates a first learning model 213 and a second learning model 214. More specifically, the generation unit 222 uses multiple datasets contained in the first training data 215 to perform machine learning on the neural network model, thereby generating the first learning model 213. Additionally, the generation unit 222 uses multiple datasets contained in the second training data 216 to perform machine learning on the neural network model, thereby generating the second learning model 214. The first learning model 213 is an example of a model that has completed learning.

[0079] More specifically, the training data for the first learning model 213 consists of multiple datasets. One dataset includes input data containing captured images, output data extracted from those captured images, and decision labels. The captured images are obtained by photographing the area including the projected image projected onto the projected object during the projector setup. The output data contains the coordinates of four feature points corresponding one-to-one with the four corners of the projected object. The decision labels are used to determine whether the positions of the four feature points match the positions of the corresponding four corners. The structure of the training data is not limited to this. Specifically, the input data may not include the projected image.

[0080] The judgment label is determined by the user to determine whether the positions of the four feature points are consistent with the positions of the four corresponding corners. More specifically, the user observes the projected image containing the extracted feature points and judges whether the positions of the four feature points are consistent with the positions of the four corresponding corners.

[0081] In this embodiment, a convolutional neural network (CNN) is used as the first learning model 213.

[0082] The training data for the second learning model 214 consists of multiple datasets. One dataset includes input data containing captured images, output data extracted from those captured images, and decision labels. The captured images are obtained by photographing the area including the projected image projected onto the projected object during projector setup. The output data includes information related to the projected image used to extract four feature points (corresponding one-to-one with the four corners of the projected object) in cases where the positions of these four feature points do not match the original positions of the four corners. The decision labels are used to determine whether the positions of the four feature points extracted from the projected image used for this purpose match the positions of the original four corners. The structure of this training data is not limited to this. Specifically, the input data may not include the projected image.

[0083] The judgment label is determined by the user to determine whether the positions of the four feature points are consistent with the positions of the four corresponding corners. More specifically, the user observes the projected image containing the extracted feature points and judges whether the positions of the four feature points are consistent with the positions of the four corresponding corners.

[0084] In this embodiment, a CNN is used as the second learning model 214.

[0085] The extraction unit 223 inputs the first captured image GS1 into the first learning model 213. Additionally, the extraction unit 223 acquires the coordinates of four feature points FP1-1 to FP1-4, output from the first learning model 213 and corresponding one-to-one with the four corners CN1-1 to CN1-4. Furthermore, the extraction unit 223 stores coordinate information 212, representing the coordinates of the four acquired feature points FP1-1 to FP1-4, in the storage device 21. The coordinate information 212 is an example of the first output data. The first captured image GS1 input into the first learning model 213 is an example of the first input data.

[0086] The decision unit 224 determines the second projected image GP2 to be projected by the projection device 14. More specifically, the decision unit 224 inputs the first captured image GS1 into the second learning model 214, and determines the second projected image GP2 based on the image information 217 related to the projected image output from the second learning model 214. Furthermore, when the image information 217 related to the projected image is output from the second learning model 214, the decision unit 224 causes the storage device 21 to store the image information 217 related to the projected image. The image information 217 may be code information indicating the type of image, or it may be information containing image data.

[0087] The communication device 23 is hardware that serves as a transceiver for communicating with other devices. The communication device 23 is also referred to as a network device, network controller, network interface card (NIC), communication module, etc. The communication device 23 may also have a connector for wired connection and an interface circuit corresponding to that connector. Additionally, the communication device 23 may also have a wireless communication interface. Examples of wired connectors and interface circuits that conform to wired LAN, IEEE 1394, and USB are possible. Examples of wireless communication interfaces that conform to wireless LAN and Bluetooth (registered trademark) are possible.

[0088] 1.5. Specific Actions in Image Display System 1

[0089] Next, refer to Figure 11 as well as Figure 12 The specific actions in the image display system 1 are explained.

[0090] 1.5.1. Operation of the image display system

[0091] Figure 11 It is shown Figure 3 The processing device 12 and Figure 10 A flowchart of the operation of the processing device 22. Hereinafter, refer to... Figure 11 The operation of the processing device 12 and the processing device 22 will be explained. Figure 11The routine, for example, begins by the user operating the operating device 15 to invoke the function of the projector 10 to prepare the settings for the projected image.

[0092] In step S11, the processing device 12 functions as a projection control unit 121 to project the first projection image GP1 onto the screen SC.

[0093] The first projected image GP1 is a completely white image.

[0094] In step S12, the processing device 12 functions as a shooting control unit 122, and uses the shooting device 13 to acquire a first captured image GS1 including the first region R1. Furthermore, the processing device 12 stores first captured image information 109 representing the acquired first captured image GS1 in the storage device 11.

[0095] In step S13, the processing device 12 functions as the first determination unit 123 to determine whether the projector 10 is connected to the network NET.

[0096] When the projector 10 is connected to the network NET, that is, when the determination result in step S13 is "yes", the processing device 12 functions as a communication control unit 124 in step S14 and sends the first captured image GS1 to the server 20 via the communication device 16.

[0097] In step S15, processing devices 12 and 22 extract feature points, temporarily terminating this routine. Details of the processing in step S15 will be described later.

[0098] On the other hand, if the projector 10 is not connected to the network NET, that is, if the determination result in step S13 is "No", the processing device 12 functions as the projection control unit 121 in step S16, projecting a third projection image GP3 onto the screen SC, including a query asking whether to connect the projector 10 to the network NET to advance the "settings". More specifically, in this case, the processing device 12 projects a dialog box DB1 onto the screen SC containing the message "If connected to the network, setup preparation can be completed faster. Connect to the network?" and selection buttons SB1 for "Yes" and "No".

[0099] In step S17, the processing device 12 functions as the second determination unit 125 to determine whether the user has connected the projector 10 to the network NET.

[0100] If it is determined that the user has connected the projector 10 to the network NET, that is, if the determination result in step S17 is "yes", the processing device 12 functions as a communication control unit 124 in step S14 and sends the captured image to the server 20. Specifically, the processing device 12 functions as a communication control unit 124 in step S14 and sends the first captured image GS1 to the server 20.

[0101] On the other hand, if it is determined that the user has not connected the projector 10 to the network NET, that is, if the determination result in step S17 is "no", the processing device 12 functions as the extraction unit 126 in step S18, extracts the feature points FP2-1 to FP2-4 that correspond one-to-one with the four corners CN1-1 to CN1-4 of the screen SC, and temporarily ends this routine.

[0102] 1.5.2. Operation of the Image Display System

[0103] Figure 12 It is shown Figure 11 The flowchart shows an example of the actions of a subroutine. See below for reference. Figure 12 The actions of the subroutine in step S15 will be explained.

[0104] In step S1501, the processing device 22 functions as an extraction unit 223, inputting the first captured image GS1 into the first learning model 213.

[0105] In step S1502, the processing device 22 functions as an extraction unit 223, acquiring the coordinates of four feature points FP1-1 to FP1-4, which correspond one-to-one with the four corners CN1-1 to CN1-4, output from the first learning model 213. Furthermore, the processing device 22 stores the coordinate information 212 representing the acquired coordinates of the four feature points FP1-1 to FP1-4 in the storage device 21.

[0106] In step S1503, the processing device 22 functions as a communication control unit 221, sending feature point information related to the four acquired feature points to the projector 10.

[0107] In step S1504, the processing device 12 functions as the projection control unit 121, projecting the fifth projection image GP5, which includes feature points FP1-1 to FP1-4, onto the screen SC.

[0108] In step S1505, the processing device 12 functions as the projection control unit 121, querying the user whether the positions of feature points FP1-1 to FP1-4 displayed on the screen SC are correct. More specifically, the processing device 12 projects a sixth projection image GP6 onto the screen SC, containing the feature points, a message "Are the positions of the displayed feature points correct?", and "Yes" and "No" selection buttons SB2. Here, "the positions of the feature points are correct" means that feature points FP1-1 to FP1-4 correspond to the four corners CN1-1 to CN1-4 of the frame FR of the screen SC, respectively.

[0109] In step S1506, the processing device 12 functions as the third determination unit 127 to determine whether the user's judgment is "correct".

[0110] If the user's judgment is deemed "correct," that is, if the judgment result in step S1506 is "yes," the processing device 12 temporarily terminates the current routine. That is, the positions of the extracted feature points FP1-1 to FP1-4 are determined as the positions of the four corners CN1-1 to CN1-4 of the frame FR of the screen SC, respectively.

[0111] On the other hand, if the user does not determine that the judgment is "correct," that is, if the judgment result in step S1506 is "no," the processing device 22 functions as a decision unit 224 in step S1507, thereby determining the second projected image GP2 to be projected by the projection device 14. More specifically, by inputting the first captured image GS1 into the second learning model 214, the second projected image GP2 is determined based on the second image information representing the "type of image" output from the second learning model 214.

[0112] In step S1508, the processing device 22 functions as a communication control unit 221, and sends the determined second image information to the projector 10 via the communication device 23.

[0113] In step S1509, the processing device 12 functions as a projection control unit 121, projecting the second projection image GP2 based on the second image information onto the screen SC.

[0114] In step S1510, the processing device 12 functions as a shooting control unit 122 and acquires a second captured image GS2. More specifically, the processing device 12 acquires the second captured image GS2 obtained by the shooting device 13 capturing a range of a second region R2, including the screen SC on which the second projected image GP2 is projected.

[0115] In step S1511, the processing device 12 functions as the extraction unit 126, extracting four sets of feature points FP2-1 to FP2-4 corresponding to the four corners CN1-1 to CN1-4 through image processing, and temporarily ending this routine.

[0116] 1.6. Effects of the first embodiment

[0117] As explained above, the feature point extraction method according to the first embodiment includes: inputting a first captured image GS1 into a first learning model 213; and obtaining coordinate information 212 representing the coordinates of four feature points FP1-1 to FP1-4, which correspond one-to-one with the four corners CN1-1 to CN1-4, output from the first learning model 213. When input data including the captured image is input, the first learning model 213 outputs output data representing the four feature points FP1-1 to FP1-4, which correspond one-to-one with the four corners, in the captured image. The captured image is obtained by capturing an area including the projection object from the projection device and the four corners of the projection object. The first captured image GS1 is an image obtained by capturing a first region R1 including the screen SC and the four corners CN1-1 to CN1-4 of the screen SC.

[0118] According to this method, when input data including a first captured image GS1 is input, the first learning model 213 outputs output data representing one feature point FP1 corresponding to a corner CN1 in the first captured image GS1, which is obtained by capturing an area including the screen SC and the corner CN1 of the screen SC, and the screen SC is an image projected from the projection device 14. Therefore, the effort required for the user to select an appropriate feature point from several candidate feature points corresponding to a corner CN1 of the screen SC is reduced, alleviating the inconvenience felt by the user. Thus, user convenience is improved. Furthermore, according to this method, feature points FP1-1 to FP1-4 are extracted at the four corners CN1-1 to CN1-4 respectively, thus improving the accuracy of geometric correction using the extracted feature points.

[0119] In addition, the feature point extraction method according to the first embodiment captures a first captured image GS1 by the shooting device 13, has a first learning model 213 in the server 20, sends the first input data from the shooting device 13 to the server 20, and sends the coordinate information 212 from the server 20 to the projection device 14.

[0120] According to this method, the computational processing for extracting feature points FP1 is not performed in the processing unit 12 of the projector 10, but is performed on an external server 20, thus reducing the processing load on the processing unit 12 of the projector 10. Furthermore, if the computational processing for extracting feature points FP1 needs to be updated, the control program 100 of the processing unit 12 of the projector 10 does not need to be updated, thus reducing the burden on the user.

[0121] Furthermore, the feature point extraction method of the first embodiment further includes: projecting the four acquired feature points FP1-1 to FP1-4 onto the screen SC; if the user answers "no" to a question regarding whether the positions of the four projected feature points FP1-1 to FP1-4 correspond to the four corresponding corners CN1-1 to CN1-4, projecting a second projected image GP2 that is different from the first projected image GP1 projected when the first captured image GS1 was captured; performing first image processing on the second captured image GS2; and obtaining coordinate information 111 extracted by the first image processing, representing the four sets of feature points FP2-1 to FP2-4 that correspond one-to-one with the four corners. The second captured image GS2 is an image obtained by capturing a second region R2 including the screen SC, the four corners CN-1 to CN1-4, and the second projected image GP2 projected onto the screen SC.

[0122] If the extracted feature point FP1 is unacceptable to the user, consider extracting the feature point again. According to this method, when extracting the feature point again, the projected image changes from the first projected image GP1 to the second projected image GP2, and image processing is used instead of the first learning model 213 for feature point extraction. Therefore, it is expected that the position of the extracted feature point will change from the original position.

[0123] 2. Variations

[0124] This invention is not limited to the embodiments described above, and various modifications can be made within the scope of this invention. Specific modifications are illustrated below. Furthermore, two or more modifications selected from the following examples can be appropriately combined without contradiction. In addition, in the modifications illustrated below, elements with the same function or effect as those described in the preceding embodiments are represented by the same reference numerals used in the above description, and their detailed descriptions are appropriately omitted.

[0125] 2.1. Variation Example 1

[0126] In the first embodiment, the first projected image GP1 is a completely white image. In this case, the determination unit 224 can determine the second projected image based on the brightness distribution of the first captured image GS1. Here, the brightness distribution represents the degree of light reflection by the screen SC in the first captured image GS1. Furthermore, the first captured image GS1 also includes the first projected image GP1. The brightness distribution here can be represented using the Y value of the XYZ values ​​or using RGB values.

[0127] For example, during the preparation phase of projector setup, by accumulating the data up to this point, it can be known that the degree of light reflection varies with the reflectivity and shape of the projected object, which affects the accuracy of feature point extraction. It can also be known that the accuracy of feature point extraction can be improved by changing the projected image. In this case, it is advisable to prepare multiple projected images in advance as candidates.

[0128] In the first embodiment, the second projected image GP2 is determined by inputting the first captured image GS1 into the second learning model 214. However, in the learning model of Variation 1, in addition to the first captured image GS1, information on the average brightness of the first projected image GP1 is also input to determine the second projected image. The average brightness of the first projected image GP1 is calculated as the average brightness of the range obtained by capturing the first region R1 in the first captured image GS1.

[0129] Therefore, the input data in the training data of the learning model of Variation 1 includes a photographed image obtained by taking pictures of the area including the projected image projected onto the projected object and the projected object during the setup preparation of the projector, as well as the average brightness of the projected image.

[0130] Based on this approach, it is expected that the accuracy of feature point extraction can be improved by changing the projected image.

[0131] 2.2. Variation Example 2

[0132] Furthermore, in Modification 1, the average brightness of the first captured image GS1 and the first projected image GP1 is used as input data for the learning model. However, in Modification 2, in addition to the average brightness of the first captured image GS1 and the first projected image GP1, the gain and exposure settings of the shooting device 13 when capturing the first captured image GS1 can also be used as input data for the learning model.

[0133] According to this method, since the learning model learns by using the shooting conditions of the shooting device 13, it is expected to further improve the accuracy of feature point extraction.

[0134] 2.3. Variation Example 3

[0135] In addition, in the first embodiment, it is configured to determine whether the positions of the four feature points are consistent with the positions of the corresponding four corners, but it can also be configured to determine whether the positions of the four feature points are consistent with the positions of the corresponding corners separately.

[0136] 2.4. Variation Example 4

[0137] Furthermore, in the first embodiment, the configuration involves determining whether the positions of the four feature points are consistent with the positions of the corresponding four corners. If the user determines that the two are inconsistent, the feature points are re-extracted. However, it can also be configured so that the positions of the feature points that the user determines are inconsistent can be corrected by the user operating the operation device 15. 2.5. Modification 5

[0138] Furthermore, while the first embodiment is configured for the user to determine whether the positions of the four feature points correspond to the positions of the four corresponding corners, it can also be configured to include multiple levels of evaluation values. For example, consider a structure with five levels of evaluation values. In a structure with multiple levels of evaluation values, the weighted average of the coordinates of the previously extracted feature points and the coordinates of the newly extracted feature points is calculated by changing the weighting distribution based on the evaluation values.

[0139] 2.6. Variation Example 6

[0140] Regarding the training data for the first learning model 213, a dataset can also be constructed using input data containing captured images and output data extracted from those images. The captured images are obtained by photographing the area including the projected image projected onto the projected object during projector setup and preparation. The output data contains the coordinates of four feature points corresponding one-to-one with the four corners of the projected object. Only the dataset from multiple datasets where the user determines that the positions of the four feature points match the positions of the corresponding four corners is used as training data. In this method, the coordinates of the extracted feature points are used as the correct labels.

[0141] 2.7. Variation Example 7

[0142] In the first embodiment, by storing the first learning model 213 and the second learning model 214 in the storage device 21 of the server 20, the processing load of the processing device 12 of the projector 10 is reduced, but this disclosure is not limited to this method. Depending on the capabilities of the processing device of the projector 10, the first learning model and / or the second learning model may also be stored in the storage device 11 of the projector 10 and processed in the processing device 12.

[0143] 2.8. Variation Example 8

[0144] In the first embodiment, if the user determines that the position of feature point FP1 extracted in the extraction unit 223 of server 20 is incorrect, the projected image is changed from the first projected image GP1 to the second projected image GP2, and feature points are extracted in the extraction unit 126 of projector 10 through image processing. However, this disclosure is not limited to this method. In variation 8, if the user determines that the position of feature point FP1 extracted in the extraction unit 223 of server 20 is incorrect, the projected image is changed from the first projected image GP1 to the second projected image GP2, and feature points are extracted again in the extraction unit 223 of server 20.

[0145] Figure 13 This is a flowchart illustrating one example of the operation of variation 8. Hereinafter, by referring to... Figure 13 The operation of variation example 8 will be explained. Furthermore, Figure 13 and Figure 12 Corresponding to the flowchart, for Figure 12 The steps shown are labeled with the same numbers.

[0146] In step S1501, the processing device 22 functions as an extraction unit 223, inputting the first captured image GS1 into the first learning model 213.

[0147] In step S1502, the processing device 22 functions as an extraction unit 223, acquiring the coordinates of four feature points FP1-1 to FP1-4, which correspond one-to-one with the four corners CN1-1 to CN1-4, output from the first learning model 213. Furthermore, the processing device 22 stores the coordinate information 212 representing the acquired coordinates of the four feature points FP1-1 to FP1-4 in the storage device 21.

[0148] In step S1503, the processing device 22 functions as a communication control unit 221, sending feature point information related to the four acquired feature points to the projector 10.

[0149] In step S1504, the processing device 12 functions as the projection control unit 121, projecting the fifth projection image GP5, which includes feature points FP1-1 to FP1-4, onto the screen SC.

[0150] In step S1505, the processing device 12 functions as the projection control unit 121, querying the user whether the positions of feature points FP1-1 to FP1-4 displayed on the screen SC are correct. More specifically, the processing device 12 projects a sixth projection image GP6 onto the screen SC, including the feature points, a message "Are the positions of the displayed feature points correct?", and selection buttons SB2 for "Yes" and "No". Here, "the positions of the feature points are correct" means that the feature points FP1-1 to FP1-4 correspond to the four corners CN1-1 to CN1-4 of the frame FR of the screen SC, respectively.

[0151] In step S1506, the processing device 12 functions as the third determination unit 127 to determine whether the user's judgment is "correct".

[0152] If the user's judgment is deemed "correct," that is, if the judgment result in step S1506 is "yes," the processing device 12 temporarily terminates the current routine. That is, the positions of the extracted feature points FP1-1 to FP1-4 are determined as the positions of the four corners CN1-1 to CN1-4 of the frame FR of the screen SC, respectively.

[0153] On the other hand, if the user's judgment is not "correct," that is, if the judgment result in step S1506 is "no," the processing device 22 functions as a decision unit 224 in step S2501, determining the second projected image GP2 to be projected by the projection device 14. More specifically, by inputting the first captured image GS1 into the second learning model 214, the second projected image GP2 is determined based on the second image information representing the "type of image" output from the second learning model 214.

[0154] In step S2502, the processing device 22 functions as a communication control unit 221, and sends the determined second image information to the projector 10 via the communication device 23.

[0155] In step S2503, the processing device 12 functions as a projection control unit 121, projecting the second projection image GP2 based on the second image information onto the screen SC.

[0156] In step S2504, the processing device 12 functions as the shooting control unit 122 and acquires the second captured image GS2. More specifically, the processing device 12 acquires the second captured image GS2 obtained by the shooting device 13 capturing a range of a second area R2, including the screen SC on which the second projected image GP2 is projected.

[0157] In step S2505, the processing device 12 functions as a communication control unit 124 and sends the second captured image GS2 to the server 20 via the communication device 16.

[0158] In step S2506, the processing device 22 functions as an extraction unit 223, acquiring the coordinates of four feature points FP2-1 to FP2-4, which correspond one-to-one with the four corners CN1-1 to CN1-4, output from the first learning model 213, and temporarily terminating the current routine. Furthermore, the processing device 22 stores the coordinate information 212 representing the acquired coordinates of the four feature points FP2-1 to FP2-4 in the storage device 21.

[0159] 2.9. Variation Example 9

[0160] In the first embodiment, an image display system 1 using one projector was described as an example, but it is not limited to this method. This disclosure can also be applied to a multi-projection system that realizes a large screen with a horizontal length or stacking based on multiple projectors.

[0161] 2.10. Variation Example 10

[0162] In the first embodiment, during the first period, the projector 10 projects projection light onto a first region R1, including the screen SC which serves as the projection surface; however, the present invention is not limited to this method. For example, instead of projecting projection light onto the first region R1, including the screen SC which serves as the projection surface, the room where the screen SC is located may be made bright. That is, when extracting feature points corresponding to the four corners CN1-1 to CN1-4 of the frame FR of the screen SC, it is also possible not to project projection light onto the first region R1, including the screen SC which serves as the projection surface.

[0163] 2.11. Variation Example 11

[0164] In the first embodiment, by inputting the first captured image GS1 into the second learning model 214, the decision unit 224 determines the second projected image GP2 to be projected by the projection device 14, but the present invention is not limited to this method. For example, the decision unit 224 may also determine the second projected image GP2 to be projected by the projection device 14 based on pre-set image information 217 related to the projected image without using the second learning model 214.

[0165] Furthermore, in the feature point extraction method of Variation 11, the second projected image GP2 is predetermined as an image projected following the first projected image GP1.

[0166] For example, if multiple projected images suitable for feature point extraction exist due to the accumulation of data up to this point, it is advisable to prepare multiple projected images as candidates in advance. According to this approach, it is expected that the accuracy of feature point extraction can be improved by changing the projected images.

[0167] 3. Notes

[0168] The following is a summary published in this note.

[0169] 3.1. Appendix 1

[0170] The feature point extraction method in Appendix 1 includes the following steps: inputting first input data into a learned model, wherein the learned model, upon receiving input data including a captured image, outputs output data, the captured image being obtained by capturing a region including a projection object from a projection device and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner, the first input data including a first captured image, the first captured image being obtained by capturing a first region including a first projection object and at least one first corner of the first projection object; and obtaining first output data output from the learned model, wherein the first output data representing at least one first feature point corresponding to the at least one first corner.

[0171] According to the feature point extraction method in Appendix 1, after the model has been trained, when given input data including a captured image, it outputs output data representing one feature point corresponding to a corner in the captured image, which is obtained by capturing an area including the screen from which the image is projected from the projection device and the corner of the screen. Therefore, this reduces the hassle for the user of selecting an appropriate feature point from several candidate feature points corresponding to a corner of the screen, thus alleviating the user's experience. Consequently, user convenience is improved.

[0172] 3.2. Appendix 2

[0173] The feature point extraction method in Appendix 2 is based on Appendix 1. The projection object contains four first corners. When the learned model is input with input data including captured images, it outputs output data. The captured images are obtained by capturing the region including the projection object and the four corners of the projection object. The output data represents the four feature points in the captured images that correspond one-to-one with the four corners. The first captured image is an image obtained by capturing the first region including the first projection object and the four first corners of the first projection object. The first output data represents the four first feature points output from the learned model that correspond one-to-one with the four first corners.

[0174] Based on the feature point extraction method in Appendix 2, feature points are extracted at the four corners respectively, thus improving the accuracy of geometric correction using the extracted feature points.

[0175] 3.3. Appendix 3

[0176] The feature point extraction method in Appendix 3 is based on Appendix 1 or Appendix 2. The first image is captured by the first imaging device, the external device has the learned model, the first input data is sent from the first imaging device to the external device, and the first output data is sent from the external device to the projection device.

[0177] According to the feature point extraction method in Appendix 3, the computational processing for feature point extraction is performed externally, instead of within the processing unit of the projection device, thus reducing the processing load on the projection device's processing unit. Furthermore, if updates to the computational processing for feature point extraction are required, updates to the control program of the projection device's processing unit are unnecessary, thereby reducing the burden on the user.

[0178] 3.4. Appendix 4

[0179] The feature point extraction method in Appendix 4, based on any one of Appendix 1 to Appendix 3, further includes the following steps: projecting the obtained at least one first feature point onto the first projection object; if the user's answer to the question of whether the position of the projected at least one first feature point is consistent with the at least one first corner is negative, projecting a second projection image different from the first projection image projected when the first captured image is taken; inputting the second captured image into the learned model, wherein the second captured image is obtained by taking a picture of a second region including the first projection object, the at least one first corner, and the second projection image projected onto the first projection object; and obtaining second output data output from the learned model, wherein the second output data represents at least one second feature point corresponding one-to-one with the at least one first corner.

[0180] Based on the feature point extraction method in Appendix 4, if the extracted feature points are unacceptable to the user, feature points should be extracted again. According to this method, the projected image during the re-extraction of feature points changes from the first projected image to the second projected image. Therefore, it is expected that the positions of the re-extracted feature points will change from the original positions.

[0181] 3.5. Appendix 5

[0182] The feature point extraction method in Appendix 5, based on any one of Appendix 1 to Appendix 3, further includes the following steps: projecting the obtained at least one first feature point onto the first projection object; if the user's answer to the question of whether the position of the projected at least one first feature point is consistent with the corresponding at least one first corner is negative, projecting a second projection image different from the first projection image projected when the first captured image is taken; performing first image processing on the second captured image, wherein the second captured image is obtained by taking a picture of a second region including the first projection object, the at least one first corner, and the second projection image projected onto the first projection object; and obtaining third output data extracted by the first image processing, wherein the third output data represents at least one third feature point corresponding one-to-one with the at least one first corner.

[0183] If the extracted feature points are unacceptable to the user, consider re-extracting the feature points. According to the feature point extraction method in Appendix 5, the projected image used during re-extraction is changed from the first projected image to the second projected image, and image processing is used instead of the learned model for feature point extraction. Therefore, it is expected that the positions of the re-extracted feature points will change from the original feature point positions.

[0184] 3.6. Appendix 6

[0185] The feature point extraction method in Appendix 6 is based on Appendix 4, wherein the second projected image is predetermined as an image projected following the first projected image.

[0186] For example, if multiple projected images suitable for feature point extraction exist due to the accumulation of data up to this point, it is advisable to prepare multiple projected images as candidates in advance. Based on the feature point extraction method in Appendix 6, it is expected that the accuracy of feature point extraction can be improved by changing the projected images.

[0187] 3.7. Appendix 7

[0188] The feature point extraction method in Appendix 7, based on Appendix 4, includes the following steps: when the first projected image is a white image, the second projected image is determined based on the brightness distribution of the first captured image, wherein the brightness distribution represents the degree of light reflection of the first projected object in the first captured image, and the first captured image also includes the first projected image.

[0189] For example, by accumulating the data up to this point, it is known that the degree of light reflection varies with the reflectivity and shape of the projected object, affecting the accuracy of feature point extraction. Furthermore, it is known that changing the projected image can improve the accuracy of feature point extraction. In this case, it is advisable to prepare multiple projected images as candidates in advance. Based on the feature point extraction method in Appendix 7, it is expected that changing the projected image can improve the accuracy of feature point extraction.

[0190] 3.8. Appendix 8

[0191] The feature point extraction method in Note 8 is based on Note 5, wherein the second projected image is predetermined as an image projected following the first projected image.

[0192] For example, if multiple projected images suitable for feature point extraction exist due to the accumulation of data up to this point, it is advisable to prepare multiple projected images as candidates in advance. Based on the feature point extraction method in Appendix 8, it is expected that the accuracy of feature point extraction can be improved by changing the projected images.

[0193] 3.9. Appendix 9

[0194] The feature point extraction method in Appendix 9, based on Appendix 5, includes the following steps: when the first projected image is a white image, the second projected image is determined based on the brightness distribution of the first captured image, wherein the brightness distribution represents the degree of light reflection of the first projected object in the first captured image, and the first captured image also includes the first projected image.

[0195] For example, by accumulating the data up to this point, it is known that the degree of light reflection varies with the reflectivity and shape of the projected object, affecting the accuracy of feature point extraction. Furthermore, it is known that changing the projected image can improve the accuracy of feature point extraction. In this case, it is advisable to prepare multiple projected images as candidates in advance. Based on the feature point extraction method in Appendix 9, it is expected that changing the projected image can improve the accuracy of feature point extraction.

[0196] 3.10. Appendix 10

[0197] The feature point extraction method in Appendix 10, based on any one of Appendix 1 to Appendix 9, further includes the following steps: projecting the obtained at least one first feature point onto the first projection object; obtaining the user's answer to a question about whether the projected at least one first feature point is consistent with the at least one first angle; and using the first captured image, the first output data, and the obtained answer as learning data, so that the learned model can undergo additional learning.

[0198] According to the feature point extraction method in Appendix 10, by appending the set of the first captured image, the first output data, and the user's response as the data for learning, the accuracy of the learned model can be improved.

[0199] 3.11. Appendix 11

[0200] The feature point extraction system in Appendix 11 comprises: an external device having a learned model that, when input data including a captured image is input, outputs output data, the captured image being obtained by an imaging device capturing a region including a projection object from which an image is projected and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner; a first projection device that projects a first projected image onto a first projection object; and a first imaging device that captures a first region including the first projected image, the first projection object, and at least one first corner of the first projection object, the first imaging device sending first input data including the first captured image obtained by capturing the first region to the external device, the external device inputting the first input data to the learned model, obtaining first output data output from the learned model representing at least one first feature point corresponding to the at least one first corner, and sending the first output data to the first projection device.

[0201] According to the feature point extraction system in Appendix 11, after the model has been trained, when given input data including a captured image, it outputs output data representing one feature point corresponding to a corner in the captured image, which is obtained by capturing an area including the screen from which the image is projected from the projection device and the corner of the screen. Therefore, the hassle of selecting an appropriate feature point from several candidates corresponding to a corner of the screen is reduced, alleviating the user's experience. Thus, user convenience is improved.

[0202] 3.12. Appendix 12

[0203] The procedure in Appendix 12 causes the computer to perform the following processing: inputting first input data representing a first captured image to the learned model, wherein the learned model, upon being input with input data containing first capture information, outputs output data, the first capture information representing a captured image obtained by capturing an area including a projection object from a projection device and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner, the first captured image including the first projection object and at least one first corner of the first projection object; and obtaining first output data output from the learned model, wherein the first output data represents at least one first feature point corresponding to the at least one first corner.

[0204] According to the procedure in Appendix 12, after the learning model is completed and given input data including a captured image, it outputs output data representing one feature point corresponding to a corner in the captured image, which is obtained by capturing an area including the screen from which the image is projected from the projection device and the corner of the screen. Therefore, the hassle of selecting an appropriate feature point from several candidates corresponding to a corner of the screen is reduced, alleviating the user's perception of inconvenience. Thus, user convenience is improved.

Claims

1. A feature point extraction method, comprising the following steps: The learned model is input with first input data, wherein the learned model, upon receiving input data including a captured image, outputs output data. The captured image is obtained by capturing a region including a projection object from a projection device and at least one corner of the projection object. The output data represents at least one feature point in the captured image corresponding one-to-one with the at least one corner. The first input data includes a first captured image, which is obtained by capturing a first region including a first projection object and at least one first corner of the first projection object. Obtain first output data from the learned model, wherein the first output data represents at least one first feature point corresponding to at least one first angle.

2. The feature point extraction method according to claim 1, wherein, The projected object contains four first corners. Once the learning model has been completed, it outputs data when given input data including captured images. These captured images are of a region including the projected object and its four corners. The output data represents four feature points in the captured image that correspond one-to-one with the four corners. The first captured image is an image obtained by capturing a first region including the first projected object and its four first corners. The first output data represents the four first feature points output from the learned model, which correspond one-to-one with the four first angles.

3. The feature point extraction method according to claim 1, wherein, The first image is captured by the first imaging device. The external device has the learned model. The first input data is sent from the first imaging device to the external device. The first output data is sent from the external device to the projection device.

4. The feature point extraction method according to claim 1, wherein, The feature point extraction method further includes the following steps: Project at least one of the obtained first feature points onto the first projection object; If the user answers no to the question of whether the position of at least one first feature point of the projection is consistent with the at least one first angle, a second projection image different from the first projection image projected when the first captured image is captured is projected. Input a second captured image into the learned model, wherein the second captured image is obtained by capturing a second region including the first projection object, the at least one first corner, and the second projected image projected onto the first projection object; and Obtain second output data from the learned model, wherein the second output data represents at least one second feature point corresponding to at least one first angle.

5. The feature point extraction method according to claim 1, wherein, The feature point extraction method further includes the following steps: Project at least one of the obtained first feature points onto the first projection object; If the user answers negatively to a question about whether the position of at least one first feature point of the projection is consistent with the corresponding at least one first angle, a second projection image different from the first projection image projected when the first captured image was captured is projected. Perform first image processing on the second captured image, wherein the second captured image is obtained by capturing a second region including the first projection object, the at least one first corner, and the second projected image projected onto the first projection object; and Obtain the third output data extracted through the first image processing, wherein the third output data represents at least one third feature point corresponding one-to-one with the at least one first corner.

6. The feature point extraction method according to claim 4, wherein, The second projected image is predetermined as an image that is projected following the first projected image.

7. The feature point extraction method according to claim 4, wherein, The feature point extraction method includes the following steps: when the first projected image is a white image, the second projected image is determined based on the brightness distribution of the first captured image. The brightness distribution represents the degree of light reflection by the first projection object in the first captured image. The first captured image also includes the first projected image.

8. The feature point extraction method according to claim 5, wherein, The second projected image is predetermined as an image that is projected following the first projected image.

9. The feature point extraction method according to claim 5, wherein, The feature point extraction method includes the following steps: when the first projected image is a white image, the second projected image is determined based on the brightness distribution of the first captured image. The brightness distribution represents the degree of light reflection by the first projection object in the first captured image. The first captured image also includes the first projected image.

10. The feature point extraction method according to claim 1, wherein, The feature point extraction method further includes the following steps: Project at least one of the obtained first feature points onto the first projection object; Obtain the user's answer to the question regarding whether at least one of the projected first feature points coincides with at least one first angle; and The first captured image, the first output data, and the obtained answer are used as learning data to enable the learned model to undergo additional learning.

11. A feature point extraction system, comprising: An external device having a learned model that, when input data including a captured image is input, outputs output data, the captured image being obtained by the capturing device capturing an area including a projection object from which an image is projected and at least one corner of the projection object, the output data representing at least one feature point in the captured image corresponding to the at least one corner; A first projection device projects a first projected image onto a first projection object; and A first imaging device captures an image of a first region including the first projected image, the first projected object, and at least one first corner of the first projected object. The first imaging device sends first input data, including a first captured image obtained by capturing the first area, to the external device. The external device inputs the first input data into the learned model, obtains the first output data output from the learned model, which represents at least one first feature point corresponding to the at least one first angle, and sends the first output data to the first projection device.

12. A program product that causes a computer to perform the following processes: The learned model is fed with first input data representing the first captured image, where... Upon receiving input data containing first image information, the learned model outputs output data. This first image information represents an image captured by photographing a region including a projection object from a projection device and at least one corner of the projection object. The output data represents at least one feature point in the image corresponding to the at least one corner. The first image includes the first projection object and at least one first corner of the first projection object. Obtain first output data from the learned model, wherein the first output data represents at least one first feature point corresponding to the at least one first angle.

Images