Photographing method, learning method, learning data generation device, and data structure

Abstract

The photographing method uses the photographing device to photograph a third image. An imaging device includes an imaging element, an optical system, and an optical element. The optical system forms an image of incident first light in a light receiving region. The optical element guides the second light to the light receiving region. An angle formed by an optical axis of the optical system of the second light and a chief ray incident on the optical system is different from that of the first light. The third image is an image in which the first image and the second image are simultaneously displayed on the display. The first image is formed by object points emitting the first light. The second image is formed by object points emitting the second light.

Description

[0001] Cross-reference to related applications

[0002] This application claims the priority of Japanese Patent Application No. 2022-112719 filed in Japan on July 13, 2022, and all the disclosed contents of the above-mentioned prior application are incorporated into this application for reference. Technical Field

[0003] The present disclosure relates to a photographing method, a learning method, a learning data generating device and a data structure. Background Art

[0004] The shooting optical system that forms an image of the observed object has various physical properties such as focal length and field of view angle. When the focal length becomes longer, a magnified image of the observed object is formed, so that detailed optical information of the observed object at a distance can be obtained, in other words, magnified optical information. The wider the field of view angle, the more optical information of the observed object located in a wide range can be obtained. However, there is a trade-off relationship between the focal length and the field of view angle. If the focal length becomes longer, the field of view angle becomes narrower, and if the focal length becomes shorter, the field of view angle becomes wider.

[0005] Therefore, the focal length is adjusted to obtain the desired optical information according to the situation. For example, the focal length is adjusted by shifting the zoom lens included in the photographing optical system. Alternatively, the focal length is adjusted by switching a plurality of single-focus lenses (see Patent Documents 1 and 2).

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 11-311832

[0009] Patent Document 2: Japanese Patent Application Publication No. 2004-279556 Summary of the invention

[0010] The shooting method of the first viewpoint uses a shooting device, which comprises: a shooting element having a light receiving area; an optical system that forms an image of incident first light in the light receiving area; and an optical element that guides second light, whose angle is different from that of the first light and whose optical axis forms an angle with a main light ray incident to the optical system, to the light receiving area; and shoots a third image that is simultaneously displayed on a display as a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light.

[0011] The learning method of the second viewpoint uses a photographing device to photograph a third image that is simultaneously displayed on a display as a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light, the photographing device comprising: a photographing element having a light receiving area; an optical system that forms an image of the incident first light in the light receiving area; and an optical element that guides the second light to the light receiving area, the angle between the optical axis of the optical system and the principal light incident on the optical system being different from that of the first light,

[0012] In a photographing method in which a first overlapping image generated by the photographing device by photographing the third image is stored in association with images corresponding to the first image and the second image, respectively, an image processing model is learned for separating a second overlapping image photographed by the photographing device into a first image component corresponding to an object point emitting the first light and a second image component corresponding to an object point emitting the second light, using the first overlapping image generated by the photographing device by photographing the third image and the images corresponding to the first image and the second image, respectively.

[0013] A learning data generating device according to a third aspect includes:

[0014] A photographing device, comprising: a photographing element having a light receiving area; an optical system for forming an image of incident first light on the light receiving area; and an optical element for guiding second light, whose angle formed by an optical axis of the optical system and a principal light incident on the optical system is different from that of the first light, to the light receiving area;

[0015] A display, located within the field of view of the camera; and

[0016] The information processing device causes the photographing device to photograph a third image that simultaneously displays a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light on the display.

[0017] A data structure of a fourth aspect, the data structure being used for learning an image processing model for separating a second overlapping image in which a first image component corresponding to an object point emitting a first light and a second image component corresponding to an object point emitting a second light are overlapped into the first image component and the second image component,

[0018] The data structure has:

[0019] a first image formed by an object point emitting the first light;

[0020] a second image formed by the object point emitting the second light;

[0021] a first overlapping image, causing the first image and the second image to overlap; and

[0022] Correspondence information, indicating a correspondence relationship among the first image, the second image, and the first overlapping image;

[0023] The image processing model is learned using a combination of the first image, the second image, the first overlapping image, and the correspondence information as training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is an external view showing a schematic configuration of a learning data generating device including an imaging device used in the imaging method of the present embodiment.

[0025] Figure 2 Yes means Figure 1 A structural diagram of the schematic structure of the shooting device.

[0026] Figure 3 It is used to illustrate Figure 2 Diagram of the physical properties of the imaging element and optical system.

[0027] Figure 4 It means arrival Figure 2 Conceptual diagram of the image of the light-receiving area.

[0028] Figure 5 It means that the formation of arrival Figure 2 Conceptual diagram of the overlapping images of the light-receiving area.

[0029] Figure 6 It is used to illustrate the use of Figure 2 Conceptual diagram of the process by which a controller generates a restored image from overlapping images.

[0030] Figure 7 Yes means Figure 1 A functional block diagram of a schematic structure of an information processing device.

[0031] Figure 8 It is used to illustrate Figure 1 FIG. 1 is a diagram of a third image displayed on a display.

[0032] Fig. 9 It is used to illustrate Figure 7 A flowchart of a first generation process performed by a controller.

[0033] Fig.10 It is used to illustrate Figure 7 Flowchart of the second generation process performed by the controller.

[0034] Fig.11 It is used to illustrate Figure 7Flowchart of the learning process performed by the controller. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the structural members shown in the following drawings, the same reference numerals are given to the same structural members.

[0036] like Figure 1 As shown, the learning data generating device 11 including the imaging device 10 used in the imaging method of one embodiment of the present disclosure is configured to include the imaging device 10, a display 12, and an information processing device 13. The learning data generating device 11 may also be configured to include a providing device 14 that provides image data.

[0037] The learning data generating device 11 can generate learning data for the image processing model applied to the imaging device 10. The image processing model separates the second overlapping image generated by the imaging device 10 into a first image component and a second image component. As described later, the learning data generating device 11 generates learning data by causing the imaging device 10 to capture an image displayed on the display 12 by the information processing device 13.

[0038] The photographing device 10 may be a mass-produced product. Before the parameters of the image processing model are assigned, the photographing device 10 may be used for the learning data generating device 11. The learning data generating device 11 using the photographing device 10 may generate learning data. The parameters of the image processing model may be generated by learning using the learning data. The parameters may be assigned to other photographing devices 10.

[0039] like Figure 2 As shown, the imaging device 10 includes an imaging optical system (optical system) 15, an imaging element 16, and an optical element 17. The imaging device 10 may further include a memory 18 and a controller 19.

[0040] The shooting optical system 15 forms an image of the incident subject light beam. In the shooting device 10, the shooting optical system 15 forms an image of the incident first light at the light receiving area of ​​the shooting element 16 described later. The first light may be light emitted from an object point within the field of view of the shooting optical system 15. Hereinafter, the field of view of the shooting optical system 15 itself, in other words, the field of view in the structure not including the optical element 17, is referred to as a direct field of view. The shooting optical system 15 may be configured as an optical element that forms images of light beams emitted from object points at different positions at different image points by itself, in other words, by a structure not including the optical element 17. Optical elements constituting the shooting optical system 15 are, for example, lenses, reflectors, apertures, and the like.

[0041] The photographing optical system 15 does not need to be telecentric on the image side. In other words, the angle of any light beam passing through the photographing optical system 15 with respect to the optical axis of the principal ray may be greater than 0°.

[0042] The imaging element 16 has a light receiving area ra. The imaging element 16 captures an image formed in the light receiving area ra via the imaging optical system 15. The imaging element 16 can capture images based on invisible light such as visible light, infrared light and ultraviolet light. The imaging element 16 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, or the like. The imaging element 16 can be a color image sensor. In other words, a plurality of pixels arranged in the light receiving area ra of the imaging element 16 can be covered by, for example, RGB color filters so as to be evenly distributed in the light receiving area ra. The imaging element 16 generates an image signal equivalent to the image captured by the imaging light. The imaging element 16 can generate an image signal at a specified frame rate such as 30fps.

[0043] In the imaging element 16, the outer edge of the light receiving area ra on the side where the optical element 17 described later is provided may be located outside the outer edge of the exit pupil of the imaging optical system 15. Outside the outer edge of the exit pupil means outside with respect to the optical axis ox of the imaging optical system 15. The light receiving area ra may be rectangular.

[0044] The optical element 17 guides the second light to the light receiving area. The angle formed by the optical axis ox of the shooting optical system 15 of the second light and the principal ray incident on the shooting optical system 15 is different from that of the first light. The second light may be light emitted from an object point located outside the field of view of the shooting optical system 15, in other words, outside the direct field of view. The principal ray may be any one of a ray passing through the center of the aperture of the shooting optical system 15, a ray passing through the center of the entrance pupil of the shooting optical system 15, and a ray at the center of a light beam emitted from any object point and incident on the shooting optical system 15. Furthermore, the optical element 17 can form an image of at least a portion of the light beam incident on the shooting optical system 15 from outside the direct field of view of the shooting optical system 15 in the light receiving area ra of the shooting element 16. The direct field of view corresponds to the light receiving area ra of the shooting element 16, and as described above, is the field of view of the shooting optical system 15 itself, and is the field of view of the range of object points that are imaged in the light receiving area ra without passing through the optical element 17.

[0045] The optical element 17 may be a reflector that reflects at least a portion of the light beam incident to the shooting optical system 15 and guides it to the light receiving area ra. The reflector may form an image of the second light incident to the shooting optical system 15 in the light receiving area ra. The reflective surface of the reflector may be parallel to the optical axis ox of the shooting optical system 15. The reflective surface of the reflector may be parallel to any side of the rectangular light receiving area ra.

[0046] The reflecting mirror may be located outside the exit pupil of the photographing optical system 15 when viewed from the optical axis ox direction of the photographing optical system 15. More specifically, the reflecting mirror may be arranged with respect to the photographing optical system 15 so that the reflecting surface is located outside the exit pupil.

[0047] The reflector may include a plurality of plane reflectors. Two plane reflectors belonging to at least one group of the plurality of plane reflectors may be located at positions where the reflective surfaces are opposite and parallel to each other. The plane reflector and the outer edge of the light receiving area ra of the imaging element 16 may be in close contact in the normal direction of the plane reflector.

[0048] like Figure 3 As shown, the intervals H between the two plane reflectors whose reflection surfaces are parallel to each other and the optical axis ox can be equal. The two plane reflectors, the photographing optical system 15 and the photographing element 16 can be designed and configured to satisfy CRA≤tan -1 (H / B). CRA is the angle of a light beam emitted from an object point pp at an angle twice the direct field angle relative to the optical axis ox of the principal light of the photographing optical system 15. B is the back focal length of the photographing optical system 15.

[0049] Through the above structure, Figure 4 As shown, the first image component ic1 corresponding to the object point within the direct field of view, in other words, the object point emitting the first light, does not reach the light receiving area ra of the imaging element 16 via the optical element 17. More specifically, the first image component ic1 corresponding to the object point within the direct field of view is equivalent to the subject image located within the direct field of view. In addition, the second image component ic2 corresponding to the object point outside the direct field of view, in other words, the object point emitting the second light, is reversed via the optical element 17 and reaches the light receiving area ra. In other words, the second image component ic2 is an image component formed in the light receiving area ra via the optical element 17. More specifically, the second image component ic2 corresponding to the object point outside the direct field of view is equivalent to the subject image located outside the direct field of view.

[0050] In addition, through the above-mentioned structure, Figure 5As shown, the first image component ic1 and the second image component ic2 inverted in the structure where the optical element 17 is a reflector overlap in the light receiving area ra. Therefore, the imaging element 16 captures a second overlapping image olim2 of the first image component ic1 and the second image component ic2 inverted in the structure where the optical element 17 is a reflector.

[0051] The memory 18 includes, for example, any storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 18 can store various programs for the controller 19 to function and various information used by the controller 19. The memory 18 can store, for example, the parameters of the above-mentioned general image processing model. However, in the imaging device 10 applied to the learning data generating device 11, the memory 18 may not store the parameters.

[0052] The controller 19 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) or a dedicated processor for specific processing. The dedicated circuit may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The controller 19 may perform image processing on the image signal obtained from the imaging element 16.

[0053] like Figure 6 As shown, the controller 19 can implement image processing to separate the second overlapping image olim2 corresponding to the image signal into the first image component ic1 and the second image component ic2 through image processing. The controller 19 can separate the second overlapping image olim2 by applying an image processing model. As described later, the image processing model is a model constructed by learning the first image and the second image as the correct answer for the first overlapping image that overlaps the first image and the second image. The image separation model can be a model that applies Pix-to-Pix, which pits a generator that generates an image like an encoder-decoder model against a discriminator that determines whether the generated image is a false image, and generates a pair of images that reflect the relationship. The controller 19 can generate a restored image rcim by combining the separated first image component ic1 and the second image component ic2.

[0054] The controller 19 can use the restored image rcim to measure the distance of the subject photographed around the shooting device 10. The controller 19 uses the restored image rcim to measure the distance, for example, based on DFD (Depth From Defocus). The controller 19 can use the restored image rcim to measure the distance based on the motion parallax method (SLAM: Simultaneous Localization and Mapping, Motion Stereo), a separation model based on deep learning, a distance measurement method based on the foot, etc. The distance measurement method based on the foot is a method of calculating three-dimensional coordinates based on image coordinates, assuming that the lower end of the subject image is located on the ground.

[0055] The controller 19 may generate a distance image based on the distances corresponding to the addresses of the restored image rcim. The distance image is an image in which the pixel value of each pixel corresponds to the distance. The controller 19 may provide the distance image to an external device.

[0056] like Figure 1 As shown, the display 12 can display images. The display 12 may include, for example, an LCD (liquid crystal display), a PDP (plasma display panel), and an OLED (organic electroluminescent display).

[0057] The display 12 is located within the field of view of the camera 10 in the learning data generating device 11. In the learning data generating device 11, the display 12 can be configured so that the display surface of the display 12 includes the entire field of view of the camera 10. The display 12 and the camera 10 can be positioned in a predetermined relative position and relative posture in the learning data generating device 11. For example, the display 12 and the camera 10 can be configured so that the optical axis ox of the camera 10 is perpendicular to the display surface of the display 12 and passes through the center of the display surface.

[0058] The providing device 14 can provide the image of a plurality of frames to the display 12. The image of a plurality of frames can be acquired in advance. Unlike the shooting device 10 of the present application, the image of a plurality of frames can be generated by using a shooting device that is not provided with an optical element 17 and can only shoot scenes within a direct field of view. The providing device 14 can, for example, store the image of a plurality of frames constituting a video. The providing device 14 can provide images to the display 12 based on the control of the information processing device 13. The providing device 14 can provide processed images to the display 12 based on the control of the information processing device 13. The providing device 14 can be integrated with the information processing device 13.

[0059] like Figure 7 As shown, the information processing device 13 may be configured to include an input / output interface 20 and a controller 21 .

[0060] The input / output interface 20 can communicate information and commands with external devices of the information processing device 13. In the learning data generating device 11, the input / output interface 20 can be connected to the photographing device 10 and the providing device 14.

[0061] The controller 21 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU, a GPU, or a dedicated processor for specific processing. The dedicated circuit may be, for example, an FPGA, an ASIC, or the like.

[0062] The controller 21 can selectively provide the first mode and the second mode as the action mode to the information processing device 13. In the first mode, the information processing device 13 can generate learning data. In the second mode, the information processing device 13 can learn the image processing model using the learning data. The controller 21 can switch the action mode to either the first mode or the second mode based on the detection of the operation input of the input device such as the keyboard provided in the information processing device 13.

[0063] The controller 21 can control the providing device 14 in the first mode so that the image is displayed on the display 12 frame by frame. Figure 8 As shown, each frame image includes a plurality of partial images. The partial images may include a first image im1 and a second image im2. An image that simultaneously displays the first image im1 and the second image im2 on the display 12 is called a third image im3.

[0064] The first image im1 is a partial image of an area formed by object points within the direct field of view angle, in other words, object points that emit the first light. For example, the first image im1 may be a partial image of the entire area within the direct field of view angle of the shooting device 10 displayed on the display 12 of the learning data generating device 11. The second image im2 is a partial image of an area formed by object points that emit light beams that are imaged in the light receiving area ra via the optical element 17 of the shooting device 10, in other words, object points that emit the second light. For example, the second image im2 may also be a partial image of an area that is continuous with the first image im1 and is included in an angle range wider than the direct field of view angle and is displayed on the display 12 of the learning data generating device 11.

[0065] As described above, the relative position and relative posture of the camera 10 with respect to the display 12 can be determined in advance. The controller 21 can identify the area corresponding to the direct field of view angle and the area outside the direct field of view angle at the designed relative position and relative posture. The controller 21 can identify the area corresponding to the direct field of view angle and the area outside the direct field of view angle by calculating the area corresponding to the direct field of view angle and the area outside the direct field of view angle based on the characteristics of the shooting optical system 15 such as the direct field of view angle and focal length of the camera 10. The controller 21 can identify the area corresponding to the direct field of view angle and the area outside the direct field of view angle by obtaining the area corresponding to the direct field of view angle and the area outside the direct field of view angle calculated in advance based on the characteristics of the shooting optical system 15 from the camera 10, etc.

[0066] The controller 21 causes the first image im1 and the second image im2 to be displayed simultaneously on the display 12. In the state where the first image im1 and the second image im2 are displayed simultaneously, the controller 21 causes the photographing device 10 to photograph the third image im3 displayed simultaneously. The controller 21 stores the first overlapping image generated by the photographing device 10 through the photographing of the third image im3 in an arbitrary memory. The arbitrary memory may include at least one of a memory possessed by the information processing device 13, an external memory connected to the information processing device 13, and the providing device 14. In the present embodiment, the arbitrary memory may also be the providing device 14.

[0067] After capturing the third image im3, the controller 21 causes the first image im1 and the second image im2 in the third image im3 to be displayed separately and in sequence on the display 12. Separate display may also mean that, when displaying the first image im1, substantially different amounts of light, i.e., optical information, emitted from different object points outside the direct field of view angle is not allowed to reach the light receiving area ra of the camera 10. For example, when displaying the first image im1, the area outside the direct field of view angle may be displayed on the display 12 as a single color. In addition, separate display may also mean that, when displaying the second image im2, substantially different amounts of light, i.e., optical information, emitted from different object points within the direct field of view angle is not allowed to reach the light receiving area ra of the camera 10.

[0068] The controller 21 can, for example, display the first image im1 alone by performing image processing to mask the area outside the direct viewing angle of the third image im3 displayed on the display 12 with black or the like. In addition, the controller 21 can, for example, display the second image im2 alone by performing image processing to mask the area within the direct viewing angle of the third image im3 displayed on the display 12 with black or the like. The controller 21 can cause the providing device 14 to perform image processing to mask a specific area instead of the controller 21.

[0069] In a state where the first image im1 is displayed alone, the controller 21 causes the photographing device 10 to photograph the first image im1. In a state where the second image im2 is displayed alone, the controller 21 causes the photographing device 10 to photograph the second image im2. The controller 21 may store the image generated by the photographing device 10 by photographing the first image im1 as an image corresponding to the first image im1 in the providing device 14 in association with a first overlapping image corresponding to the third image im3 including the first image im1. The controller 21 may store the image generated by the photographing device 10 by photographing the second image im2 as an image corresponding to the second image im2 in the providing device 14 in association with a first overlapping image corresponding to the third image im3 including the second image im2.

[0070] The controller 21 may obtain the first image im1 and the second image im2 included in the third image im from the providing device 14 each time the third image im3 is captured, instead of separately displaying and capturing the first image im1 and the second image im2. The controller 21 may store the first image im1 acquired from the providing device 14 in the providing device 14 as an image corresponding to the first image im1, in association with a first overlapping image corresponding to the third image im3 containing the first image im1. The controller 21 may store the second image im2 acquired from the providing device 14 in the providing device 14 as an image corresponding to the second image im2, in association with a first overlapping image corresponding to the third image im3 containing the second image im2.

[0071] The controller 21 can generate the first overlapping image frame by frame in a group of images of a plurality of frames like a video, and store the first overlapping image, an image corresponding to the first image im1, and an image corresponding to the second image im2 in association with each other. The controller 21 associates the first overlapping image, the image corresponding to the first image im1, and the images corresponding to the second image im2, and generates a set as learning data.

[0072] The controller 21 learns the image processing model in the second mode using the learning data stored in the providing device 14. In other words, the controller 21 learns the image processing model using the first overlapping image and the images generated by the imaging device 10 by separately capturing the first image im1 and the second image im2. Alternatively, the controller 21 learns the image processing model in the second mode using the first overlapping image, the first image im1, and the second image im2.

[0073] Next, use Fig. 9 The flowchart of FIG. 1 illustrates the first generation process executed by the controller 21 in this embodiment. The first generation process starts when the operation mode is switched to the first mode. The first generation process is a process for generating learning data by independently displaying and photographing the first image im1 and the second image im2.

[0074] In step S100, the controller 21 identifies a video for generating learning data from the videos stored in the providing device 14. The video identification is based on, for example, detection of an operation input to an input device such as a keyboard provided in the information processing device 13. After identification, the process proceeds to step S101.

[0075] In step S101, the controller 21 selects one frame of image from the plurality of frames constituting the video identified in step S100. The frames may be selected in the order constituting the video. After the selection, the process proceeds to step S102.

[0076] In step S102, the controller 21 causes the display 12 to simultaneously display the first image im1 and the second image im2 of the frame selected in step S101. In other words, the controller 21 causes the display 12 to display the third image im3. After the display, the process proceeds to step S103.

[0077] In step S103, the controller 21 generates a first superimposed image by causing the imaging device 10 to capture the third image im3. After the capturing, the process proceeds to step S104.

[0078] In step S104, the controller 21 stores the first superimposed image generated in step S103 in the providing device 14. After storage, the process proceeds to step S105.

[0079] In step S105, the controller 21 causes the display 12 to display only the first image im1 among the images of the frame selected in step S101. After the display, the process proceeds to step S106.

[0080] In step S106, the controller 21 generates an image corresponding to the first image im1 by causing the imaging device 10 to capture the first image im1. After capturing, the process proceeds to step S107.

[0081] In step S107, the controller 21 stores the image corresponding to the first image im1 generated in step S106 in the providing device 14 in association with the first superimposed image stored in the providing device 14 in the previous step S104. After storage, the process proceeds to step S108.

[0082] In step S108, the controller 21 causes the display 12 to display only the second image im2 of the image of the frame selected in step S101. After the display, the process proceeds to step S109.

[0083] In step S109, the controller 21 generates an image corresponding to the second image im2 by causing the imaging device 10 to capture the second image im2. After the capturing, the process proceeds to step S110.

[0084] In step S110, the controller 21 stores the image corresponding to the second image im2 generated in step S109 in association with the first superimposed image stored in the providing device 14 in the previous step S104 in the providing device 14. After storage, the process proceeds to step S111.

[0085] In step S111, the controller 21 determines whether the image of the previous frame constituting the video identified in step S100 has been shot. If the shooting of all frames has been completed, the first generation process is terminated. If the shooting of the previous frame has not been completed, the process proceeds to step S112.

[0086] In step S112, the controller 21 selects images of other frames of the captured frame images of the first image im1, the second image im2, and the third image im3. After the selection, the process returns to step S102.

[0087] Next, refer to Fig.10 The flowchart of FIG. 1 illustrates the second generation process executed by the controller 21 in this embodiment. The second generation process starts when the operation mode is switched to the first mode. The second generation process is a process for generating learning data without displaying and capturing the first image im1 and the second image im2 individually.

[0088] In steps S200 to S204, the controller 21 performs the same processing as steps S100 to S104 in the first generation processing. After the first superimposed image is stored in step S204, the process proceeds to step S205.

[0089] In step S205, the controller 21 performs image processing to invert the second image im2 and shift the second image im2 so as to include it in the light receiving area ra of the imaging device 10. After the image processing, the process proceeds to step S206.

[0090] In step S206, the controller 21 stores the first image im1 of the frame selected in step S201 as an image corresponding to the first image im1 in the providing device 14 in association with the first overlapping image stored in the providing device 14 in the previous step S204. Furthermore, the controller 21 stores the second image im2 subjected to image processing in step S205 as an image corresponding to the second image im2 in association with the first overlapping image stored in the providing device 14 in the previous step S204 in the providing device 14. After storage, the process proceeds to step S207.

[0091] In steps S207 and S208 , the controller 21 performs the same processing as steps S111 and S112 in the first generation processing.

[0092] Next, use Fig.11 The flowchart of FIG. 1 illustrates the learning process executed by the controller 21 in the present embodiment. The learning process starts when the operation mode is switched to the second mode.

[0093] In step S300, the controller 21 reads the learning data stored in the providing device 14 by the first generation process or the second generation process. After reading, the process proceeds to step S301.

[0094] In step S301, the controller 21 learns the image processing model using the learning data read in step S300. The controller 21 generates parameters of the learned image processing model. After learning, the process proceeds to step S302.

[0095] In step S302, the controller 21 stores the parameters generated in step S301 in the providing device 14. After storage, the learning process ends.

[0096] In the photographing method of the information processing device 13 of the present embodiment using the above structure, the photographing device 10 is used, and the photographing device 10 has an optical element 17 that forms an image of at least a part of the light beam incident on the photographing optical system 15 from outside the direct field angle of the photographing optical system 15 corresponding to the light receiving area ra of the photographing element 16 in the light receiving area ra, and photographs a first image im1 formed by an object point in the direct field angle and a second image im2 formed by an object point that emits a light beam formed in the light receiving area ra via the optical element 17, which are simultaneously displayed on the display 12. With such a structure, the photographing method can generate a first superimposed image by causing the photographing device 10 to photograph an image that has been acquired in advance as information. The superimposed image obtained by performing a process of separating the image that has been acquired in advance and performing a process of reversing and superimposing a part is an ideal superimposed image generated by the designed photographing device 10. Therefore, the superimposed image does not include the influence of the mounting errors of the photographing optical system 15, the photographing element 16, and the optical element 17 in the photographing device 10 actually manufactured. Therefore, in the learning of the image processing model using the overlapped image, there is a limit to the improvement of the separation accuracy of the second overlapped image generated by the actually manufactured imaging device 10. On the other hand, since the imaging method can generate the first overlapped image, the image processing model can be learned in a manner that further improves the separation accuracy of the second overlapped image compared to the case of using the overlapped image based on image processing. Therefore, the imaging method can manufacture the imaging device 10 that generates a wide range of amplified optical information.

[0097] In addition, in the shooting method of this embodiment, the first image im1 and the second image im2 displayed separately on the display 12 are shot using the shooting device 10. With such a structure, the shooting method can obtain a non-overlapping image whose pixel position is consistent with the first overlapping image. Therefore, the shooting method can learn the image processing model in a manner that further improves the separation accuracy of the second overlapping image.

[0098] In addition, in the shooting method of this embodiment, the first overlapped image is stored in association with the images corresponding to the first image im1 and the second image im2 included in the third image im3. According to such a structure, when generating the first overlapped image of a plurality of frames, the shooting method can easily use each overlapped image as learning data.

[0099] In one embodiment, (1) a shooting method, wherein a shooting device is used to shoot a third image that is simultaneously displayed on a display as a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light, the shooting device comprising: a shooting element having a light receiving area; an optical system that forms an image of the incident first light in the light receiving area; and an optical element that guides the second light to the light receiving area, the angle between the optical axis of the optical system and the main light incident on the optical system being different from that of the first light.

[0100] (2) In the photographing method of (1) above, wherein:

[0101] The first image and the second image displayed separately on the display are also photographed using the photographing device.

[0102] (3) In the photographing method of (1) or (2) above,

[0103] A first overlapping image generated by the photographing device by photographing the third image is stored in association with images corresponding to the first image and the second image, respectively.

[0104] (4) In the learning method using the photographing method of (3) above,

[0105] Using the first overlapping image generated by the shooting device by shooting the third image in the shooting method and the images corresponding to the first image and the second image respectively, an image processing model is learned, and the image processing model is used to separate the second overlapping image that overlaps the first image component corresponding to the object point emitting the first light and the second image component corresponding to the object point emitting the second light into the first image component and the second image component.

[0106] (5) In the learning method to which the photographing method of (4) above is applied,

[0107] The images corresponding to the first image and the second image, respectively, are images generated by the imaging device by imaging the first image and the second image separately.

[0108] (6) In the learning method using the photographing method of (4) above,

[0109] The images corresponding to the first image and the second image, respectively, are the first image and the second image.

[0110] In one embodiment, (7) a learning data generating device, wherein:

[0111] have:

[0112] A photographing device, comprising: a photographing element having a light receiving area; an optical system for forming an image of incident first light on the light receiving area; and an optical element for guiding second light, whose angle formed by an optical axis of the optical system and a principal light incident on the optical system is different from that of the first light, to the light receiving area;

[0113] A display, located within the field of view of the camera; and

[0114] The information processing device causes the photographing device to photograph a third image that simultaneously displays a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light on the display.

[0115] In one embodiment, (8) a data structure, wherein:

[0116] The data structure is used for learning an image processing model for separating a second overlapping image in which a first image component corresponding to an object point emitting a first light and a second image component corresponding to an object point emitting a second light are overlapped into the first image component and the second image component,

[0117] The data structure has:

[0118] a first image formed by an object point emitting the first light;

[0119] a second image formed by the object point emitting the second light;

[0120] a first overlapping image, causing the first image and the second image to overlap; and

[0121] Correspondence information, indicating a correspondence relationship among the first image, the second image, and the first overlapping image;

[0122] The image processing model is learned using a combination of the first image, the second image, the first overlapping image, and the correspondence information as training data.

[0123] The above describes an implementation of a shooting method using the shooting device 10, but as an implementation of the present disclosure, in addition to a method or program for implementing the device, an implementation using a storage medium (for example, a CD, a magneto-optical disk, CD-ROM, CD-R, CD-RW, a magnetic tape, a hard disk, or a memory card, etc.) on which a program is recorded can be adopted.

[0124] In addition, the installation method of the program is not limited to an application program such as an object code compiled by a compiler or a program code executed by an interpreter, and may be a program module compiled into an operating system. Furthermore, the program may be configured so that all processing is performed only in the CPU on the control substrate. The program may be configured so that part or all of it is performed by an expansion board installed on the substrate or other processing units installed in the expansion unit as needed.

[0125] The drawings for explaining the embodiments of the present disclosure are schematic diagrams, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual dimensional ratios.

[0126] Although the embodiments of the present disclosure are described based on the drawings and embodiments, it should be noted that those skilled in the art can make various deformations or changes according to the present disclosure. Therefore, it should be noted that these deformations or changes are included in the scope of the present disclosure. For example, the functions included in each structural part can be reconfigured in a logically non-contradictory manner, and multiple structural parts can be combined or divided.

[0127] For example, in this embodiment, the information processing device 13 generates learning data and learns the image processing model. However, the information processing device 13 may not learn the image processing model, but another information processing device may learn the image processing model.

[0128] All structural elements and / or all steps of all methods or processes disclosed in the present disclosure can be combined in any combination except for mutually exclusive combinations of these features. In addition, the features recorded in the present disclosure can be replaced with alternative features that work for the same purpose, equivalent purpose, or for similar purposes, unless explicitly denied. Therefore, unless explicitly denied, each feature disclosed is only an example of a series of identical or equivalent features included.

[0129] Furthermore, the embodiments of the present disclosure are not limited to any specific structure of the above-mentioned embodiment mode. The embodiments of the present disclosure can be extended to all new features, their combinations or all new methods, processing steps or their combinations recorded in the present disclosure.

[0130] In the present disclosure, the records of "first" and "second" etc. are identifiers for distinguishing the structure. The structures distinguished by the records of "first" and "second" etc. in the present disclosure can exchange the numbers in the structure. For example, the first image component can exchange "first" and "second" as identifiers with the second image component. The exchange of identifiers is performed simultaneously. The structure can also be distinguished after the exchange of identifiers. The identifiers can also be deleted. The structure from which the identifiers are deleted is distinguished by the figure marks. The records based solely on the identifiers of "first" and "second" etc. in the present disclosure should not be used for the interpretation of the order of the structure, or the basis for the existence of small numbered identifiers.

[0131] Description of Reference Numerals

[0132] 10. Filming Device

[0133] 11 Learning Data Generation Device

[0134] 12 Displays

[0135] 13Information processing device

[0136] 14 Provide device

[0137] 15 Photography Optical System

[0138] 16 camera elements

[0139] 17 Optical Elements

[0140] 18 Memory

[0141] 19 Controller

[0142] 20 Input and output interfaces

[0143] 21 Controller

[0144] CRA is the angle of the light beam emitted from the object point at an angle twice the direct field angle relative to the optical axis of the principal ray of the photographic optical system.

[0145] ic1 first image component

[0146] ic2 second image component

[0147] im1 first image

[0148] im2 second image

[0149] im3 third image

[0150] olim2 second overlapping image

[0151] ox optical axis

[0152] pp Object point at an angle twice the direct field of view

[0153] ra light receiving area

[0154] rcim restored image

Claims

1. A shooting method, wherein: photographing a third image by using a photographing device so that a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light are simultaneously displayed on a display, The photographing device comprises: A photographing element having a light receiving area; An optical system for imaging the incident first light in the light receiving area; as well as The optical element guides second light to the light receiving area, wherein the angle formed between the optical axis of the optical system and the principal light incident on the optical system of the second light is different from that of the first light.

2. The shooting method according to claim 1, wherein: The first image and the second image displayed separately on the display are also photographed using the photographing device.

3. The photographing method according to claim 1 or 2, wherein: A first superimposed image generated by the imaging device by imaging the third image is stored in association with images corresponding to the first image and the second image respectively included in the third image.

4. A learning method, wherein: Using the first overlapping image generated by the shooting device by shooting the third image in the shooting method described in claim 3 and the images corresponding to the first image and the second image respectively, an image processing model is learned, and the image processing model is used to separate the second overlapping image that overlaps the first image component corresponding to the object point emitting the first light and the second image component corresponding to the object point emitting the second light into the first image component and the second image component.

5. The learning method according to claim 4, wherein: The images corresponding to the first image and the second image, respectively, are images generated by the imaging device by imaging the first image and the second image separately.

6. The learning method according to claim 4, wherein: The images corresponding to the first image and the second image, respectively, are the first image and the second image.

7. A learning data generating device, wherein: have: The imaging device comprises: an imaging element having a light receiving area; and an optical system for imaging an incident first light in the light receiving area; and an optical element for guiding a second light having an angle different from the first light between the optical axis of the optical system and the principal light incident on the optical system to the light receiving area; A display, located within the field of view of the camera; as well as The information processing device causes the photographing device to photograph a third image that simultaneously displays a first image formed by an object point emitting the first light and a second image formed by an object point emitting the second light on the display.

8. A data structure, wherein The data structure is used for learning an image processing model, the image processing model being for separating a second overlapping image in which a first image component corresponding to an object point emitting a first light and a second image component corresponding to an object point emitting a second light are overlapped into the first image component and the second image component, The data structure has: a first image formed by an object point emitting the first light; a second image formed by the object point emitting the second light; a first overlapping image, causing the first image and the second image to overlap; as well as Correspondence information, indicating a correspondence relationship among the first image, the second image, and the first overlapping image; The image processing model is learned using a combination of the first image, the second image, the first overlapping image, and the correspondence information as training data.

Images