Image processing method, neural network training method, three-dimensional image display method, image processing system, neural network training system, and three-dimensional image display system

Abstract

An image processing method for synthesizing a free viewpoint image using a plurality of captured images on a display projection plane on the basis of viewpoint information, which is executed by a computer: an image acquisition step of acquiring the plurality of captured images by each of a plurality of cameras; a residual estimation step of inputting the plurality of captured images and the viewpoint information, and estimating residual information of a projection surface by machine learning, the residual information of the projection surface indicating a difference between a predefined bowl-shaped predetermined projection surface and the display projection surface; and a mapping step of mapping the plurality of captured images to the display projection surface using information related to the predetermined projection surface, the residual information, and the viewpoint information to obtain the free viewpoint image.

Description

Technical Field

[0001] The present invention relates to an image processing method, a neural network training method, a three-dimensional image display method, an image processing system, a neural network training system and a three-dimensional image display system. Background Art

[0002] There is known an image processing system that uses a plurality of captured images captured by a plurality of cameras to synthesize a free viewpoint image, which is a three-dimensional image that can be displayed by freely moving the viewpoint.

[0003] For example, a technique is known in which a bowl-shaped (mortar-shaped) projection surface is predefined and images captured by multiple cameras are projected onto the projection surface to synthesize a free-viewpoint image (see Non-Patent Document 1). Furthermore, a technique is known in which distance information measured by a three-dimensional sensor such as LiDAR (Laser Imaging Detection and Ranging) is used to calculate the projection surface and images captured by multiple cameras are projected onto the projection surface to synthesize a free-viewpoint image (see Non-Patent Document 2).

[0004] Non-Patent Document 1: Seiya Shimizu et al., “Wraparound View System for Motor Vehicles,” Fujitsu Scientific & Technical Journal 46(1):95-102, 2010

[0005] Non-patent document 2: "The beginning of the world! The development of the vehicle-mounted 3-dimensional image synthesis technology for the vehicle-mounted 3D image synthesis technology, such as the vehicle-mounted 3D image synthesis technology, and the vehicle-mounted 3D image synthesis technology." Fujitsu PRESS RELEASE, 2013, [Online], Fujitsu, Internet <URL: https: / / pr.fujitsu.com / jp / news / 2013 / 10 / 9-2.html>, [Retrieved March 24, 1992] Summary of the Invention

[0006] [Problems to be Solved by the Invention]

[0007] The technology disclosed in Non-Patent Document 1 has a problem in that distortion occurs in a composite image projected onto the projection surface due to a mismatch between a predefined projection surface and an actual three-dimensional structure.

[0008] As in the technology disclosed in Non-Patent Document 2, distortion of the synthesized image can be suppressed by adding a three-dimensional sensing device such as LiDAR. However, there is a problem in that the cost increases due to the addition of the three-dimensional sensing device.

[0009] One embodiment of the present invention has been made in view of the above-mentioned problems, and can synthesize a free viewpoint image with less distortion in an image processing system that uses a plurality of captured images to synthesize a free viewpoint image without relying on a three-dimensional sensor device.

[0010] [Solutions to solve the problem]

[0011] In order to solve the above problems, an image processing method of one embodiment of the present invention is an image processing method for synthesizing a free viewpoint image using multiple captured images on a display projection surface based on viewpoint information, which is executed by a computer: an image acquisition step, acquiring the multiple captured images through each of a plurality of cameras; a residual inference step, inputting the multiple captured images and the viewpoint information, and inferring the residual information of the projection surface through machine learning, wherein the residual information of the projection surface represents the difference between a predefined bowl-shaped established projection surface and the display projection surface; and a mapping step, using information related to the established projection surface, the residual information and the viewpoint information, mapping the multiple captured images to the display projection surface to obtain the free viewpoint image.

[0012] [Effects of the Invention]

[0013] According to one embodiment of the present invention, in an image processing system for synthesizing a free viewpoint image using a plurality of captured images, it is possible to synthesize a free viewpoint image with less distortion without relying on a three-dimensional sensor device. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing an example of a system configuration of an image processing system according to one embodiment.

[0015] Figure 2 This is a diagram for explaining an overview of image processing according to one embodiment.

[0016] Figure 3 This is a diagram showing an example of the hardware configuration of a computer according to one embodiment.

[0017] Figure 4 This is a diagram showing an example of the functional configuration of an image processing device according to one embodiment.

[0018] Figure 5 This is a flowchart illustrating an example of image processing according to one embodiment.

[0019] Figure 6 This is a diagram for explaining an overview of the training process according to the first embodiment.

[0020] Figure 7 This is a diagram showing an example of the functional configuration of the image processing device (during training) according to the first embodiment.

[0021] Figure 8 This is a flowchart showing an example of the training process according to the first embodiment.

[0022] Figure 9 This is a diagram for explaining an overview of the training process according to the second embodiment.

[0023] Figure 10 This is a diagram showing an example of the functional configuration of the image processing device (during training) according to the second embodiment.

[0024] Figure 11 This is a flowchart showing an example of the training process according to the second embodiment.

[0025] Figure 12 This is a flowchart showing an example of calculation processing of residual information according to the second embodiment.

[0026] Figure 13 It is a diagram showing a configuration example of a residual estimation model according to the third embodiment.

[0027] Figure 14 This is a diagram showing an example of the functional configuration of an image processing device according to the third embodiment.

[0028] Figure 15 This is a diagram for explaining an overview of the training process according to the third embodiment.

[0029] Figure 16 This is a flowchart (1) showing an example of the training process according to the third embodiment.

[0030] Figure 17 This is a flowchart (2) showing an example of the training process according to the third embodiment.

[0031] Figure 18 1 is a diagram showing an example of the system configuration of a three-dimensional image display system according to a fourth embodiment.

[0032] Figure 19 This is a diagram showing an example of the hardware configuration of an edge device according to the fourth embodiment.

[0033] Figure 20 It is a diagram showing an example of the functional configuration of a three-dimensional image display system according to the fourth embodiment.

[0034] Figure 21 It is a sequence diagram showing an example of three-dimensional image display processing according to the fourth embodiment. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0036] The image processing system according to this embodiment is a system for synthesizing a free viewpoint image by using multiple images captured by multiple cameras. The free viewpoint image is a three-dimensional image that can be displayed by freely moving the viewpoint. For example, the image processing system according to this embodiment can be applied to the surrounding monitoring of mobile objects such as automobiles, robots, or drones, or AR (augmented reality) / VR (virtual reality) technology. Here, as an example, an example of a case where the image processing system according to this embodiment is installed in a vehicle such as an automobile is described.

[0037] System Structure

[0038] Figure 1 FIG. 1 is a diagram showing an example of a system configuration of an image processing system according to an embodiment. Figure 1 In the example of FIG, the image processing system 100 includes an image processing device 10, a plurality of cameras 12, and a display device 16 mounted on a vehicle 1 such as a car. These components are communicably connected via an in-vehicle network, a wired cable, or wireless communication.

[0039] It should be noted that vehicle 1 is an example of a mobile object equipped with the image processing system 100 of this embodiment. The mobile object is not limited to vehicle 1 and may also include various devices or machines with mobility functions, such as robots that move using legs, manned aircraft, unmanned aircraft, etc.

[0040] The camera 12 is a photographing device that photographs the surroundings of the vehicle 1 and acquires photographed images. Figure 1 In the example shown in FIG. 1 , four cameras 12A to 12D are installed in vehicle 1 facing different imaging areas E1 to E4. Note that in the following description, "camera 12" is used when referring to any of the four cameras 12A to 12D. Furthermore, "imaging area E" is used when referring to any of the four imaging areas E1 to E4. Figure 1 The numbers of cameras 12 and photographing areas E shown are just examples, and may be other numbers of two or more.

[0041] exist Figure 1 In the example, as an example, the camera 12A is set toward the imaging area E1 in front of the vehicle 1, and the camera 12B is set toward the imaging area E2 on the side of the vehicle 1. In addition, the camera 12C is set toward the imaging area E3 on the other side of the vehicle 1, and the camera 12D is set toward the imaging area E4 behind the vehicle 1.

[0042] The display device 16 may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence), or various devices or apparatuses having a display function of displaying various information.

[0043] Image processing device 10 is a computer that executes a predetermined program to perform image processing. This image processing uses multiple images captured by cameras 12A to 12D to synthesize a free viewpoint image on a display projection surface. The free viewpoint image is a three-dimensional image that can be displayed by freely moving the viewpoint using multiple images captured by multiple cameras.

[0044] (Overview of the process)

[0045] Figure 2 The image processing device 10 includes projection surface information 230 , which is information on a bowl-shaped (or mortar-shaped) projection surface (hereinafter referred to as a “predetermined projection surface 231 ”) predefined around the vehicle 1 .

[0046] Furthermore, the image processing device 10 inputs a plurality of captured images 210 obtained by the plurality of cameras 12 capturing the periphery of the vehicle 1 and viewpoint information 240 indicating the viewpoints of the free viewpoint images into a residual estimation model, and estimates residual information 220 of the projection plane through machine learning (step S1). The residual estimation model is a trained neural network that uses the plurality of captured images 210 and viewpoint information 240 as input data and outputs residual information 220 of the projection plane indicating the difference between the projection plane onto which the free viewpoint images are projected (hereinafter referred to as the "display projection plane") and the predetermined projection plane 231.

[0047] Furthermore, the image processing device 10 generates a free viewpoint image 250 that maps the plurality of captured images 210 onto the display projection surface using the projection surface information 230, the residual information 220, and the viewpoint information 240, which are information about the predetermined projection surface 231 (step S2). Here, since the residual information 220 is information indicating the difference between the display projection surface and the predetermined projection surface 231, as described above, the image processing device 10 can calculate the display projection surface based on the projection surface information 230 and the residual information 220.

[0048] It should be noted that the residual estimation model has been pre-machined to estimate the difference between the predetermined projection plane 231 and the display projection plane based on multiple training captured images, viewpoint information 240 and three-dimensional information of one or more three-dimensional objects reflected in the multiple training captured images.

[0049] Therefore, according to the present embodiment, in the image processing system 100 that synthesizes the free viewpoint image 250 using a plurality of captured images 210 , it is possible to synthesize a free viewpoint image with less distortion without relying on a three-dimensional sensor device.

[0050] It should be noted that Figure 1The system structure of the image processing system 100 shown is an example. For example, the image processing system 100 may be a wearable device such as AR goggles or VR goggles that is worn by a user and includes a plurality of cameras 12 and a display device 16 .

[0051] <Hardware Structure>

[0052] The image processing device 10 has, for example, Figure 3 The hardware structure of the computer 300 is shown.

[0053] Figure 3 The computer 300 includes, for example, a processor 301 , a memory 302 , a storage device 303 , an I / F (interface) 304 , an input device 305 , an output device 306 , a communication device 307 , and a bus 308 .

[0054] The processor 301 is a computing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) that executes predetermined processing by executing programs stored in a storage medium such as the storage device 303. The memory 302 includes, for example, RAM (Random Access Memory), which is a volatile memory used as a work area for the processor 301, and ROM (Read Only Memory), which is a non-volatile memory that stores programs for starting the processor 301. The storage device 303 is a large-capacity non-volatile storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The I / F 304 includes various interfaces for connecting external devices such as the camera 12 and the display device 16 to the computer 300.

[0055] The input device 305 includes various devices for receiving external input (e.g., a keyboard, touchpad, pointing device, microphone, switch, button, or sensor). The output device 206 includes various devices for performing external output (e.g., a display, speaker, indicator, etc.). The communication device 307 includes various communication devices for communicating with other devices via wired or wireless networks. The bus 308 connects to each of the above components and, for example, transmits address signals, data signals, and various control signals.

[0056] Functional Structure

[0057] Figure 4 FIG. 1 is a diagram showing an example of a functional configuration of an image processing apparatus according to an embodiment. Figure 3The image processing program is executed in the processor 301 to implement the image acquisition unit 401, residual estimation unit 402, mapping unit 403, display control unit 404, setting unit 405 and storage unit 406. It should be noted that at least part of the above functional structures can also be implemented by hardware.

[0058] The image acquisition unit 401 executes image acquisition processing to acquire multiple captured images 210 using each of the multiple cameras 12. For example, the image acquisition unit 401 acquires multiple captured images 210 obtained by capturing the periphery of the vehicle 1 using the multiple cameras 12A, 12B, 12C, and 12D.

[0059] The residual estimation unit 402 performs residual estimation processing, wherein the plurality of captured images 210 acquired by the image acquisition unit 401 and the viewpoint information 240 are input, and residual information 220 of the projection surface representing the difference between the predetermined bowl-shaped projection surface 231 and the display projection surface is estimated through machine learning.

[0060] Preferably, the residual estimation unit 402 includes a residual estimation model 410, which is trained to infer the difference between the predetermined projection plane 231 and the display projection plane based on multiple training captured images, viewpoint information 240, and three-dimensional information of one or more three-dimensional objects reflected in the multiple training captured images. The residual estimation model 410 is a trained neural network (hereinafter referred to as "NN") that uses the multiple captured images 210 and viewpoint information 240 as input data and outputs residual information 220 of the projection plane, which represents the difference between the display projection plane projecting the free viewpoint image and the predetermined projection plane 231. In this embodiment, the trained NN among the NNs that uses the multiple captured images 210 and viewpoint information 240 as input data and outputs residual information 220 is referred to as the residual estimation model 410.

[0061] The residual estimation unit 402 inputs the plurality of captured images 210 and viewpoint information 240 acquired by the image acquisition unit 401 into the residual estimation model 410, and acquires the residual information 220 output by the residual estimation model 410. Viewpoint information 240 is coordinate information representing the viewpoint of the free viewpoint image generated by the image processing device 10, expressed, for example, in Cartesian coordinates or polar coordinates. Furthermore, the three-dimensional information is data including three-dimensional distance information of objects around the vehicle 1, such as three-dimensional point cloud information measured by a three-dimensional sensing device such as LiDAR (Laser Imaging Detection and Ranging), or a depth image containing depth information.

[0062] The mapping unit 403 performs a mapping process to obtain the free viewpoint image 250 by mapping the acquired plurality of captured images 210 onto the display projection plane using the projection plane information 230 associated with the predetermined projection plane 231 and the residual information 220 estimated by the residual estimation unit 402. As described above, since the residual information 220 represents the difference between the display projection plane and the predetermined projection plane 231, the mapping unit 403 can calculate the display projection plane based on the projection plane information 230, which is information about the predetermined projection plane 231, and the residual information 220. Furthermore, the process of obtaining the free viewpoint image 250 by mapping the plurality of captured images onto the calculated display projection plane can apply known techniques such as those described in Non-Patent Documents 1 and 2.

[0063] The display control unit 404 performs display control processing for displaying the free viewpoint image 250 and the like generated by the mapping unit 403 on the display device 16 and the like.

[0064] The setting unit 405 executes a setting process for setting information such as the projection surface information 230 and the viewpoint information 240 in the image processing device 10 .

[0065] The storage unit 406 is implemented by, for example, a program executed by the processor 301 , the storage device 303 , and the memory 302 , and performs storage processing for storing various information (or data) including the captured image 210 , the projection surface information 230 , and the viewpoint information 240 .

[0066] It should be noted that Figure 4 The functional configuration of the image processing apparatus 10 shown is an example. For example, the functional configurations included in the image processing apparatus 10 may be distributed among a plurality of computers 300 .

[0067] <Processing Flow>

[0068] Next, the flow of processing of the image processing method according to this embodiment will be described.

[0069] (Image Processing)

[0070] Figure 5 This is a flowchart showing an example of image processing according to one embodiment. Figure 4 The image processing device 10 described above performs the Figure 2 A specific example of image processing is described.

[0071] In step S501 , the image acquisition unit 401 acquires a plurality of captured images 210 obtained by capturing the periphery of the vehicle 1 using, for example, a plurality of cameras 12 .

[0072] In step S502 , the residual estimation unit 402 inputs the plurality of captured images 210 acquired by the image acquisition unit 401 and the viewpoint information 240 indicating the viewpoint of the free viewpoint image 250 into the residual estimation model 410 to estimate the residual information 220 of the projection plane.

[0073] In step S503, the mapping unit 403 uses Figure 2 The projection plane information 230 of the predetermined projection plane 231 and the residual information 220 estimated by the residual estimation unit 402 are used to calculate the display projection plane on which the plurality of captured images 210 are projected. For example, the mapping unit 403 calculates the display projection plane by reflecting the residual information 220 on the predetermined projection plane 231.

[0074] In step 504 , the mapping unit 403 maps the plurality of captured images 210 acquired by the image acquisition unit 401 onto the display projection surface to generate the free viewpoint image 250 .

[0075] In step S505 , the display control unit 404 displays the free viewpoint image 250 generated by the mapping unit 403 on the display device 16 or the like.

[0076] pass Figure 5 In the processing in the image processing system 100 that uses a plurality of captured images 210 to synthesize a free viewpoint image 250, the image processing device 10 can synthesize a free viewpoint image 250 with less distortion without relying on a three-dimensional sensing device.

[0077] About training processing

[0078] Next, a training process for subjecting the residual estimation model 410 to machine learning will be described.

[0079] [First embodiment]

[0080] (Overview of the process)

[0081] Figure 6 1 is a diagram for explaining an overview of the training process of the first embodiment. The process shows an example of a training process in which the image processing device 10 causes a residual training model to perform machine learning, and the residual training model is a NN that takes a plurality of captured images 210 and viewpoint information 240 as input data and outputs residual information 220. It should be noted that, in this embodiment, the NN before or during training among the NNs that take a plurality of captured images 210 and viewpoint information 240 as input data and outputs residual information 220 is referred to as a residual training model. In addition, the image processing device 10 that performs the training process can be either the same as the one that performs the reference training process or the one that performs the reference training process. Figures 2 to 4 The computer 300 for image processing described above is the same computer, but may be another computer.

[0082] The image processing device 10 obtains a plurality of captured images 210 captured by a plurality of cameras 12 and three-dimensional information (e.g., three-dimensional point cloud information, depth image, etc.) obtained by a three-dimensional sensor such as LiDAR. Furthermore, the image processing device 10 restores a three-dimensional image of the plurality of captured images 210 based on the obtained three-dimensional information and generates (renders) a teacher image 602 (i.e., a teacher free viewpoint image) based on the input viewpoint information 240 (step S11).

[0083] Furthermore, the image processing device 10 inputs the plurality of captured images 210 and the viewpoint information 240 into the residual training model to obtain residual information output by the residual training model (hereinafter referred to as "training residual information 601") (step S12). Subsequently, the image processing device 10 maps the plurality of captured images 210 onto the display projection surface using the obtained training residual information, the projection surface information 230, and the viewpoint information 240, thereby generating a training free viewpoint image (hereinafter referred to as "training image 603") (step S13).

[0084] Furthermore, the image processing device 10 trains the residual training model (NN) so as to reduce the error between the generated teaching image 602 and the training image 603 (step S14 ).

[0085] Functional Structure

[0086] Figure 7 1 is a diagram showing an example of the functional structure of the image processing device (during training) according to the first embodiment. Figure 3 The processor 301 executes a program for training processing to implement the captured image preparation unit 701, the three-dimensional information preparation unit 702, the teacher image preparation unit 703, the training unit 704, the setting unit 705, and the storage unit 706. It should be noted that at least part of the above functional structures can be implemented by hardware.

[0087] The captured image preparation unit 701 performs captured image preparation processing to prepare a plurality of captured images 210 for training, captured by each of the plurality of cameras 12. It should be noted that the captured image preparation unit 701 may acquire the plurality of captured images 210 in real time using the plurality of cameras 12, or may acquire the plurality of captured images 210 required for training processing from captured images 711 previously captured and stored in the storage unit 706 or the like.

[0088] The 3D information preparation unit 702 performs 3D information preparation processing to obtain 3D information (e.g., 3D point cloud information or depth images) corresponding to the plurality of captured images 210 for training prepared by the captured image preparation unit 701. As an example, the 3D information preparation unit 702 uses a 3D sensor such as LiDAR 707 to obtain 3D point cloud information around the vehicle 1 at the same timing (synchronously) as the captured images 210. It should be noted that the 3D information preparation unit 702 can obtain 3D information using, for example, a stereo camera, a depth camera that captures depth images, or other 3D sensors such as wireless sensor devices.

[0089] As another example, the three-dimensional information preparation unit 702 can utilize Visual SLAM (Simultaneous Localization and Mapping) technology to obtain three-dimensional information indicating the positions of surrounding three-dimensional objects from the plurality of captured images 210 for training stored in the storage unit 706 or the like. In short, the three-dimensional information preparation unit 702 can be any method as long as it prepares three-dimensional information indicating the positions of three-dimensional objects around the vehicle 1 in synchronization with the plurality of captured images 210 for training prepared by the captured image preparation unit 701.

[0090] The teacher image preparation unit 703 performs teacher image preparation processing, wherein the three-dimensional image of multiple captured images 210 is restored based on the three-dimensional information prepared by the three-dimensional information preparation unit 702, and the teacher image 602 (i.e., the teacher free viewpoint image) is generated (rendered) based on the viewpoint information 240.

[0091] The training unit 704 uses a plurality of captured images 210, viewpoint information 240, projection surface information 230, and a teacher image 602 to perform a training process for training the residual training model (NN) 710. For example, the training unit 704 inputs the plurality of captured images 210 and the viewpoint information 240 into the residual training model (NN) 710 to obtain training residual information 601, and uses the training residual information 601 and the projection surface information 230 to calculate the display projection surface for training. In addition, the training unit 704 generates a training image 603 (i.e., a training free viewpoint image) by mapping the plurality of captured images 210 to the calculated display projection surface using the viewpoint information 240. In addition, the training unit 704 trains the residual training model (NN) 710 in a manner that reduces the error between the generated teacher image 602 and the training image 603.

[0092] The setting unit 705 executes a setting process for setting various information such as the projection surface information 230 and the viewpoint information 240 in the image processing device 10 .

[0093] The storage unit 706 is implemented by, for example, a program executed by the processor 301 , the storage device 303 , and the memory 302 , and stores various information (or data) such as the captured image 711 , three-dimensional information 712 , projection surface information 230 , and viewpoint information 240 .

[0094] It should be noted that Figure 7 The functional configuration of the image processing apparatus 10 shown is an example. For example, the functional configurations included in the image processing apparatus 10 may be distributed among a plurality of computers 300 .

[0095] <Processing Flow>

[0096] Next, the processing flow of the neural network training method according to the first embodiment will be described.

[0097] Figure 8 This is a flowchart showing an example of the training process of the first embodiment. Figure 6 A specific example of the training process described herein is described, which is performed by referring to Figure 7 The image processing device 10 described above performs

[0098] In step S801a, the captured image preparation unit 701 prepares a plurality of captured images 210 for training, each captured by the plurality of cameras 12. For example, the captured image preparation unit 701 acquires a plurality of captured images 210 captured around the vehicle 1 using the plurality of cameras 12.

[0099] In step S801b, the 3D information preparation unit 702 obtains 3D information corresponding to the plurality of captured images 210 for training prepared by the captured image preparation unit 701. For example, the 3D information preparation unit 702 and the captured image preparation unit 701 obtain 3D information (e.g., 3D point cloud information) around the vehicle 1 at the same time.

[0100] In step S802 , the image processing device 10 prepares viewpoint information 240 indicating a viewpoint to be trained. For example, the setting unit 705 sets the coordinates of the viewpoint to be trained for the residual training model 710 in the viewpoint information 240 .

[0101] In step S803 , the teaching image preparation unit 703 restores the three-dimensional images of the plurality of captured images 210 based on the three-dimensional information prepared by the three-dimensional information preparation unit 702 , and generates (renders) the teaching image 602 based on the viewpoint information 240 .

[0102] In step S804 , in parallel with the process of step S803 , the training unit 704 inputs the plurality of captured images 210 and the viewpoint information 240 into the residual training model (NN) 710 to obtain the training residual information 601 .

[0103] In step S805 , the training unit 704 calculates the display projection plane based on the training residual information 601 and the projection plane information 230 , and maps the plurality of captured images 210 onto the display projection plane based on the viewpoint information 240 to generate the training image 603 .

[0104] In step S806, the training unit 704 trains the residual training model 710 in a manner that minimizes the difference between the generated teaching image 602 and the training image 603. For example, the training unit 704 calculates the weights of the residual training model 710 that minimize the difference between the two images (for example, the sum of the differences in the pixel values ​​of all pixels) and sets the calculated weights in the residual training model 710.

[0105] In step S807, the training unit 704 determines whether the training has ended. For example, the training unit 704 may determine that the training has ended when the process of steps S801 to S806 has been executed a predetermined number of times. Alternatively, the training unit 704 may determine that the training has ended when the difference between the teaching image 602 and the training image 603 falls below a predetermined value.

[0106] If the training is not completed, the training unit 704 returns the process to steps S801a and S801b. On the other hand, if the training is completed, the training unit 704 ends the process. Figure 8 processing.

[0107] In reference Figure 4 The image processing device 10 described above can process the Figure 8 The NN (residual training model 710) trained by the processing is used as the residual estimation model 410 to perform reference Figure 5 Image processing is illustrated.

[0108] [Second embodiment]

[0109] (Overview of the process)

[0110] Figure 9 This diagram is used to explain an overview of the training process of the second embodiment. This process illustrates another example of a training process in which the image processing device 10 causes a residual training model to perform machine learning. The residual training model is a neural network that receives a plurality of captured images 210 and viewpoint information 240 as input data and outputs residual information 220. It should be noted that detailed descriptions of the same processes as those of the first embodiment are omitted here.

[0111] The image processing device 10 generates an uncorrected free viewpoint image (hereinafter referred to as “uncorrected image 901 ”) obtained by mapping the plurality of captured images 210 onto a predetermined projection plane 231 using the plurality of captured images 210 , the projection plane information 230 , and the viewpoint information 240 (step S21 ).

[0112] Furthermore, the image processing device 10 acquires three-dimensional information, restores the plurality of captured images 210 using the acquired three-dimensional information, and generates (renders) a teaching image 602 (ie, a free viewpoint image for teaching) based on the viewpoint information 240 (step S22 ).

[0113] The image processing device 10 then compares the generated uncorrected image 901 with the training image 602 to obtain residual information between the two images (step S23). Furthermore, the image processing device 10 inputs the plurality of captured images 210 and the viewpoint information 240 into the residual training model 710 to obtain training residual information 601 (step S24). Next, the image processing device 10 trains the residual training model 710 to minimize the difference between the residual information between the two images and the training residual information 601 (step S25).

[0114] Functional Structure

[0115] Figure 10 : is a diagram showing an example of the functional structure of the image processing device (during training) of the second embodiment. Figure 10 As shown, the image processing device 10 of the second embodiment is Figure 7 In addition to the functional configuration of the image processing apparatus 10 of the first embodiment described above, the apparatus further includes an uncorrected image preparation unit 1001 and a residual calculation unit 1002. Figure 9 As described above, the training unit 704 performs a training process that is different from that in the first embodiment.

[0116] The uncorrected image preparation unit 1001 is implemented, for example, by a program executed by the processor 301. The uncorrected image preparation unit 1001 performs uncorrected image preparation processing to generate an uncorrected image 901 by mapping the plurality of captured images 210 onto a predetermined projection plane 231 using the plurality of captured images 210, the projection plane information 230, and the viewpoint information 240.

[0117] The residual calculation unit 1002 is realized by, for example, a program executed by the processor 301 , and performs a residual calculation process of comparing the generated uncorrected image 901 with the teaching image 602 to calculate residual information between the two images.

[0118] The training unit 704 of the second embodiment performs a training process for training the residual training model 710 so as to minimize the difference between the residual information of two images calculated by the residual calculation unit 1002 and the training residual information 601 output by the residual training model 710 .

[0119] It should be noted that the functional structures other than those mentioned above are Figure 7The functional configuration of the image processing device 10 of the first embodiment described above is the same, and therefore the description thereof is omitted here.

[0120] <Processing Flow>

[0121] Next, the processing flow of the neural network training method according to the second embodiment will be described.

[0122] Figure 11 This is a flowchart showing an example of the training process of the second embodiment. Figure 9 A specific example of the training process described herein is described, which is performed by referring to Figure 10 The image processing device 10 is described. Figure 11 In the process shown, the processes of steps S801a, S801b, and S802 are the same as those of steps S801a, S801b, and S802. Figure 8 The training process is the same as that of the first embodiment described above, so the description thereof is omitted here.

[0123] In step S1101 , the uncorrected image preparation unit 1001 generates the uncorrected image 901 obtained by mapping the plurality of captured images 210 onto a predetermined projection plane 231 using the plurality of captured images 210 , the projection plane information 230 , and the viewpoint information 240 .

[0124] In step S1102 , the teaching image preparation unit 703 restores the three-dimensional images of the plurality of captured images 210 based on the three-dimensional information prepared by the three-dimensional information preparation unit 702 , and generates (renders) the teaching image 602 based on the viewpoint information 240 .

[0125] In step S1103 , the residual calculation unit 1002 compares the generated uncorrected image 901 with the teaching image 602 to calculate residual information between the two images.

[0126] In step S1104 , for example, in parallel with the processing of steps S1101 to S1103 , the training unit 704 inputs the plurality of captured images 210 and the viewpoint information 240 into the residual training model (NN) 710 to obtain the training residual information 601 .

[0127] In step S1105, the training unit 704 trains the residual training model 710 so as to minimize the difference between the residual information of the two images calculated by the residual calculation unit 1002 and the residual information for training. For example, the training unit 704 calculates the weights of the residual training model 710 that minimize the difference between the two residual information and sets the calculated weights in the residual training model 710.

[0128] In step S1106, the training unit 704 determines whether the training is completed. If the training is not completed, the training unit 704 returns the process to steps S801a and S801b. On the other hand, if the training is completed, the training unit 704 ends the process. Figure 11 processing.

[0129] Reference Figure 4 The image processing device 10 described above can process the Figure 11 The NN (residual training model 710) trained by the processing is used as the residual estimation model 410 to perform reference Figure 5 Image processing is illustrated.

[0130] (Calculation and processing of residual information)

[0131] Figure 12 This is a flowchart showing an example of the calculation process of the residual information of the second embodiment. Figure 11 This is an example of the calculation process of the residual information performed by the residual calculation unit 1002 in step S1103.

[0132] In step S1201 , the residual calculation unit 1002 calculates the difference between each pixel of the uncorrected image 901 and the teaching image 602 (for example, the difference in pixel value of each pixel).

[0133] In step S1202, the residual calculation unit 1002 determines whether the calculated difference is less than a predetermined value. If the difference is less than the predetermined value, the residual calculation unit 1002 proceeds to step S1207 and uses the current projection plane residual as the residual information between the two images. On the other hand, if the difference is not less than the predetermined value, the residual calculation unit 1002 proceeds to step S1203.

[0134] After moving to step S1203 , the residual calculation unit 1002 obtains a position on the image where the difference is large, and obtains the coordinates of the corresponding projection plane.

[0135] In step S1204 , the residual calculation unit 1002 sets the residual information of the projection plane so as to reduce the difference near the acquired coordinates.

[0136] In step S1205 , the residual calculation unit 1002 reflects the set residual information to generate a free viewpoint image.

[0137] In step S1206 , the residual calculation unit 1002 calculates the difference between each pixel value of the generated free viewpoint image and the teaching image, and returns the process to step S1202 .

[0138] The residual calculation unit 1002 repeatedly executes Figure 12The residual information of the two images can be obtained by performing the processing until the difference between the pixel values ​​of the two images is less than a predetermined value. However, the method for the residual calculation unit 1002 to obtain the residual information of the two images is not limited to this.

[0139] It should be noted that the training process of the first embodiment utilizes the error back propagation method (backpropagation) to train (update weights) the residual training model 710 using image errors as a starting point. As a prerequisite, the residual calculation process must be in an order that allows each calculation to be differentiable within the training system.

[0140] On the other hand, the training process of the second embodiment directly uses the residual information of the two images to train the residual training model 710, so it is possible to obtain the advantage that the residual calculation process does not need to be in a differentiable order.

[0141] For example, developers etc. can select the training process of the first embodiment or the second embodiment based on whether to take advantage of the ability to use direct images as teacher data or the ability to relax the conditions of the back propagation of error method.

[0142] [Third embodiment]

[0143] In the third embodiment, a preferred configuration example of the residual estimation model 410 and the residual training model will be described.

[0144] Figure 13 This figure shows an example structure of a residual estimation model according to the third embodiment. The residual estimation model 410 can also be separated into multiple camera characteristic correction models 1301-1, 1301-2, 1301-3, ..., and a base model 1302 shared by each of the camera characteristic correction models. In the following description, "camera characteristic correction model 1301" is used to refer to any of the multiple camera characteristic correction models 1301-1, 1301-2, 1301-3, ....

[0145] In this case, for example, the image processing device 10 switches the camera characteristic correction model 1301 according to the user's settings. Specifically, an independent variable specifying the camera characteristic correction model 1301 is set in the user API (Application Programming Interface), and a database defining multiple camera characteristic correction models 1301 that are referenced in conjunction with the independent variable is set in the user SDK.

[0146] The camera characteristic calibration model 1301 is the network component of the residual estimation model 410 that primarily processes image inputs and learns weight data sensitive to camera characteristic parameters (focal length, etc.). The camera characteristic calibration model 1301 is an example of a camera model inference engine, which is trained to infer feature map information of feature points of one or more 3D objects from multiple training images and 3D information about the objects reflected in the training images.

[0147] The basic model 1302 learns weight data common to each camera characteristic correction model 1301, which is less susceptible to camera characteristic parameters. The basic model 1302 is an example of a basic model inference engine trained to infer the difference between the predetermined projection plane 231 and the display projection plane based on the feature map information output from the camera characteristic correction model 1301 and the viewpoint information 240.

[0148] It should be noted that residual estimation model 410 is an example of an inference engine structured separately into multiple camera model inference engines and a basic model inference engine. The trained weight data of the camera model inference engine is more influenced by the characteristic parameters of multiple cameras than the trained weight data of the basic model inference engine.

[0149] Functional Structure

[0150] Figure 14 FIG. 1 is a diagram showing an example of the functional configuration of an image processing apparatus according to the third embodiment. Figure 14 As shown, the image processing device 10 of the third embodiment is Figure 4 In addition to the functional configuration of the image processing device 10 according to the embodiment described above, a calibration model DB (Database) 1401 is stored in the storage unit 406 .

[0151] The calibration model DB 1401 is a database in which a plurality of camera characteristic calibration models 1301 - 1 , 1301 - 2 , 1301 - 3 , . . . are defined.

[0152] For example, when the setting unit 405 displays the camera group setting screen and accepts the user's camera group settings, it obtains the camera characteristic calibration model 1301 corresponding to the accepted camera group from the calibration model DB 1401. The setting unit 405 also sets the obtained camera characteristic calibration model 1301 as the residual estimation model 410.

[0153] Thus, for example, when the user sets the first camera group, the image processing apparatus 10 uses the residual estimation model 410 including the camera characteristic correction model 1301-1 and the basic model 1302 corresponding to the first camera group to perform reference Figure 5 Similarly, for example, when the user sets the second camera group, the image processing device 10 uses the residual estimation model 410 including the camera characteristic correction model 1301-2 corresponding to the second camera group and the basic model 1302 to perform reference Figure 5 Image processing is illustrated.

[0154] Training Process

[0155] Next, the training process of the third embodiment will be described.

[0156] (Overview of the process)

[0157] Figure 15 3 is a diagram for explaining an outline of the training process according to the third embodiment. In the image processing device 10 , as a first training process, the first camera characteristic correction model 1301 - 1 and the basic model 1302 are trained (step S31 ).

[0158] Furthermore, in the image processing device 10 , as a second training process, the second camera characteristic correction model 1301 - 2 is combined with the basic model 1302 trained in the first training process to train a camera characteristic correction model 1302 - 2 (step S32 ).

[0159] In the same manner, in the image processing device 10 , as the n-th training process, the n-th camera characteristic correction model 1301 - n is combined with the basic model 1302 trained in the first training process to train the camera characteristic correction model 1301 - n .

[0160] (Training Process 1)

[0161] Figure 16 This is a flowchart (1) showing an example of a training process of the third embodiment. This process shows the process of applying the third embodiment to a reference Figure 7 An example of the training process in the case of the image processing device 10 according to the first embodiment has been described. Note that detailed description of the same processes as those in the first embodiment will be omitted here.

[0162] In step S1601 , the image processing apparatus 10 initializes a counter n to 1 to execute the process of step S1602 .

[0163] In step S1602, the image processing apparatus 10 refers to Figure 8 The training process of the first embodiment described above trains the residual training model 710 including the first camera characteristic correction model 1301 - 1 and the basic model 1302 .

[0164] For example, refer to Figure 8, in step S801a , the captured image preparation section 701 performs a first captured image preparation process in which a first plurality of captured images are prepared by each of the first plurality of cameras (first camera group) 12 .

[0165] In step S801b, the three-dimensional information preparation unit 702 performs a first three-dimensional information preparation process of preparing first three-dimensional information of one or more three-dimensional objects reflected in the first plurality of captured images.

[0166] In step S803 , the teaching image preparation unit 703 performs a first teaching image preparation process of restoring a three-dimensional image of the first plurality of captured images based on the first three-dimensional information and generating a first teaching image based on the input viewpoint information.

[0167] In step S804 , the training unit 704 inputs the first plurality of captured images and characteristic parameters related to at least one camera 12 of the first plurality of cameras 12 into the first camera characteristic calibration model 1301 - 1 to obtain first training residual information.

[0168] In step S805 , the training unit 704 generates first training residual information and a first training image of the projection plane information 23 .

[0169] In step S806 , the training unit 704 trains both the camera characteristic correction model 1301 - 1 and the basic model 1302 so as to reduce the error between the first teaching image and the first training image.

[0170] Here, return to Figure 16 The processing starting from step S1603 is described. In step S1603, the image processing device 10 determines whether n≤N (N represents the number of camera characteristic calibration models 1301). If n≤N is not satisfied, the image processing device 10 moves the processing to step S1604. On the other hand, if n≤N is satisfied, the image processing device 10 ends the processing. Figure 16 training process.

[0171] After moving to step S1604 , the image processing apparatus 10 adds 1 to n to execute the process of step S1605 .

[0172] In step S1605, the image processing apparatus 10 fixes the basic model 1302 and Figure 8 The training process of the first embodiment described trains the n-th camera characteristic correction model and returns the process to step S1603.

[0173] For example, in the case of n=2, refer to Figure 8, in step S801a, the captured image preparation section 701 performs a second captured image preparation process in which a second plurality of captured images is prepared using each of the second plurality of cameras (second camera group) 12.

[0174] In step S801b, the three-dimensional information preparation unit 702 performs a second three-dimensional information preparation process of preparing second three-dimensional information of one or more three-dimensional objects reflected in the second plurality of captured images.

[0175] In step S803 , the teaching image preparation unit 703 performs a second teaching image preparation process of restoring the three-dimensional image of the second plurality of captured images using the second three-dimensional information and generating a second teaching image based on the input viewpoint information.

[0176] In step S804 , the training unit 704 inputs the second plurality of captured images and characteristic parameters related to at least one camera 12 of the second plurality of cameras 12 into the second camera characteristic calibration model 1301 - 2 to obtain second training residual information.

[0177] In step S805 , the training unit 704 generates second training images by mapping the second plurality of captured images onto the display projection plane using the second training residual information, the projection plane information 230 , and the viewpoint information 240 .

[0178] In step S806 , the training unit 704 fixes the basic model 1302 to train the camera characteristic correction model 1301 - 2 so as to reduce the error between the second teaching image and the second training image.

[0179] pass Figure 16 The training process shown in FIG. 1 can obtain the following Figure 13 The residual estimation model 410 shown includes multiple camera characteristic correction models 1301 - 1 , 1301 - 2 , 1302 - 3 , . . . and a basic model 1302 .

[0180] (Training Process 2)

[0181] Figure 17 This is a flowchart (2) showing an example of the training process of the third embodiment. This process shows the application of the third embodiment to the reference Figure 10 An example of the training process in the case of the image processing device 10 according to the second embodiment has been described. Note that detailed description of the same processes as those in the second embodiment will be omitted here.

[0182] In step S1701 , the image processing apparatus 10 initializes a counter n to 1 to execute the process of step S1702 .

[0183] In step S1702, the image processing apparatus 10 refers to Figure 11 The training process of the second embodiment described above trains the residual training model 710 including the first camera characteristic correction model 1301 - 1 and the basic model 1302 .

[0184] For example, refer to Figure 11 , in step S801 a , the captured image preparation section 701 performs a first captured image preparation process in which a first plurality of captured images is prepared by each of the first plurality of cameras (first camera group) 12 .

[0185] In step S801b, the three-dimensional information preparation unit 702 performs a first three-dimensional information preparation process of preparing first three-dimensional information of one or more three-dimensional objects reflected in the first plurality of captured images.

[0186] In step S1101 , the uncorrected image preparation unit 1101 performs a first uncorrected image preparation process, in which a first plurality of captured images is mapped onto a predetermined projection plane, and a first uncorrected image is generated based on input viewpoint information.

[0187] In step S1102 , the teaching image preparation unit 703 performs a first teaching image preparation process of restoring the three-dimensional image of the first plurality of captured images using the first three-dimensional information and generating a first teaching image based on the input viewpoint information 240 .

[0188] In step S1103 , the residual calculation unit 1002 performs a first residual calculation process in which the first uncorrected image and the first teaching image are compared to prepare first residual information.

[0189] In step S1104, the training unit 704 inputs the first plurality of captured images and the viewpoint information 240 into the residual training model 710 being trained to obtain first training residual information. Specifically, the training unit 704 inputs the first plurality of captured images and characteristic parameters associated with at least one of the first plurality of cameras into the first camera characteristic calibration model 1301-1 to obtain first feature map information. Furthermore, the training unit 704 inputs the obtained first feature map information and the viewpoint information 240 into the base model 1302 to obtain first training residual information.

[0190] In step S1105, the training unit 704 trains the residual training model 710 to minimize the difference between the first residual information calculated by the residual calculation unit 1002 and the first training residual information. In this way, the training unit 704 simultaneously trains both the first camera characteristic correction model 1301-1 and the base model 1302 using the first residual information as training data.

[0191] Here, return to Figure 17 The processing starting from step S1703 is described. In step S1703, the image processing device 10 determines whether n≤N (N represents the number of camera characteristic calibration models 1301). If n≤N is not satisfied, the image processing device 10 moves the processing to step S1704. On the other hand, if n≤N is satisfied, the image processing device 10 ends the processing. Figure 17 training process.

[0192] After moving to step S1704 , the image processing apparatus 10 adds 1 to n to execute the process of step S1705 .

[0193] In step S1705, the image processing apparatus 10 fixes the basic model 1302 and Figure 11 The training process of the second embodiment described above trains the n-th camera characteristic correction model and returns the process to step S1703.

[0194] For example, in the case of n=2, refer to Figure 11 , in step S801a, the captured image preparation section 701 performs a second captured image preparation process in which a second plurality of captured images is prepared using each of the second plurality of cameras (second camera group) 12.

[0195] In step S801b, the three-dimensional information preparation unit 702 performs a second three-dimensional information preparation process of preparing second three-dimensional information of one or more three-dimensional objects reflected in the second plurality of captured images.

[0196] In step S1101 , the uncorrected image preparation unit 1101 performs a second uncorrected image preparation process, in which a second plurality of captured images is projected onto the predetermined projection plane 231 , and a second uncorrected image is generated based on the input viewpoint information 240 .

[0197] In step S1102 , the teaching image preparation unit 703 performs a second teaching image preparation process of restoring the three-dimensional image of the second plurality of captured images based on the second three-dimensional information and generating a second teaching image based on the input viewpoint information 240 .

[0198] In step S1103 , the residual calculation unit 1002 performs a second residual calculation process in which the second uncorrected image is compared with the teaching image 21 to prepare second residual information.

[0199] In step S1104, the training unit 704 inputs the second plurality of captured images and the viewpoint information 240 into the residual training model 710 being trained to obtain second training residual information. Specifically, the training unit 704 inputs the second plurality of captured images and characteristic parameters associated with at least one of the second plurality of cameras into the second camera characteristic calibration model 1301-2 to obtain second feature map information. Furthermore, the training unit 704 inputs the obtained second feature map information and the viewpoint information 240 into the base model 1302 to obtain second training residual information.

[0200] In step S1105, the training unit 704 trains the second camera characteristics correction model 1301-2 by fixing the base model 1302 so as to minimize the difference between the second residual information calculated by the residual calculation unit 1002 and the second training residual information. Thus, the training unit 704 trains the second camera characteristics correction model 1301-2 using the second residual information as training data.

[0201] pass Figure 17 The training process shown in FIG. 1 can obtain Figure 13 The residual estimation model 410 shown includes multiple camera characteristic correction models 1301 - 1 , 1301 - 2 , 1302 - 3 , . . . and a basic model 1302 .

[0202] [Fourth embodiment]

[0203] In the above embodiments, examples have been described in which the image processing system 100 is installed in a vehicle 1 such as an automobile. In the fourth embodiment, an example will be described in which the image processing system 100 is applied to a three-dimensional image display system that displays a three-dimensional image on an edge device such as AR goggles.

[0204] Figure 18 An example of a system structure of a three-dimensional image display system according to a fourth embodiment is shown. A three-dimensional image display system 1800 includes an edge device 1801 such as AR goggles and a server 1802 capable of communicating with the edge device 1801 via a communication network N such as the Internet or a LAN (Local Area Network).

[0205] The edge device 1801 includes, for example, one or more peripheral cameras, a three-dimensional sensor, a display device, and a communication I / F, and transmits images captured by the peripheral cameras and three-dimensional information acquired by the three-dimensional sensor to the server 1802 .

[0206] The server 1802 includes one or more computers 300 and generates a 3D image using the captured images and 3D information received from the edge device 1801 by executing a predetermined program, and transmits the generated 3D image to the edge device 1801. It should be noted that the server 1802 is an example of a remote processing device.

[0207] The edge device 1801 displays a three-dimensional image of the surrounding area by displaying the three-dimensional image received from the server 1802 on a display device.

[0208] However, in conventional systems, there is a problem in that the edge device 1801 cannot display a 3D image until it receives the 3D image from the server 1802 after transmitting the captured image and 3D information to the server 1802 .

[0209] Therefore, according to this embodiment, the edge device 1801 sends the captured image and three-dimensional information to the server 1802 and receives the three-dimensional image from the server 1802, for example, by displaying the reference image. Figure 5 Thus, according to the three-dimensional image display system 1800 of this embodiment, the edge device 1801 can perform virtual space display before receiving the three-dimensional image from the server 1802.

[0210] <Hardware Structure>

[0211] Figure 19 This figure shows an example of the hardware configuration of an edge device according to the fourth embodiment. The edge device 1801 includes a computer configuration, such as a processor 1901, a memory 1902, a storage device 1903, a communication interface 1904, a display device 1905, multiple peripheral cameras 1906, an IMU 1907, a three-dimensional sensor 1908, and a bus 1909.

[0212] The processor 1901 is a computing device such as a CPU or a GPU that executes predetermined processing by executing a program stored in a storage medium such as the storage device 1903. The memory 1902 includes, for example, a RAM (volatile memory used as a work area for the processor 1901), a ROM (non-volatile memory storing programs for starting the processor 1901), etc. The storage device 1903 is a large-capacity non-volatile storage device such as an SSD or an HDD.

[0213] Communication I / F 1904 is a communication device such as a WAN (Wide Area Network) or LAN (Local Area Network) that connects edge device 1801 to communication network N and communicates with server 1802. Display device 1905 is a display device such as an LCD or organic EL. Multiple peripheral cameras 1906 are cameras that capture images of the surroundings of edge device 1801.

[0214] The IMU (Inertial Measurement Unit) 1907 is an inertial measurement device that uses, for example, a gyroscope and an accelerometer to detect three-dimensional angular velocity and acceleration. The three-dimensional sensor 1908 is a sensor that acquires three-dimensional information, such as a LiDAR, stereo camera, depth camera, or wireless sensor device. The bus 1909 connects to each of these components, transmitting, for example, address signals, data signals, and various control signals.

[0215] Functional Structure

[0216] Figure 20 It is a diagram showing the functional configuration of a three-dimensional image display system according to the fourth embodiment.

[0217] (Functional Structure of Edge Device)

[0218] The edge device 1801 executes a predetermined program through the processor 1901, except for the reference Figure 4 In addition to the functional structure of the image processing device 10 described above, the edge device 1801 further includes a three-dimensional information acquisition unit 2001, a sending unit 2002, and a receiving unit 2003. In addition, the edge device 1801 includes a display control unit 2004 instead of the display control unit 404. It should be noted that the image acquisition unit 401, the residual estimation unit 402, the mapping unit 403, the setting unit 405, and the storage unit 406 are connected to the reference unit 406. Figure 4 The functional configurations of the image processing apparatus 10 described above are the same, and therefore their description is omitted here.

[0219] The 3D information acquisition unit 2001 acquires 3D information around the edge device 1801 using the 3D sensor 1908. The transmission unit 2002 transmits the 3D information acquired by the 3D information acquisition unit 2001 and the plurality of captured images acquired by the image acquisition unit 401 to the server 1802.

[0220] The receiving unit 2003 receives the three-dimensional image transmitted by the server 1802 in response to the three-dimensional information and the plurality of captured images transmitted by the transmitting unit 2002. Before the receiving unit 2003 completes reception of the three-dimensional image, the display control unit 2004 displays the free viewpoint image 250 generated by the mapping unit 403 on the display device 16 or the like. After the receiving unit 2003 completes reception of the three-dimensional image, the display control unit 2004 displays the received three-dimensional image on the display device 16 or the like.

[0221] (Functional structure of the server)

[0222] The server 1802 implements the receiving unit 2011 , the three-dimensional image generating unit 2012 , the transmitting unit 2013 , and the like by executing predetermined programs on one or more computers 300 .

[0223] The receiving unit 2011 receives the three-dimensional information and the plurality of captured images transmitted by the edge device 1801 using, for example, the communication device 307 .

[0224] The 3D image generation unit 2012 uses the 3D information received by the receiving unit 2011 and the multiple captured images to render the multiple captured images into a 3D space, thereby generating a 3D image of the area surrounding the edge device 1801. It should be noted that in this embodiment, the server 1802 may use any method to generate the 3D image.

[0225] The transmitting unit 2013 transmits the three-dimensional image generated by the three-dimensional image generating unit 2012 to the edge device using, for example, the communication device 307 .

[0226] <Processing Flow>

[0227] Figure 21 It is a sequence diagram showing an example of three-dimensional image display processing according to the fourth embodiment.

[0228] In step S2101 , the image acquisition unit 401 of the edge device 1801 acquires a plurality of captured images obtained by capturing the periphery of the edge device 1801 using each of the plurality of periphery cameras 1906 .

[0229] In step S2102, the 3D information acquisition unit 2001 of the edge device 1801 acquires 3D information of one or more 3D objects reflected in the plurality of captured images using the 3D sensor 1908. For example, the 3D information acquisition unit 2001 acquires 3D point cloud information around the edge device 1801.

[0230] In step S2103 , the transmitting unit 2002 of the edge device 1801 transmits the plurality of captured images acquired by the image acquiring unit 401 and the three-dimensional information acquired by the three-dimensional information acquiring unit 2001 to the server 1802 .

[0231] In step S2104, the 3D image generation unit 2012 of the server 1802 performs 3D image generation processing, rendering the multiple captured images into a 3D space to generate a 3D image using the 3D information and the multiple captured images received from the edge device 1801. However, this processing takes time and may vary depending on factors such as the communication status with the edge device 1801 and the load on the server 1802.

[0232] In step S2105, the edge device 1801 performs a reference operation in parallel with the process of step S2104. Figure 5 The image processing described above is performed to generate a free viewpoint image by mapping multiple captured images onto a display projection surface, and the image is displayed on the display device 1905. It should be noted that this processing can be performed in a shorter time than the three-dimensional image generation processing performed by the server 1802. Since this processing is not affected by communication information with the server 1802, the load on the server 1802, etc., the image around the edge device 1801 can be displayed in a shorter time.

[0233] In step S2106 , when the 3D image generation unit 2012 of the server 1802 completes the generation of the 3D image, the transmission unit 2013 of the server 1802 transmits the generated 3D image to the edge device 1801 .

[0234] In step S2107 , upon receiving the three-dimensional image from the server 1802 , the display control unit 2004 of the edge device 1801 displays the received three-dimensional image on the display device 1905 .

[0235] pass Figure 21 In the processing, the three-dimensional image display system 1800 can display a virtual space after sending a plurality of captured images and three-dimensional information to the server 1802 and before receiving a three-dimensional image from the server 1802.

[0236] As described above, according to the embodiments of the present invention, in an image processing system for synthesizing a free viewpoint image using a plurality of captured images, it is possible to synthesize a free viewpoint image with less distortion without relying on a three-dimensional sensor device.

[0237] Description of Reference Numerals

[0238] 1 Image Processing System

[0239] 10 Image processing device

[0240] 12, 12A to 12D cameras

[0241] 16 Display device

[0242] 210 Capture Image

[0243] 130 Projection surface information

[0244] 220 Residual Information

[0245] 230 Projection surface information (information related to the given projection surface)

[0246] 231 Predetermined projection surface

[0247] 240 Viewpoint Information

[0248] 250 free viewpoint images

[0249] 300 Computer

[0250] 401 Image Acquisition Unit

[0251] 402 Residual Estimation

[0252] 403 Mapping Department

[0253] 404, 2004 Display Control Unit

[0254] 410 Residual Inference Model (Inference Engine)

[0255] 601 Residual information for training

[0256] 602 Teacher Images

[0257] 603 training images

[0258] 701 Image Preparation Department

[0259] 702 Three-dimensional Information Preparation Department

[0260] 703 Teacher Image Preparation Department

[0261] 704 Training Department

[0262] 405, 705 setting department

[0263] 710 Residual Training Model (Neural Network)

[0264] 901 Uncorrected Image

[0265] 1001 Uncorrected Image Preparation Department

[0266] 1002 Residual Calculation Department

[0267] 1301, 1301-1 to 1301-3 camera characteristic calibration model (camera model inference engine)

[0268] 1302 Basic Model (Basic Model Inference Engine)

[0269] 1800 3D Image Display System

[0270] 2001 3D Information Acquisition Department

[0271] 2002 Sending Department

[0272] 2003 Receiving Department.

Claims

1. An image processing method for synthesizing a free viewpoint image using a plurality of captured images on a display projection surface, wherein the image processing method comprises executing, by a computer: An image acquisition step, acquiring the plurality of captured images by each of the plurality of cameras; a residual estimation step, inputting the plurality of captured images and viewpoint information, and estimating residual information of the projection surface by machine learning, wherein the residual information of the projection surface represents a difference between a predefined bowl-shaped predetermined projection surface and the display projection surface; and A mapping step is performed to map the plurality of captured images onto the display projection surface using information related to the predetermined projection surface, the residual information and the viewpoint information to obtain the free viewpoint image.

2. The image processing method according to claim 1, wherein: In the residual estimation step, The residual information is estimated using an inference engine, which is trained to estimate the difference between the predetermined projection plane and the display projection plane based on multiple training captured images, the viewpoint information, and three-dimensional information of one or more solid objects projected in the multiple training captured images.

3. The image processing method according to claim 1, wherein: In the residual estimation step, Using the camera model inference engine and the basic model inference engine to infer the residual information, The camera model inference engine is trained in a manner of inferring feature map information of feature points of the three-dimensional object based on a plurality of training captured images and three-dimensional information of one or more three-dimensional objects reflected in the plurality of training captured images. The basic model inference engine is trained in a manner to infer the difference between the predetermined projection plane and the display projection plane based on the feature map information and the viewpoint information output by the camera model inference engine.

4. The image processing method according to claim 3, wherein: The trained weight data of the camera model inference engine is affected by the characteristic parameters of the multiple cameras to a greater extent than the trained weight data of the basic model inference engine is affected by the characteristic parameters of the multiple cameras.

5. The image processing method according to claim 3 or 4, wherein: The camera model inference engine is input with characteristic parameters related to at least one camera among the multiple cameras.

6. The image processing method according to claim 3 or 4, wherein: For the camera model inference engine, the camera model inference engine used to infer the feature map information can be selected from candidates of multiple camera model inference engines that have been trained based on different characteristic parameters.

7. A method for training a neural network, wherein the neural network infers residual information based on a plurality of captured images, wherein the residual information represents a difference between a predetermined bowl-shaped projection surface and a projection surface reflecting three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images, wherein a computer executes: a captured image preparation step of preparing the plurality of captured images captured by each of the plurality of cameras; A three-dimensional information preparation step, preparing the three-dimensional information; a teaching image preparation step of restoring the three-dimensional image of the plurality of captured images based on the three-dimensional information and generating a teaching image as a free viewpoint image for teaching based on the input viewpoint information; and A training step, inputting the multiple captured images and the viewpoint information into the neural network to obtain training residual information, and using the training residual information, information related to the predetermined projection surface and the viewpoint information, mapping the multiple captured images to a display projection surface to generate a training image as a training free viewpoint image, and training the neural network in a manner that reduces the error between the teacher image and the training image.

8. A method for training a neural network, wherein the neural network infers residual information based on a plurality of captured images, wherein the residual information represents a difference between a predetermined bowl-shaped projection surface and a projection surface reflecting three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images, wherein a computer executes: an image preparation step of preparing the plurality of captured images by each of the plurality of cameras; an uncorrected image preparation step of mapping the plurality of captured images to the predetermined projection surface and generating an uncorrected image as an uncorrected free viewpoint image based on the input viewpoint information; A three-dimensional information preparation step, preparing the three-dimensional information; a teacher image preparation step of restoring the three-dimensional image of the plurality of captured images based on the three-dimensional information and generating a teacher image as a teacher free viewpoint image based on the input viewpoint information; a residual calculation step of comparing the uncorrected free viewpoint image with the teacher image to prepare the residual information; as well as The training step inputs the plurality of captured images and viewpoint information into the neural network, and uses the residual information prepared in the residual calculation step as teacher data to train the neural network.

9. The neural network training method according to claim 7 or 8, wherein: The neural network includes a camera model inference network and a basic model inference network, The camera model inference network is input with the plurality of captured images and characteristic parameters related to at least one camera among the plurality of cameras, The output of the camera model inference network and the viewpoint information are input into the basic model inference network.

10. A method for training a neural network, wherein the neural network infers residual information based on a plurality of captured images, wherein the residual information represents a difference between a predetermined bowl-shaped projection surface and a projection surface reflecting three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images, The neural network includes a first camera model inference network or a second camera model inference network, and a basic model inference network, The training method of the neural network includes: a first captured image preparation step of preparing a first plurality of captured images by each of the first plurality of cameras; a second captured image preparation step of preparing a second plurality of captured images by each of the second plurality of cameras; A first three-dimensional information preparing step of preparing first three-dimensional information of one or more three-dimensional objects reflected in the first plurality of captured images; A second three-dimensional information preparing step of preparing second three-dimensional information of one or more three-dimensional objects reflected in the second plurality of captured images; a first teaching image preparation step of restoring the three-dimensional image of the first plurality of captured images based on the first three-dimensional information and generating a first teaching image as a first teaching free viewpoint image based on the input viewpoint information; a second teaching image preparation step of restoring the three-dimensional image of the second plurality of captured images based on the second three-dimensional information and generating a second teaching image as a second teaching free viewpoint image based on the viewpoint information; as well as The training steps are: Using first training residual information, information related to the predetermined projection surface and the viewpoint information, the first plurality of captured images are mapped to a display projection surface to generate a first training image as a first training free viewpoint image, and the first camera model inference network and the basic model inference network are trained simultaneously in a manner that reduces the error between the first teacher image and the first training image, wherein the first training residual information is obtained by inputting the first plurality of captured images and characteristic parameters related to at least one camera of the first plurality of cameras into the first camera model inference network, and inputting the output of the first camera model inference network and the viewpoint information into the basic model inference network, and The second plurality of captured images are mapped to a display projection surface using second training residual information, information related to the predetermined projection surface, and the viewpoint information to generate a second training image as a second training free viewpoint image, and the second camera model inference network is trained in a manner that reduces the error between the second teacher image and the second training image, wherein the second training residual information is obtained by inputting the second plurality of captured images and characteristic parameters related to at least one of the second plurality of cameras into the second camera model inference network, and inputting the output of the second camera model inference network and the viewpoint information into the basic model inference network that has been trained.

11. A method for training a neural network, wherein the neural network infers residual information of a projection surface for reflecting three-dimensional position information of one or more three-dimensional objects reflected in the plurality of captured images on a predetermined bowl-shaped projection surface based on a plurality of captured images, wherein: The neural network includes a first camera model inference network or a second camera model inference network, and a basic model inference network, The training method of the neural network includes: a first captured image preparation step of preparing a first plurality of captured images by each of the first plurality of cameras; a second captured image preparation step of preparing a second plurality of captured images by each of the second plurality of cameras; a first uncorrected image preparation step of mapping the first plurality of captured images to the predetermined projection surface, and generating a first uncorrected image as a first uncorrected free viewpoint image based on the input viewpoint information; a second uncorrected image preparation step of mapping the second plurality of captured images to the predetermined projection surface, and generating a second uncorrected image as a second uncorrected free viewpoint image based on the input viewpoint information; A first three-dimensional information preparing step of preparing first three-dimensional information of one or more three-dimensional objects reflected in the first plurality of captured images; A second three-dimensional information preparing step of preparing second three-dimensional information of one or more three-dimensional objects reflected in the second plurality of captured images; a first teaching image preparation step of restoring the three-dimensional image of the first plurality of captured images based on the first three-dimensional information and generating a first teaching image as a first teaching free viewpoint image based on the input viewpoint information; a second teaching image preparation step of restoring the three-dimensional image of the second plurality of captured images based on the second three-dimensional information and generating a second teaching image as a second teaching free viewpoint image based on the viewpoint information; a first residual calculation step of comparing the first uncorrected image with the first teacher image to prepare first residual information; a second residual calculation step of comparing the second uncorrected image with the second teacher image to prepare second residual information; and The training steps are: inputting the first plurality of captured images and characteristic parameters related to at least one camera among the first plurality of cameras into the first camera model inference network, inputting the output of the first camera model inference network and viewpoint information into the basic model inference network, and using the first residual information prepared in the first residual calculation step as teacher data to simultaneously train both the first camera model inference network and the basic model inference network, and The second plurality of captured images and characteristic parameters related to at least one of the second plurality of cameras are input into the second camera model inference network, the output and viewpoint information of the second camera model inference network are input into the basic model inference network that has been trained, and the second residual information prepared in the second residual calculation step is used as teacher data to train the second camera model inference network.

12. A three-dimensional image display method for displaying a free viewpoint image synthesized from a plurality of captured images on a display projection surface, wherein a computer executes: a captured image acquisition step, acquiring the plurality of captured images by each of the plurality of cameras; a three-dimensional information acquisition step of acquiring three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images; A residual estimation step, inputting the plurality of captured images and viewpoint information, and estimating residual information of the projection surface by machine learning, wherein the residual information of the projection surface represents a difference between a predefined bowl-shaped predetermined projection surface and the display projection surface; A mapping step, using the information related to the predetermined projection surface, the residual information and the viewpoint information, mapping the plurality of captured images to a display projection surface to obtain a free viewpoint image; a sending step of sending the plurality of captured images and the three-dimensional information to a remote processing device; a receiving step of receiving, from the remote processing device, a three-dimensional image of the plurality of captured images restored in the remote processing device based on the three-dimensional information; as well as a display control step of displaying the free viewpoint image on a display device before completing reception of the three-dimensional image in the receiving step, and displaying the three-dimensional image on the display device after completing reception of the three-dimensional image in the receiving step.

13. An image processing system for synthesizing a free viewpoint image using a plurality of captured images on a display projection surface, the image processing system comprising: an image acquisition unit configured to acquire the plurality of captured images through each of the plurality of cameras; A residual estimation unit configured to input the plurality of captured images and viewpoint information, and estimate residual information of a projection surface by machine learning, wherein the residual information of the projection surface represents a difference between a predefined bowl-shaped predetermined projection surface and the display projection surface; as well as The mapping unit is configured to map the plurality of captured images onto the display projection plane using the information related to the predetermined projection plane, the residual information, and the viewpoint information, so as to acquire the free viewpoint image.

14. A neural network training system, wherein the neural network infers residual information based on a plurality of captured images, wherein the residual information represents a difference between a predetermined bowl-shaped projection surface and a projection surface reflecting three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images, the neural network training system comprising: a captured image preparation unit configured to prepare the plurality of captured images by each of the plurality of cameras; a three-dimensional information preparation unit, configured to prepare the three-dimensional information; a teacher image preparation unit configured to restore the three-dimensional image of the plurality of captured images based on the three-dimensional information, and to generate a teacher image as a teacher free viewpoint image based on the input viewpoint information; as well as The training unit is configured to input the multiple captured images and the viewpoint information into the neural network to obtain training residual information, and use the training residual information, information related to the predetermined projection surface and the viewpoint information to map the multiple captured images to a display projection surface to generate a training image as a training free viewpoint image, and train the neural network in a manner that reduces the error between the teacher image and the training image.

15. A neural network training system, wherein the neural network infers residual information based on a plurality of captured images, wherein the residual information represents a difference between a predetermined bowl-shaped projection surface and a projection surface reflecting three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images, the neural network training system comprising: an image preparation unit configured to prepare the plurality of captured images by each of the plurality of cameras; an uncorrected image preparation unit configured to map the plurality of captured images to the predetermined projection plane and generate an uncorrected image as an uncorrected free viewpoint image based on the input viewpoint information; a three-dimensional information preparation unit, configured to prepare the three-dimensional information; a teacher image preparation unit configured to restore the three-dimensional image of the plurality of captured images based on the three-dimensional information, and to generate a teacher image as a teacher free viewpoint image based on the input viewpoint information; a residual calculation unit configured to compare the uncorrected free viewpoint image with the teaching image to prepare the residual information; as well as The training unit is configured to input the plurality of captured images and viewpoint information into the neural network, and train the neural network using the residual information prepared by the residual calculation unit as teacher data.

16. A three-dimensional image display system for displaying a free viewpoint image synthesized using a plurality of captured images on a display projection surface, the three-dimensional image display system comprising: an image acquisition unit configured to acquire the plurality of captured images through each of the plurality of cameras; a three-dimensional information acquisition unit configured to acquire three-dimensional information of one or more three-dimensional objects reflected in the plurality of captured images; A residual estimation unit configured to input the plurality of captured images and viewpoint information, and estimate residual information of a projection surface by machine learning, wherein the residual information of the projection surface represents a difference between a predefined bowl-shaped predetermined projection surface and the display projection surface; a mapping unit configured to map the plurality of captured images onto a display projection surface using information related to the predetermined projection surface, the residual information, and the viewpoint information to acquire a free viewpoint image; A sending unit configured to send the plurality of captured images and the three-dimensional information to a remote processing device; a receiving unit configured to receive, from the remote processing device, a three-dimensional image of the plurality of captured images restored in the remote processing device based on the three-dimensional information; as well as The display control unit is configured to display the free viewpoint image on a display device before the receiving unit completes receiving the three-dimensional image, and to display the three-dimensional image on the display device after the receiving unit completes receiving the three-dimensional image.

Images