Surrounding image generation device, surrounding image generation method and storage medium

Abstract

The present invention provides a surrounding image generation device, a surrounding image generation method and a storage medium. The surrounding image generation device (1) comprises: an image input unit (10) for inputting surrounding image data captured by a plurality of cameras (20); an image synthesis unit (12) for synthesizing the surrounding image data to generate a synthetic image observed from a specified viewpoint; a stereoscopic shape estimation unit (13) for estimating the stereoscopic shape of an object located in the surrounding area based on the surrounding image data; an occlusion area estimation unit (14) for estimating an occlusion area in the synthetic image that cannot be observed from a specified viewpoint using the result of the stereoscopic shape estimation; an inference unit (15) for inferring an image of the occlusion area using deep learning; and an image overlapping unit (16) for overlapping the image inferred by the inference unit (15) with the occlusion area of ​​the synthetic image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to patent application number 2019-147985 filed on August 9, 2019, and the entire contents are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a surrounding image generation device, a surrounding image generation method, and a program. Background Art

[0004] Conventionally, in order to correctly identify the conditions around a vehicle, there is known a device that provides the driver with images captured by a camera mounted on the vehicle. For example, Patent Document 1 discloses an invention of a peripheral image generating device that corrects images captured by cameras installed at the front, rear, left, and right sides of the vehicle into a top view format and provides the images to the driver.

[0005] Patent Document 1: Japanese Patent Application No. 2012-514558

[0006] In the above-mentioned top-view image, if an attempt is made to synthesize an area of ​​several meters or more, an area blocked by surrounding objects becomes a steep image, and such a situation becomes more difficult to intuitively grasp as it approaches the edge of the screen. Summary of the invention

[0007] In view of the above background, an object of the present disclosure is to provide a surrounding image generation device and the like that can generate a surrounding image without a sense of discomfort regardless of the presence of an object.

[0008] The present disclosure adopts the following technical means to solve the above-mentioned problems. The reference numerals in parentheses in the claims are examples of the correspondence relationship with specific elements described in the embodiments described later as one mode, and do not limit the technical scope of the present disclosure.

[0009] The surrounding image generation device involved in the present disclosure comprises: an image input unit, which inputs surrounding image data captured by multiple cameras; an image synthesis unit, which synthesizes the surrounding image data to generate a synthetic image observed from a specified viewpoint; a stereoscopic shape estimation unit, which estimates the stereoscopic shape of an object located in the surrounding based on the surrounding image data; an occlusion area estimation unit, which estimates the occlusion area in the synthetic image that cannot be observed from the specified viewpoint using the estimation result of the stereoscopic shape; an inference unit, which infers the image of the occlusion area using deep learning; and an image overlapping unit, which overlaps the image inferred by the inference unit with the occlusion area of ​​the synthetic image. In addition, the stereoscopic shape estimation unit can also estimate the stereoscopic shape based on the detection data detected by the ranging sensor.

[0010] According to the present disclosure, even in a region blocked by an object located in the periphery, an image without any sense of discomfort can be displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a diagram showing the configuration of the surrounding image generating device according to the first embodiment.

[0012] Figure 2 It is a diagram showing the structure of GAN.

[0013] Figure 3 It is a diagram showing the operation of the surrounding image generating device according to the first embodiment.

[0014] Figure 4 It is a diagram showing the configuration of a surrounding image generating device according to the second embodiment.

[0015] Figure 5 It is a diagram showing the operation of the surrounding image generating device according to the second embodiment.

[0016] Figure 6 It is a diagram showing the configuration of a surrounding image generating device according to a third embodiment.

[0017] Figure 7 It is a diagram showing the operation of the surrounding image generating device according to the third embodiment. DETAILED DESCRIPTION

[0018] Hereinafter, the surrounding image generation device of the embodiment of the present disclosure will be described with reference to the accompanying drawings. The surrounding image generation device of the embodiment described below is mounted on a vehicle and is used to generate and display a top view image of the surroundings of the vehicle. In the embodiment, as an example, a scene of generating a surrounding image in a parking lot is described. In addition, the use of the surrounding image generation device of the present disclosure is not limited to vehicles, and can also be used in other applications.

[0019] (First Embodiment)

[0020] Figure 1 1 is a diagram showing the configuration of a surrounding image generation device 1 according to a first embodiment. The surrounding image generation device 1 is connected to four cameras 20 mounted on a vehicle and a display 21. The four cameras 20 are cameras 20 for capturing images of the front, rear, left, and right sides of the vehicle, respectively. The display 21 can be used together with the display 21 of a navigation device to display images captured by the cameras 20.

[0021] The surrounding image generation device 1 comprises: an image input unit 10 for inputting surrounding image data captured by four cameras 20; an image processing unit 11 for processing the surrounding image data input to the image input unit 10 to generate a top view image; an image output unit 17 for outputting the top view image; a memory 18; and a power supply unit 19. The image output unit 17 sends the image data to a display 21.

[0022] The image processing unit 11 includes: an image synthesis unit 12, which synthesizes surrounding image data to generate a top-view image observed from above the vehicle; a three-dimensional shape estimation unit 13, which estimates the three-dimensional shape of objects located in the surrounding area based on the surrounding image data; an occlusion area estimation unit 14, which uses the estimation result of the three-dimensional shape to estimate the occlusion area in the synthesized image that cannot be observed from a specified viewpoint; an inference unit 15, which uses deep learning to infer the image of the occlusion area; and an image overlapping unit 16, which overlaps the image inferred by the inference unit 15 with the occlusion area of ​​the synthesized image.

[0023] The image synthesis unit 12 synthesizes the image data from the four cameras 20 input to the image input unit 10 to generate a top view image. The image synthesis unit 12 can use a known technique such as that described in Patent Document 1, for example.

[0024] The stereoscopic shape estimation unit 13 estimates the stereoscopic shape of the object captured in the image using the SfM (Structure from Motion) technique for the image data from each camera 20 input to the image input unit 10. SfM is a technique described in, for example, Kazuo Oda's "Explanation: Structure from Motion (SfM) 1st Issue Overview of SfM and Bundle Adjustment" Photometry and Remote Sensing, Vol. 55, No. 3. The stereoscopic shape estimation unit 13 estimates the stereoscopic shape of the object located around the vehicle by superimposing the results estimated from the image data of each camera 20.

[0025] The occlusion region estimation unit 14 estimates an occlusion region that is hidden and cannot be seen in the top-view image due to the object, using the information on the estimated three-dimensional shape of the object, and masks the occlusion region.

[0026] The estimation unit 15 estimates the image of the occluded region using a generator created by deep learning. In this embodiment, GAN (Generative Adversarial Network) is used as deep learning.

[0027] Figure 2This is a diagram used to explain GAN. GAN has a Generator (hereinafter also referred to as "G") and a Discriminator (hereinafter also referred to as "D"). "G" is a neural network model that generates images that deceive "D", and "D" is a neural network model that distinguishes between images generated by "G" (pseudo data) and correct images (real data). In GAN, the learning of "G" and "D" is performed alternately. Here, first, the learning of "G" is described. The vector z is sampled and given to "G", and an image (pseudo data) is output. The image is given to "D" to make a true / false judgment. The parameters of "G" are updated and learning is performed so that it can be judged as true by "D". In this way, "G" generates an image that deceives "D".

[0028] Next, the parameters of "G" are fixed, and the learning of "D" is performed. The parameters of "D" are updated so that the difference between the output when "D" is given the teacher data (real data) and the output when it is given the image (pseudo data) generated in "G" is increased. In this way, the image (pseudo data) generated in "G" can be identified.

[0029] By alternately repeating the learning of "G" and "D" described above, "G" can generate images close to real objects.

[0030] The inference unit 15 has a generator generated in advance by GAN. In addition, the learning of the generator based on GAN can also use the image of the actual parking lot, but it is also possible to use the image of the parking lot made by CG and the teacher data of the occlusion area automatically marked in the CG image. According to the method of generating the image of the parking lot by CG, a large amount of teacher data can be easily prepared.

[0031] The estimation unit 15 estimates the image of the blocked occlusion region using the generator, as a problem of filling the region blocked by the occlusion region estimation unit 14 as a missing portion.

[0032] The image superimposing unit 16 superimposes the image of the occlusion area estimated by the estimation unit 15 on the top view image. The image superimposing unit 16 superimposes the image estimated by the estimation unit 15 in a display mode different from that of the top view image to clarify the unobservable area. The different display mode is, for example, coloring the estimated image semi-transparently.

[0033] The above is a description of the configuration of the surrounding image generation device 1 of the present embodiment. An example of the hardware of the surrounding image generation device 1 is an ECU having a CPU, a RAM, a ROM, a hard disk, and a communication interface. A program having modules for implementing the above functions is stored in the RAM or the ROM in advance, and the surrounding image generation device 1 is implemented by executing the program by the CPU. Such a program is also included in the scope of the present disclosure. Other embodiments described below can also be implemented by the program in the same manner.

[0034] Figure 3 1 is a diagram showing the operation of the surrounding image generation device 1 according to the first embodiment. When the surrounding image generation device 1 receives image data from four cameras 20 (S10), it synthesizes the image data of the four cameras 20 to generate a top view image (S11). In parallel with this, the surrounding image generation device 1 estimates the three-dimensional shape of the object captured in the image using the SfM technique for the image data (S12).

[0035] Next, the surrounding image generation device 1 estimates the occluded area that cannot be observed in the top view image using the information of the three-dimensional shape of the object (S13). Next, the surrounding image generation device 1 infers the image of the occluded area through GAN (S14), and overlaps the inferred image with the top view image (S15).

[0036] The configuration and operation of the surrounding image generating device 1 according to the first embodiment have been described above.

[0037] The surrounding image generation device 1 of the first embodiment infers the image of the obstruction area that is blocked and cannot be observed by the object, and overlaps the inferred image so that a top view image without a sense of incongruity can be displayed. In addition, the inferred image is overlapped in a display mode different from the top view image, so that the driver can recognize the obstruction area that is actually not observed. In this way, the risk of the driver trusting the inferred image to perform driving operations can be avoided. In addition, there is also an effect of reminding attention to the obstruction area where there is a possibility of a person, a motorcycle, etc. suddenly appearing.

[0038] In this embodiment, an example of using GAN is described as a means of inferring the image of the occluded area, but the image can also be inferred by other methods other than GAN. For example, the image of the occluded area can also be inferred using a variational autoencoder (VAE) or an autoregressive model.

[0039] Furthermore, if a suitable scenario for applying the surrounding image generation device 1 of this embodiment is described, it is automatic parking in which the vehicle is parked by automatic driving. This is because in this scenario, the driver does not drive, so there is no problem even if the inferred image overlaps with the blocked area, so it is important to have an image that does not feel out of place.

[0040] (Second Embodiment)

[0041] Figure 4 1 is a diagram showing the configuration of the surrounding image generation device 2 of the second embodiment. The surrounding image generation device 2 of the second embodiment further includes a detection data acquisition unit 23 that acquires detection data from the LIDAR 22 in addition to the configuration described in the first embodiment. In the second embodiment, the three-dimensional shape estimation unit 13 estimates the three-dimensional shape of an object located around the vehicle using the detection data acquired from the LIDAR 22.

[0042] Figure 5 It is a diagram showing the operation of the surrounding image generating device 2 according to the second embodiment.

[0043] When the surrounding image generation device 2 receives image data from the four cameras 20 (S20), it synthesizes the image data from the four cameras 20 to generate a top view image (S21). In parallel with this, the surrounding image generation device 2 acquires data from the LIDAR 22 (S22), and estimates the three-dimensional shape of objects located around the vehicle based on the acquired data (S23).

[0044] Next, the surrounding image generation device 2 estimates the occlusion area that cannot be observed in the top view image using the information of the three-dimensional shape of the object (S24). Next, the surrounding image generation device 2 infers the image of the occlusion area through GAN (S25), and overlaps the inferred image with the top view image (S26).

[0045] The above is a description of the configuration and operation of the surrounding image generation device 2 of the second embodiment. The surrounding image generation device 2 of the second embodiment is the same as the first embodiment, and can display a top view image without any sense of incongruity. In addition, in the second embodiment, the three-dimensional shape of the object can be estimated with good accuracy using the data obtained from the LIDAR 22.

[0046] In the present embodiment, an example is given in which the data of the LIDAR 22 is used to estimate the three-dimensional shape of the object, but a distance measuring sensor other than the LIDAR 22 may be used. For example, radar, ultrasonic sonar, millimeter wave radar, etc. may be used.

[0047] (Third Embodiment)

[0048] Figure 6: is a diagram showing the configuration of the surrounding image generation device 3 of the third embodiment. The surrounding image generation device 3 of the third embodiment is different in that it does not use deep learning to estimate the image of the occlusion region, but uses an image of the environment where the vehicle is placed to generate the image.

[0049] The surrounding image generation device 3 of the third embodiment includes a communication unit 24, and communicates with a parking lot management device 30 that manages a parking lot. The parking lot management device 30 includes a storage unit that stores an image of the parking lot that it manages. When the parking lot management device 30 requests transmission of an image of the parking lot from the surrounding image generation device 3 mounted on the vehicle, the parking lot management device 30 transmits the image to the surrounding image generation device 3.

[0050] The image processing unit 11 of the surrounding image generating device 3 of the third embodiment includes an occlusion area image generating unit 25 instead of the estimation unit 15 provided in the surrounding image generating device 1 of the first embodiment. The occlusion area image generating unit 25 generates an image of the occlusion area by processing the image of the parking lot received from the parking lot management device 30. The shape of the occlusion area may be cut out from the image of the parking lot, or the image of the parking lot may be filtered to blur the image and then the shape of the occlusion area may be cut out.

[0051] Figure 7 1 is a diagram showing the operation of the surrounding image generation device 3 according to the third embodiment. When the surrounding image generation device 3 receives image data from four cameras 20 (S30), it synthesizes the image data of the four cameras 20 to generate a top view image (S31). In parallel with this, the surrounding image generation device 3 estimates the three-dimensional shape of the object captured in the image using the SfM technique for the image data (S32).

[0052] Next, the surrounding image generation device 3 estimates the occlusion area that cannot be observed in the top view image using the information of the three-dimensional shape of the object (S33). Next, the surrounding image generation device 3 obtains an image of the parking lot from the parking lot management device 30 (S34), generates an image of the occlusion area using the obtained image (S35), and overlaps the generated image with the top view image (S36).

[0053] The configuration and operation of the surrounding image generating device 3 according to the third embodiment have been described above.

[0054] The surrounding image generation device 3 of the third embodiment is the same as the above-mentioned embodiment, and can display a top view image without any sense of incongruity. In addition, in the third embodiment, an image of a parking lot is used, so it is easy to generate an image of an obstructed area. The obstructed area is an area where image data cannot be acquired by the camera 20, so by displaying the situation where image data cannot be acquired without any sense of incongruity, the risk caused by the obstructed area can be reduced.

[0055] In addition, in the present embodiment, an example of obtaining an image of a parking lot where the vehicle is actually located as an image of the environment in which the vehicle is placed is given, but it is also possible to generate an image of the occluded area by using an image of an environment such as a parking lot where the vehicle is placed, instead of using the image of the parking lot itself where the vehicle is actually placed.

[0056] The present disclosure is useful as a device for generating an image of the surroundings, and can be used to generate an image of the surroundings of a vehicle, for example.

Claims

1. A surrounding image generating device, comprising: An image input unit, inputting peripheral image data captured by a plurality of cameras; An image synthesis unit synthesizes the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; A three-dimensional shape estimating unit estimates the three-dimensional shape of an object located in the surrounding area based on the surrounding image data; an occlusion region estimating unit that estimates an occlusion region in the synthetic image that is not observable from the predetermined viewpoint, using the estimation result of the three-dimensional shape; An inference unit, which uses deep learning to infer the image of the occluded area and generates an inferred image; an image overlapping unit, which overlaps the image inferred and generated by the inference unit with the occluded area of ​​the synthetic image; and The image output unit outputs the image generated by the image overlapping unit.

2. The surrounding image generation device according to claim 1, wherein: The above-mentioned multiple cameras are mounted on the vehicle. The image synthesis unit synthesizes a top view image from above the vehicle.

3. The surrounding image generation device according to claim 1, wherein: The image superimposing unit superimposes the image estimated and generated by the estimation unit on the occlusion region in a display mode different from that of the synthetic image.

4. The surrounding image generation device according to claim 2, wherein: The image superimposing unit superimposes the image estimated and generated by the estimation unit on the occlusion region in a display mode different from that of the synthetic image.

5. The surrounding image generation device according to claim 1, wherein: The occlusion region estimation unit masks the occlusion region. The estimating unit estimates the blocked area.

6. The surrounding image generation device according to claim 1, wherein: The inference unit uses a generative adversarial network as the model of the deep learning, forming a generator that generates images close to reality, and uses the image of the masked occluded area as a filling-in-the-blank problem to perform inference using the generator.

7. A surrounding image generating device, comprising: An image input unit, inputting peripheral image data captured by a plurality of cameras; An image synthesis unit synthesizes the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; A detection data acquisition unit, which acquires the detection data detected by the distance measuring sensor; a three-dimensional shape estimating unit that estimates the three-dimensional shape of an object located in the surrounding area based on the detection data; an occlusion region estimating unit that estimates an occlusion region in the synthetic image that is not observable from the predetermined viewpoint, using the estimation result of the three-dimensional shape; An inference unit, which uses deep learning to infer the image of the occluded area and generates an inferred image; an image overlapping unit, which overlaps the image inferred and generated by the inference unit with the occluded area of ​​the synthetic image; and The image output unit outputs the image generated by the image overlapping unit.

8. The surrounding image generation device according to claim 7, wherein: The above-mentioned multiple cameras are mounted on the vehicle. The image synthesis unit synthesizes a top view image from above the vehicle.

9. The surrounding image generation device according to claim 7, wherein: The image superimposing unit superimposes the image estimated and generated by the estimation unit on the occlusion region in a display mode different from that of the synthetic image.

10. The surrounding image generation device according to claim 8, wherein: The image superimposing unit superimposes the image estimated and generated by the estimation unit on the occlusion region in a display mode different from that of the synthetic image.

11. A surrounding image generating device, comprising: An image input unit that inputs surrounding image data captured by a plurality of cameras mounted on the vehicle; An image synthesis unit synthesizes the peripheral image data to generate a top view image from above the vehicle; A three-dimensional shape estimation unit estimates the three-dimensional shape of an object located in the surrounding area based on the surrounding image data or the detection data of the ranging sensor; an occlusion region estimating unit that estimates an occlusion region that cannot be observed from above the vehicle in the top view image, using the estimation result of the three-dimensional shape; an occlusion area image generating unit, which acquires an image of the environment from a storage device storing an image of the environment in which the vehicle is placed, and generates an image of the occlusion area based on the image of the environment; an image superimposing unit that superimposes the image generated by the occlusion area image generating unit on the occlusion area of ​​the top view image in a display mode different from that of the top view image; and The image output unit outputs the image generated by the image superimposing unit.

12. The surrounding image generation device according to claim 11, wherein: The blocked area image generating unit acquires an image of the environment from a storage device outside the vehicle storing an image of the environment in which the vehicle is placed, and generates the image of the blocked area based on the image of the environment.

13. A method for generating a surrounding image is a method for generating a surrounding image, comprising: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; estimating the three-dimensional shape of the object located in the surrounding area based on the surrounding image data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

14. A method for generating a surrounding image is a method for generating a surrounding image, comprising: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; Acquire detection data detected by the distance measuring sensor; estimating the three-dimensional shape of the object located in the surrounding area based on the detection data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

15. A computer-readable storage medium storing a program, wherein when the program is executed by a processor, the following surrounding image generation process is performed, namely: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; estimating the three-dimensional shape of the object located in the surrounding area based on the surrounding image data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

16. A computer-readable storage medium storing a program, wherein when the program is executed by a processor, the following surrounding image generation process is performed, namely: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; Acquire detection data detected by the distance measuring sensor; estimating the three-dimensional shape of the object located in the surrounding area based on the detection data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the following method steps, namely: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; estimating the three-dimensional shape of the object located in the surrounding area based on the surrounding image data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

18. A computer program product, comprising a computer program, which, when executed by a processor, implements the following method steps, namely: Inputting surrounding image data captured by multiple cameras; synthesizing the peripheral image data to generate a synthetic image observed from a predetermined viewpoint; Acquire detection data detected by the distance measuring sensor; estimating the three-dimensional shape of the object located in the surrounding area based on the detection data; Using the estimation result of the three-dimensional shape, estimating an occlusion area in the synthetic image that cannot be observed from the predetermined viewpoint; Use deep learning to infer the image of the above-mentioned occluded area and generate the inferred image; Overlapping the inferred and generated image with the above-mentioned occluded area of ​​the above-mentioned synthetic image; as well as Output the image generated by overlapping.

Images