Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method capable of accurately improving image quality even when the image quality is remarkably lowered.SOLUTION: An image processing device comprises: an image acquisition unit 100 for acquiring a first image and a second image captured by a first camera and a second camera; an information storage unit 102 for storing camera position information relating to installation positions of the first camera and the second camera; a distance information acquisition unit 101 for acquiring distance information in at least a partial region of the first image; a viewpoint conversion unit 103 for generating a first viewpoint converted image by viewpoint-converting the first image to a viewpoint of the second camera on the basis of the camera position information and the distance information; and an image quality improvement unit 104 for improving the image quality of the second image on the basis of the second image and the first viewpoint converted image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Autonomous driving is expected to reduce traffic accidents and improve logistics efficiency. Autonomous driving is expected to use cameras, which can detect three-dimensional objects such as vehicles and pedestrians by analyzing images. However, a challenge in detecting three-dimensional objects is the degradation of image quality. For example, when a camera is installed outside the vehicle, image quality can be degraded by raindrops, mud, and other contaminants adhering to the camera and being captured in the image. Even when a camera is installed inside the vehicle, image quality can be degraded by indirectly capturing dirt on the windshield. When image quality is degraded in this way, vehicles, pedestrians, and other objects are captured in images that differ from their intended appearance, resulting in reduced detection accuracy. As a result, vehicle control cannot be performed correctly, and the reliability of the autonomous driving system is reduced. Therefore, Patent Document 1 discloses a method for generating an image in which dirt has been removed from an image containing dirt using a machine-learned generative model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-83841 Summary of the Invention [Problem to be solved by the invention]

[0004] By using the method disclosed in Patent Document 1, it is possible to correctly remove dirt such as raindrops when their transparency is high and the degradation of image quality is minimal. On the other hand, if mud or other impurities with low transparency are attached in a way that completely obscures a vehicle in an image, the method disclosed in Patent Document 1 can remove the mud itself, but it also removes the vehicle behind it. As a result, it is not possible to detect the vehicle that should be detected, and it is not possible to control the vehicle appropriately.

[0005] In view of the above problems, an object of the present invention is to provide an image processing device and an image processing method that can accurately improve image quality even when the image quality has significantly deteriorated. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention has an image acquisition unit that acquires a first image and a second image captured by a first camera and a second camera, an information storage unit that stores camera position information regarding the installation positions of the first camera and the second camera, a distance information acquisition unit that acquires distance information in at least a portion of the first image, a viewpoint conversion unit that generates a first viewpoint converted image by viewpoint converting the first image to the viewpoint of the second camera based on the camera position information and the distance information, and an image quality improvement unit that improves the image quality of the second image based on the second image and the first viewpoint converted image.

[0007] In addition, the image processing method of the present invention is an image processing method executed by a computer, and includes an image acquisition step of acquiring a first image and a second image captured by a first camera and a second camera, a distance information acquisition step of acquiring distance information in at least a portion of the first image, a viewpoint conversion step of generating a first viewpoint-converted image by viewpoint-converting the first image to the viewpoint of the second camera based on camera position information regarding the installation positions of the first camera and the second camera and the distance information, and an image quality improvement step of improving the image quality of the second image based on the second image and the first viewpoint-converted image. [Effects of the Invention]

[0008] According to the present invention, even when the image quality is significantly degraded, the image quality can be accurately improved.

[0009] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional block diagram showing the configuration of an image processing device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing the positional relationship between a vehicle and a camera in the first embodiment. [Figure 3] FIG. 2 is a processing flow diagram in the image processing device of the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating a method for calculating a distance. [Figure 5] FIG. 10 is a diagram illustrating a method for improving image quality. [Figure 6] FIG. 10 is a functional block diagram in which a cause identification unit is added to the functional blocks of the first embodiment. [Figure 7] FIG. 7 is a processing flow diagram of the image processing device of the functional blocks of FIG. 6. [Figure 8] FIG. 10 is a diagram illustrating a method of allocating pixels in a viewpoint conversion image. [Figure 9]FIG. 10 is a diagram showing the relationship between the installation positions of the cameras and the angle of view when five cameras are used. [Figure 10] FIG. 10 is a functional block diagram showing the configuration of an image processing device according to a second embodiment of the present invention. [Figure 11] FIG. 10 is a processing flow diagram in the image processing device of the second embodiment. [Figure 12] FIG. 10 is a diagram illustrating the processing content of a region specifying unit. [Figure 13] 10A and 10B are diagrams illustrating a method for improving image quality using the output of a region specifying unit. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] [Embodiment 1] 1 is a functional block diagram showing the configuration of an image processing device 1 according to the first embodiment. In the configuration of the first embodiment, a plurality of cameras are connected to the image processing device 1. In the following description, it is assumed that the image processing device 1 is mounted on a vehicle.

[0013] The image processing device 1 of this embodiment is composed of a camera, a computer, a memory, a storage device, etc., and operates as various functional units by the computer executing a control program stored in the memory, etc.

[0014] As shown in FIG. 1, the image processing device 1 has an image acquisition unit 100, a distance information acquisition unit 101, an information storage unit 102, a viewpoint conversion unit 103, and an image quality improvement unit 104 as functional units realized by the operation of a camera and a computer.

[0015] The image acquisition unit 100 acquires images from a plurality of cameras connected to the image processing device 1.

[0016] The distance information acquisition unit 101 acquires distance information in at least a partially overlapping region of the camera images acquired by the image acquisition unit 100. The distance information may be calculated by multi-view stereo using multiple cameras, or may be calculated using a three-dimensional sensor such as LiDAR.

[0017] Information on the relative positions and orientations of multiple cameras (camera position information) is stored in the information storage unit 102. Here, the positions and orientations refer to the relative relationships between the cameras. In addition, when a three-dimensional sensor is used, the positions and orientations between the cameras and the three-dimensional sensor are also stored.

[0018] A viewpoint conversion unit 103 converts the viewpoint of the image captured by the image acquisition unit 100, using the distance information acquired by the distance information acquisition unit 101 and the positions and orientations between the cameras and between the camera and the three-dimensional sensor stored in the information storage unit 102. A specific method will be described later.

[0019] The image quality improvement unit 104 improves the image quality of an image whose image quality has been degraded by dirt or the like, by using the image acquired by the image acquisition unit 100 and the image after viewpoint conversion calculated by the viewpoint conversion unit 103. A machine learning model can be used to improve the image quality.

[0020] Hereinafter, an example of the operation of the image processing device 1 of this embodiment will be described in detail with reference to the flowchart of FIG. 3. The image processing device 1 of this embodiment improves image quality by generating an image from which dirt such as raindrops or mud has been removed when the camera is installed outside the vehicle cabin and dirt such as raindrops or mud has adhered to the camera's lens, resulting in a degradation of image quality, or when the camera is installed inside the vehicle cabin and dirt on the windshield or other surface has been indirectly captured in the image, resulting in a degradation of image quality. The following example of operation will discuss a method for improving image quality by generating an image from which dirt such as raindrops or mud has been removed when the camera is installed outside the vehicle cabin and dirt such as raindrops or mud has adhered to the camera's lens and has been captured, resulting in a degradation of image quality. Here, as shown in FIG. 2, it is assumed that three cameras (C1, C2, and C3) are installed on a vehicle V1 to capture the environment surrounding the vehicle V1, and that raindrops on the lens of C1 degrade the image quality, while no raindrops are attached to C2 or C3, resulting in no degradation of image quality. It is also assumed that C2 and C3 are stereo cameras. Note that as long as the distance calculation process P102 described below can be executed, the camera is not limited to a stereo camera, and may be a multi-camera consisting of three or more cameras installed on a vehicle. When a multi-camera is connected to the image processing device 1, the distance calculation process P102 can be applied to images captured by any two camera pairs having overlapping imaging areas where at least a portion of the fields of view overlap.

[0021] In the image acquisition process P101, images captured by the cameras C1, C2, and C3 are acquired.

[0022] In the distance calculation process P102, the distance is calculated using two cameras, C2 and C3, whose image quality is not degraded. FIG. 4 shows the distance calculation method. Here, the image captured by C2 is 301, and the image captured by C3 is 300. Stereo matching is used to calculate the distance. When calculating the distance to position 302(x,y) of image 301 captured by C2, block matching is performed from position 303(x,y) to position 304(x+D,y) of image 300 captured by C3, and the position d at which the ZSAD (Zero Mean Sum of Absolute Difference) is minimized is defined as the parallax. Here, D is the search width in stereo matching. Using the calculated d, the distance Z can be calculated by f*B / d. Here, f is the focal length of C2 and C3, and B is the baseline. From the distance calculation results for each position on image 301, a parallax image (distance image) 301D can be generated, which includes distance information for areas where at least a portion of image 301 overlaps.

[0023] In viewpoint conversion processing P103, the viewpoint is converted so that image 301 captured by camera C2 becomes an image captured by camera C1 (to the viewpoint of camera C1). Here, using the distance Z calculated in distance calculation processing P102, each pixel of image 301 can be treated as a 3D point group X. Furthermore, if R and t are respectively a rotation matrix and a translation vector from camera C2 to camera C1, and K is an internal parameter matrix of camera C1, then by calculating K[R|t]X, image 301 captured by camera C2 can be converted into an image captured from the viewpoint of camera C1.

[0024] In the image quality improvement process P104, an image is generated from which raindrops adhering to camera C1 have been removed. The details of this process are shown in Figure 5. In Figure 5, a generative model trained to generate an image with improved image quality is used as the deep learning model. Here, image 500 is the image captured by camera C1, and image 501 is the image captured by camera C2, which has been converted to the viewpoint of camera C1 in viewpoint conversion process P103 (viewpoint converted image). Area 510 in image 500 indicates the area where raindrops are adhering. Image 501 is the result of viewpoint conversion of an image that does not contain raindrops, so it does not contain raindrops like those contained in image 500. In the image quality improvement process P104, these two images, image 500 and image 501, are input into the deep learning model to generate image 502 from which the dirt has been removed. The respective images are combined in the channel direction and input. The deep learning model is a convolutional neural network trained using the training method described below. It is an encoder-decoder type convolutional neural network consisting of five stages (1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32). Each stage consists of two 3x3 convolutional layers, and after each convolution, it is activated by a batch normalization layer and a rectified linear unit. However, the final layer of the network is activated by a sigmoid function rather than a rectified linear unit. Next, we will explain the training method for the deep learning model. Similarly, during training, an image with reduced image quality due to raindrops or mud (equivalent to 500 in Figure 5) and a perspective-transformed image (equivalent to 501 in Figure 5) are input. An image without reduced image quality due to raindrops or mud (equivalent to 502 in Figure 5) is used as the ground truth. The loss is |v-v'|. Here, v is the network output, and v' is the pixel value of the ground truth, which is the image without reduced image quality. By minimizing this loss function, the image quality of images with dirt or other contaminants can be improved.

[0025] As described above, the image processing device 1 of this embodiment uses a clean camera to improve the image quality of a dirty camera. As a result, even if the camera is covered with dirt with poor permeability, such as mud or soil, and a vehicle or the like is present behind the dirt, the image information from the clean camera can be used, allowing for accurate image quality improvement.

[0026] Furthermore, the image processing device 1 of this embodiment improves image quality using a deep learning model. For example, as a method of improving image quality without using a deep learning model, the viewpoint converted image, which is the output of the viewpoint conversion process, can be used as the improved image quality result. However, since the distance information used in the viewpoint conversion itself contains errors, the improved result will contain errors. On the other hand, by using a deep learning model, even if the viewpoint conversion itself contains errors, the final image quality can be improved by taking those errors into account, thereby enabling high-precision image improvement.

[0027] The image processing device 1 of this embodiment can also include a factor identification unit 105, as shown in Fig. 6. The factor identification unit 105 identifies the factor that caused the degradation in image quality, such as whether it was caused by raindrops or mud. The factor identification unit 105 can use a machine learning model that uses deep learning, or can identify the factor by classical image processing.

[0028] Hereinafter, a series of processing steps in a configuration with the factor identification unit 105 added will be described with reference to Fig. 7. Here, the image acquisition process P101, distance calculation process P102, and viewpoint conversion process P103 are the same as those described in Fig. 3, so detailed explanations will be omitted. In addition, in the factor identification process P105, a case will be described in which two patterns of factors, raindrops and mud, are identified as factors that cause degradation of image quality.

[0029] The factor identification process (P105) uses a deep learning machine learning model to identify the causes of image quality degradation. Specifically, it uses a convolutional neural network consisting of a five-stage (1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32) feature extraction block, global average pooling, and a classification layer. Each stage of the feature extraction block has two 3x3 convolution layers, and after each convolution, activation is performed using a batch normalization layer and a rectified linear unit. The classification layer is composed of a fully connected layer. It is also possible that a single image may contain both raindrops and mud. Therefore, since these two causes of image quality degradation must be handled simultaneously, the output nodes corresponding to each class are activated using a sigmoid function, enabling multi-label classification. Next, we describe the network training method. The network uses labels representing the images with degraded image quality and their causes. For example, if the cause is raindrops, the network trains so that the activation value corresponding to raindrops approaches 1, and the activation value corresponding to mud approaches 0. Also, if there are two factors, raindrops and mud, the activation values ​​corresponding to raindrops and mud are trained to both approach 1. In the factor identification process P105, the network trained in this way is used to identify the factors that cause deterioration in image quality.

[0030] In the image quality improvement process P104, the image quality is improved using the image whose quality has been degraded by dirt or the like, the viewpoint converted image, and the information on the cause of image quality degradation output by the factor identification process P105. These images and the information on the cause of image quality degradation are combined in the channel direction and input to the network. Here, the information on the cause of image quality degradation is tiled in the spatial direction using the output of the factor identification process P105 and combined with other images. The network used in the image quality improvement process P104 is trained using the loss function described above.

[0031] The image processing device 1 of this embodiment identifies factors that cause image quality degradation in an image and improves image quality using the factor information. In this way, the image quality improvement process can remove dirt by taking the factors into account, thereby achieving more accurate dirt removal. In the above description, the image quality improvement process P104 directly uses the output of the factor identification process P105. However, a threshold may be introduced for the score of each factor output by the factor identification process P105, and the score may be converted to a binary representation of 0 or 1 and input. Furthermore, while the factor identification process P105 deals with a classification problem of identifying factors that cause image quality degradation in an input image, it can also deal with semantic segmentation, which identifies factors that cause image quality degradation for each pixel of the input image.

[0032] The image processing device 1 of this embodiment has been described so far as a case where a pixel value is assigned to each pixel of the viewpoint-converted image. Hereinafter, a case where a pixel value is not assigned to all pixels of the viewpoint-converted image due to viewpoint conversion being performed based on the reprojection of the 3D point cloud X will be described. FIG. 8 shows a viewpoint-converted image (802). Black pixels in the figure indicate areas where no pixel value is assigned. In such cases, the value of the pixels to which no pixel value is assigned is set to 0. That is, 0 is assigned to pixels in the viewpoint-converted image (802) to which no pixel value is assigned. By doing so, the image quality improvement process can determine whether a pixel is not assigned a pixel value and therefore has no information by focusing on the value 0. As a result, areas to which pixel values ​​are assigned can be actively utilized, resulting in more accurate image quality improvement. Furthermore, while the above description has been given of an example in which distance is calculated by stereo matching using two cameras, the present invention may also acquire distance information using a 3D sensor such as LiDAR. When a 3D sensor such as LiDAR is used, the distance information that can be acquired becomes sparse, resulting in an increase in pixels to which no pixel value is assigned in the viewpoint-converted image. Even in such a case, by applying a value of 0 to pixels to which no pixel value has been assigned, it is possible to ensure the accuracy of image quality improvement while using a three-dimensional sensor.

[0033] Furthermore, although the image processing device 1 of this embodiment has been described so far as using three cameras, a configuration with five or more cameras can also be applied. Hereinafter, a description will be given with reference to FIG. 9. In FIG. 9, five cameras (C1, C2, C3, C4, and C5) are installed on a vehicle V1 to capture images of the environment surrounding the vehicle V1. It is assumed that raindrops on the lens of C1 degrade the image quality, while no raindrops are attached to C2, C3, C4, and C5, resulting in no degradation in image quality. It is also assumed that C2 and C3 are stereo cameras. Furthermore, as shown on the right side of FIG. 9, it is assumed that, of the image (I9) captured by C1, the left angle of view (R2) is shared with camera C2, and the right angle of view (R4) is shared with camera C4. A series of processing steps in this configuration will be described with reference to FIG. 3.

[0034] In the image acquisition process P101, images captured by the cameras C1, C2, C3, C4, and C5 are acquired.

[0035] In distance calculation processing P102, distance information for camera C2 is calculated using the camera pair (C2 and C3), and the distance for camera C4 is calculated using the camera pair (C4 and C5). Here, when calculating distance information using the camera pair (C2 and C3), distance information is calculated based on stereo matching as already described. When calculating distance using the camera pair (C4 and C5), distance is calculated using a known multi-view stereo method.

[0036] In the viewpoint conversion process P103, the viewpoint is converted so that the image captured by camera C2 becomes the image captured by camera C1 (to the viewpoint of camera C1), and the viewpoint is also converted so that the image captured by camera C4 becomes the image captured by camera C1 (to the viewpoint of camera C1). As already mentioned, the viewpoint conversion method utilizes the rotation and translation between the cameras, the internal parameters of the cameras, and distance information to convert the viewpoints of the images from camera C2 and camera C4 so that they each become the viewpoint of camera C1.

[0037] As already mentioned, the image quality improvement process P104 uses a convolutional neural network to improve image quality using the image from camera C1 and the viewpoint-converted image from camera C2 as input, and also to improve image quality using the image from camera C1 and the viewpoint-converted image from camera C4 as input. This allows two images with improved image quality to be calculated, and the two images are integrated based on the information on the areas shown on the right side of Figure 9. Specifically, the image quality improvement result generated from the image from camera C1 and the viewpoint-converted image from camera C2 is a result corresponding to area R2, and the image quality improvement result generated from the image from camera C1 and the viewpoint-converted image from camera C4 is a result corresponding to area R4, and the integration of these two results is the final output result.

[0038] In this way, the image processing device 1 of this embodiment improves the image quality of a single camera image by linking multiple cameras. In other words, since the image quality of camera images can be improved by sharing it among multiple cameras, the image quality can be improved even when the cameras are installed in various positions and orientations.

[0039] The image processing device 1 of the present embodiment described so far uses camera images with no degradation in image quality. Therefore, if dirt is experimentally attached to a camera with no degradation in image quality, the improvement in image quality will change. By experimentally attaching dirt to the camera in this way, the image quality will change, and therefore behavior such as object detection will also change, and ultimately the vehicle control results will also change. By conducting such experimental experiments and analyzing the behavior of vehicle control, it is possible to determine whether the present invention is applicable.

[0040] [Embodiment 2] 10 is a functional block diagram showing the configuration of an image processing device 2 of embodiment 2. The configuration of embodiment 2 includes an image acquisition unit 100, a distance information acquisition unit 101, an information storage unit 102, a viewpoint conversion unit 103, an image quality improvement unit 104, and an area identification unit 106. The following description will focus on the configuration and processing flow added to embodiment 1.

[0041] The region identification unit 106 identifies regions in an image where image quality has deteriorated. That is, the region identification unit 106 identifies regions where raindrops, mud, etc., which are factors that deteriorate image quality, are attached. The region identification unit 106 identifies regions where image quality has deteriorated using a machine learning model based on deep learning or classical image processing based on brightness values ​​and the amount of blur.

[0042] Hereinafter, an example of the operation of the image processing device 2 of this embodiment will be described in detail with reference to the flowchart of FIG. 11. The image processing device 2 of this embodiment improves image quality by generating an image from which dirt such as raindrops or mud has been removed when the camera is installed outside the vehicle cabin and dirt such as raindrops or mud has adhered to the camera (lens) and captured in the image, resulting in a degradation of image quality, or when the camera is installed inside the vehicle cabin and dirt on the windshield or the like has been indirectly captured in the image, resulting in a degradation of image quality. The following example of operation will discuss a method for improving image quality by generating an image from which dirt such as raindrops or mud has been removed when the camera is installed outside the vehicle cabin and dirt such as raindrops or mud has adhered to the camera (lens) and captured in the image, resulting in a degradation of image quality. Here, as shown in FIG. 2, the image processing device 2 is assumed to have three cameras (C1, C2, and C3) connected to a vehicle V1 to capture the environment surrounding the vehicle V1, and the image from C1 has deteriorated image quality due to dirt, while the images from C2 and C3 have no deterioration in image quality.

[0043] In the image acquisition process P101, images captured by the cameras C1, C2, and C3 are acquired.

[0044] In the region identification process P106, regions of degraded image quality are identified in the image from camera C1. As shown in Figure 12, a region identification model is used to identify regions (510) where image quality has been degraded due to dirt or other factors, and a labeled image (120) is output, in which the relevant pixels are assigned a value of 1 and all other pixels are assigned a value of 0. In other words, the labeled image (120) is an image in which the region (510) where image quality has been degraded is identified at the pixel level. The region identification model outputs the labeled image by performing semantic segmentation. The model structure is an encoder-decoder type convolutional neural network consisting of five stages (1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32). Each stage has two 3x3 convolution layers, and after each convolution, activation is performed by a batch normalization layer and a rectified linear unit. However, only the final layer of the network is activated by a SoftMax function. Here, the region identification model is trained using the correct value for the presence or absence of image quality degradation. Specifically, the loss function used is the cross-entropy that uses the network output and the correct answer value.

[0045] In process P107, the number of pixels in the area identified in area identification process P106 is counted, and processing is switched depending on the count value. Specifically, if the number of pixels is equal to or greater than a threshold (for example, 10 pixels or more) (i.e., if at least a portion of the area in the image of camera C1 is identified as an area where image quality has deteriorated), distance calculation process P102, viewpoint conversion process P103, and image quality improvement process P104 are executed in this order as described in embodiment 1, and the image quality of camera C1 is improved. On the other hand, if the number of pixels is less than the threshold, it is determined that there is almost no deterioration in image quality of camera C1, and processing is terminated.

[0046] In this way, the image processing device 2 of this embodiment executes the image quality improvement process only when the image quality of (at least a part of) the image has deteriorated. By doing so, the process is not executed when the image quality has not deteriorated, thereby saving calculation resources and reducing power consumption.

[0047] Furthermore, the image processing device 2 of this embodiment can also perform processing as shown in FIG. 13 in the image quality improvement process P104. In FIG. 13, an image 502 is obtained by removing dirt using the image from camera C1 and the viewpoint conversion image from camera C2. The labeled image output by the region identification process P106 is 120, and the image from camera C1 with reduced image quality is 500. As described above, the value of the portions of the labeled image 120 with reduced image quality is 1, and the other portions are 0. In the image quality improvement process P104, based on the information in the labeled image 120, the output (502) of the generative model is used for the portions with reduced image quality, and the image 500 is used for the portions with no reduced image quality, to generate a final image (503). Specifically, the element product of image 502 and labeled image 120 is calculated, and then the labeled image 120 is inverted (changing 1 to 0 and 0 to 1), and the element product of the inverted labeled image and image 500 is calculated. The results of these two element products are added together to generate the final output.

[0048] In this way, the image processing device 2 of this embodiment improves the image quality only in areas where image quality has deteriorated by using the output (502) of the generative model. This makes it possible to limit the influence of the generative model to only areas where image quality has deteriorated, and remove dirt while maintaining the original appearance of the image.

[0049] Furthermore, the region identification process P106 in the image processing device 2 described above uses a neural network that uses only a 3x3 convolution kernel to identify areas of image quality degradation at the pixel level. However, in addition to calculations using a 3x3 convolution kernel, global average pooling can also be used. Specifically, global average pooling is applied to a feature map with 1 / 32x resolution, and the output result of global average pooling is spatially tiled. The tiled feature map is then combined in the channel direction of the feature map with 1 / 32x resolution. By modifying the neural network encoder in this way, global information about the entire image can be propagated to each pixel of the feature map. That is, this neural network calculates feature values ​​extracted from the entire image and uses these feature values ​​to identify areas of image quality degradation at the pixel level. This makes it possible to use information about dirt adhering to a certain area of ​​the image to detect other dirt, thereby more accurately identifying areas of image quality degradation such as dirt.

[0050] In addition to global average pooling, a self-attention structure can also be used. Specifically, a query, key, and value are calculated from a feature map with 1 / 32x resolution, and an attention map (relevance) is calculated using the query and key. The calculated attention map is then used as a weight to integrate the value features. In other words, this neural network calculates the relevance between each pixel, integrates the features for each pixel based on that relevance, and uses the integrated features to identify areas of poor image quality at the pixel level. This allows information about the entire image to be conveyed to a particular pixel while taking into account the relevance between pixels. Information about dirt on a particular part of the image can be used to detect other dirt, further preventing undetected areas.

[0051] Furthermore, in the above explanation, an example has been given in which only camera C1's image quality has deteriorated, but in a real environment, it is conceivable that raindrops or the like may also adhere to camera C2, causing a deterioration in image quality. In such a case, in the image processing device 2 of this embodiment, the area identification process P106 (area identification unit 106) and the viewpoint conversion process P103 (viewpoint conversion unit 103) can be changed as follows.

[0052] In the area identification process P106, areas of reduced image quality are identified not only in the image from camera C1 but also in the image from camera C2. Then, in the viewpoint conversion process P103, the image from camera C2 is converted to the viewpoint of camera C1, but only areas determined to have no reduced image quality in the area identification process P106 are converted to the viewpoint. In other words, the viewpoint conversion is performed on areas excluding areas of reduced image quality in the image from camera C2 (areas of reduced image quality included in the image from camera C2). In this way, the image quality of camera C1 can be improved using only information on areas of camera C2 that have no reduced image quality, and the image quality of camera C1 can be improved even when the image quality of camera C2 has reduced.

[0053] Furthermore, in the first and second embodiments described above, the improvement of image quality was described using camera images taken at the same time. However, the present invention can also use camera images taken at different times. Specifically, by using the amount of vehicle movement, it is possible to change a camera image taken at one time to the viewpoint of a camera taken at a different time. This makes it possible to improve image quality by using images taken at different times.

[0054] Furthermore, in the above-described first and second embodiments, the degradation of image quality due to raindrops, mud, etc. has been described as an example, but other possible causes of degradation of image quality are the headlights of oncoming vehicles, a sudden change in brightness at the exit of a tunnel, etc. These causes of degradation of image quality are also within the scope of the present invention, and the present invention is not limited to the removal of dirt such as raindrops and mud.

[0055] [summary] As described above, the image processing device of this embodiment includes an image acquisition unit 100 that acquires a first image and a second image captured by a first camera (camera C2) and a second camera (camera C1) (that capture the environment surrounding vehicle V1), an information storage unit 102 that stores camera position information regarding the installation positions of the first camera (camera C2) and the second camera (camera C1), a distance information acquisition unit 101 that acquires distance information in at least a portion of the first image, a viewpoint conversion unit 103 that generates a first viewpoint converted image by viewpoint converting the first image to the viewpoint of the second camera (camera C1) based on the camera position information and the distance information, and an image quality improvement unit 104 that improves the image quality of the second image based on the second image and the first viewpoint converted image.

[0056] The image processing device of this embodiment further includes an area identification unit 106 that identifies areas of reduced image quality contained in the second image, and the image quality improvement unit 104 improves the image quality of the second image only when the area identification unit 106 identifies at least a portion of the area of ​​the second image as an area of ​​reduced image quality (embodiment 2).

[0057] In the image processing device of this embodiment, the image quality improving unit 104 improves the image quality of the second image only in the area identified by the area identifying unit 106 as a location of degraded image quality.

[0058] In the image processing device of this embodiment, the area identification unit 106 identifies areas of reduced image quality contained in the first image in addition to the second image, and the viewpoint conversion unit 103 converts the viewpoint of the area excluding the areas of reduced image quality contained in the first image.

[0059] In the image processing device of this embodiment, the region identification unit 106 is a neural network that identifies areas of degraded image quality at the pixel level. The neural network calculates feature amounts extracted from the entire image and identifies areas of degraded image quality at the pixel level using the feature amounts. Alternatively, the neural network calculates the correlation between pixels, integrates the feature amounts for each pixel based on the correlation, and identifies areas of degraded image quality at the pixel level using the integrated feature amounts.

[0060] The image processing device of this embodiment further includes a factor identification unit 105 that identifies factors that degrade image quality contained in the second image, and the image quality improvement unit 104 improves the image quality of the second image based on the factors that degrade image quality in addition to the second image and the first viewpoint conversion image (a modified form of embodiment 1).

[0061] In the image processing device of this embodiment, the image acquisition unit 100 acquires a third image captured by a third camera (camera C3) whose field of view overlaps with that of the first camera (camera C2) and the second camera (camera C1) (which capture the surrounding environment of the vehicle V1), and the distance information acquisition unit 101 acquires the distance information by a multi-view stereo method using the first image and the third image.

[0062] In the image processing device of this embodiment, the viewpoint conversion unit 103 assigns 0 to pixels of the first viewpoint converted image to which pixel values ​​of the first image are not assigned.

[0063] In the image processing device of this embodiment, the image quality improvement unit 104 is a deep learning model (generative model) that is trained to generate an image after image quality improvement.

[0064] In the image processing device of this embodiment, the image quality improving unit 104 improves the image quality by generating an image from which dirt (raindrops, mud, etc.) contained in the second image has been removed.

[0065] The image processing method of this embodiment is an image processing method executed by a computer, and includes an image acquisition step (image acquisition process P101) of acquiring a first image and a second image captured by a first camera (camera C2) and a second camera (camera C1) (which capture the environment surrounding the vehicle V1), a distance information acquisition step (distance calculation process P102) of acquiring distance information in at least a portion of the first image, a viewpoint conversion step (viewpoint conversion process P103) of generating a first viewpoint converted image by viewpoint converting the first image to the viewpoint of the second camera (camera C1) based on camera position information regarding the installation positions of the first camera (camera C2) and the second camera (camera C1) and the distance information, and an image quality improvement step (image quality improvement process P104) of improving the image quality of the second image based on the second image and the first viewpoint converted image.

[0066] According to this embodiment, even when image quality is significantly degraded, it is possible to accurately improve image quality. As a result, for example, even when dirt with poor permeability such as mud or earth adheres to the camera and a vehicle or the like is present behind the dirt, information on the camera image without the dirt is used to generate a camera image from which the dirt has been removed, thereby accurately improving the image quality of the camera image with the dirt, making it possible to detect vehicles or the like that should be detected, and appropriately control the vehicle.

[0067] Although the present invention has been described above with reference to Embodiments 1 and 2, the present invention is not limited to the above embodiments. In particular, the network structure described in the present invention has been described using an example, but it is easy to imagine changing the number of stages, the number of convolutional layers between stages, and other such changes. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0068] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely realized by hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be realized by software, in which a processor interprets and executes a program that realizes each function. Information such as the program, tape, and file that realizes each function can be stored in a memory, a recording device such as a hard disk or solid state drive (SSD), or a recording medium such as an IC card, SD card, or DVD.

[0069] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0070] 1. Image processing device 2. Image processing device 100 Image acquisition unit 101 Distance information acquisition section 102 Information storage unit 103 Viewpoint conversion unit 104 Image Quality Improvement Section 105 Factor Identification Unit 106 Area identification part

Claims

1. an image acquisition unit that acquires a first image and a second image captured by the first camera and the second camera; an information storage unit in which camera position information relating to the installation positions of the first camera and the second camera is stored; a distance information acquisition unit that acquires distance information in at least a partial region of the first image; a viewpoint conversion unit that converts the first image into a viewpoint of the second camera based on the camera position information and the distance information, thereby generating a first viewpoint-converted image; an image quality improving unit that improves the image quality of the second image based on the second image and the first viewpoint converted image.

2. 2. The image processing device according to claim 1, further comprising an area specifying unit for specifying an area of ​​reduced image quality included in the second image; an image processing device, characterized in that the image quality improvement unit improves the image quality of the second image only when the area identification unit identifies at least a part of the area of ​​the second image as an area of ​​reduced image quality.

3. 3. The image processing device according to claim 2, The image processing device is characterized in that the image quality improvement unit improves the image quality of only the area of ​​the second image that the area specification unit specifies as a location of reduced image quality.

4. 3. The image processing device according to claim 2, the region specifying unit specifies a portion of the first image where image quality has deteriorated in addition to the second image; The image processing device, wherein the viewpoint conversion unit converts the viewpoint of an area of ​​the first image excluding areas of reduced image quality.

5. 3. The image processing device according to claim 2, The image processing device is characterized in that the area specifying unit is a neural network and specifies areas where image quality has deteriorated at the pixel level.

6. 6. The image processing device according to claim 5, The image processing device is characterized in that the neural network calculates feature amounts extracted from the entire image and identifies areas where image quality has deteriorated at the pixel level using the feature amounts.

7. 6. The image processing device according to claim 5, The neural network calculates the degree of association between each pixel, integrates the feature values ​​for each pixel based on the degree of association, and identifies areas of reduced image quality at the pixel level using the integrated feature values.

8. 2. The image processing device according to claim 1, a factor specifying unit for specifying a factor of image quality degradation included in the second image; The image processing device is characterized in that the image quality improvement unit improves the image quality of the second image based on the second image, the first viewpoint converted image, and the cause of image quality degradation.

9. 2. The image processing device according to claim 1, the image acquisition unit acquires a third image captured by a third camera whose field of view overlaps with each of the first camera and the second camera; The image processing device, wherein the distance information acquisition unit acquires the distance information by a multi-view stereo method using the first image and the third image.

10. 2. The image processing device according to claim 1, The image processing device, wherein the viewpoint conversion unit assigns 0 to pixels of the first viewpoint converted image that are not assigned a pixel value of the first image.

11. 2. The image processing device according to claim 1, The image processing device, wherein the image quality improvement unit is a deep learning model trained to generate an image after image quality improvement.

12. 2. The image processing device according to claim 1, The image processing device, wherein the image quality improving unit improves image quality by generating an image from which dirt contained in the second image has been removed.

13. 1. A computer-implemented image processing method, comprising: an image acquisition step of acquiring a first image and a second image captured by a first camera and a second camera; a distance information acquisition step of acquiring distance information in at least a partial region of the first image; a viewpoint conversion step of converting the first image into a viewpoint of the second camera based on camera position information relating to installation positions of the first camera and the second camera and the distance information, thereby generating a first viewpoint-converted image; an image quality improvement step of improving the image quality of the second image based on the second image and the first viewpoint converted image.

Images