Electronic device, image processing method, and image processing program

Abstract

This invention provides an electronic device, an image processing method, and an image processing program that can easily increase the variety of training images. [Solution] The electronic device 10 includes a processing unit 14 that inputs a pre-processing image to a water droplet effect reproduction model and obtains a processed image from the water droplet effect reproduction model by processing the pre-processing image so that it becomes an image of what would happen if water were attached to the path of the imaging light incident on the camera 20, and an output unit 16 that outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera 20 with a second image data captured by the camera 20 with water attached to the path of the imaging light incident on the camera 20.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, an image processing method, and an image processing program.

Background Art

[0002] As described in Patent Document 1, an apparatus that stops a recognition operation when raindrops adhere to the lens of an in-vehicle camera is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a learned model that uses object detection in an image as a use case, the adhesion of water droplets to the lens of a camera or the like affects the detection accuracy. It is required to easily increase the variations of images taken in a state where water droplets adhere to a lens or the like as learning images used for learning to generate a learned model so that objects are less likely to be misdetected while the detection operation continues.

[0005] An object of the present disclosure is to provide an electronic device, an image processing method, and an image processing program that can easily increase the variations of learning images.

Means for Solving the Problems

[0006] An electronic device according to one embodiment of the present disclosure comprises a processing unit and an output unit. The processing unit inputs a pre-processing image to a water droplet effect reproduction model and obtains a processed image from the water droplet effect reproduction model by processing the pre-processing image to represent an image of what would occur if water were adhering to the path of the imaging light incident on the camera. The output unit outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

[0007] An image processing method according to one embodiment of the present disclosure includes inputting a pre-processing image to a water droplet effect reproduction model, obtaining a processed image from the water droplet effect reproduction model obtained by processing the pre-processing image to represent an image of water adhering to the path of imaging light incident on the camera, and outputting the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates first image data captured by the camera with second image data captured by the camera with water adhering to the path of imaging light incident on the camera.

[0008] An image processing program according to one embodiment of the present disclosure causes a processor to input a pre-processing image into a water droplet effect reproduction model, process the pre-processing image to resemble an image of water adhering to the path of imaging light incident on the camera, obtain a processed image from the water droplet effect reproduction model, and output the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of imaging light incident on the camera. [Effects of the Invention]

[0009] According to an electronic device, an image processing method, and an image processing program according to one embodiment of this disclosure, the variations of training images can be easily increased. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows a schematic configuration example of an image processing system according to one embodiment. [Figure 2] This figure shows an example of training data, consisting of pairs of images showing no water droplet effect and images showing water droplet effect. [Figure 3A] This figure shows an example of an image unaffected by water droplets, including visible and infrared images. [Figure 3B] This figure shows an example of an image affected by water droplets, including visible and infrared images. [Figure 4] This figure shows an example of a processed image generated by processing an image before processing. [Figure 5] This block diagram shows an example of the learning process for generating a water droplet effect reproduction model, and an example of the inference process for processing using the generated water droplet effect reproduction model. [Figure 6] This flowchart shows an example of the procedure for an image processing method according to one embodiment. [Figure 7] This block diagram shows an example of the learning process for generating an object detection model, and the inference process for object detection using the generated object detection model. [Modes for carrying out the invention]

[0011] Improving object detection accuracy from images is crucial. For example, increasing the accuracy of object detection from images captured by in-vehicle cameras enhances vehicle safety. Furthermore, improving object detection accuracy from images is beneficial not only for in-vehicle cameras but also in various other fields.

[0012] When generating a learned model applied to object detection by learning, learning data in which a learning image in which an object to be detected appears is associated with correct data of the position of the object appearing in the learning image is used for learning. The greater the variation in the learning data, the higher the accuracy of object detection using the learned model generated by performing learning using the learning data.

[0013] In object detection from an image, when the influence of water droplets adhering to a lens or the like of a camera that captures the image appears in the image, the detection accuracy of the object may decrease due to the influence of the water droplets. Even when the influence of the water droplets appears in the image, it is conceivable to perform learning using a learning image in which the influence of the water droplets appears so as to maintain or improve the detection accuracy. The greater the variation in the influence of the water droplets in the learning image, the more the detection accuracy of the object from the image in which the influence of the water droplets appears is maintained or improved by the learned model generated by performing learning using the learning image.

[0014] When actually capturing and generating a learning image, it is realistically difficult to capture by comprehensively reproducing the mode of adhesion of water droplets. Then, it is difficult to increase the variation in the influence of water droplets as a variation in the learning image. There is a need to easily generate a learning image with an increased variation in the influence of water droplets.

[0015] Hereinafter, in the present disclosure, an example of an embodiment of an electronic device 10 (see FIG. 1), an image processing method, and an image processing program that can easily increase the variation in learning images will be described.

[0016] (Configuration example of image processing system 1) As shown in FIG. 1, an image processing system 1 according to an embodiment of the present disclosure includes an electronic device 10 and a camera 20.

[0017] <Electronic device 10> The electronic device 10 includes an acquisition unit 12, a processing unit 14, an output unit 16, a learning unit 18, and a storage unit 22.

[0018] The acquisition unit 12 acquires image data from the camera 20. The acquisition unit 12 may acquire various other data or information. The acquisition unit 12 may include a communication interface for communicating with the camera 20 or other devices, either wired or wirelessly. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0019] The acquisition unit 12 may include an input device for receiving input from the user. The input device may include, for example, a keyboard or physical keys, or may include a pointing device such as a touch panel, touch sensor, or mouse. The input device is not limited to these examples and may be configured to include various other devices. The acquisition unit 12 may be configured to communicate with an external input device. The electronic device 10 may include the input device as an input unit separate from the acquisition unit 12.

[0020] The processing unit 14 processes the image data acquired by the acquisition unit 12 to generate a learning image. Further, the processing unit 14 generates learning data including the learning image.

[0021] The processing unit 14 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The general-purpose processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The dedicated circuit may include, for example, an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The processing unit 14 may be implemented as a single integrated circuit (IC). The processing unit 14 may be implemented as a plurality of communicably connected integrated circuits or discrete circuits. The processing unit 14 may include either a System-on-a-Chip (SoC) or a System-in-a-Package (SiP) in which one or more processors cooperate. The processing unit 14 may be implemented based on various other known technologies.

[0022] The storage unit 22 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit 22 stores various types of information. The storage unit stores programs executed by a general-purpose processor or the like that functions as the processing unit 14, or various types of data or information. The storage unit 22 may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the processing unit 14. At least a part of the storage unit 22 may be configured integrally with the processing unit 14.

[0023] The output unit 16 outputs the training image or training data generated by the processing unit 14 to the training unit 18 or an external training device. The output unit 16 may also output various other data or information. The output unit 16 may be equipped with a communication interface for wired or wireless communication with the training unit 18, an external training device, or other devices. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0024] The output unit 16 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) or an inorganic EL display. The electronic device 10 may also include the display device as a separate display unit from the output unit 16.

[0025] The output unit 16 is not limited to these examples and may include various other output devices such as an audio output unit that outputs sound, or a vibration output unit that outputs vibration.

[0026] The learning unit 18 generates a trained model by performing training using training images or training data. The learning unit 18 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The learning unit 18 may be configured identically to or similarly to the processing unit 14. The learning unit 18 may be included in the processing unit 14. If the learning unit 18 is included in the processing unit 14, the processing unit 14 may implement the functions of the learning unit 18.

[0027] The electronic device 10 relating to this disclosure may be various types of devices. For example, the electronic device 10 according to one embodiment may be a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC (Personal Computer), computer, or server. The electronic device 10 may be configured in a cloud computing environment or in an on-premise environment. The functions of each part of the electronic device 10 may be realized on separate servers connected via a wired or wireless network. The electronic device 10 may be mounted on a mobile device such as a vehicle including a bus or truck, a drone, an airplane, a bicycle, a motorcycle, a tractor, a delivery robot, or a security robot.

[0028] <Camera 20> The camera 20 includes an image sensor. The image sensor may include, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor may be configured to capture at least one of infrared light or visible light. The camera 20 may include both an infrared image sensor and a visible light image sensor. The infrared light may include, for example, light in the wavelength range from 780 nm to 1000 nm. The visible light may include, for example, light in the wavelength range from 380 nm to 780 nm.

[0029] Camera 20 includes an optical system that images incoming light onto an image sensor. The optical system may include lenses or mirrors. The image sensor can capture light that has been imaged within the imaging range by the optical system. Of the light arriving towards camera 20, the amount of light that is imaged within the imaging range by the optical system is determined according to the configuration of the optical system. The light that is imaged within the imaging range by the optical system and captured by the image sensor is also called imaging light.

[0030] The camera 20 may have a cover on the outside of the optical system that transmits imaging light. The cover may be included in the optical system as the outermost element of the optical system.

[0031] The image generated by camera 20 when it captures imaging light with its image sensor is affected by water droplets adhering to the path through which the imaging light passes. For example, water droplets adhering to the part of the camera 20's optical system or cover through which the imaging light passes will cause refraction, reflection, or scattering of the imaging light. Also, if camera 20 is mounted inside the vehicle's windshield, water droplets adhering to the part of the windshield through which the imaging light passes will cause refraction, reflection, or scattering of the imaging light. Due to the refraction, reflection, or scattering of the imaging light, the image captured when water droplets are present will differ from the image captured when water droplets are absent.

[0032] The number of cameras 20 is not limited to one; there may be two or more. Cameras 20 may be mounted on moving objects such as vehicles, or on devices such as roadside units used in traffic systems. Cameras 20 may also be included in electronic equipment 10.

[0033] (Example of operation of image processing system 1) In the image processing system 1 of this disclosure, the electronic device 10 generates training images used for training to generate an object detection model. As described above, if an image to be detected is affected by water droplets attached to the lens of the camera 20 that captures the image, the water droplets may cause the object to not be detected or to be falsely detected, reducing the accuracy of object detection. In other words, the electronic device 10 generates training images with increased variations in the effects of water droplets so that the object detection model can maintain or improve the accuracy of object detection from images affected by water droplets.

[0034] In this disclosure, the acquisition unit 12 of the electronic device 10 acquires a pre-processing image that will be the basis for the training image. The pre-processing image may be image data captured by the camera 20. The pre-processing image may also be image data generated by an external device of the image processing system 1. The pre-processing image may include an image taken when there are no water droplets attached to the lens or cover of the camera 20 or the windshield of a vehicle, etc. The pre-processing image may include an image taken when there are almost no water droplets attached. The state of having almost no water droplets attached may include a state where there are few attached water droplets or where the attached water droplets are small.

[0035] <Processing> The processing unit 14 of the electronic device 10 inputs the pre-processing image to the water droplet effect reproduction model and acquires the image output from the water droplet effect reproduction model as the processed image. The water droplet effect reproduction model accepts the pre-processing image as input, processes the input pre-processing image to reproduce the effect of additional water droplets that were not present in the path of the imaging light when the pre-processing image was captured, and outputs the processed image.

[0036] In other words, the processed image is an image obtained by processing the original image to reproduce the effect of water droplets that have additionally adhered to the path of the imaging light incident on the camera 20. The processed image is used as a training image for generating a water droplet effect reproduction model. The processed image may include images that have changed due to the effect of water droplets compared to the original original image. The processed image may include images in which the number of areas affected by water droplets has increased, or images in which the area affected by water droplets has expanded, compared to the original image.

[0037] The water droplet effect reproduction model may be configured to accept input for the position of the light source in the pre-processed image. The light source may include a self-illuminating object such as a light fixture. The light source may include an object that reflects light emitted from other objects toward the camera 20. The light source may include an object that directs light stronger than that of surrounding objects toward the camera 20. The light source position may include, for example, the position where the light fixture is installed.

[0038] The water droplet effect reproduction model relating to this disclosure may be a CycleGAN model, which is a type of GAN (Generative Adversarial Network). A CycleGAN model is a model that learns the domain relationship between two image datasets and transforms image data belonging to one image dataset into image data belonging to the other image dataset. A domain is a concept that distinguishes the fields or regions of images belonging to an image dataset. Specifically, the training of a CycleGAN model may be performed such that the difference between the image before and after transformation, i.e., the periodic consistency loss, is small when an image belonging to the first domain is transformed into an image belonging to the second domain and then transformed back into an image belonging to the first domain. A CycleGAN model can be trained without requiring paired data by using a periodic consistency loss. In other words, a CycleGAN model can transform from one region to another without a one-to-one mapping between source and target regions.

[0039] In the water droplet effect reproduction model relating to this disclosure, the first domain corresponds to an image without the effect of water droplets. The second domain corresponds to an image with the effect of water droplets. The water droplet effect reproduction model is a model generated by performing training to minimize the difference between the original image and the transformed image, i.e., the periodic consistency loss, when an image without the effect of water droplets is transformed into an image with the effect of water droplets and then transformed back into an image without the effect of water droplets.

[0040] The processing unit 14 acquires training data to generate a cycleGAN model as a water droplet effect reproduction model. The training data for the cycleGAN model includes an image dataset without water droplet effects and an image dataset with water droplet effects. Each of the image datasets, the one without water droplet effects and the one with water droplet effects, contains at least one image data. The image data included in the image dataset without water droplet effects is also called a water droplet-free image. The image data included in the image dataset with water droplet effects is also called a water droplet-affected image. The processing unit 14 pairs the water droplet-free images and water droplet-affected images and performs training to generate a water droplet effect reproduction model.

[0041] The image dataset affected by water droplets may include image data that does not match any other condition besides the presence or absence of water droplets compared to all image data in the image dataset not affected by water droplets. The processing unit 14 may perform training by pairing images that do not match any other condition besides the presence or absence of water droplets. In this case, the training data does not need to include combinations of images with and without water droplet effects that match any other condition besides the presence or absence of water droplets. If images that match any other condition besides the presence or absence of water droplets are required as training data, the work required to prepare such images may become excessive. By eliminating the need to prepare images that match any other condition besides the presence or absence of water droplets, the preparation of training data becomes easier. As a result, training to generate a water droplet effect reproduction model can be performed more simply.

[0042] The training data may include combinations of images with and without water droplet effects, where all other conditions are the same except for the presence or absence of water droplet effects.

[0043] The processing unit 14 may perform training by pairing one image without water droplet effect and two images with water droplet effect, as shown in the image dataset illustrated in Figure 2. In this case, the processing unit 14 can perform training using two pairs of images without water droplet effect and images with water droplet effect. If the training data includes two images without water droplet effect and two images with water droplet effect, the processing unit 14 can perform training using four pairs (2x2). The number of images without water droplet effect included in the training data may be three or more. The number of images with water droplet effect included in the training data may be three or more.

[0044] The image with water droplet effect, illustrated in Figure 2, differs from the image without water droplet effect in that the water droplet effect region 34 is visible. The image without water droplet effect may be an image with no water droplet effect at all, or an image with less water droplet effect than the image with water droplet effect. The magnitude of the water droplet effect may be expressed as the size of the water droplet effect region 34, or as the magnitude of the change in brightness of the water droplet effect region 34. The magnitude of the water droplet effect may also be expressed as the degree of contrast reduction in the water droplet effect region 34.

[0045] In addition, the image with water droplet effects, as exemplified in Figure 2, differs from the image without water droplet effects in terms of the people depicted. In other words, the processing unit 14 may perform training by pairing images with and without water droplet effects, where the conditions other than the presence or absence of water droplet effects do not match.

[0046] The water droplet-affected region 34 tends to appear over a wide area or with high brightness around the light source. Therefore, learning can be performed efficiently by performing learning that takes the light source position into consideration. Specifically, the processing unit 14 may perform learning using data that associates the light source position in the image without water droplet effect with the image without water droplet effect, and data that associates the light source position in the image with water droplet effect with the image with water droplet effect, as training data. In the example in Figure 2, a light source 32, which is a type of light source, is visible in both the image without water droplet effect and the image with water droplet effect. The processing unit 14 may perform learning using data that associates information about the position of the light source 32 with both the image without water droplet effect and the image with water droplet effect, as training data.

[0047] In the images without and with water droplet effects illustrated in Figure 2, the light source 32 is shown in the same position. The processing unit 14 may perform training by pairing images where the light source position matches, as in the example in Figure 2. The processing unit 14 may also perform training by pairing images where the difference in light source position falls within a predetermined range. In other words, the processing unit 14 may perform training by pairing images where the light source positions are close together. By performing training by pairing images where the light source positions are close together, the learning of how the light source is depicted under the influence of water droplets can be performed efficiently. Conversely, if training is performed by pairing images where the difference in light source position is outside the predetermined range, i.e., images where the light source position is far apart, the way the effect of water droplets manifests may differ significantly from reality, potentially reducing the contribution to the water droplet effect reproduction model.

[0048] The image dataset may be divided into an image dataset containing visible images and an image dataset containing infrared images. The processing unit 14 may use an image dataset containing visible images unaffected by water droplets and an image dataset containing visible images affected by water droplets as training data. The processing unit 14 may use an image dataset containing infrared images unaffected by water droplets and an image dataset containing infrared images affected by water droplets as training data. In other words, the processing unit 14 may perform training by pairing images unaffected by water droplets with visible images, or infrared images with infrared images.

[0049] The training data may include image data consisting of a set of visible and infrared images captured under the same conditions, as illustrated in Figures 3A and 3B. Figure 3A shows an example of a set of visible and infrared images without the effect of water droplets. Figure 3B shows an example of a set of visible and infrared images with the effect of water droplets.

[0050] The set of visible and infrared images with water droplet effects, as exemplified in Figure 3B, differs from the set of visible and infrared images without water droplet effects, as exemplified in Figure 3A, in that the water droplet-affected region 34 is visible. In addition, the image with water droplet effects exemplified in Figure 3B differs from the image without water droplet effects exemplified in Figure 3A in terms of the person depicted. Even when the processing unit 14 uses image data consisting of a set of visible and infrared images as training data, it may perform training by pairing image data that do not match in any condition other than the presence or absence of water droplet effects.

[0051] The processing unit 14 may generate a processed image that reproduces the effect of water droplet adhesion in the path of the imaging light in the pre-processed image by performing inference using the water droplet effect reproduction model generated as described above. Specifically, the processing unit 14 inputs the pre-processed image into the water droplet effect reproduction model generated by learning. The processing unit 14 may also input the light source position into the water droplet effect reproduction model.

[0052] The processing unit 14 acquires the image output from the water droplet effect reproduction model as the processed image. In other words, the processing unit 14 generates the output image from the water droplet effect reproduction model as the processed image. In this way, the processing unit 14 can generate a processed image using the water droplet effect reproduction model.

[0053] The processing unit 14 may input the pre-processing image into a water droplet effect reproduction model, as illustrated in Figure 4, and acquire the image output from the water droplet effect reproduction model as the processed image. In the processed image of Figure 4, a water droplet effect region 34 is added that reproduces the effect of water droplet adhesion. When inputting the pre-processing image into the water droplet effect reproduction model, the processing unit 14 may also input the position of the light source 32 into the water droplet effect reproduction model.

[0054] The processing described above may be divided into a learning phase in which a water droplet effect reproduction model is generated, and an inference phase in which a processed image is generated using the generated water droplet effect reproduction model, as illustrated in Figure 5. The processing unit 14 may perform learning in the learning phase using an image without water droplet effect, an image with water droplet effect, and the light source position as training data. In the inference phase, the processing unit 14 may input the unprocessed image and the light source position into the water droplet effect reproduction model and acquire the image output from the water droplet effect reproduction model as the processed image.

[0055] The processing unit 14 may obtain a water droplet effect reproduction model generated by another device, rather than generating the water droplet effect reproduction model itself. In other words, the processing unit 14 may execute only the inference phase without executing the learning phase.

[0056] In the block diagram illustrated in Figure 5, training data including the light source position is used for training, but training data that does not include the light source position may also be used. When training data that does not include the light source position is used for training, the water droplet effect reproduction model may be configured to output a processed image from the unprocessed image even without accepting input for the light source position. Even if the water droplet effect reproduction model does not accept input for the light source position, it can estimate the light source position shown in the unprocessed image and generate a processed image that reflects the effect of water droplets.

[0057] On the other hand, if the water droplet effect reproduction model accepts input for the light source position, improving the accuracy of the light source position can improve the accuracy of reproducing the effect of water droplet adhesion in the processed image.

[0058] <<Example of processing procedure>> The processing unit 14 may generate a processed image that reproduces the effect of water droplets in the original image by executing an image processing method that includes the steps of the flowchart illustrated in Figure 6. The image processing method may be implemented as an image processing program to be executed by the processing unit 14. The image processing program may be stored on a non-temporary computer-readable medium.

[0059] The processing unit 14 acquires the pre-processing image (step S1). The processing unit 14 may acquire the pre-processing image from the camera 20 or an external device. The processing unit 14 may store the pre-processing image acquired from the camera 20 or the like in the storage unit and acquire the pre-processing image from the storage unit. The processing unit 14 inputs the pre-processing image into the water droplet effect reproduction model (step S2). The processing unit 14 acquires the image output from the water droplet effect reproduction model as the processed image (step S3). The processing unit 14 outputs the processed image from the output unit 16 (step S4). After executing the procedure in step S4, the processing unit 14 finishes executing the procedure in the flowchart of Figure 6.

[0060] <<Summary of Processing>> As described above, the electronic device 10 of this disclosure can easily generate an image that reproduces the effect of water droplets from an image before processing. As a result, the variations of images that reproduce the effect of water droplets can be easily increased.

[0061] <Object detection processing> As described above, the processing unit 14 of the electronic device 10 according to this disclosure can generate processed images that reproduce the effect of water droplets adhering to the path of imaging light captured by the camera 20 in various variations, as training images used to generate an object detection model. The electronic device 10 may perform training to generate an object detection model, as illustrated in Figure 7, and perform object detection inference using the generated object detection model. Training to generate an object detection model may be performed by the training unit 18 of the electronic device 10, or by a training device different from the electronic device 10. Object detection inference using the object detection model may be performed by a detection device different from the electronic device 10 or the training device. Inference using the object detection model may be performed by the device that generated the object detection model, i.e., the electronic device 10 or the training device.

[0062] <<Training an object detection model>> The following describes an example of how the learning unit 18 of the electronic device 10 performs training on an object detection model. The learning unit 18 acquires the processed image generated by the processing unit 14 from the output unit 16 of the electronic device 10 as a training image. The learning unit 18 generates an object detection model by performing training using training data in which the training image is associated with the correct information of the object positions to be detected. The correct information of the object positions is the position information of an object that has been confirmed to be present in the training image. The correct information of the object positions may include information on whether the object to be detected is present in the training image. The learning unit 18 may acquire the correct information of the object positions through input from the user of the electronic device 10.

[0063] When the learning unit 18 acquires a learning image from the output unit 16, it may acquire the learning image as learning data in which the correct information of object positions is associated with the learning image. In this case, when the processing unit 14 generates a processed image, it may generate information that associates the correct information of object positions in the processed image with the processed image. The output unit 16 may output the information that associates the correct information of object positions with the processed image to the learning unit 18.

[0064] The object positions in the pre-processed image and the processed image are the same. Therefore, the processing unit 14 may acquire information on the object positions in the pre-processed image and generate information that associates this information with the processed image as the correct object position information in the processed image.

[0065] The examples of object detection model training described above may also be applied to learning devices.

[0066] <<Inference using an object detection model>> The following describes an example of operation in which a detection device different from the electronic device 10 or the learning device performs object detection inference using an object detection model. The detection device obtains the object detection model from the output unit 16 of the electronic device 10 when the learning unit 18 of the electronic device 10 generates the object detection model, and obtains the object detection model from the learning device when the learning device generates the object detection model.

[0067] The detection device inputs a detection image into the object detection model. The detection image is the image targeted for object detection. The detection device acquires the object detection result output from the object detection model. In this way, the detection device can acquire object detection results using the object detection model.

[0068] <<Summary of Object Detection Process>> As described above, the object detection model is generated by performing training using processed images as training images that reproduce the effects of water droplets adhering to the path of imaging light captured by the camera 20 in various variations. By performing object detection inference using the object detection model generated in this way, it is expected that the accuracy of object detection from the detected image will be maintained or improved, even when the detected image is affected by water droplets adhering to the lens, etc.

[0069] (summary) As described above, the electronic device 10, image processing method, and image processing program relating to this disclosure make it possible to easily generate images that reproduce the effect of water droplets adhering to the path of imaging light. As a result, the variety of images that reproduce the effect of water droplet adhesion can be easily increased.

[0070] Furthermore, an object detection model is generated by performing training using diverse variations of training images that reproduce the effects of water droplet adhesion. By performing object detection inference using this generated object detection model, it is expected that the accuracy of object detection from the detected image will be maintained or improved, even when the detected image is affected by water droplet adhesion on lenses, etc.

[0071] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0072] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. These are also to be understood as being included within the scope of this disclosure.

[0073] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0074] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0075] Vehicles relating to this disclosure may include, for example, automobiles, industrial vehicles, railway vehicles, residential vehicles, or fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture or construction. Industrial vehicles may include, for example, forklifts or golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. Vehicles of the same nature may be included in multiple classifications.

[0076] While embodiments of the image processing method using the image processing system 1 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0077] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. In addition, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0078] In one embodiment, (1) the electronic device includes a processing unit that inputs a pre-processing image to a water droplet effect reproduction model and obtains a processed image from the water droplet effect reproduction model by processing the pre-processing image so that it becomes an image of what would occur if water were adhering to the path of the imaging light incident on the camera; and an output unit that outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using learning data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

[0079] (2) In the electronic device described in (1) above, the water droplet effect reproduction model may be a model generated by performing learning using data in which the position of the first light source when the first image data is captured is associated with the first image data, and the position of the second light source when the second image data is captured is associated with the second image data.

[0080] (3) In the electronic device described in (2) above, the water droplet effect reproduction model may be a model generated by performing learning using data in which the difference between the position of the first light source and the position of the second light source is within a predetermined range.

[0081] (4) In the electronic device described in any one of (1) to (3) above, the camera may be configured to capture infrared light.

[0082] (5) In the electronic device described in any one of (1) to (4) above, the camera may be configured to capture visible light.

[0083] (6) The electronic device described in any one of (1) to (5) above may further include a learning unit that performs learning to generate an object detection model for detecting objects in an image. The output unit may output the processed image to the learning unit as learning data for generating the object detection model.

[0084] In one embodiment, (7) the image processing method includes inputting a pre-processing image to a water droplet effect reproduction model, obtaining a processed image from the water droplet effect reproduction model obtained by processing the pre-processing image so that it becomes an image of what would occur if water were adhering to the path of the imaging light incident on the camera, and outputting the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

[0085] In one embodiment, (8) the image processing program inputs a pre-processing image to a water droplet effect reproduction model, processes the pre-processing image to resemble an image of water adhering to the path of imaging light incident on the camera, obtains a processed image from the water droplet effect reproduction model, and outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of imaging light incident on the camera. [Explanation of Symbols]

[0086] 1. Image Processing System 10 Electronic devices (12: acquisition unit, 14: processing unit, 16: output unit, 18: learning unit, 22: memory unit) 20 cameras 32 light source 34 Water droplet influence area

Claims

1. A processing unit inputs a pre-processing image into a water droplet effect reproduction model, processes the pre-processing image to resemble an image of water adhering to the path of imaging light incident on the camera, and obtains a processed image from the water droplet effect reproduction model. The output unit outputs the processed image. Equipped with, The aforementioned water droplet effect reproduction model is an electronic device, which is a model generated by performing training using training data that associates first image data captured by the camera with second image data captured by the camera when water is adhering to the path of the imaging light incident on the camera.

2. The electronic device according to claim 1, wherein the water droplet effect reproduction model is a model generated by performing learning using data in which the position of the first light source when the first image data is captured is associated with the first image data, and the position of the second light source when the second image data is captured is associated with the second image data.

3. The electronic device according to claim 2, wherein the water droplet effect reproduction model is a model generated by performing learning using data in which the difference between the position of the first light source and the position of the second light source is within a predetermined range.

4. The camera is configured to capture infrared light, as described in any one of claims 1 to 3.

5. The camera is configured to capture visible light, as described in any one of claims 1 to 3.

6. It further includes a learning unit that performs training to generate an object detection model for detecting objects in an image, The output unit outputs the processed image to the learning unit as training data for generating the object detection model. The electronic device according to any one of claims 1 to 3.

7. The process involves inputting the pre-processing image into a water droplet effect reproduction model, processing the pre-processing image to resemble the image of water adhering to the path of the imaging light incident on the camera, and then obtaining a processed image from the water droplet effect reproduction model. Outputting the processed image mentioned above Includes, The water droplet effect reproduction model is an image processing method, wherein the model is generated by performing training using training data that associates first image data captured by the camera with second image data captured by the camera when water is adhering to the path of the imaging light incident on the camera.

8. The process involves inputting the pre-processing image into a water droplet effect reproduction model, processing the pre-processing image to resemble the image of water adhering to the path of the imaging light incident on the camera, and then obtaining a processed image from the water droplet effect reproduction model. Outputting the processed image mentioned above Make the processor execute it, The aforementioned water droplet effect reproduction model is an image processing program, which is a model generated by performing training using training data that associates first image data captured by the camera with second image data captured by the camera when water is adhering to the path of the imaging light incident on the camera.

Images