Electronic device, image processing method, and image processing program

Abstract

This electronic device comprises: a processing unit that inputs a pre-processing image to a water droplet impact reproduction model, and acquires, from the water droplet impact reproduction model, a post-processing image obtained by processing the pre-processing image so as to create an image of a case in which water is adhered on the path of imaging light incident on a camera; and an output unit that outputs the post-processing image. The water droplet impact reproduction model is a model generated by executing learning using learning data in which data of a first image captured by the camera is associated with data of a second image captured by the camera in a state in which water is adhered on the path of imaging light incident on the camera.

Description

Electronic device, image processing method, and image processing program Cross-reference to related applications

[0001] This application claims the priority of Japanese Patent Application No. 2024-168889 (filed on September 27, 2024), and the entire disclosure of the application is incorporated herein by reference for that purpose.

[0002] This disclosure relates to an electronic device, an image processing method, and an image processing program.

[0003] As described in Patent Document 1, a device that stops the recognition operation when raindrops adhere to the lens of an in-vehicle camera is known.

[0004] Japanese Patent Application Laid-Open No. 2015-26987

[0005] An electronic device according to an embodiment of this disclosure includes a processing unit and an output unit. The processing unit inputs a pre-processed image into a water droplet influence reproduction model, and acquires a post-processed image obtained by processing the pre-processed image so as to be an image when water adheres to the path of the imaging light incident on the camera from the water droplet influence reproduction model. The output unit outputs the post-processed image. The water droplet influence reproduction model is a model generated by performing learning using learning data that associates first image data captured by the camera with second image data captured by the camera in a state where water adheres to the path of the imaging light incident on the camera.

[0006] An image processing method according to an embodiment of this disclosure includes inputting a pre-processed image into a water droplet influence reproduction model, and acquiring a post-processed image obtained by processing the pre-processed image so as to be an image when water adheres to the path of the imaging light incident on the camera from the water droplet influence reproduction model, and outputting the post-processed image. The water droplet influence reproduction model is a model generated by performing learning using learning data that associates first image data captured by the camera with second image data captured by the camera in a state where water adheres to the path of the imaging light incident on the camera.

[0007] An image processing program according to one embodiment of the present disclosure causes a processor to input a pre-processing image into a water droplet effect reproduction model, process the pre-processing image to resemble an image of water adhering to the path of imaging light incident on the camera, obtain a processed image from the water droplet effect reproduction model, and output the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of imaging light incident on the camera.

[0008] This is a block diagram showing a schematic configuration example of an image processing system according to one embodiment. This is a block diagram showing an example of training data consisting of pairs of images without water droplet effects and images with water droplet effects. This is a block diagram showing an example of an image without water droplet effects, including visible and infrared images. This is a block diagram showing an example of an image with water droplet effects, including visible and infrared images. This is a block diagram showing an example of a processed image generated by processing an image before processing. This is a block diagram showing an example of the learning flow for generating a water droplet effect reproduction model and the inference flow for processing using the generated water droplet effect reproduction model. This is a flowchart showing an example of the procedure for an image processing method according to one embodiment. This is a block diagram showing an example of the learning flow for generating an object detection model and the inference flow for object detection using the generated object detection model.

[0009] In a trained model used for object detection in images, the presence of water droplets on the camera lens, etc., affects detection accuracy. To reduce false detections while maintaining detection operation, it is necessary to easily increase the variety of training images used for training the trained model, which include images taken with water droplets on the lens, etc. According to an embodiment of the electronic device, image processing method, and image processing program of this disclosure, the variety of training images can be easily increased.

[0010] Improving object detection accuracy from images is crucial. For example, increasing the accuracy of object detection from images captured by in-vehicle cameras enhances vehicle safety. Furthermore, improving object detection accuracy from images is beneficial not only for in-vehicle cameras but also in various other fields.

[0011] When generating a pre-trained model for object detection through training, the training data used is a sequence of training images containing the target object and ground truth data of the object's position in the training image. The greater the variety of training data, the higher the accuracy of object detection using the pre-trained model generated by training with that data.

[0012] In object detection from images, if water droplets adhering to the camera lens or other parts used to capture the image affect the image, the accuracy of object detection may decrease. To maintain or improve detection accuracy even when water droplets are present in the image, it is advisable to perform training using training images that show the effects of water droplets. The greater the variety of water droplet effects in the training images, the better the trained model generated by training with those training images will be able to maintain or improve the accuracy of object detection from images showing the effects of water droplets.

[0013] When generating training images by actually taking photographs, it is practically difficult to comprehensively reproduce and capture all patterns of water droplet adhesion. Therefore, it is difficult to increase the variations in the effects of water droplets as training images. There is a need for a way to easily generate training images with a wider variety of water droplet effects.

[0014] Hereinafter, an example of an embodiment of an electronic device 10 (see Figure 1), an image processing method, and an image processing program that can easily increase the variety of training images will be described.

[0015] (Example of the configuration of the image processing system 1) As shown in Figure 1, the image processing system 1 according to one embodiment of the present disclosure comprises an electronic device 10 and a camera 20.

[0016] <Electronic device 10> The electronic device 10 comprises an acquisition unit 12, a processing unit 14, an output unit 16, a learning unit 18, and a storage unit 22.

[0017] The acquisition unit 12 acquires image data from the camera 20. The acquisition unit 12 may acquire various other data or information. The acquisition unit 12 may be equipped with a communication interface for wired or wireless communication with the camera 20 or other devices. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0018] The acquisition unit 12 may include an input device that accepts input from the user. The input device may include, for example, a keyboard or physical keys, or a pointing device such as a touch panel, touch sensor, or mouse. The input device is not limited to these examples and may include various other devices. The acquisition unit 12 may be configured to communicate with an external input device. The electronic device 10 may include an input device as a separate input unit from the acquisition unit 12.

[0019] The processing unit 14 processes the image data acquired by the acquisition unit 12 to generate training images. The processing unit 14 also generates training data that includes the training images.

[0020] The processing unit 14 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The general-purpose processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The dedicated circuit may include, for example, an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD). The PLD may include a Field-Programmable Gate Array (FPGA). The processing unit 14 may be implemented as a single integrated circuit (IC). The processing unit 14 may be implemented as a plurality of communicably connected integrated circuits or discrete circuits. The processing unit 14 may include either a System-on-a-Chip (SoC) or a System-in-a-Package (SiP) in which one or more processors cooperate. The processing unit 14 may be implemented based on various other known technologies.

[0021] The storage unit 22 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit 22 stores various types of information. The storage unit stores programs executed by a general-purpose processor or the like that functions as the processing unit 14, or various types of data or information. The storage unit 22 may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the processing unit 14. At least a part of the storage unit 22 may be configured integrally with the processing unit 14.

[0022] The output unit 16 outputs the training image or training data generated by the processing unit 14 to the training unit 18 or an external training device. The output unit 16 may also output various other data or information. The output unit 16 may be equipped with a communication interface for wired or wireless communication with the training unit 18, an external training device, or other devices. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0023] The output unit 16 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) or an inorganic EL display. The electronic device 10 may also include the display device as a separate display unit from the output unit 16.

[0024] The output unit 16 is not limited to these examples and may include various other output devices such as an audio output unit that outputs sound, or a vibration output unit that outputs vibration.

[0025] The learning unit 18 generates a trained model by performing training using training images or training data. The learning unit 18 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The learning unit 18 may be configured identically to or similarly to the processing unit 14. The learning unit 18 may be included in the processing unit 14. If the learning unit 18 is included in the processing unit 14, the processing unit 14 may implement the functions of the learning unit 18.

[0026] The electronic device 10 relating to this disclosure may be various types of devices. For example, the electronic device 10 according to one embodiment may be a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC (Personal Computer), computer, or server. The electronic device 10 may be configured in a cloud computing environment or in an on-premise environment. The functions of each part of the electronic device 10 may be realized on separate servers connected via a wired or wireless network. The electronic device 10 may be mounted on a mobile device such as a vehicle including a bus or truck, a drone, an airplane, a bicycle, a motorcycle, a tractor, a delivery robot, or a security robot.

[0027] <Camera 20> Camera 20 includes an image sensor. The image sensor may be configured to include, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor may be configured to capture at least one of infrared light or visible light. Camera 20 may include both an infrared image sensor and a visible light image sensor. Infrared light may include, for example, light in the wavelength range from 780 nm to 1000 nm. Visible light may include, for example, light in the wavelength range from 380 nm to 780 nm.

[0028] Camera 20 includes an optical system that images incoming light onto an image sensor. The optical system may include lenses or mirrors. The image sensor can capture light that has been imaged within the imaging range by the optical system. Of the light arriving towards camera 20, the amount of light that is imaged within the imaging range by the optical system is determined according to the configuration of the optical system. The light that is imaged within the imaging range by the optical system and captured by the image sensor is also called imaging light.

[0029] The camera 20 may have a cover on the outside of the optical system that transmits imaging light. The cover may be included in the optical system as the outermost element of the optical system.

[0030] The image generated by the camera 20 when it captures the imaging light with its image sensor is affected by water droplets adhering to the path through which the imaging light passes. For example, water droplets adhering to the part of the camera 20's optical system or cover through which the imaging light passes will cause refraction, reflection, or scattering of the imaging light. Also, if the camera 20 is mounted inside the vehicle's windshield, water droplets adhering to the part of the windshield through which the imaging light passes will cause refraction, reflection, or scattering of the imaging light. Due to the refraction, reflection, or scattering of the imaging light, the image captured when water droplets are present will differ from the image captured when water droplets are absent.

[0031] The number of cameras 20 is not limited to one; there may be two or more. The cameras 20 may be mounted on a moving object such as a vehicle, or on a device such as a roadside unit used in a traffic system. The cameras 20 may also be included in the electronic device 10.

[0032] (Example of operation of image processing system 1) In the image processing system 1 according to this disclosure, the electronic device 10 generates training images used for training to generate an object detection model. As described above, if an image to be detected is affected by water droplets attached to the lens of the camera 20 that captures the image, the water droplets may cause the object to not be detected or to be falsely detected, reducing the accuracy of object detection. In other words, the electronic device 10 generates training images with increased variations in the effect of water droplets so that the object detection model can maintain or improve the accuracy of object detection from images affected by water droplets.

[0033] In this disclosure, the acquisition unit 12 of the electronic device 10 acquires a pre-processing image that will be the basis for the training image. The pre-processing image may be image data captured by the camera 20. The pre-processing image may also be image data generated by an external device of the image processing system 1. The pre-processing image may include an image taken when there are no water droplets attached to the lens or cover of the camera 20 or the windshield of a vehicle, etc. The pre-processing image may include an image taken when there are almost no water droplets attached. The state of having almost no water droplets attached may include a state where there are few attached water droplets or where the attached water droplets are small.

[0034] <Processing> The processing unit 14 of the electronic device 10 inputs the pre-processing image to the water droplet effect reproduction model and acquires the image output from the water droplet effect reproduction model as the processed image. The water droplet effect reproduction model accepts the pre-processing image as input, processes the input pre-processing image to reproduce the effect of additional water droplets that were not present in the path of the imaging light when the pre-processing image was captured, and outputs the processed image.

[0035] In other words, the processed image is an image obtained by processing the original image to reproduce the effect of water droplets that have additionally adhered to the path of the imaging light incident on the camera 20. The processed image is used as a training image for generating a water droplet effect reproduction model. The processed image may include images that have been changed by the effect of water droplets compared to the original original image. The processed image may include images in which the number of areas affected by water droplets has increased, or images in which the area affected by water droplets has expanded, compared to the original image.

[0036] The water droplet effect reproduction model may be configured to accept input for the position of the light source in the pre-processed image. The light source may include a self-illuminating object such as a light fixture. The light source may include an object that reflects light emitted from other objects toward the camera 20. The light source may include an object that directs light stronger than that of surrounding objects toward the camera 20. The light source position may include, for example, the position where the light fixture is installed.

[0037] The water droplet effect reproduction model relating to this disclosure may be a CycleGAN model, which is a type of GAN (Generative Adversarial Network). A CycleGAN model is a model that learns the domain relationship between two image datasets and converts image data belonging to one image dataset to image data belonging to the other image dataset. A domain is a concept that distinguishes the fields or regions of images belonging to an image dataset. Specifically, the training of a CycleGAN model may be performed such that the difference between the image before and after conversion, i.e., the periodic consistency loss, is small when an image belonging to the first domain is converted to an image belonging to the second domain and then converted back to an image belonging to the first domain. A CycleGAN model can be trained using a periodic consistency loss without requiring paired data. In other words, a CycleGAN model does not need to perform a one-to-one mapping between source and target regions and can convert from one region to another.

[0038] In the water droplet effect reproduction model relating to this disclosure, the first domain corresponds to an image without the effect of water droplets. The second domain corresponds to an image with the effect of water droplets. The water droplet effect reproduction model is a model generated by performing training so that the difference between the image before and after transformation, i.e., the periodic consistency loss, is small when an image without the effect of water droplets is transformed into an image with the effect of water droplets and then transformed back into an image without the effect of water droplets.

[0039] The processing unit 14 acquires training data to generate a cycleGAN model as a water droplet effect reproduction model. The training data for the cycleGAN model includes an image dataset without water droplet effects and an image dataset with water droplet effects. Each of the image datasets, the one without water droplet effects and the one with water droplet effects, contains at least one image data. The image data included in the image dataset without water droplet effects is also called a water droplet-free image. The image data included in the image dataset with water droplet effects is also called a water droplet-affected image. The processing unit 14 pairs the water droplet-free images and water droplet-affected images and performs training to generate a water droplet effect reproduction model.

[0040] The image dataset affected by water droplets may include image data that does not match any other condition besides the presence or absence of water droplets compared to all image data in the image dataset not affected by water droplets. The processing unit 14 may perform training by pairing images that do not match any other condition besides the presence or absence of water droplets. In this case, the training data does not need to include combinations of images with and without water droplet effects that match any other condition besides the presence or absence of water droplets. If images that match any other condition besides the presence or absence of water droplets are required as training data, the work required to prepare such images may become excessive. By eliminating the need to prepare images that match any other condition besides the presence or absence of water droplets, the preparation of training data becomes easier. As a result, training to generate a water droplet effect reproduction model can be performed more simply.

[0041] The training data may include combinations of images with and without water droplet effects, where all other conditions are the same except for the presence or absence of water droplet effects.

[0042] The processing unit 14 may perform learning by pairing each single image without water droplet influence and two images with water droplet influence included in the image dataset illustrated in FIG. 2. In this case, the processing unit 14 can perform learning using two types of pairs: an image without water droplet influence and an image with water droplet influence. When the learning data includes two images without water droplet influence and two images with water droplet influence, the processing unit 14 can perform learning using four pairs of 2×2. The number of images without water droplet influence included in the learning data may be three or more. The number of images with water droplet influence included in the learning data may be three or more.

[0043] The image with water droplet influence illustrated in FIG. 2 is different from the image without water droplet influence in that the water droplet influence region 34 appears. The image without water droplet influence may be an image without any influence of water droplets, or may be an image with less influence of water droplets than the image with water droplet influence. The magnitude of the influence of water droplets may be represented as the size of the water droplet influence region 34 or the magnitude of the change in luminance of the water droplet influence region 34. The magnitude of the influence of water droplets may also be represented as the degree of decrease in contrast in the water droplet influence region 34.

[0044] In addition, the image with water droplet influence illustrated in FIG. 2 is different from the image without water droplet influence with respect to the person shown. That is, the processing unit 14 may perform learning by pairing an image without water droplet influence and an image with water droplet influence in which conditions other than the presence or absence of the influence of water droplets do not match.

[0045] The water droplet influence region 34 tends to appear widely or with high luminance around the light source. Therefore, by performing learning considering the light source position, the learning is efficiently performed. Specifically, the processing unit 14 may perform learning using, as learning data, data associating the light source position shown in the image without water droplet influence with the image without water droplet influence, and data associating the light source position shown in the image with water droplet influence with the image with water droplet influence. A light source 32, which is a type of light source, is shown in each of the image without water droplet influence and the image with water droplet influence illustrated in FIG. 2. The processing unit 14 may perform learning using, as learning data, data associating information on the position of the light source 32 with each of the image without water droplet influence and the image with water droplet influence.

[0046] In the image without the influence of water droplets and the image with the influence of water droplets illustrated in FIG. 2, the light source 32 appears at the same position. The processing unit 14 may perform learning by pairing images in which the light source positions match as in the example of FIG. 2. The processing unit 14 may perform learning by pairing images in which the difference in the light source positions falls within a predetermined range. That is, the processing unit 14 may perform learning by pairing images with close light source positions. By performing learning by pairing images with close light source positions, learning of the appearance of the light source affected by water droplets is efficiently performed. Conversely, when learning is performed by pairing images in which the difference in the light source positions is outside the predetermined range, that is, images with far light source positions, the contribution to the water droplet influence reproduction model may be small because the appearance of the influence of water droplets is significantly different from the actual situation.

[0047] The image dataset may be divided into an image dataset including visible images and an image dataset including infrared images. The processing unit 14 may use, as learning data, an image dataset including visible images without the influence of water droplets and an image dataset including visible images with the influence of water droplets. The processing unit 14 may use, as learning data, an image dataset including infrared images without the influence of water droplets and an image dataset including infrared images with the influence of water droplets. That is, the processing unit 14 may perform learning by pairing an image without the influence of water droplets and an image with the influence of water droplets among visible images or among infrared images.

[0048] The learning data may include image data in which visible images and infrared images captured under the same conditions are set, as illustrated in FIGS. 3A and 3B. An image without the influence of water droplets in which a visible image and an infrared image are set is illustrated in FIG. 3A. An image with the influence of water droplets in which a visible image and an infrared image are set is illustrated in FIG. 3B.

[0049] The set of visible and infrared images with water droplet effects, illustrated in Figure 3B, differs from the set of visible and infrared images without water droplet effects, illustrated in Figure 3A, in that the water droplet-affected region 34 is visible. Additionally, the image with water droplet effects, illustrated in Figure 3B, differs from the image without water droplet effects, illustrated in Figure 3A, in terms of the person depicted. Even when the processing unit 14 uses image data consisting of a set of visible and infrared images as training data, it may perform training using pairs of image data where conditions other than the presence or absence of water droplet effects do not match.

[0050] The processing unit 14 may generate a processed image that reproduces the effect of water droplet adhesion in the path of the imaging light in the pre-processed image by performing inference using the water droplet effect reproduction model generated as described above. Specifically, the processing unit 14 inputs the pre-processed image into the water droplet effect reproduction model generated by learning. The processing unit 14 may also input the light source position into the water droplet effect reproduction model.

[0051] The processing unit 14 acquires the image output from the water droplet effect reproduction model as a processed image. In other words, the processing unit 14 generates a processed image from the output image of the water droplet effect reproduction model. In this way, the processing unit 14 can generate a processed image using the water droplet effect reproduction model.

[0052] As illustrated in Figure 4, the processing unit 14 may input the pre-processing image into a water droplet effect reproduction model and acquire the output image from the water droplet effect reproduction model as the post-processing image. In the post-processing image of Figure 4, a water droplet effect region 34 is added that reproduces the effect of water droplet adhesion. When inputting the pre-processing image into the water droplet effect reproduction model, the processing unit 14 may also input the position of the light source 32 into the water droplet effect reproduction model.

[0053] The processing described above may be divided into a learning phase in which a water droplet effect reproduction model is generated, and an inference phase in which a processed image is generated using the generated water droplet effect reproduction model, as illustrated in Figure 5. The processing unit 14 may perform learning in the learning phase using an image without water droplet effect, an image with water droplet effect, and the light source position as learning data. In the inference phase, the processing unit 14 may input the unprocessed image and the light source position into the water droplet effect reproduction model and acquire the image output from the water droplet effect reproduction model as the processed image.

[0054] The processing unit 14 may obtain a water droplet effect reproduction model generated by another device, rather than generating the water droplet effect reproduction model itself. In other words, the processing unit 14 may execute only the inference phase without executing the learning phase.

[0055] In the block diagram illustrated in Figure 5, training data including the light source position is used for training, but training data that does not include the light source position may also be used. When training data that does not include the light source position is used for training, the water droplet effect reproduction model may be configured to output a processed image from the unprocessed image even without accepting input for the light source position. Even if the water droplet effect reproduction model does not accept input for the light source position, it can estimate the light source position shown in the unprocessed image and generate a processed image that reflects the effect of water droplets.

[0056] On the other hand, if the water droplet effect reproduction model accepts input for the light source position, improving the accuracy of the light source position can improve the accuracy of reproducing the effect of water droplet adhesion in the processed image.

[0057] <<Example of Processing Procedure>> The processing unit 14 may generate a processed image that reproduces the effect of water droplets in the pre-processing image by executing an image processing method that includes the steps of the flowchart illustrated in Figure 6. The image processing method may be implemented as an image processing program to be executed by the processing unit 14. The image processing program may be stored on a non-temporary computer-readable medium.

[0058] The processing unit 14 acquires the pre-processing image (step S1). The processing unit 14 may acquire the pre-processing image from the camera 20 or an external device. The processing unit 14 may store the pre-processing image acquired from the camera 20 or the like in the storage unit and acquire the pre-processing image from the storage unit. The processing unit 14 inputs the pre-processing image into the water droplet effect reproduction model (step S2). The processing unit 14 acquires the image output from the water droplet effect reproduction model as the processed image (step S3). The processing unit 14 outputs the processed image from the output unit 16 (step S4). After executing the procedure in step S4, the processing unit 14 finishes executing the procedure in the flowchart of Figure 6.

[0059] <<Summary of Processing>> As described above, the electronic device 10 of this disclosure can easily generate an image that reproduces the effect of water droplets from the image before processing. As a result, the variations of images that reproduce the effect of water droplets can be easily increased.

[0060] <Object Detection Processing> As described above, the processing unit 14 of the electronic device 10 according to this disclosure can generate processed images that reproduce the effect of water droplets adhering to the path of imaging light captured by the camera 20 in various variations, as training images used to generate an object detection model. The electronic device 10 may perform learning to generate an object detection model, as illustrated in Figure 7, and perform object detection inference using the generated object detection model. The learning to generate the object detection model may be performed by the learning unit 18 of the electronic device 10, or by a learning device different from the electronic device 10. The object detection inference using the object detection model may be performed by a detection device different from the electronic device 10 or the learning device. The inference using the object detection model may be performed by the device that generated the object detection model, i.e., the electronic device 10 or the learning device.

[0061] <<Learning the Object Detection Model>> The following describes an example of how the learning unit 18 of the electronic device 10 performs learning of an object detection model. The learning unit 18 acquires the processed image generated by the processing unit 14 from the output unit 16 of the electronic device 10 as a learning image. The learning unit 18 generates an object detection model by performing learning using learning data in which the correct information of the object positions to be detected is associated with the learning image. The correct information of the object positions is the position information of an object that has been confirmed to be present in the learning image. The correct information of the object positions may include information on whether the object to be detected is present in the learning image. The learning unit 18 may acquire the correct information of the object positions by input from the user of the electronic device 10.

[0062] When the learning unit 18 acquires a learning image from the output unit 16, it may acquire the learning image as learning data in which the correct information of the object's position is associated with the learning image. In this case, when the processing unit 14 generates a processed image, it may generate information that associates the correct information of the object's position in the processed image with the processed image. The output unit 16 may output the information that associates the correct information of the object's position with the processed image to the learning unit 18.

[0063] In both the pre-processing image and the post-processing image, the positions of the objects in each image are the same. Therefore, the processing unit 14 may acquire information on the positions of the objects in the pre-processing image and generate information that associates this information with the post-processing image as the correct information for the positions of the objects in the post-processing image.

[0064] The examples of object detection model training described above may also be applied to learning devices.

[0065] <<Inference using an object detection model>> The following describes an example of operation in which a detection device different from the electronic device 10 or the learning device performs object detection inference using an object detection model. The detection device acquires the object detection model from the output unit 16 of the electronic device 10 when the learning unit 18 of the electronic device 10 generates the object detection model, and acquires the object detection model from the learning device when the learning device generates the object detection model.

[0066] The detection device inputs a detection image into the object detection model. The detection image is the image targeted for object detection. The detection device obtains the object detection result output from the object detection model. In this way, the detection device can obtain object detection results using the object detection model.

[0067] <<Summary of Object Detection Process>> As described above, the object detection model is generated by performing training using processed images as training images that reproduce the effects of water droplets adhering to the path of imaging light captured by the camera 20 in various variations. By performing object detection inference using the object detection model generated in this way, it is expected that the accuracy of object detection from the detected image will be maintained or improved, even when the detected image is affected by water droplets adhering to the lens, etc.

[0068] (Summary) As described above, the electronic device 10, image processing method, and image processing program according to this disclosure can easily generate images that reproduce the effect of water droplets adhering to the path of imaging light. As a result, the variety of images that reproduce the effect of water droplet adhesion can be easily increased.

[0069] Furthermore, an object detection model is generated by performing training using diverse variations of training images that reproduce the effects of water droplet adhesion. By performing object detection inference using this generated object detection model, it is expected that the accuracy of object detection from the detected image will be maintained or improved, even when the detected image is affected by water droplet adhesion on lenses, etc.

[0070] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0071] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. These are also to be understood as being included within the scope of this disclosure.

[0072] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0073] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0074] Vehicles relating to this disclosure may include, for example, automobiles, industrial vehicles, railway vehicles, residential vehicles, or fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture or construction. Industrial vehicles may include, for example, forklifts or golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. Vehicles of the same nature may be included in multiple classifications.

[0075] While embodiments of the image processing method using the image processing system 1 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0076] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. Moreover, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0077] In one embodiment, (1) the electronic device includes a processing unit that inputs a pre-processing image to a water droplet effect reproduction model and obtains a processed image from the water droplet effect reproduction model by processing the pre-processing image so that it becomes an image of what would occur if water were adhering to the path of the imaging light incident on the camera; and an output unit that outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using learning data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

[0078] (2) In the electronic device described in (1) above, the water droplet effect reproduction model may be a model generated by performing learning using data in which the position of the first light source when the first image data is captured is associated with the first image data, and the position of the second light source when the second image data is captured is associated with the second image data.

[0079] (3) In the electronic device described in (2) above, the water droplet effect reproduction model may be a model generated by performing learning using data in which the difference between the position of the first light source and the position of the second light source is within a predetermined range.

[0080] (4) In the electronic device described in any one of (1) to (3) above, the camera may be configured to capture infrared light.

[0081] (5) In the electronic device described in any one of (1) to (4) above, the camera may be configured to capture visible light.

[0082] (6) The electronic device described in any one of (1) to (5) above may further include a learning unit that performs learning to generate an object detection model for detecting objects in an image. The output unit may output the processed image to the learning unit as learning data for generating the object detection model.

[0083] In one embodiment, (7) the image processing method includes inputting a pre-processing image to a water droplet effect reproduction model, obtaining a processed image from the water droplet effect reproduction model obtained by processing the pre-processing image so that it becomes an image of what would occur if water were attached to the path of the imaging light incident on the camera, and outputting the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water attached to the path of the imaging light incident on the camera.

[0084] In one embodiment, (8) the image processing program inputs a pre-processing image to a water droplet effect reproduction model, processes the pre-processing image to make it appear as if water is adhering to the path of the imaging light incident on the camera, obtains a processed image from the water droplet effect reproduction model, and outputs the processed image. The water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

[0085] 1 Image processing system 10 Electronic equipment (12: acquisition unit, 14: processing unit, 16: output unit, 18: learning unit, 22: memory unit) 20 Camera 32 Light source 34 Water droplet affected area

Claims

1. An electronic device comprising: a processing unit that inputs a pre-processing image into a water droplet effect reproduction model, processes the pre-processing image to make it appear as if water is adhering to the path of the imaging light incident on the camera, and obtains a processed image from the water droplet effect reproduction model; and an output unit that outputs the processed image, wherein the water droplet effect reproduction model is a model generated by performing learning using learning data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of the imaging light incident on the camera.

2. The electronic device according to claim 1, wherein the water droplet effect reproduction model is a model generated by performing learning using data in which the position of the first light source when the first image data is captured is associated with the first image data, and the position of the second light source when the second image data is captured is associated with the second image data.

3. The electronic device according to claim 2, wherein the water droplet effect reproduction model is a model generated by performing learning using data in which the difference between the position of the first light source and the position of the second light source is within a predetermined range.

4. The electronic device according to any one of claims 1 to 3, wherein the camera is configured to capture infrared light.

5. The electronic device according to any one of claims 1 to 4, wherein the camera is configured to capture visible light.

6. The electronic device according to any one of claims 1 to 5, further comprising a learning unit that performs training for generating an object detection model for detecting objects in an image, wherein the output unit outputs the processed image to the learning unit as training data for generating the object detection model.

7. An image processing method comprising: inputting a pre-processing image into a water droplet effect reproduction model; obtaining a processed image from the water droplet effect reproduction model by processing the pre-processing image so that it resembles an image of water adhering to the path of imaging light incident on the camera; and outputting the processed image, wherein the water droplet effect reproduction model is a model generated by performing learning using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of imaging light incident on the camera.

8. An image processing program that causes a processor to input an unprocessed image into a water droplet effect reproduction model, process the unprocessed image to resemble an image of water adhering to the path of imaging light incident on the camera, obtain a processed image from the water droplet effect reproduction model, and output the processed image, wherein the water droplet effect reproduction model is a model generated by performing training using training data that associates a first image data captured by the camera with a second image data captured by the camera with water adhering to the path of imaging light incident on the camera.

Images