Electronic device, image processing method, and image processing program

Abstract

This invention provides an electronic device, an image processing method, and an image processing program that can easily increase the variety of training images. [Solution] The electronic device 10 includes a processing unit 14 that processes a pre-processing image based on processing information to generate a post-processing image, and an output unit 16 that outputs the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information about at least one of the position of the light source, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source, as seen in the pre-processing image. The environmental information includes information about the environment of the space seen in the pre-processing image.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, an image processing method, and an image processing program.

Background Art

[0002] As described in Patent Document 1, there is known an apparatus that generates a learning image according to a use case of AI (Artificial Intelligence) by processing an image based on information related to a camera (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a learned model that uses object detection in an image as a use case, the detection accuracy decreases due to misdetection of virtual images such as ghosts or flares shown in the image. It is required to easily increase the variations of images showing virtual images such as ghosts or flares as learning images used for learning to generate a learned model so that virtual images are less likely to be misdetected.

[0005] An object of the present disclosure is to provide an electronic device, an image processing method, and an image processing program that can easily increase the variations of learning images.

Means for Solving the Problems

[0006] An electronic device according to one embodiment of the present disclosure comprises a processing unit that processes a pre-processing image based on processing information to generate a post-processing image, and an output unit that outputs the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image.

[0007] An image processing method according to one embodiment of the present disclosure includes an electronic device processing a pre-processing image based on processing information to generate a post-processing image, and the electronic device outputting the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image.

[0008] An image processing program according to one embodiment of the present disclosure causes an electronic device to process a pre-processing image based on processing information to generate a post-processing image, and to output the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image. [Effects of the Invention]

[0009] According to an electronic device, an image processing method, and an image processing program according to one embodiment of this disclosure, the variations of training images can be easily increased. [Brief explanation of the drawing]

[0010] [Figure 1]This block diagram shows a schematic configuration example of an image processing system according to one embodiment. [Figure 2] This figure shows an example of an image in which ghosting and flare occurred. [Figure 3] This figure shows an example of an image before processing. [Figure 4] This figure shows an example of a processed image generated by processing an image before processing. [Figure 5] This figure shows an example of a processed image generated using cGAN from an original image. [Figure 6] This block diagram shows an example of the learning process for generating a processing model and the inference process for processing using the generated processing model. [Figure 7] This flowchart shows an example of the procedure for an image processing method according to one embodiment. [Figure 8] This block diagram shows an example of the learning process for generating an object detection model, and the inference process for object detection using the generated object detection model. [Modes for carrying out the invention]

[0011] Improving object detection accuracy from images is crucial. For example, increasing the accuracy of object detection from images captured by in-vehicle cameras enhances vehicle safety. Furthermore, improving object detection accuracy from images is beneficial not only for in-vehicle cameras but also in various other fields.

[0012] When generating a pre-trained model for object detection through training, the training data used is a sequence of training images containing the target object and ground truth data of the object's position in the training image. The greater the variety of training data, the higher the accuracy of object detection using the pre-trained model generated by training with that data.

[0013] In object detection from an image, when a virtual image such as a ghost or flare appears in the image, an object corresponding to the virtual image may be detected. An object corresponding to a virtual image is an object that does not actually exist and is an object that has been erroneously detected. In order to reduce the possibility that a virtual image such as a ghost or flare is erroneously detected as an object, it is conceivable to perform learning using a training image in which a virtual image such as a ghost or flare appears. The greater the variations in the appearance of a ghost or flare or other virtual image in the training image, the lower the possibility that the virtual image such as a ghost or flare will be erroneously detected by the trained model generated by performing learning using that training image.

[0014] When actually shooting and generating a training image, it is difficult to reproduce and shoot the training image so that a virtual image such as a ghost or flare has a desired appearance. This is because the appearance of a virtual image such as a ghost or flare varies depending on factors such as the optical system of the camera or the imaging device, the factors of the light source appearing in the image, or factors such as rain or fog in the space appearing in the image. Then, it is difficult to increase the variations in the appearance of a ghost or flare or other virtual image in the training image. There is a need to simply generate a training image with increased variations in the appearance of a ghost or flare or other virtual image.

[0015] Hereinafter, in the present disclosure, an example of an embodiment of an electronic device 10 (see FIG. 1), an image processing method, and an image processing program that can easily increase the variations of training images will be described.

[0016] (Configuration example of image processing system 1) As shown in FIG. 1, an image processing system 1 according to an embodiment of the present disclosure includes an electronic device 10 and a camera 20.

[0017] <Electronic device 10> The electronic device 10 includes an acquisition unit 12, a processing unit 14, an output unit 16, a learning unit 18, and a storage unit 22.

[0018] The acquisition unit 12 acquires image data from the camera 20. The acquisition unit 12 may acquire various other data or information. The acquisition unit 12 may include a communication interface for wired or wireless communication with the camera 20 or other devices. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0019] The acquisition unit 12 may include an input device for receiving input from the user. The input device may include, for example, a keyboard or physical keys, or may include a pointing device such as a touch panel, touch sensor, or mouse. The input device is not limited to these examples and may be configured to include various other devices. The acquisition unit 12 may be configured to communicate with an external input device. The electronic device 10 may include the input device as an input unit separate from the acquisition unit 12.

[0020] The processing unit 14 processes the image data acquired by the acquisition unit 12 to generate a learning image. The processing unit 14 also generates learning data including the learning image.

[0021] [[ID=十一年]] The processing unit 14 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The general-purpose processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The dedicated circuit may include, for example, an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The processing unit 14 may be implemented as a single integrated circuit (IC). The processing unit 14 may be implemented as a plurality of communicably connected integrated circuits or discrete circuits. The processing unit 14 may include either a System-on-a-Chip (SoC) or a System-in-a-Package (SiP) in which one or more processors cooperate. The processing unit 14 may be implemented based on various other known technologies.

[0022] The storage unit 22 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit 22 stores various types of information. The storage unit 22 stores programs executed by a general-purpose processor or the like that functions as the processing unit 14, or various types of data or information. The storage unit 22 may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the processing unit 14. At least a part of the storage unit 22 may be configured integrally with the processing unit 14.

[0023] The output unit 16 outputs the training image or training data generated by the processing unit 14 to the training unit 18 or an external training device. The output unit 16 may also output various other data or information. The output unit 16 may be equipped with a communication interface for wired or wireless communication with the training unit 18, an external training device, or other devices. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0024] The output unit 16 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) or an inorganic EL display. The electronic device 10 may also include the display device as a separate display unit from the output unit 16.

[0025] The output unit 16 is not limited to these examples and may include various other output devices such as an audio output unit that outputs sound, or a vibration output unit that outputs vibration.

[0026] The learning unit 18 generates a trained model by performing training using training images or training data. The learning unit 18 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The learning unit 18 may be configured identically to or similarly to the processing unit 14. The learning unit 18 may be included in the processing unit 14. If the learning unit 18 is included in the processing unit 14, the processing unit 14 may implement the functions of the learning unit 18.

[0027] The electronic device 10 relating to this disclosure may be various types of devices. For example, the electronic device 10 according to one embodiment may be a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC (Personal Computer), computer, or server. The electronic device 10 may be configured in a cloud computing environment or in an on-premise environment. The functions of each part of the electronic device 10 may be realized on separate servers connected via a wired or wireless network. The electronic device 10 may be mounted on a mobile device such as a vehicle including a bus or truck, a drone, an airplane, a bicycle, a motorcycle, a tractor, a delivery robot, or a security robot.

[0028] <Camera 20> The camera 20 includes an image sensor. The image sensor may be configured to capture at least one of infrared light or visible light. The camera 20 may include both an infrared image sensor and a visible light image sensor. The infrared light may include, for example, light in the wavelength range from 780 nm to 1000 nm. The visible light may include, for example, light in the wavelength range from 380 nm to 780 nm.

[0029] The image sensor may include, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0030] The number of cameras 20 is not limited to one; there may be two or more. Cameras 20 may be mounted on moving objects such as vehicles, or on devices such as roadside units used in traffic systems. Cameras 20 may also be included in electronic equipment 10.

[0031] (Example of operation of image processing system 1) In the image processing system 1 relating to this disclosure, the electronic device 10 generates training images used for training to generate an object detection model. As described above, if a virtual image such as a ghost or flare is present in the image to be detected as an object, the virtual image may be mistakenly detected as an object. The electronic device 10 generates training images that increase the variations in how virtual images such as ghosts or flares appear so as to reduce the false detection of virtual images such as ghosts or flares by the object detection model. In this disclosure, images containing virtual images may include, for example, images that show images generated when light is reflected by dust or snow in the air.

[0032] In this disclosure, a ghost is a virtual image of an object where light is projected at a location where the object does not actually exist, due to diffuse reflection of light within the camera 20, for example, in the optical system or image sensor. A flare is a virtual image of an object that appears larger than its actual size, due to an excessive amount of light incident on the camera 20, or due to scattering of light emitted from a light source before it enters the camera 20.

[0033] Ghosting and flare can appear in images taken of a vehicle 30 with its headlights 32 (the light source) illuminated, positioned above the road surface 40, as shown in Figure 2, for example. Flare appears as a virtual image that makes the headlights 32 appear larger than their actual size. Ghosting appears as a virtual image that makes it seem as if a light source exists in a location where no light source such as the headlights 32 actually exists.

[0034] In this disclosure, the acquisition unit 12 of the electronic device 10 acquires a pre-processing image that will be the basis for the training image. The pre-processing image may be image data captured by the camera 20. The pre-processing image may also be image data generated by an external device of the image processing system 1. The pre-processing image may include an image in which no virtual images such as ghosts or flares are captured. The pre-processing image may include an image in which very few virtual images are captured. An image in which very few virtual images are captured may include an image in which the number of virtual images captured is less than or equal to the set threshold when a threshold for the number of virtual images is set. An image in which the number of virtual images is faint may include an image in which the light intensity of the captured virtual images is less than or equal to the set threshold when a predetermined threshold for the light intensity of the virtual images is set.

[0035] <Processing> The processing unit 14 of the electronic device 10 generates a processed image by processing the pre-processing image based on processing information. The processed image is an image obtained by processing the pre-processing image and is used as a training image. The processed image may include images in which the number of virtual images has increased or the virtual images have become denser compared to the pre-processing image from which the processing was performed. The processing information is information used to increase the variations in the appearance of virtual images such as ghosts or flares, and includes, for example, light source information, environmental information, or shooting information.

[0036] Light source information includes information about light sources visible in the pre-processed image. Light sources may include self-illuminating objects such as lights. Light sources may include objects that reflect light emitted from other objects toward the camera 20. Light sources may include objects that project light stronger than that of surrounding objects toward the camera 20.

[0037] Light source information may include information about the position of the light source. The position of the light source may include, for example, the location where the lights are installed on a vehicle. Light source information may include information about the direction of the light source. The direction of the light source may include, for example, information that identifies the direction in which the lights installed on a vehicle are emitting light. The direction in which the lights installed on a vehicle are emitting light may include, for example, the direction of travel of the vehicle, the rear, or the side. Light source information may include information about the angle of the optical axis of the light source. The optical axis of the light source is defined as the center line of the spread of light emitted from the light source. The angle of the optical axis of the light source is defined as the angle between the optical axis and a reference, such as the ground. If the light source is a vehicle headlight, the angle of the optical axis of the light source will differ depending on whether the headlight is emitting light in so-called high beam mode or so-called low beam mode. Also, the angle of the optical axis of the light source may change due to adjustments made by the vehicle driver. Light source information may include information about the pattern of light emitted by the light source. The pattern of light may include, for example, how the light emitted by the light source spreads. The way light spreads may be specified by the solid angle over which the light spreads. The way light spreads may be specified by the shape of the area over which the light spreads. If the light source is a vehicle headlight, the light pattern may include the headlight's beam pattern. The headlight's beam pattern may include the cutoff line when the headlight emits light in so-called low beam mode. Light source information may include the type of light source. The type of light source may include, but is not limited to, an LED (Light Emitting Diode) or a halogen lamp. Light source information may include the wavelength or spectrum of the light emitted by the light source. Light source information may include the intensity of the light as electromagnetic waves emitted by the light source, or the luminous intensity of the light source. Light source information may include various other information about the light source, not limited to the examples described above.

[0038] The environmental information includes information about at least one of the conditions of rain or fog in the space captured in the unprocessed image. The environmental information may include information about at least one of the rainfall amount or fog density. The environmental information may include information about objects that scatter or diffusely reflect light emitted from a light source toward the camera 20 in the unprocessed image. The environmental information may include information about at least one of the following: temperature, humidity, snowfall amount, wind direction, wind speed, photochemical oxidant concentration related to photochemical smog, or amount of yellow dust.

[0039] The imaging information includes information about the position of the camera 20, the direction of the optical axis of the camera 20, or at least one of the design data of the optical system of the camera 20. The design data of the optical system may include information about at least one of the following: the material, shape, curvature, thickness or refractive index of the lens, focal length, image brightness, field of view, resolution, size of the optical system, light source used, or image sensor.

[0040] The following describes an example of how the processing unit 14 generates a processed image from a pre-processing image based on processing information. In this example, the image shown in Figure 3 is used as the pre-processing image. The pre-processing image in Figure 3 is an image of the space above the road surface 40 where a vehicle 30, which has a headlight 32 as a light source, is located.

[0041] <<Rule-based processing>> The processing unit 14 may, as an example, process the pre-processing image using rule-based processing such as masking to generate a post-processing image. Masking may include a process of generating a mask corresponding to a virtual image such as ghosting or flare by performing a simulation based on processing conditions, and superimposing the generated mask onto the pre-processing image. An example of a post-processing image generated by rule-based processing from the pre-processing image in Figure 3 is shown in Figure 4. Multiple post-processing images can be created from a single pre-processing image, reproducing variations of virtual images such as ghosting or flare caused by the scattering or diffuse reflection of light emitted by the headlights 32 in the pre-processing image.

[0042] The processing unit 14 acquires the pre-processing image. The processing unit 14 acquires basic information of the pre-processing image. The basic information of the pre-processing image includes information about the conditions under which the pre-processing image was taken. The basic information may include information about the position and direction of the camera 20 when the pre-processing image was taken with the camera 20. The basic information may include information about the position of the light source or the angle of the optical axis of the light source that is visible in the pre-processing image. The basic information may include information about the date and time the pre-processing image was taken. The basic information may include information about the weather when the pre-processing image was taken. The weather information may include information about at least one of the following conditions: rain or fog. The weather information may include at least one of the following: temperature, humidity, snowfall amount, wind direction, or wind speed.

[0043] The processing unit 14 may acquire the basic information of the pre-processing image as information input by the user of the electronic device 10. The processing unit 14 may also estimate the basic information of the pre-processing image from the content of the pre-processing image.

[0044] The processing unit 14 sets the processing information. The processing unit 14 may set at least one of the light source information or environmental information included in the processing information. The processing unit 14 may further set the shooting information included in the processing information.

[0045] The processing information may include information that has been modified for application to processing the pre-processed image, with respect to at least one item included in the basic information of the pre-processed image. The light source information included in the processing information may include information that has been changed in the position or angle of the optical axis of the light source applied to the processing. The environmental information included in the processing information may include information that has been changed in the shooting date and time or the weather at the time of shooting applied to the processing. The shooting information included in the processing information may include information that has been changed in the position or direction of the camera 20 applied to the processing.

[0046] The processing information may include information that represents the changes applied to the processing of the pre-processed image as a difference from the basic information of the pre-processed image, for at least one item included in the basic information of the pre-processed image. The light source information may include information that represents the changes in the position or optical axis angle of the light source applied to the processing as a difference from the position or optical axis angle of the light source in the basic information of the pre-processed image. The environmental information may include information that represents the changes in the shooting date and time or weather at the time of shooting applied to the processing as a difference from the shooting date and time or weather at the time of shooting in the basic information of the pre-processed image. The shooting information may include information that represents the changes in the position or direction of the camera 20 applied to the processing as a difference from the position or direction of the camera 20 in the basic information of the pre-processed image.

[0047] The processing unit 14 may acquire information entered by the user of the electronic device 10 as processing information and set the acquired processing information. The processing unit 14 may also set the processing information to change the basic information of the pre-processing image within a predetermined range.

[0048] The processing unit 14 may perform a simulation of light rays incident on the image sensor of the camera 20 based on the basic information and processing information. Based on the simulation results, the processing unit 14 may perform processing to add a virtual image such as a ghost or flare to the pre-processed image and generate a post-processed image. Based on the simulation results, the processing unit 14 may generate a mask image corresponding to the virtual image such as a ghost or flare that reflects the effects of the changes made by the processing information, and perform processing to add a virtual image to the pre-processed image by superimposing the mask image onto the pre-processed image.

[0049] For example, if the processing unit 14 sets processing information to change at least one of the rain or fog conditions in the environmental information, it may perform processing to add a virtual image such as a ghost or flare caused by the scattering of light by at least one of the rain or fog. For example, if the processing unit 14 sets processing information to change the position of the light source in the light source information, it may perform processing to change and add the position of the ghost caused by the diffuse reflection of light inside the camera 20 according to the position of the light source.

[0050] <<Processing using a processing model>> As another example, the processing unit 14 may process the pre-processing image using a processing model to generate a post-processing image. The processing model may include a conditional GAN ​​(Generative Adversarial Network) model, which is a type of GAN. A conditional GAN ​​differs from a general GAN ​​in that it adds a condition between the generator and the discriminator. The generator outputs data that has been conditionally generated based on the input information when random noise and a condition are input. The discriminator outputs a result that distinguishes between real data and generated data based on the input information when a condition and data are input. Figure 5 shows an example of a post-processing image generated from the pre-processing image in Figure 3 using a conditional GAN ​​model. By using a GAN model, it is possible to generate a post-processing image that includes objects that are not present in the pre-processing image. Also, regardless of the resolution of the pre-processing image, it is possible to generate a post-processing image with a higher resolution than the pre-processing image.

[0051] The processing unit 14 may perform learning to generate a machining model, as illustrated in Figure 6, and perform machining inference using the generated machining model. The processing unit 14 may also obtain a machining model generated by another device instead of generating the machining model itself.

[0052] <<<Training the processed model>>> The processing unit 14 acquires training data to be used for training in order to generate a processing model. The training data is data that associates images without virtual images with images with virtual images and correct processing information. The correct processing information represents the correct values ​​of processing information to be applied when processing an image without virtual images into an image with virtual images.

[0053] The training data may include data that associates an image containing a virtual image with an image in which the virtual image has increased or become more intense, and the correct values ​​of processing information applied to the processing that increases or intensifies the virtual image.

[0054] The correct processing information may be generated based on various conditions when images without and with virtual images were taken, respectively.

[0055] The processing unit 14 uses the aforementioned training data to train a conditional GAN ​​model and generate a modified model.

[0056] <<<Inference using a fabricated model>>> The processing unit 14 processes the unprocessed image and generates a processed image by performing inference using a processing model. Specifically, the processing unit 14 inputs the unprocessed image into the processing model generated by training. The processing unit 14 may also input processing information into the processing model. The processing information may be included in the processing model. In other words, the processing model may be generated to apply predetermined processing information.

[0057] The processing unit 14 obtains the processed image as the output result after processing the pre-processing image output from the processing model. In other words, the processing unit 14 generates the processed image from the output image of the processing model. In this way, the processing unit 14 can generate a processed image using the processing model.

[0058] <<Processing method combining rule-based and conditional GANs>> The processing unit 14 may generate the processed image using a conditional GAN ​​that has been trained by pairing an image with variations of virtual images, obtained by processing the pre-processing image using rule-based processing, with an image of the virtual image variations that was actually captured. In other words, the processing unit 14 may use a model in which conditions have been added to cycleGAN.

[0059] The training data used to generate a cycleGAN model with added conditions may include data that associates the original image with the processed image processed by rule-based processing and the correct processing information. In this case, the correct processing information may be the processing information applied when processing the original image with rule-based processing.

[0060] <<Example of processing procedure>> The processing unit 14 may process the pre-processing image based on processing conditions to generate a post-processing image by executing an image processing method that includes the steps of the flowchart illustrated in Figure 7. The image processing method may be implemented as an image processing program to be executed by the processing unit 14. The image processing program may be stored on a non-temporary computer-readable medium.

[0061] The processing unit 14 acquires a pre-processing image (step S1). The processing unit 14 may acquire a pre-processing image from the camera 20 or an external device. The processing unit 14 may store the pre-processing image acquired from the camera 20 or the like in the storage unit and acquire the pre-processing image from the storage unit. The processing unit 14 sets the processing information (step S2). The processing unit 14 may accept input of processing information from the user of the electronic device 10 and set the processing information. The processing unit 14 processes the pre-processing image based on the processing information (step S3). The processing unit 14 outputs the processed image generated by processing the pre-processing image from the output unit 16 (step S4). After executing the procedure in step S4, the processing unit 14 finishes executing the procedure in the flowchart of Figure 7.

[0062] <<Summary of Processing>> As described above, the electronic device 10 according to this disclosure can easily generate an image that reproduces a desired appearance of a virtual image by processing the pre-processing image based on processing information. As a result, it is possible to easily increase the variations of images that contain virtual images such as ghosts or flares.

[0063] <Object detection processing> As described above, the processing unit 14 of the electronic device 10 according to this disclosure can generate a processed image to which variations of virtual images such as ghosts or flares are added as training images used to generate an object detection model. The electronic device 10 may perform training to generate an object detection model, as illustrated in Figure 8, and perform object detection inference using the generated object detection model. Training to generate an object detection model may be performed by the training unit 18 of the electronic device 10, or by a training device different from the electronic device 10. Object detection inference using the object detection model may be performed by a detection device different from the electronic device 10 or the training device. Inference using the object detection model may be performed by the device that generated the object detection model, i.e., the electronic device 10 or the training device.

[0064] <<Training an object detection model>> The following describes an example of how the learning unit 18 of the electronic device 10 performs training on an object detection model. The learning unit 18 acquires the processed image generated by the processing unit 14 from the output unit 16 of the electronic device 10 as a training image. The learning unit 18 generates an object detection model by performing training using training data in which the training image is associated with the correct information of the object positions to be detected. The correct information of the object positions is the position information of an object that has been confirmed to be present in the training image. The correct information of the object positions may include information on whether the object to be detected is present in the training image. The learning unit 18 may acquire the correct information of the object positions through input from the user of the electronic device 10.

[0065] When the learning unit 18 acquires a learning image from the output unit 16, it may acquire the learning image as learning data in which the correct information of object positions is associated with the learning image. In this case, when the processing unit 14 generates a processed image, it may generate information that associates the correct information of object positions in the processed image with the processed image. The output unit 16 may output the information that associates the correct information of object positions with the processed image to the learning unit 18.

[0066] The object positions in the pre-processed image and the processed image are the same. Therefore, the processing unit 14 may acquire information on the object positions in the pre-processed image and generate information that associates this information with the processed image as the correct object position information in the processed image.

[0067] The examples of object detection model training described above may also be applied to learning devices.

[0068] <<Inference using an object detection model>> The following describes an example of operation in which a detection device different from the electronic device 10 or the learning device performs object detection inference using an object detection model. The detection device obtains the object detection model from the output unit 16 of the electronic device 10 when the learning unit 18 of the electronic device 10 generates the object detection model, and obtains the object detection model from the learning device when the learning device generates the object detection model.

[0069] The detection device inputs a detection image into the object detection model. The detection image is the image targeted for object detection. The detection device obtains the object detection result output from the object detection model. In this way, the detection device can obtain object detection results using the object detection model.

[0070] <<Summary of Object Detection Process>> As described above, object detection models are generated by training them using a wide variety of training images that include virtual images such as ghosts or flares. By performing object detection inference using the object detection model generated in this way, it is expected that the frequency of false detections of virtual images such as ghosts or flares in the detection images as objects will decrease.

[0071] (summary) As described above, the electronic device 10, image processing method, and image processing program relating to this disclosure make it possible to easily generate images that reproduce a desired type of virtual image. As a result, the variations in images containing virtual images such as ghosts or flares can be easily increased.

[0072] Furthermore, an object detection model is generated by training it using many variations of training images that contain virtual images such as ghosts or flares. By performing object detection inference using this generated object detection model, it is expected that the frequency of false detections of virtual images such as ghosts or flares in the detection image as objects will decrease. As a result, the accuracy of object detection will improve.

[0073] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0074] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. These are also to be understood as being included within the scope of this disclosure.

[0075] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0076] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0077] Vehicles relating to this disclosure may include, for example, automobiles, industrial vehicles, railway vehicles, residential vehicles, or fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture or construction. Industrial vehicles may include, for example, forklifts or golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. Vehicles of the same nature may be included in multiple classifications.

[0078] While embodiments of the image processing method using the image processing system 1 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0079] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. In addition, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0080] In one embodiment, (1) the electronic device comprises a processing unit that processes a pre-processing image based on processing information to generate a post-processing image, and an output unit that outputs the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image.

[0081] (2) In the electronic device described in (1) above, the environmental information may include information relating to at least one of the conditions of rain or fog in the space shown in the pre-processed image.

[0082] (3) In the electronic device described in (1) or (2) above, the processing information may further include photographic information. The photographic information may include information relating to at least one of the position of the camera that took the pre-processing image, the direction of the optical axis of the camera, or the design data of the optical system of the camera.

[0083] (4) In the electronic device described in any one of (1) to (3) above, the processing unit may generate an image as the processed image by adding at least one of flare or ghost to the pre-processing image.

[0084] (5) In the electronic device described in any one of (1) to (4) above, the processing unit may input the pre-processing image to a processing model that processes and outputs an input image based on the processing information, and generate the output image of the processing model as the post-processing image.

[0085] (6) In the electronic device described in any one of (1) to (5) above, the output unit may output the processed image as training data for generating an object detection model for detecting objects in the image to a learning device that generates the object detection model.

[0086] (7) The electronic device described in any one of (1) to (6) above may further include a learning unit that performs learning to generate an object detection model for detecting objects in an image. The output unit may output the processed image to the learning unit as learning data for generating the object detection model.

[0087] In one embodiment, (8) the image processing method includes the electronic device processing a pre-processing image based on processing information to generate a post-processing image, and the electronic device outputting the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image.

[0088] In one embodiment, (9) an image processing program causes an electronic device to process a pre-processing image based on processing information to generate a post-processing image, and to output the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of a light source in the pre-processing image, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source. The environmental information includes information regarding the environment of the space in the pre-processing image. [Explanation of Symbols]

[0089] 1. Image Processing System 10 Electronic devices (12: acquisition unit, 14: processing unit, 16: output unit, 18: learning unit, 22: memory unit) 20 cameras 30 vehicles (32: headlights) 40 Road surface

Claims

1. A processing unit that processes the pre-processing image based on processing information to generate a post-processing image, The output unit outputs the processed image. Equipped with, The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of the light source, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source, The aforementioned environmental information includes information about the environment of the space captured in the pre-processed image. electronic equipment.

2. The electronic device according to claim 1, wherein the environmental information includes information relating to at least one of the conditions of rain or fog in the space captured in the pre-processing image.

3. The aforementioned processing information further includes photographic information, The aforementioned shooting information includes information regarding at least one of the following: the position of the camera that captured the pre-processed image, the direction of the optical axis of the camera, or the design data of the camera's optical system. The electronic device according to claim 1.

4. The electronic device according to claim 1, wherein the processing unit generates an image as the processed image by adding at least one of flare or ghosting to the pre-processing image.

5. The electronic device according to any one of claims 1 to 4, wherein the processing unit inputs the pre-processing image to a processing model that processes and outputs an input image based on the processing information, and generates the output image of the processing model as the post-processing image.

6. The electronic device according to any one of claims 1 to 4, wherein the output unit outputs the processed image as training data for generating an object detection model for detecting objects in the image to a learning device that generates the object detection model.

7. It further includes a learning unit that performs training to generate an object detection model for detecting objects in an image, The output unit outputs the processed image to the learning unit as training data for generating the object detection model. The electronic device according to any one of claims 1 to 4.

8. The electronic device processes the pre-processing image based on processing information to generate a post-processing image, and the electronic device outputs the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of the light source, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source, The aforementioned environmental information includes information about the environment of the space captured in the pre-processed image. Image processing methods.

9. The electronic device is made to perform the following actions: process the pre-processing image based on processing information to generate a post-processing image, and output the post-processing image. The processing information includes at least one of light source information or environmental information. The light source information includes information regarding at least one of the position of the light source, the direction of the light source, the angle of the optical axis of the light source, or the pattern of light emitted by the light source, The aforementioned environmental information includes information about the environment of the space captured in the pre-processed image. Image processing program.

Images