Electronic device, image processing method, and image processing program

Abstract

The present invention provides an electronic device, an image processing method, and an image processing program that can improve the detection accuracy of objects with a light source. [Solution] The electronic device 10 includes a learning unit 16. The learning unit 16 generates an object detection model that detects the position of a second object 62 from a second image 60 that shows a second object 62 having a second light source 64, where at least a portion of the second object 62 is hidden by a second occluder 68. This is achieved by performing learning using learning data that associates a first image 50 showing a first object 32 having a first light source 34 with correct position information of the first object 32 in the first image 50.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, an image processing method, and an image processing program.

Background Art

[0002] As described in Patent Document 1, there is known an apparatus that estimates the driving lane of another vehicle based on the magnitude of the average luminance inside a detection range set in an overhead image around a vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When detecting an object having a light source based on the average luminance of an image, the presence of the object may be erroneously detected due to the influence of other light sources. It is required to improve the detection accuracy of an object having a light source.

[0005] An object of the present disclosure is to provide an electronic device, an image processing method, and an image processing program capable of improving the detection accuracy of an object having a light source.

Means for Solving the Problems

[0006] An electronic device according to an embodiment of the present disclosure includes a learning unit that generates an object detection model for detecting the position of a second object from a second image depicting a state in which at least a part of the second object having a second light source is hidden by a second shielding object by performing learning using learning data in which a first image depicting a state in which at least a part of a first object having a first light source is hidden by a first shielding object is associated with correct information on the position of the first object in the first image.

[0007] An image processing method according to one embodiment of the present disclosure includes an electronic device performing learning using training data that associates a first image showing a state in which at least a portion of a first object having a first light source is hidden by a first occluder with ground truth information of the position of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a portion of a second object having a second light source is hidden by a second occluder.

[0008] An image processing program according to one embodiment of the present disclosure includes causing an electronic device to perform learning using training data that associates a first image showing a state in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a portion of a second object having a second light source is hidden by a second occluder. [Effects of the Invention]

[0009] According to an electronic device, an image processing method, and an image processing program according to one embodiment of the present disclosure, the detection accuracy of an object having a light source can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example of the schematic configuration of the image processing system related to this disclosure. [Figure 2] This block diagram shows an example of the learning process for generating an object detection model, and the inference process for object detection using the generated object detection model. [Figure 3] This figure shows an example of an object occlusion image. [Figure 4] This figure shows an example of a detected image. [Figure 5] This block diagram shows the process flow for generating an object occlusion image from an object image and an occlusion image, and the process flow for obtaining the correct object position information from the object image. [Figure 6]This diagram shows an example of generating an object occlusion image from an object image and an occlusion image. [Figure 7] This flowchart shows an example of the procedure for generating an object detection model. [Modes for carrying out the invention]

[0011] Improving object detection accuracy from images is crucial. For example, increasing the accuracy of object detection from images captured by in-vehicle cameras enhances vehicle safety. Furthermore, improving object detection accuracy from images is beneficial not only for in-vehicle cameras but also in various other fields.

[0012] Furthermore, it is beneficial to be able to detect an object from an image even if at least a part of it is obscured by other objects. For example, light emitted from a light source of the object to be detected may be visible in the image. Even if all or most of the object to be detected is obscured by other objects in the image, it becomes possible to detect the object that has the light source by detecting the light visible in the image.

[0013] When detecting objects with light sources using a pre-trained model, the training data used to generate the pre-trained model consists of training images containing objects with light sources and ground truth data of the positions of the objects in those training images. The training images include images in which at least a portion of the object with a light source, i.e., all or part of the object with a light source, is obscured by other objects. A pre-trained model generated using such training images as training data can detect objects with light sources from images even when at least a portion of the object with a light source is obscured by other objects. Furthermore, by using images with a greater variation in the state in which all or part of the object with a light source is obscured by other objects as training images, the accuracy of object detection by pre-trained models generated by training with training data containing those training images can be improved.

[0014] Hereinafter, an example of an embodiment of an electronic device 10 (see FIG. 1), an image processing method, and an image processing program that can improve the detection accuracy of an object having a light source will be described in the present disclosure.

[0015] (Configuration example of image processing system 1) As shown in FIG. 1, an image processing system 1 according to an embodiment of the present disclosure includes an electronic device 10 and a camera 20.

[0016] <Electronic device 10> The electronic device 10 includes an acquisition unit 12, a generation unit 14, a learning unit 16, an output unit 18, and a storage unit 22.

[0017] The acquisition unit 12 acquires image data from the camera 20. The acquisition unit 12 may acquire various other data or information from the camera 20 or an external device. The acquisition unit 12 may include a communication interface for communicating with the camera 20 or an external device by wire or wirelessly. The communication interface may be configured to communicate by a communication method based on various communication standards. The communication interface may be configured based on known communication technologies.

[0018] The acquisition unit 12 may include an input device that receives an input from a user. The input device may include, for example, a keyboard or physical keys, or may include a pointing device such as a touch panel, a touch sensor, or a mouse. The input device is not limited to these examples and may be configured to include various other devices. The acquisition unit 12 may be configured to communicate with an external input device. The electronic device 10 may include the input device as an input unit separate from the acquisition unit 12.

[0019] The generation unit 14 may generate a learning image using the image data acquired by the acquisition unit 12. Further, the generation unit 14 may generate learning data including the learning image.

[0020] The learning unit 16 performs learning using the training data to generate a trained model for object detection. The learning unit 16 may perform learning using the training data generated by the generation unit 14. The learning unit 16 may also perform learning using training data acquired from an external device. If the learning unit 16 uses training data acquired from an external device, the electronic device 10 does not need to include the generation unit 14.

[0021] The generation unit 14 or learning unit 16 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The general-purpose processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The dedicated circuit may include, for example, an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The generation unit 14 or learning unit 16 may be implemented as a single integrated circuit (IC). The generation unit 14 may be implemented as a plurality of communicably connected integrated circuits or discrete circuits. The generation unit 14 or learning unit 16 may include either a System-on-a-Chip (SoC) or a System-in-a-Package (SiP) in which one or more processors cooperate. The generation unit 14 or the learning unit 16 may be implemented based on various other known technologies. The generation unit 14 and the learning unit 16 may be configured as a single unit.

[0022] The storage unit 22 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit 22 stores various types of information. The storage unit 22 stores programs executed by a general-purpose processor that functions as the generation unit 14 or the learning unit 16, or various types of data or information. The storage unit 22 may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the generation unit 14 or the learning unit 16. At least a part of the storage unit 22 may be configured integrally with the generation unit 14 or the learning unit 16.

[0023] The electronic device 10 may include an inference unit 24. The inference unit performs inference using an object detection model. The inference unit 24 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The inference unit 24 may be configured identically or similarly to the generation unit 14 or the learning unit 16.

[0024] The output unit 18 outputs the training images or training data generated by the generation unit 14, or the object detection model generated by the learning unit 16, to an external device. The output unit 18 may also output the object detection results from the inference unit 24 to an external device. The output unit 18 may also output various other data or information. The output unit 18 may be equipped with a communication interface for wired or wireless communication with an external device. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0025] The output unit 18 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) or an inorganic EL display. The electronic device 10 may also include the display device as a separate display unit from the output unit 18.

[0026] The output unit 18 is not limited to these examples and may include various other output devices.

[0027] The electronic device 10 relating to this disclosure may be various types of devices. For example, the electronic device 10 according to one embodiment may be a specially designed terminal, or it may be a general-purpose smartphone, tablet, phablet, notebook PC (Personal Computer), computer, or server. The electronic device 10 may be configured in a cloud computing environment or in an on-premise environment. The electronic device 10 may be mounted on a mobile device such as a vehicle.

[0028] <Camera 20> The camera 20 includes an image sensor. The image sensor may include, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor may be configured to capture at least one of infrared light or visible light. The camera 20 may include both an infrared image sensor and a visible light image sensor. The infrared light may include, for example, light in the wavelength range from 780 nm to 1000 nm. The visible light may include, for example, light in the wavelength range from 380 nm to 780 nm.

[0029] The number of cameras 20 is not limited to one; there may be two or more. Cameras 20 may be mounted on moving objects such as vehicles. Cameras 20 may be mounted on devices such as roadside units used in traffic systems. Cameras 20 may be included in electronic equipment 10.

[0030] <Object detection device> The image processing system 1 may include an object detection device that performs object detection inference using an object detection model. The object detection device may include a processor that performs object detection inference using an object detection model. The object detection device may include a communication interface for acquiring an image to be input to the object detection model, i.e., an image to be used for object detection. The object detection device may include an output device that outputs the object detection results from the image obtained by the object detection model. The object detection device is not limited to these, and may include components that are the same as or similar to the components included in the electronic device 10.

[0031] The object detection device may be mounted on a moving object such as a vehicle. The object detection device may be mounted on a roadside device or other device used in a traffic system. The object detection device may also be an external device not included in the image processing system 1.

[0032] (Example of operation of image processing system 1) In the image processing system 1 according to this disclosure, as illustrated in Figure 2, a learning phase for generating an object detection model and an inference phase for object detection using the generated object detection model are performed. The object detection model is configured to detect an object from an image and output the position of the object when an image is input in which at least a part of the object to be detected, which has a light source, is hidden. In other words, by using the object detection model, the position of the object to be detected can be detected from an image in which at least a part of the object to be detected is hidden. The learning phase is performed by the learning unit 16 of the electronic device 10. The inference phase may be performed by the inference unit of the electronic device 10 or by an object detection device different from the electronic device 10. The inference phase may also be performed by an external device not included in the image processing system 1. Examples of the operation of the learning phase and the inference phase will be described below.

[0033] <Learning Phase> The learning unit 16 of the electronic device 10 generates an object detection model by performing learning using the learning data. The learning data includes, for example, data that associates an object occlusion image 50 shown in Figure 3 with the correct information of the object's position in the space depicted in the object occlusion image 50.

[0034] <<Object occlusion image 50>> The object occlusion image 50 is an image of the space in which the first object 32 exists, with at least a portion of the first object 32 hidden by the first occluding object 42. The object occlusion image 50 is also referred to as the first image. In the object occlusion image 50, i.e., the first image, illustrated in Figure 3, the entire first object 32 is hidden by the first occluding object 42. At least a portion or all of the first image illustrated in Figure 3 may be configured to include images of various types, such as images of real objects captured by a camera, images created using computer graphics, images automatically generated by AI, or handwritten images, or combinations thereof.

[0035] The first object 32 has a first light source 34. In other words, the first image is an image in which at least a part of the first object 32 having the first light source 34 is hidden by the first occluding object 42. In the first image illustrated in Figure 3, the first object 32 is assumed to be an automobile. The first light source 34 is assumed to be the headlight of the automobile. The first object 32 is not limited to an automobile, but may be a vehicle such as a motorcycle or bicycle. If the first object 32 is a vehicle, the first light source 34 may include the vehicle's headlight or taillight, etc. The first object 32 is not limited to a vehicle, but may be a drone or a pedestrian, etc. If the first object 32 is a drone, the first light source 34 may include the drone's searchlight, etc. If the first object 32 is a pedestrian, the first light source 34 may include a flashlight or hand light carried by the pedestrian. Vehicles, drones, pedestrians, etc. are also collectively referred to as mobile objects. The first light source 34 may be a lighting device owned by a mobile object.

[0036] The first shield 42 may include an object that blocks light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include an object that blocks light from a continuous range of a portion of the first object 32. The first shield 42 may include an object that blocks light from most of the first object 32 while allowing light to pass through from multiple narrow ranges of the first object 32, such as a hedge or a lattice fence. The first shield 42 may include an object that attenuates light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include an object that scatters light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include, for example, at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0037] The first image is not limited to an image showing the entire first object 32 hidden by the first occluder 42, as illustrated in Figure 3, but may include an image showing a state in which only a part of the first object 32 is hidden by the first occluder 42. For example, the first image may include an image showing a state in which the first light source 34 of the first object 32 is hidden by the first occluder 42, while at least a part of the other parts of the first light source 34 is not hidden by the first occluder 42. The first image may include an image showing a state in which the other parts of the first light source 34 of the first object 32 are hidden by the first occluder 42, while the first light source 34 is not hidden by the first occluder 42.

[0038] The first image may be an image showing the state in which at least one feature portion of the first object 32 is hidden by the first occlusion 42. The feature portion of the first object 32 may be at least a part of the contour of the first object 32. The feature portion of the first object 32 may be a portion representing the texture of the first object 32. The feature portion of the first object 32 is not limited to these examples and may be various parts of the first object 32.

[0039] The first image may be an infrared image captured using infrared light. The first image may be a visible light image captured using visible light. In other words, the training data may include at least one of either an infrared image or a visible light image.

[0040] <<Correct information on object position>> The correct object position information is information that represents the correct position of the first object 32 as it appears in the first image. If the entire first object 32 is hidden by the first occluder 42, the correct object position information is information that represents the correct position of the first object 32 that is not visible in the first image because it is hidden by the first occluder 42. In the first image illustrated in Figure 3, the correct object position information is information that represents the correct position where the first object 32, which is a car, exists.

[0041] Assume that the first image is an image of the first illumination range 36 illuminated by the first light source 34. The first illumination range 36 is useful information for estimating the position of the first object 32 that is not visible in the first image when the entire first object 32 is hidden by the first occlusion 42. Furthermore, the first illumination range 36 is useful information for improving the accuracy of estimating the position of the first object 32 that is only partially visible in the first image when a part of the first object 32 is hidden by the first occlusion 42. Furthermore, the first illumination range 36 is useful information for improving the accuracy of estimating the position of the first object 32 that is not visible in the first image when a feature part of the first object 32 is hidden by the first occlusion 42.

[0042] The correct information for the object's position may be generated by detecting the object's position from an image obtained by removing the occluding object that obscures at least a portion of the object in the object occluding image 50. If the object occluding image 50 is an image generated by compositing an occluding object onto an image without an occluding object as the original image, then the image obtained by removing the occluding object that obscures at least a portion of the object in the object occluding image 50 corresponds to the original image used to generate the object occluding image 50. Therefore, the correct information for the object's position can be easily generated by detecting the object's position from the original image.

[0043] The correct object position information may be generated by input from a user who knows the correct position of an object within the shooting range of the object occluded image 50. Alternatively, the correct object position information may be generated by detecting the position of an object in an image taken from a different direction than the shooting direction of the object occluded image 50, i.e., an image in which the object is not obscured.

[0044] <<Learning>> As shown in Figure 2 as the learning phase, the learning unit 16 generates an object detection model by performing learning using learning data that associates object occlusion images 50, i.e., the first image, with the correct information of the object's position.

[0045] The training data may include data in which ground truth information of object positions is associated with at least one of the infrared or visible images. The training data may also include data in which ground truth information of object positions is associated with infrared and visible images from the same imaging range.

[0046] The electronic device 10 may output the object detection model generated by the learning unit 16 to the object detection device via the output unit 18. The electronic device 10 may use the object detection model generated by the learning unit 16 to execute the inference phase in the inference unit, as described later.

[0047] Object detection accuracy can be improved by generating an object detection model using object occlusion images 50, which reproduce various variations of the state in which an object is hidden by an occluding object, as training data. Furthermore, object detection accuracy in images in which the feature parts of an object are hidden by an occluding object can be improved by generating an object detection model using object occlusion images 50, which reproduce the state in which the feature parts of an object are hidden by an occluding object, as training data. In addition, object detection accuracy can be improved if the object occlusion images 50 include at least one of an infrared image or a visible image.

[0048] <Inference Phase> The object detection device of the image processing system 1 performs object detection inference using an object detection model to detect the position of the second object 62 from the detection image 60 shown in Figure 4, for example.

[0049] <<Detected Image 60>> The detected image 60 is an image of the space in which the second object 62 exists, with at least a portion of the second object 62 obscured by the second occluding object 68. The detected image 60 is also referred to as the second image. In the detected image 60, i.e., the second image, illustrated in Figure 4, the entire second object 62 is obscured by the second occluding object 68. At least a portion or all of the second image illustrated in Figure 4 may consist of appropriate types of images, such as images of real objects captured by a camera, images created using computer graphics, images automatically generated by AI, or handwritten images, or combinations thereof.

[0050] The second object 62 has a second light source 64. In other words, the second image is an image in which at least a part of the second object 62 having the second light source 64 is hidden by the second occluding object 68. In the second image illustrated in Figure 4, the second object 62 is assumed to be an automobile. The second light source 64 is assumed to be the headlight of the automobile. The second object 62 is not limited to an automobile, but may be a vehicle such as a motorcycle or bicycle. If the second object 62 is a vehicle, the second light source 64 may include the vehicle's headlight or taillight, etc. The second object 62 is not limited to a vehicle, but may be a drone or a pedestrian, etc. If the second object 62 is a drone, the second light source 64 may include the drone's searchlight, etc. If the second object 62 is a pedestrian, the second light source 64 may include a flashlight or hand light carried by the pedestrian. The second light source 64 may be a lighting device owned by a moving object such as a vehicle, drone, or pedestrian.

[0051] The second shield 68 may include an object that blocks light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include an object that blocks light from a continuous range of a portion of the second object 62. The second shield 68 may include an object that blocks light from most of the second object 62 while allowing light to pass through from multiple narrow ranges of the second object 62, such as a hedge or a lattice fence. The second shield 62 may include an object that attenuates light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include an object that scatters light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include, for example, at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0052] The second image is not limited to an image showing the entire second object 62 hidden by the second occluder 68, as illustrated in Figure 4, but may include an image showing a state in which only a part of the second object 62 is hidden by the second occluder 68. For example, the second image may include an image showing the second light source 64 of the second object 62 hidden by the second occluder 68, while at least a part of the other part of the second light source 64 is not hidden by the second occluder 68. The second image may also include an image showing the other part of the second light source 64 of the second object 62 hidden by the second occluder 68, while the second light source 64 is not hidden by the second occluder 62.

[0053] The second image may be an image showing the state in which at least one feature portion of the second object 62 is hidden by the second occlusion 68. The feature portion of the second object 62 may be at least a part of the contour of the second object 62. The feature portion of the second object 62 may be a portion representing the texture of the second object 62. The feature portion of the second object 62 may be any of the various parts of the second object 62, and is not limited to these examples.

[0054] The second image is an image of the second illumination area 66 illuminated by the second light source 64. The second illumination area 66 is useful information for estimating the position of the second object 62 that is not visible in the second image when the entire second object 62 is hidden by the second occlusion 68. Furthermore, the second illumination area 66 is useful information for improving the accuracy of estimating the position of the second object 62 that is only partially visible in the second image when a part of the second object 62 is hidden by the second occlusion 68. Furthermore, the second illumination area 66 is useful information for improving the accuracy of estimating the position of the second object 62 that is not visible in the second image when a feature part of the second object 62 is hidden by the second occlusion 68.

[0055] The second image may be an infrared image captured using infrared light. The second image may be a visible light image captured using visible light.

[0056] <<Object Detection Inference>> As shown in Figure 2 as the inference phase, the object detection device inputs the detected image 60, i.e., the second image, into the object detection model generated in the learning phase. The object detection model detects the position of the second object 62 from the input second image and outputs an object detection result. The object detection device acquires the object detection result output from the object detection model.

[0057] The object detection device may input at least one of an infrared image or a visible image as a second image into the object detection model. The object detection device may also input a set of infrared and visible images from the same imaging range as a second image.

[0058] <Summary> As described above, the image processing system 1 or electronic device 10 relating to this disclosure can improve the detection accuracy of objects that are captured in the detection image 60 while at least partially obscured by an occluding object, by performing an object detection inference phase using an object detection model generated from a training image in which the object is captured while obscured by an occluding object.

[0059] In the image processing system 1 of the present disclosure, the learning unit 16 of the electronic device 10 generates an object detection model using object occlusion images 50, which reproduce the state in which an object is hidden by an occlusion in various ways, as training data, thereby improving the accuracy of object detection.

[0060] <Generating training data> As described above, the training data is data that associates the object occlusion image 50, i.e., the first image, with the correct information of the object's position. In the image processing system 1 according to this disclosure, the training data may be generated by the generation unit 14 of the electronic device 10. In this case, the learning unit 16 of the electronic device 10 may acquire the training data from the generation unit 14 and execute the training phase. The training data may also be generated by an external device. In this case, the learning unit 16 may acquire the training data from the external device and execute the training phase.

[0061] The following describes an example of how the generation unit 14 generates training data. The generation unit 14 may generate an object occlusion image 50, i.e., a first image, from an object image 30 (see Figure 6) and an occlusion image 40 (see Figure 6), as illustrated in the block diagram in Figure 5. The generation unit 14 may generate correct information about the object's position from the object image 30.

[0062] Object image 30 is an image in which the first object 32 is not obscured by an occluding object, as illustrated in Figure 6. Object image 30 is also referred to as the third image.

[0063] In the object image 30, i.e., the third image, illustrated in Figure 6, the first object 32 is assumed to be an automobile. The first light source 34 is assumed to be the automobile's headlight. The first illumination range 36 is assumed to be the range illuminated by the automobile's headlight, which is the first light source 34. The first object 32 may be various moving objects, as described in the explanation of the object occlusion image 50. The first light source 34 may be a light source possessed by the moving object.

[0064] The object image 30 may be an image actually captured by the camera 20 or the like. At least a portion of the object image 30 may be an image virtually generated as CG (Computer Graphics) or the like. In other words, at least a portion of the pixels of the object image 30 may be virtually generated pixels.

[0065] The occlusion image 40 is an image of the first occlusion 42, as illustrated in Figure 6. The shooting range of the occlusion image 40 corresponds to the shooting range of the object image 30. The first occlusion 42 may be various objects, as described in the explanation of the object occlusion image 50.

[0066] The occluding object image 40 may be an image actually captured by the camera 20 or the like. At least a portion of the occluding object image 40 may be an image virtually generated as computer graphics or the like. In other words, at least a portion of the pixels of the occluding object image 40 may be virtually generated pixels. The occluding object image 40 may be generated by the generation unit 14.

[0067] The generation unit 14 may generate an object occlusion image 50, i.e., a first image, by combining the object image 30 and the occlusion image 40, as illustrated in Figure 6. In the first image illustrated in Figure 6, the first object 32 and the first light source 34, which were visible in the object image 30, are shown hidden by the first occlusion 42. The first image may be generated such that at least a portion of the first illumination range 36 is not hidden by the first occlusion 42. By using an image in which at least a portion of the first illumination range 36 is visible as training data, the accuracy of the object detection model's estimation of the position of the first object 32 can be improved.

[0068] The generation unit 14 may generate correct object position information from the object image 30, as illustrated in the block diagram in Figure 5. In the object image 30, the first object 32 is visible without being obscured by an occluder. The generation unit 14 may use a model that detects objects without considering occluders to detect the position of the first object 32 from the object image 30 and generate correct object position information.

[0069] The generation unit 14 generates training data by associating the object occlusion image 50 generated from the object image 30 with the correct object position information detected from the object image 30. As mentioned above, the correct object position information detected from the object image 30 is expected to be generated with high accuracy because the first object 32 is captured without being obscured by an occlusion object. Therefore, the object detection accuracy is improved in the object detection model generated using the object occlusion image 50, which is obtained by combining the object image 30 with the occlusion image 40, as training data.

[0070] Furthermore, by combining an object image 30 with various occluding object images 40, it is possible to generate object occluding images 50 that reproduce a variety of states in which the first object 32 is hidden by an occluding object from a single object image 30. By using these generated object occluding images 50 as training data to generate an object detection model, it is expected that the detection accuracy of objects that are at least partially hidden will improve.

[0071] The generation unit 14 may output the generated training data to the learning unit 16. The learning unit 16 may use the training data generated by the generation unit 14 to perform the learning phase described above and generate an object detection model. The generation unit 14 may also output the generated training data to an external device that performs the learning phase.

[0072] <Example of the procedure for generating an object detection model> The electronic device 10 may generate an object detection model by executing an image processing method that includes the steps of the flowchart illustrated in Figure 7. The image processing method may be implemented as an image processing program to be executed by the generation unit 14 or the learning unit 16 of the electronic device 10. The image processing program may be stored on a non-temporary computer-readable medium.

[0073] The generation unit 14 acquires an object image 30 (step S1). The generation unit 14 may acquire the object image 30 from a camera 20 or the like. The generation unit 14 may also acquire an object image 30 that has been previously stored in the storage unit from the storage unit.

[0074] The generation unit 14 obtains the correct information of the object position of the first object 32 from the object image 30 (step S2).

[0075] The generation unit 14 generates an object occlusion image 50 from the object image 30 (step S3). The generation unit 14 may generate the object occlusion image 50 by acquiring an occlusion image 40 from a camera 20 or the like and compositing it with the object image 30. The generation unit 14 may also generate an occlusion image 40 and composite it with the object image 30 to generate the object occlusion image 50.

[0076] The generation unit 14 generates training data by associating the object occlusion image 50 with the correct information of the object's position (step S4). The generation unit 14 outputs the generated training data to the learning unit 16.

[0077] The learning unit 16 generates an object detection model by performing learning using the training data (step S5). After performing the procedure in step S5, the electronic device 10 finishes performing the procedure in the flowchart in Figure 7.

[0078] (summary) As described above, the electronic device 10, image processing method, and image processing program relating to this disclosure generate an object detection model using images as training data in which at least a portion of an object with a light source is obscured by an occluding object. By performing object detection inference using the object detection model thus generated, the accuracy of object detection from images in which at least a portion of an object with a light source is obscured by an occluding object can be improved.

[0079] Furthermore, training data is generated by compositing occluders onto images in which objects are not obscured by occluders. In this way, the accuracy of the ground truth information for object positions is improved in the generated training data. As a result, the accuracy of object detection by inference using the object detection model is improved.

[0080] Furthermore, by combining the object image 30 with the occlusion image 40 to generate an object occlusion image 50, it is possible to generate an object occlusion image 50 that reproduces various variations of the state in which the first object 32 is hidden by the occlusion from a single object image 30. If an image of an object hidden by an occlusion were to be actually captured and generated, it would be practically difficult to capture and reproduce all states in which the object is hidden by the occlusion. According to this disclosure, an object occlusion image 50 that reproduces various variations of the state in which the first object 32 is hidden by the occlusion can be easily generated. As a result, an improvement in the detection accuracy of objects that are at least partially hidden is expected.

[0081] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0082] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. These are also to be understood as being included within the scope of this disclosure.

[0083] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0084] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0085] Vehicles relating to this disclosure may include, for example, automobiles, industrial vehicles, railway vehicles, residential vehicles, or fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture or construction. Industrial vehicles may include, for example, forklifts or golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. Vehicles of the same nature may be included in multiple classifications.

[0086] While embodiments of the image processing method using the image processing system 1 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0087] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. In addition, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0088] In one embodiment, (1) the electronic device includes a learning unit that generates an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by performing learning using learning data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with correct information on the position of the first object in the first image.

[0089] (2) In the electronic device described in (1) above, at least one of the first object or the second object may be a moving body. At least one of the first light source or the second light source may be an illumination device of the moving body.

[0090] (3) The electronic device described in (1) or (2) above may further include a generation unit that generates the first image by combining an image of the first occluding object with a third image which shows the entirety of the first object having the first light source.

[0091] (4) In the electronic device described in (3) above, at least some of the pixels of the third image may be virtually generated pixels.

[0092] (5) In the electronic device described in any one of (1) to (4) above, at least some of the pixels of the first image may be virtually generated pixels.

[0093] (6) In the electronic device described in any one of (1) to (5) above, the first shield may include at least one of the following: a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0094] (7) In the electronic device described in any one of (1) to (6) above, the second shield may include at least one of the following: a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0095] (8) In the electronic device described in any one of (1) to (7) above, at least one of the first image or the second image may include an infrared image captured from infrared light.

[0096] (9) In the electronic device described in any one of (1) to (8) above, at least one of the first image or the second image may include a visible image captured in visible light.

[0097] (10) In the electronic device described in any one of (1) to (9) above, the first image may include an image showing the state in which at least one feature part of the first object is hidden by the first occlusion.

[0098] (11) In the electronic device described in any one of (1) to (10) above, the second image may include an image showing the state in which at least one feature portion of the second object is hidden by the second occlusion.

[0099] In one embodiment, (12) the image processing method includes the electronic device performing learning using training data that associates a first image showing a state in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a portion of a second object having a second light source is hidden by a second occluder.

[0100] In one embodiment, (13) the image processing program causes the electronic device to perform learning using training data that associates a first image showing a state in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a portion of a second object having a second light source is hidden by a second occluder. [Explanation of Symbols]

[0101] 1. Image Processing System 10 Electronic devices (12: Acquisition unit, 14: Generation unit, 16: Learning unit, 18: Output unit, 22: Memory unit, 24: Inference unit) 20 cameras 30 Object image (32: 1st object, 34: 1st light source, 36: 1st irradiation area) 40 Obstruction Image (42: First Obstruction) 50 Object occlusion images 60 Detected image (62: second object, 64: second light source, 66: second illumination area, 68: second shielding object)

Claims

1. Electronic device comprising a learning unit that generates an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by performing learning using learning data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image.

2. The electronic device according to claim 1, wherein at least one of the first object or the second object is a mobile body, and at least one of the first light source or the second light source is a lighting device of the mobile body.

3. The electronic device according to claim 1, further comprising a generation unit that generates the first image by combining an image of a first occluding object with a third image which shows the entirety of the first object having the first light source.

4. The electronic device according to claim 3, wherein at least some of the pixels in the third image are virtually generated pixels.

5. The electronic device according to any one of claims 1 to 3, wherein at least some of the pixels in the first image are virtually generated pixels.

6. The electronic device according to any one of claims 1 to 3, wherein the first shield includes at least one of a building, wall, fence, guardrail, plant, street tree, traffic light, sign, billboard, vehicle, aircraft, pedestrian, animal, smoke, fog, or rain.

7. The electronic device according to any one of claims 1 to 3, wherein the second shield includes at least one of a building, wall, fence, guardrail, plant, street tree, traffic light, sign, billboard, vehicle, aircraft, pedestrian, animal, smoke, fog, or rain.

8. The electronic device according to any one of claims 1 to 3, wherein at least one of the first image or the second image includes an infrared image obtained by capturing infrared light.

9. The electronic device according to any one of claims 1 to 3, wherein at least one of the first image or the second image includes a visible image captured in visible light.

10. The electronic device according to any one of claims 1 to 3, wherein the first image includes an image showing a state in which at least one feature portion of the first object is hidden by the first occluder.

11. The electronic device according to any one of claims 1 to 3, wherein the second image includes an image showing a state in which at least one feature portion of the second object is hidden by the second occlusion.

12. An image processing method comprising an electronic device performing training using training data that associates a first image showing a first object having a first light source in which at least a portion is obscured by a first occluder with ground truth information on the position of the first object in the first image, thereby generating an object detection model that detects the position of a second object having a second light source in which at least a portion is obscured by a second occluder.

13. An image processing program that includes causing an electronic device to perform training using training data that associates a first image showing a first object having a first light source in which at least a portion is hidden by a first occluder with the correct position information of the first object in the first image, thereby generating an object detection model that detects the position of a second object having a second light source in which at least a portion is hidden by a second occluder.

Images