Electronic apparatus, image processing method, and image processing program

Abstract

This electronic apparatus comprises a training unit. The training unit executes training using training data in which a first image, which is obtained by capturing a state in which at least a portion of a first object having a first light source is hidden by a first shielding object, is associated with correct answer information about the position of the first object in the first image, thereby generating an object detection model for detecting the position of a second object from a second image obtained by capturing a state in which at least a portion of the second object having a second light source is hidden by a second shielding object.

Description

Electronic device, image processing method, and image processing program Cross-reference to related applications

[0001] This application claims the priority of Japanese Patent Application No. 2024-168895 (filed on September 27, 2024), and the entire disclosure of the application is incorporated herein by reference for that purpose.

[0002] This disclosure relates to an electronic device, an image processing method, and an image processing program.

[0003] As described in Patent Document 1, there is known a device that estimates the driving lane of another vehicle based on the magnitude of the average luminance inside a detection range set in an overhead image around a vehicle.

[0004] Japanese Unexamined Patent Application Publication No. 2016-110484

[0005] A learning unit of an electronic device according to an embodiment of the present disclosure executes learning using learning data in which a first image depicting a state in which at least a part of a first object having a first light source is hidden by a first shielding object is associated with correct position information of the first object in the first image, and generates an object detection model for detecting the position of a second object from a second image depicting a state in which at least a part of a second object having a second light source is hidden by a second shielding object.

[0006] An image processing method according to an embodiment of the present disclosure includes an electronic device generating an object detection model for detecting the position of a second object from a second image depicting a state in which at least a part of a second object having a second light source is hidden by a second shielding object by executing learning using learning data in which a first image depicting a state in which at least a part of a first object having a first light source is hidden by a first shielding object is associated with correct position information of the first object in the first image.

[0007] An image processing program according to one embodiment of the present disclosure includes causing an electronic device to perform learning using training data that associates a first image showing a state in which at least a portion of a first object having a first light source is hidden by a first occluder with correct information on the position of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a portion of a second object having a second light source is hidden by a second occluder.

[0008] This is a block diagram showing an example of the schematic configuration of the image processing system related to this disclosure. This is a block diagram showing an example of the learning flow for generating an object detection model and the inference flow for object detection using the generated object detection model. This is a diagram showing an example of an object occluded image. This is a diagram showing an example of a detected image. This is a block diagram showing the processing flow for generating an object occluded image from an object image and an occluded image, and the processing flow for obtaining correct information on the object position from the object image. This is a diagram showing an example of generating an object occluded image from an object image and an occluded image. This is a flowchart showing an example of the procedure for generating an object detection model.

[0009] When detecting objects with a light source based on the average brightness of an image, the presence of an object may be falsely detected due to the influence of other light sources. Therefore, there is a need to improve the detection accuracy of objects with light sources. According to an electronic device, image processing method, and image processing program of one embodiment of this disclosure, the detection accuracy of objects with light sources can be improved.

[0010] Improving object detection accuracy from images is crucial. For example, increasing the accuracy of object detection from images captured by in-vehicle cameras enhances vehicle safety. Furthermore, improving object detection accuracy from images is beneficial not only for in-vehicle cameras but also in various other fields.

[0011] Furthermore, it is beneficial to be able to detect an object from an image even if at least a part of it is obscured by other objects. For example, light emitted from a light source of the object to be detected may be visible in the image. Even if all or most of the object to be detected is obscured by other objects in the image, it becomes possible to detect the object that has the light source by detecting the light visible in the image.

[0012] When detecting objects with light sources using a pre-trained model, the training data used to generate the pre-trained model consists of training images containing objects with light sources and ground truth data of the positions of the objects in those training images. The training images include images in which at least a portion of the object with a light source, i.e., all or part of the object with a light source, is obscured by other objects. A pre-trained model generated using such training images as training data can detect objects with light sources from images even when at least a portion of the object with a light source is obscured by other objects. Furthermore, by using images with a greater variation in the state in which all or part of the object with a light source is obscured by other objects as training images, the accuracy of object detection by pre-trained models generated by training with training data containing those training images can be improved.

[0013] Hereinafter, an example of an embodiment of an electronic device 10 (see Figure 1), an image processing method, and an image processing program that can improve the detection accuracy of an object having a light source will be described.

[0014] (Example of the configuration of the image processing system 1) As shown in Figure 1, the image processing system 1 according to one embodiment of the present disclosure comprises an electronic device 10 and a camera 20.

[0015] <Electronic device 10> The electronic device 10 includes an acquisition unit 12, a generation unit 14, a learning unit 16, an output unit 18, and a storage unit 22.

[0016] The acquisition unit 12 acquires image data from the camera 20. The acquisition unit 12 may acquire various other data or information from the camera 20 or an external device. The acquisition unit 12 may be equipped with a communication interface for wired or wireless communication with the camera 20 or an external device. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0017] The acquisition unit 12 may include an input device that accepts input from the user. The input device may include, for example, a keyboard or physical keys, or a pointing device such as a touch panel, touch sensor, or mouse. The input device is not limited to these examples and may include various other devices. The acquisition unit 12 may be configured to communicate with an external input device. The electronic device 10 may include an input device as a separate input unit from the acquisition unit 12.

[0018] The generation unit 14 may generate training images using image data acquired by the acquisition unit 12. The generation unit 14 may also generate training data that includes the training images.

[0019] The learning unit 16 performs learning using the training data to generate a trained model as an object detection model. The learning unit 16 may perform learning using the training data generated by the generation unit 14. The learning unit 16 may also perform learning using training data acquired from an external device. If the learning unit 16 uses training data acquired from an external device, the electronic device 10 does not need to include the generation unit 14.

[0020] The generation unit 14 or learning unit 16 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The general-purpose processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The dedicated circuit may include, for example, an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD). The PLD may include a Field-Programmable Gate Array (FPGA). The generation unit 14 or learning unit 16 may be implemented as a single integrated circuit (IC). The generation unit 14 may be implemented as a plurality of communicably connected integrated circuits or discrete circuits. The generation unit 14 or learning unit 16 may include either a System-on-a-Chip (SoC) or a System-in-a-Package (SiP) in which one or more processors cooperate. The generation unit 14 or the learning unit 16 may be implemented based on various other known technologies. The generation unit 14 and the learning unit 16 may be configured as a single unit.

[0021] The storage unit 22 may include an electromagnetic storage medium such as a magnetic disk, or it may include a memory such as a semiconductor memory or magnetic memory. The storage unit 22 stores various types of information. The storage unit 22 stores programs executed by a general-purpose processor that functions as the generation unit 14 or the learning unit 16, or various types of data or information. The storage unit 22 may be configured as a non-temporary readable medium. The storage unit may function as a work memory for the generation unit 14 or the learning unit 16. At least a part of the storage unit 22 may be configured integrally with the generation unit 14 or the learning unit 16.

[0022] The electronic device 10 may include an inference unit 24. The inference unit performs inference using an object detection model. The inference unit 24 may include at least one general-purpose processor or at least one dedicated circuit to provide control and processing capabilities for performing various functions. The inference unit 24 may be configured identically to or similarly to the generation unit 14 or the learning unit 16.

[0023] The output unit 18 outputs the training images or training data generated by the generation unit 14, or the object detection model generated by the learning unit 16, to an external device. The output unit 18 may also output the object detection results from the inference unit 24 to an external device. The output unit 18 may output various other data or information. The output unit 18 may be equipped with a communication interface for wired or wireless communication with an external device. The communication interface may be configured to communicate using communication methods based on various communication standards. The communication interface may be configured based on known communication technologies.

[0024] The output unit 18 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) or an inorganic EL display. The electronic device 10 may also include the display device as a separate display unit from the output unit 18.

[0025] The output unit 18 is not limited to these examples and may include various other output devices.

[0026] The electronic device 10 relating to this disclosure may be various types of devices. For example, the electronic device 10 according to one embodiment may be a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC (Personal Computer), computer, or server. The electronic device 10 may be configured in a cloud computing environment or in an on-premise environment. The electronic device 10 may be mounted on a mobile device such as a vehicle.

[0027] <Camera 20> Camera 20 includes an image sensor. The image sensor may be configured to include, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor may be configured to capture at least one of infrared light or visible light. Camera 20 may include both an infrared image sensor and a visible light image sensor. Infrared light may include, for example, light in the wavelength range from 780 nm to 1000 nm. Visible light may include, for example, light in the wavelength range from 380 nm to 780 nm.

[0028] The number of cameras 20 is not limited to one; there may be two or more. Cameras 20 may be mounted on moving objects such as vehicles. Cameras 20 may be mounted on devices such as roadside units used in traffic systems. Cameras 20 may be included in electronic equipment 10.

[0029] <Object Detection Device> The image processing system 1 may include an object detection device that performs object detection inference using an object detection model. The object detection device may include a processor that performs object detection inference using an object detection model. The object detection device may include a communication interface that acquires an image to be input to the object detection model, i.e., an image to be used for object detection. The object detection device may include an output device that outputs the object detection result from the image obtained by the object detection model. The object detection device is not limited to these, and may include components that are the same as or similar to the components included in the electronic device 10.

[0030] The object detection device may be mounted on a moving object such as a vehicle. The object detection device may be mounted on a roadside device or other device used in a traffic system. The object detection device may also be an external device not included in the image processing system 1.

[0031] (Example of operation of Image Processing System 1) In the Image Processing System 1 according to this disclosure, as illustrated in Figure 2, a learning phase for generating an object detection model and an inference phase for object detection using the generated object detection model are performed. The object detection model is configured to detect an object from an image and output the position of the object when an image is input in which at least a part of the object to be detected, which has a light source, is hidden. In other words, by using the object detection model, the position of the object to be detected can be detected from an image in which at least a part of the object to be detected is hidden. The learning phase is performed by the learning unit 16 of the electronic device 10. The inference phase may be performed by the inference unit of the electronic device 10 or by an object detection device different from the electronic device 10. The inference phase may also be performed by an external device not included in the Image Processing System 1. Examples of operation of the learning phase and the inference phase will be described below.

[0032] <Learning Phase> The learning unit 16 of the electronic device 10 generates an object detection model by performing learning using the learning data. The learning data includes, for example, data that associates the object occlusion image 50 shown in Figure 3 with the correct information of the object positions in the space captured in the object occlusion image 50.

[0033] <<Object Occlusion Image 50>> The object occlusion image 50 is an image of the space in which the first object 32 exists, with at least a part of the first object 32 hidden by the first occlusion 42. The object occlusion image 50 is also referred to as the first image. In the object occlusion image 50, i.e., the first image, illustrated in Figure 3, the entire first object 32 is hidden by the first occlusion 42. At least a part or all of the first image illustrated in Figure 3 may be configured to include images of a real object captured by a camera, images created using computer graphics, images automatically generated by AI, or handwritten images, or combinations thereof, or any other appropriate type of image.

[0034] The first object 32 has a first light source 34. In other words, the first image is an image in which at least a part of the first object 32 having the first light source 34 is hidden by the first occluding object 42. In the first image illustrated in Figure 3, the first object 32 is assumed to be an automobile. The first light source 34 is assumed to be the headlight of the automobile. The first object 32 is not limited to an automobile, but may be a motorcycle, bicycle, or other vehicle. If the first object 32 is a vehicle, the first light source 34 may include the vehicle's headlight or taillight, etc. The first object 32 is not limited to a vehicle, but may be a drone or a pedestrian, etc. If the first object 32 is a drone, the first light source 34 may include the drone's searchlight, etc. If the first object 32 is a pedestrian, the first light source 34 may include a flashlight or hand light carried by the pedestrian. Vehicles, drones, pedestrians, etc. are also collectively referred to as mobile objects. The first light source 34 may be a lighting device owned by a mobile object.

[0035] The first shield 42 may include an object that blocks light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include an object that blocks light from a continuous range of a portion of the first object 32. The first shield 42 may include an object that blocks light from most of the first object 32 while allowing light to pass through from multiple narrow ranges of the first object 32, such as a hedge or a lattice fence. The first shield 42 may include an object that attenuates light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include an object that scatters light from at least a portion of the first object 32 toward a device such as a camera 20 that takes a first image. The first shield 42 may include, for example, at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0036] The first image is not limited to an image showing the entire first object 32 hidden by the first occluder 42, as illustrated in Figure 3, but may include an image showing a state in which only a part of the first object 32 is hidden by the first occluder 42. For example, the first image may include an image showing a state in which the first light source 34 of the first object 32 is hidden by the first occluder 42, while at least a part of the other parts of the first light source 34 is not hidden by the first occluder 42. The first image may include an image showing a state in which the other parts of the first light source 34 of the first object 32 are hidden by the first occluder 42, while the first light source 34 is not hidden by the first occluder 42.

[0037] The first image may be an image showing the state in which at least one feature portion of the first object 32 is hidden by the first occlusion 42. The feature portion of the first object 32 may be at least a part of the contour of the first object 32. The feature portion of the first object 32 may be a portion representing the texture of the first object 32. The feature portion of the first object 32 is not limited to these examples and may be various parts of the first object 32.

[0038] The first image may be an infrared image captured using infrared light. The first image may be a visible light image captured using visible light. In other words, the training data may include at least one of either an infrared image or a visible light image.

[0039] <<Correct Object Position Information>> The correct object position information is information that represents the correct position of the first object 32 as it appears in the first image. If the entire first object 32 is hidden by the first occluder 42, the correct object position information is information that represents the correct position of the first object 32 that is not visible in the first image because it is hidden by the first occluder 42. In the first image illustrated in Figure 3, the correct object position information is information that represents the correct position where the first object 32, which is the automobile, exists.

[0040] Assume that the first image is an image of the first illumination range 36 illuminated by the first light source 34. The first illumination range 36 is useful information for estimating the position of the first object 32 that is not visible in the first image when the entire first object 32 is hidden by the first shield 42. Furthermore, the first illumination range 36 is useful information for improving the accuracy of estimating the position of the first object 32 that is only partially visible in the first image when a part of the first object 32 is hidden by the first shield 42. Furthermore, the first illumination range 36 is useful information for improving the accuracy of estimating the position of the first object 32 that is not visible in the first image when a feature part of the first object 32 is hidden by the first shield 42.

[0041] The correct information for the object's position may be generated by detecting the object's position from an image obtained by removing the occluding object that obscures at least a portion of the object in the object occluding image 50. If the object occluding image 50 is an image generated by combining an occluding object with an image without an occluding object as the original image, then the image obtained by removing the occluding object that obscures at least a portion of the object in the object occluding image 50 corresponds to the original image used to generate the object occluding image 50. Therefore, the correct information for the object's position can be easily generated by detecting the object's position from the original image.

[0042] The correct object position information may be generated by input from a user who knows the correct position of an object within the shooting range of the object occluded image 50. The correct object position information may also be generated by detecting the position of an object in an image taken from a different direction than the shooting direction of the object occluded image 50, i.e., an image in which the object is not obscured.

[0043] <<Learning>> As shown in Figure 2 as the learning phase, the learning unit 16 generates an object detection model by performing learning using learning data that associates the object occlusion image 50, i.e., the first image, with the correct information of the object's position.

[0044] The learning data may include data in which correct information on the object position is associated with at least one of an infrared image and a visible image. The learning data may include data in which correct information on the object position is associated with an infrared image and a visible image within the same imaging range.

[0045] The electronic device 10 may output the object detection model generated by the learning unit 16 from the output unit 18 to the object detection device. The electronic device 10 may use the object detection model generated by the learning unit 16 to execute the inference phase in the inference unit as described later.

[0046] By using an object occlusion image 50 that reproduces in various variations the state in which an object is hidden by an occluder as learning data to generate an object detection model, the object detection accuracy can be improved. Also, by using an object occlusion image 50 that reproduces the state in which a feature portion of an object is hidden by an occluder as learning data to generate an object detection model, the object detection accuracy in an image in which the feature portion of the object appears in a state hidden by an occluder can be improved. Further, by including at least one of an infrared image and a visible image in the object occlusion image 50, the object detection accuracy can be improved.

[0047] <Inference Phase> The object detection device of the image processing system 1 executes inference of object detection using the object detection model, and detects the position of the second object 62 from a detection image 60 shown in, for example, FIG. 4.

[0048] <<Detection Image 60>> The detection image 60 is an image obtained by imaging the space where the second object 62 exists in a state where at least a part of the second object 62 is hidden by the second occluder 68. The detection image 60 is also referred to as the second image. In the detection image 60 illustrated in FIG. 4, that is, the second image, the entire second object 62 is hidden by the second occluder 68. Note that at least a part or all of the second image illustrated in FIG. 4 may be configured to include an appropriate type of image such as an image obtained by imaging a real object with a camera, an image created using computer graphics, an image automatically generated by AI, or an image handwritten by a person, or a combination thereof.

[0049] The second object 62 has a second light source 64. In other words, the second image is an image in which at least a part of the second object 62 having the second light source 64 is hidden by the second occluding object 68. In the second image illustrated in Figure 4, the second object 62 is assumed to be an automobile. The second light source 64 is assumed to be the headlight of the automobile. The second object 62 is not limited to an automobile, but may be a vehicle such as a motorcycle or bicycle. If the second object 62 is a vehicle, the second light source 64 may include the vehicle's headlight or taillight, etc. The second object 62 is not limited to a vehicle, but may be a drone or a pedestrian, etc. If the second object 62 is a drone, the second light source 64 may include the drone's searchlight, etc. If the second object 62 is a pedestrian, the second light source 64 may include a flashlight or hand light carried by the pedestrian. The second light source 64 may be a lighting device owned by a moving object such as a vehicle, drone, or pedestrian.

[0050] The second shield 68 may include an object that blocks light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include an object that blocks light from a continuous range of a portion of the second object 62. The second shield 68 may include an object that blocks light from most of the second object 62 while allowing light to pass through from multiple narrow ranges of the second object 62, such as a hedge or a lattice fence. The second shield 62 may include an object that attenuates light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include an object that scatters light from at least a portion of the second object 62 toward a device such as a camera 20 that takes a second image. The second shield 68 may include, for example, at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0051] The second image is not limited to an image showing the entire second object 62 hidden by the second occluder 68, as illustrated in Figure 4, but may include an image showing a state in which only a part of the second object 62 is hidden by the second occluder 68. For example, the second image may include an image showing a state in which the second light source 64 of the second object 62 is hidden by the second occluder 68, while at least a part of the other part of the second light source 64 is not hidden by the second occluder 68. The second image may include an image showing a state in which the other part of the second light source 64 of the second object 62 is hidden by the second occluder 68, while the second light source 64 is not hidden by the second occluder 62.

[0052] The second image may be an image showing the state in which at least one feature portion of the second object 62 is hidden by the second occlusion 68. The feature portion of the second object 62 may be at least a part of the contour of the second object 62. The feature portion of the second object 62 may be a portion representing the texture of the second object 62. The feature portion of the second object 62 is not limited to these examples and may be various parts of the second object 62.

[0053] The second image is an image of the second illumination range 66 illuminated by the second light source 64. The second illumination range 66 is useful information for estimating the position of the second object 62 that is not visible in the second image when the entire second object 62 is hidden by the second occlusion 68. The second illumination range 66 is also useful information for improving the accuracy of estimating the position of the second object 62 that is only partially visible in the second image when a part of the second object 62 is hidden by the second occlusion 68. Furthermore, the second illumination range 66 is useful information for improving the accuracy of estimating the position of the second object 62 that is not visible in the second image when a feature part of the second object 62 is hidden by the second occlusion 68.

[0054] The second image may be an infrared image captured using infrared light. The second image may be a visible light image captured using visible light.

[0055] <<Object Detection Inference>> As shown in Figure 2 as the inference phase, the object detection device inputs the detected image 60, i.e., the second image, into the object detection model generated in the learning phase. The object detection model detects the position of the second object 62 from the input second image and outputs the object detection result. The object detection device acquires the object detection result output from the object detection model.

[0056] The object detection device may input at least one of an infrared image or a visible image as a second image into the object detection model. The object detection device may also input a set of infrared and visible images from the same shooting range as a second image.

[0057] <Summary> As described above, the image processing system 1 or electronic device 10 relating to this disclosure can improve the detection accuracy of objects that are captured in the detection image 60 while at least partially obscured by an obstruction, by performing an object detection inference phase using an object detection model generated from a training image in which the object is captured while obscured by an obstruction.

[0058] In the image processing system 1 according to this disclosure, the learning unit 16 of the electronic device 10 generates an object detection model using object occlusion images 50, which reproduce the state in which an object is hidden by an occlusion in various ways, as training data, thereby improving the accuracy of object detection.

[0059] <Generation of Training Data> As described above, training data is data that associates the object occlusion image 50, i.e., the first image, with the correct information of the object's position. In the image processing system 1 according to this disclosure, training data may be generated by the generation unit 14 of the electronic device 10. In this case, the learning unit 16 of the electronic device 10 may acquire training data from the generation unit 14 and execute the learning phase. Training data may also be generated by an external device. In this case, the learning unit 16 may acquire training data from the external device and execute the learning phase.

[0060] The following describes an example of how the generation unit 14 generates training data. The generation unit 14 may generate an object occlusion image 50, i.e., a first image, from an object image 30 (see Figure 6) and an occlusion image 40 (see Figure 6), as illustrated in the block diagram in Figure 5. The generation unit 14 may generate correct information about the object's position from the object image 30.

[0061] The object image 30 is an image in which the first object 32 is not obscured by an occluding object, as illustrated in Figure 6. The object image 30 is also referred to as the third image.

[0062] In the object image 30, i.e., the third image, illustrated in Figure 6, the first object 32 is assumed to be an automobile. The first light source 34 is assumed to be the headlight of the automobile. The first illumination range 36 is assumed to be the range illuminated by the headlight of the automobile, which is the first light source 34. The first object 32 may be various moving objects, as described in the explanation of the object occlusion image 50. The first light source 34 may be a light source possessed by the moving object.

[0063] The object image 30 may be an image actually captured by the camera 20 or the like. At least a portion of the object image 30 may be an image virtually generated as computer graphics (CG) or the like. In other words, at least a portion of the pixels of the object image 30 may be virtually generated pixels.

[0064] The occluding object image 40 is an image of the first occluding object 42, as illustrated in Figure 6. The shooting range of the occluding object image 40 corresponds to the shooting range of the object image 30. The first occluding object 42 may be various objects, as described in the explanation of the object occluding image 50.

[0065] The obstruction image 40 may be an image actually captured by the camera 20 or the like. At least a portion of the obstruction image 40 may be an image virtually generated as computer graphics or the like. In other words, at least a portion of the pixels of the obstruction image 40 may be virtually generated pixels. The obstruction image 40 may also be generated by the generation unit 14.

[0066] The generation unit 14 may generate an object occlusion image 50, i.e., a first image, by combining the object image 30 and the occlusion image 40, as illustrated in Figure 6. In the first image illustrated in Figure 6, the first object 32 and the first light source 34, which were visible in the object image 30, are shown hidden by the first occlusion 42. The first image may be generated such that at least a portion of the first illumination range 36 is not hidden by the first occlusion 42. By using an image in which at least a portion of the first illumination range 36 is visible as training data, the accuracy of the object detection model's estimation of the position of the first object 32 can be improved.

[0067] The generation unit 14 may generate correct object position information from the object image 30, as illustrated in the block diagram in Figure 5. In the object image 30, the first object 32 is visible without being obscured by an occluder. The generation unit 14 may use a model that detects objects without considering occluders to detect the position of the first object 32 from the object image 30 and generate correct object position information.

[0068] The generation unit 14 generates training data by associating the object occlusion image 50 generated from the object image 30 with the correct information of the object's position detected from the object image 30. As mentioned above, the correct information of the object's position detected from the object image 30 is expected to be generated with high accuracy because the first object 32 is captured without being obscured by an occlusion. Therefore, the object detection accuracy is improved in an object detection model generated using the object occlusion image 50, which is created by combining the object image 30 with the occlusion image 40, as training data.

[0069] Furthermore, by combining an object image 30 with various occluding object images 40, it is possible to generate an object occluding image 50 from a single object image 30 that reproduces a variety of states in which the first object 32 is hidden by an occluding object. By using the object occluding image 50 thus generated as training data to generate an object detection model, it is expected that the detection accuracy of objects that are at least partially hidden will improve.

[0070] The generation unit 14 may output the generated training data to the learning unit 16. The learning unit 16 may use the training data generated by the generation unit 14 to perform the above-described learning phase and generate an object detection model. The generation unit 14 may also output the generated training data to an external device that performs the learning phase.

[0071] <Example of procedure for generating an object detection model> The electronic device 10 may generate an object detection model by executing an image processing method that includes the steps of the flowchart illustrated in Figure 7. The image processing method may be implemented as an image processing program to be executed by the generation unit 14 or learning unit 16 of the electronic device 10. The image processing program may be stored on a non-temporary computer-readable medium.

[0072] The generation unit 14 acquires an object image 30 (step S1). The generation unit 14 may acquire the object image 30 from a camera 20 or the like. The generation unit 14 may also acquire an object image 30 that has been previously stored in the storage unit from the storage unit.

[0073] The generation unit 14 obtains the correct information of the object position of the first object 32 from the object image 30 (step S2).

[0074] The generation unit 14 generates an object occlusion image 50 from the object image 30 (step S3). The generation unit 14 may generate the object occlusion image 50 by acquiring an occlusion image 40 from a camera 20 or the like and compositing it with the object image 30. Alternatively, the generation unit 14 may generate an occlusion image 40 and composite it with the object image 30 to generate the object occlusion image 50.

[0075] The generation unit 14 generates training data by associating the object occlusion image 50 with the correct information of the object's position (step S4). The generation unit 14 outputs the generated training data to the learning unit 16.

[0076] The learning unit 16 generates an object detection model by performing learning using the training data (step S5). After performing the procedure in step S5, the electronic device 10 finishes performing the procedure in the flowchart in Figure 7.

[0077] (Summary) As described above, the electronic device 10, image processing method, and image processing program relating to this disclosure generate an object detection model using images as training data in which at least a portion of an object with a light source is obscured by an occluding object. By performing object detection inference using the object detection model thus generated, the accuracy of object detection from images in which at least a portion of an object with a light source is obscured by an occluding object can be improved.

[0078] Furthermore, training data is generated by compositing occluders onto images in which objects are not obscured by occluders. In this way, the accuracy of the ground truth information for object positions is improved in the generated training data. As a result, the accuracy of object detection by inference using the object detection model is improved.

[0079] Furthermore, by combining the object image 30 with the occlusion image 40 to generate an object occlusion image 50, it is possible to generate an object occlusion image 50 that reproduces various variations of the state in which the first object 32 is hidden by the occlusion from a single object image 30. If an image of an object hidden by an occlusion were to be generated by actually taking a photograph, it would be practically difficult to comprehensively reproduce and photograph all states in which the object is hidden by the occlusion. According to this disclosure, an object occlusion image 50 that reproduces various variations of the state in which the first object 32 is hidden by the occlusion can be easily generated. As a result, an improvement in the detection accuracy of objects that are at least partially hidden is expected.

[0080] The diagrams illustrating the embodiments described herein are schematic. Dimensions and proportions shown in the drawings do not necessarily correspond to actual dimensions.

[0081] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. These are also to be understood as being included within the scope of this disclosure.

[0082] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0083] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0084] Vehicles relating to this disclosure may include, for example, automobiles, industrial vehicles, railway vehicles, residential vehicles, or fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture or construction. Industrial vehicles may include, for example, forklifts or golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, or road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. Vehicles of the same nature may be included in multiple classifications.

[0085] While embodiments of the image processing method using the image processing system 1 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the apparatus, but also a storage medium on which the program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0086] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. Moreover, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0087] In one embodiment, (1) the electronic device includes a learning unit that generates an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by performing learning using learning data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with correct information on the position of the first object in the first image.

[0088] (2) In the electronic device described in (1) above, at least one of the first object or the second object may be a moving body. At least one of the first light source or the second light source may be a lighting device possessed by the moving body.

[0089] (3) The electronic device described in (1) or (2) above may further include a generation unit that generates the first image by combining a third image which shows the entirety of the first object having the first light source with an image of the first occluding object.

[0090] (4) In the electronic device described in (3) above, at least some of the pixels of the third image may be virtually generated pixels.

[0091] (5) In the electronic device described in any one of (1) to (4) above, at least some of the pixels of the first image may be virtually generated pixels.

[0092] (6) In the electronic device described in any one of (1) to (5) above, the first shield may include at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0093] (7) In the electronic device described in any one of (1) to (6) above, the second shield may include at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

[0094] (8) In the electronic device described in any one of (1) to (7) above, at least one of the first image or the second image may include an infrared image captured from infrared light.

[0095] (9) In the electronic device described in any one of (1) to (8) above, at least one of the first image or the second image may include a visible image captured in visible light.

[0096] (10) In the electronic device described in any one of (1) to (9) above, the first image may include an image showing the state in which at least one feature part of the first object is hidden by the first occluder.

[0097] (11) In the electronic device described in any one of (1) to (10) above, the second image may include an image showing the state in which at least one feature portion of the second object is hidden by the second occlusion.

[0098] In one embodiment, (12) the image processing method includes the electronic device performing learning using training data that associates a first image showing a state in which at least a part of a first object having a first light source is hidden by a first occluder with correct information on the position of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a part of a second object having a second light source is hidden by a second occluder.

[0099] In one embodiment, (13) the image processing program causes the electronic device to perform learning using training data that associates a first image showing a state in which at least a part of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image, thereby generating an object detection model that detects the position of a second object from a second image showing a state in which at least a part of a second object having a second light source is hidden by a second occluder.

[0100] 1 Image Processing System 10 Electronic Equipment (12: Acquisition Unit, 14: Generation Unit, 16: Learning Unit, 18: Output Unit, 22: Memory Unit, 24: Inference Unit) 20 Camera 30 Object Image (32: First Object, 34: First Light Source, 36: First Illumination Area) 40 Obstruction Image (42: First Obstruction) 50 Object Obstruction Image 60 Detection Image (62: Second Object, 64: Second Light Source, 66: Second Illumination Area, 68: Second Obstruction)

Claims

1. An electronic device comprising a learning unit that generates an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by performing learning using learning data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image.

2. The electronic device according to claim 1, wherein at least one of the first object or the second object is a mobile body, and at least one of the first light source or the second light source is a lighting device of the mobile body.

3. The electronic device according to claim 1 or 2, further comprising a generation unit that generates the first image by combining an image of the first occluding object with a third image which shows the entirety of the first object having the first light source.

4. The electronic device according to claim 3, wherein at least some of the pixels in the third image are virtually generated pixels.

5. The electronic device according to any one of claims 1 to 4, wherein at least some of the pixels of the first image are virtually generated pixels.

6. The electronic device according to any one of claims 1 to 5, wherein the first shield includes at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

7. The electronic device according to any one of claims 1 to 6, wherein the second shield includes at least one of a building, a wall, a fence, a guardrail, a plant, a street tree, a traffic light, a sign, a billboard, a vehicle, an aircraft, a pedestrian, an animal, smoke, fog, or rain.

8. The electronic device according to any one of claims 1 to 7, wherein at least one of the first image or the second image includes an infrared image obtained by capturing infrared light.

9. The electronic device according to any one of claims 1 to 8, wherein at least one of the first image or the second image includes a visible image captured in visible light.

10. The electronic device according to any one of claims 1 to 9, wherein the first image includes an image showing the state in which at least one feature portion of the first object is hidden by the first occluding object.

11. The electronic device according to any one of claims 1 to 10, wherein the second image includes an image showing a state in which at least one feature portion of the second object is obscured by the second occluding object.

12. An image processing method comprising an electronic device generating an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by performing training using training data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with ground truth information of the position of the first object in the first image.

13. An image processing program that generates an object detection model for detecting the position of a second object from a second image in which at least a portion of a second object having a second light source is hidden by a second occluder, by having an electronic device perform training using training data that associates a first image in which at least a portion of a first object having a first light source is hidden by a first occluder with the correct position information of the first object in the first image.

Images