Detection program, detection method, detection device, and training program

Abstract

This makes it possible to suppress a decrease in vehicle detection when images are taken under predetermined conditions. [Solution] The detection device acquires an input image from a camera and inputs the input image into a first detection model that performs object detection, thereby obtaining a first detection result. The detection device uses the parameters of the first detection model as initial values ​​and inputs the input image into multiple detection models that have been trained on different training datasets, thereby obtaining multiple detection results. The detection device generates a detection result that aggregates the first detection result and the multiple detection results.

Description

[Technical Field]

[0001] This invention relates to a detection program, etc. [Background technology]

[0002] <000000> Conventional detection devices for detecting objects from images captured by cameras include YOLO (You Only Look Once), Faster R-CNN (Faster Region-based CNN), and SSD (Single Shot Multibox Detector). Such detection devices are used in technological fields such as autonomous driving.

[0003] For example, when an image is input to the detection device, the "bounding box" and "category" of an object (e.g., a vehicle) present in the image data are identified. Figure 19 is a diagram illustrating the prior art. In the example shown in Figure 19, by inputting image data Im1 to the detection device, bounding box 10a and bounding box 10b are identified. The detection device also identifies that the category of bounding boxes 10a and 10b is "car". [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ben Nassi, et al., “Phantom of the ADAS: Securing Advanced Driver-Assistance Systems from Split-Second Phantom Attacks” [online], [Accessed July 17, 2024], Internet<URL:https: / / dl.acm.org / doi / 10.1145 / 3372297.<3423359> [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the conventional technology described above has the problem that the vehicle detection rate decreases when images are taken under predetermined conditions.

[0006] In the following explanation, vehicles equipped with red or blue flashing lights will be referred to as "emergency vehicles." Emergency vehicles include police vehicles, fire engines, ambulances, etc. The flashing of red or blue lights (or alternating red and blue lights) will simply be referred to as "the flashing lights are on."

[0007] For example, the image taken under the specified conditions described above is an image taken at night while the warning lights of an emergency vehicle are illuminated.

[0008] In one aspect, the present invention aims to provide a program, method, and apparatus that can prevent a decrease in the vehicle detection rate for images taken under predetermined conditions. [Means for solving the problem]

[0009] In the first approach, the computer performs the following steps: The computer acquires an input image from a camera and inputs it into a first detection model that performs object detection, thereby obtaining a first detection result. The computer uses the parameters of the first detection model as initial values ​​and inputs the input image into multiple detection models trained on different training datasets, thereby obtaining multiple detection results. The computer generates a detection result that aggregates the first detection result and the multiple detection results. [Effects of the Invention]

[0010] This makes it possible to suppress a decrease in vehicle detection when images are taken under predetermined conditions. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a diagram (1) illustrating the problems of the prior art. [Figure 2]Figure 2 is a diagram (2) illustrating the problems of the prior art. [Figure 3] Figure 3 is a diagram illustrating the Caracetamol framework. [Figure 4] Figure 4 shows an example of an MFA image. [Figure 5] Figure 5 shows an example of a GAN image. [Figure 6] Figure 6 is a diagram illustrating an example of the NMS algorithm. [Figure 7] Figure 7 shows an example of the evaluation results. [Figure 8] Figure 8 is a diagram that provides supplementary explanation for the training phase process (2). [Figure 9] Figure 9 is a functional block diagram showing the configuration of the detection device according to this embodiment. [Figure 10] Figure 10 is a functional block diagram (1) showing the configuration of the training device according to this embodiment. [Figure 11] Figure 11 is a functional block diagram (2) showing the configuration of the training device according to this embodiment. [Figure 12] Figure 12 is a flowchart showing the processing procedure of the detection device according to this embodiment. [Figure 13] Figure 13 is a flowchart (1) showing the processing procedure of the training device according to this embodiment. [Figure 14] Figure 14 is a flowchart (2) showing the processing procedure of the training device according to this embodiment. [Figure 15] Figure 15 shows an example of the configuration of an automated driving system. [Figure 16] Figure 16 shows an example of a computer hardware configuration that achieves similar functionality to the detection device described in the embodiment. [Figure 17] Figure 17 is a diagram (1) showing an example of a computer hardware configuration that achieves similar functions to the training device according to the embodiment. [Figure 18] Figure 18 is a diagram (2) showing an example of a computer hardware configuration that achieves similar functions to the training device according to the embodiment. [Figure 19] FIG. 19 is a diagram for explaining the prior art.

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the program, method, and apparatus disclosed in the present application will be described in detail based on the drawings. Note that the present invention is not limited by this embodiment.

EXAMPLE

[0013] Before describing this embodiment, the problems of the prior art detection device will be described more specifically.

[0014] As described above, for an image captured by a camera at night and captured during a period when the warning light of an emergency vehicle is on, when the prior art detection device detects an object, there is a problem that the detection rate decreases. Note that an image captured at night is an image whose average brightness is less than a threshold value (for example, 60). In the following description, an image captured by a camera at night and captured during a period when the warning light of an emergency vehicle is on will be referred to as an "image captured under predetermined conditions".

[0015] FIG. 1 is a diagram (1) for explaining the problems of the prior art. Image Im2a in FIG. 1 is an image of an emergency vehicle with the warning light off taken at night. The RGB (Red Green Blue) histogram for image Im2a is as shown in graph G2a. The horizontal axis of graph G2a is the axis corresponding to Pixel Intensity, and the vertical axis is the axis corresponding to Nourmalized counts.

[0016] When image Im2a is input to the prior art detection device, the bounding box 11a of the vehicle and the category "car" are specified. Also, the confidence score by the detection device is 0.96.

[0017] On the other hand, image Im2b in Figure 1 is an image taken under predetermined conditions. The RGB histogram for image Im2b is shown in graph G2b. The explanation of the horizontal and vertical axes in graph G2b is the same as the explanation given for graph G2a. Comparing graph G2a and graph G2b, the distributions of the Red, Green, and Blue channels are significantly different.

[0018] When image Im2b is input to a conventional detection device, the vehicle's bounding box 11b and the category "car" are identified. The confidence score from the detection device is 0.06.

[0019] Here, the confidence score obtained when an image is input to the detection device and a bounding box is identified represents the probability that the detected object actually exists. To prevent false positives, if the confidence score is lower than a pre-set threshold (for example, 0.7), the detection result is ignored, and therefore the vehicle detected in image Im2b will be ignored.

[0020] In other words, while conventional detection devices can detect vehicles from image Im2a, they are unable to properly detect vehicles included in image Im2b, which is taken under predetermined conditions.

[0021] Figure 2 is a diagram (2) illustrating the problems of the prior art. Image Im3a in Figure 2 is an image of an emergency vehicle with its warning lights illuminated, taken during the daytime. When image Im3a is input to the prior art detection device, the vehicle's bounding box 12a and the category "car" are identified. The confidence score of the detection device is 0.95.

[0022] On the other hand, image Im3b in Figure 2 is an image taken under predetermined conditions. When image Im3b is input to a conventional detection device, the vehicle's bounding box 12b and the category "car" are identified. The confidence score obtained by the detection device is 0.59.

[0023] In other words, while conventional detection devices can detect vehicles from image Im3a, they are unable to properly detect vehicles included in image Im3b, which is taken under predetermined conditions.

[0024] Although not described in this specification, the inventors conducted verifications from various perspectives other than those shown in Figures 1 and 2, and discovered that the detection rate of the conventional detection device decreased for images taken under predetermined conditions.

[0025] Next, this embodiment will be described. In this embodiment, the process will be explained in two parts: the detection phase and the training phase. The detection phase is the phase in which objects are detected from an image. The training phase is the phase in which each model used in the detection phase is trained.

[0026] First, the detection phase processing will be explained. In this embodiment, the detection device performs the detection phase processing. In the following description, the detection device according to this embodiment will be referred to as "detection device 100". The detection device 100 is connected to the camera by wire or wireless. The camera captures images and outputs the image data to the detection device 100. The image data includes time-series frames. One frame is one still image contained in the image data.

[0027] The detection device 100 uses a "Caracetamol framework" to detect objects from a frame. Figure 3 is a diagram illustrating the Caracetamol framework. As shown in Figure 3, the Caracetamol framework 20 has a detection model 30, a classification layer 21, a detectors layer 22, and a combiner layer 23. In the example shown in Figure 3, frame F1 is used for explanation.

[0028] Detection model 30 is a model trained using the same training dataset as detection models used in conventional detection devices. Detection model 30 is a model such as NN (Neural Network) or SVM (Support-Vector Machine). For example, when training detection model 30, the explanatory variable is the frame (image), and the target variable is the coordinates of the bounding box and the class label (category). Similarly, detection models 31, 32, and 33, which will be described later, are also models such as NN and SVM.

[0029] When frame F1 is input, the detection model 30 outputs the detection results to the aggregation layer 23. For example, the detection results include information about the bounding box, confidence score, and category identified from frame F1.

[0030] The classification layer 21 has a classification model 21a. The classification model 21a is a model that determines whether or not a flash is present in frame F1. A flash is the light of a warning light, etc. If the classification model 21a determines that a flash is present in frame F1, the classification layer 21 outputs frame F1 to the detection layer 22.

[0031] On the other hand, in the classification layer 21, if the classification model 21a determines that no flash is present in frame F1, frame F1 is not output to the detection layer 22. In this case, frame F1 is processed only by the detection model 30.

[0032] The detection layer 22 includes a denoiser model 22a and detection models 31, 32, and 33. Frames F1 received from the classification layer 21 are input to detection model 31, denoiser model 22a, and detection model 33, respectively.

[0033] The removal model 22a removes the flash contained in frame F1 and outputs the flash-removed "frame F1'" to the detection model 32.

[0034] The detection model 31 is a fine-tuned model using the parameters of the trained detection model 30 as initial values ​​and the first training dataset. When frame F1 is input, the detection model 31 outputs the detection result to the aggregation layer 23. For example, the detection result of the detection model 31 includes information about the bounding box, confidence score, and category identified from frame F1.

[0035] The first training dataset includes nighttime images (hereinafter referred to as MFA images) into which a simulated flash has been synthesized by a user through manual operation. For example, a user generates an MFA image using an image generator that synthesizes a simulated light mimicking a warning light. Figure 4 shows an example of an MFA image. In the example shown in Figure 4, the MFA image Im4b is generated by synthesizing a simulated light 5 with the nighttime image Im4a.

[0036] Returning to the explanation of Figure 3, the detection model 32 is a fine-tuned model using the first training dataset, with the parameters of the trained detection model 30 as initial values. The detection model 32 receives the input of the frame F1' from the removal model 22a, from which the flash has been removed. Upon receiving the frame F1', the detection model 32 outputs the detection result to the aggregation layer 23. For example, the detection result of the detection model 32 includes information about the bounding box, confidence score, and category identified from the frame F1'.

[0037] The detection model 33 is a fine-tuned model using the parameters of the trained detection model 30 as initial values ​​and the second training dataset. When frame F1 is input, the detection model 33 outputs the detection result to the aggregation layer 23. For example, the detection result of the detection model 33 includes information about the bounding box, confidence score, and category identified from frame F1.

[0038] The second training dataset includes nighttime images (hereinafter referred to as GAN images) with pseudo-flashes synthesized using a trained CycleGAN (Cycle-Consistent Generative Adversarial Networks). For example, a GAN image is generated by inputting a nighttime image into the generator of a trained CycleGAN. Figure 5 shows an example of a GAN image. In the example shown in Figure 5, pseudo-lights 6 are automatically synthesized into the nighttime image. Pseudo-lights 6 are such as the light from a warning light.

[0039] Returning to the explanation of Figure 3, the aggregation layer 23 aggregates the detection results of detection models 30-33 and outputs the aggregated results. The detection results of detection models 30-33 each include information about the bounding box, confidence score, and category identified from frame F1. The processing of the aggregation layer 23 differs depending on whether frame F1 contains a flash or not.

[0040] First, let's explain the case where frame F1 contains a flash. If frame F1 contains a flash, frame F1 is output from the classification layer 21 to the detection layer 22, and the aggregation layer 23 obtains the detection results from detection models 30 to 33. The aggregation layer 23 aggregates the detection results from detection models 30 to 33 using the NMS (Non Maximum Suppression) algorithm and outputs the aggregated result to the higher-level processing unit, etc.

[0041] Figure 6 is a diagram illustrating an example of the NMS algorithm. First, the explanation will be given using bounding boxes bb1-1, bb1-2, bb1-3, and bb1-4 identified for object Ob1. Bounding boxes bb1-1 to bb1-4 are bounding boxes corresponding to the detection results of detection models 30 to 33.

[0042] The aggregation layer 23 identifies the bounding box from bounding boxes bb1-1 to bb1-4 that has a confidence score equal to or greater than the threshold, and that has the highest confidence score. For example, the bounding box that has the highest confidence score and is equal to or greater than the threshold is described as bounding box bb1-4.

[0043] The aggregation layer 23 deletes bounding box bb1-1 if the overlap between bounding box bb1-4 and bounding box bb1-1 exceeds a threshold (significant overlap). The aggregation layer 23 deletes bounding box bb1-2 if the overlap between bounding box bb1-4 and bounding box bb1-2 exceeds a threshold (significant overlap). The aggregation layer 23 deletes bounding box bb1-3 if the overlap between bounding box bb1-4 and bounding box bb1-3 exceeds a threshold (significant overlap). As a result, the detection results for object Ob1 that remain are bounding box bb1-4 and the confidence score and category of bounding box bb1-4.

[0044] The explanation will be given using bounding boxes bb2-1, bb2-2, and bb2-3 identified for object Ob2. Bounding boxes bb2-1 to bb2-3 are bounding boxes corresponding to the detection results of detection models 30 to 32.

[0045] The aggregation layer 23 identifies the bounding box with the highest confidence score from bounding boxes bb2-1 to bb2-3, where the confidence score is equal to or greater than the threshold. For example, if there are no bounding boxes with a confidence score equal to or greater than the threshold, the aggregation layer 23 deletes bounding boxes bb2-1 to bb2-3.

[0046] The aggregation layer 23 repeats the above process for the bounding boxes of the other objects Ob3, Ob4, Ob5, Ob6, Ob7, and Ob8, thereby obtaining aggregation result 23-1. Note that the bounding box for object Ob7 is a false positive.

[0047] For example, in aggregate result 23-1, bounding box bb1-4 is set for object Ob1. Bounding box bb3-1 is set for object Ob3. Bounding box bb5-1 is set for object Ob5. Bounding box bb6-1 is set for object Ob6.

[0048] Next, we will explain the case where frame F1 does not contain a flash. If frame F1 does not contain a flash, frame F1 is not output from the classification layer 21 to the detection layer 22, so the aggregation layer 23 obtains the detection result of the detection model 30. In this case, the aggregation layer 23 outputs the detection result of the detection model 30 as the aggregated result.

[0049] The Caracetamol framework used by the detection device 100 has been described above. As described above, in the detection device 100, if the frame contains a flash, the classification layer 21 inputs the frame to the detection layer 22, and the aggregation layer 23 aggregates the detection results of the detection models 30 to 31 to obtain the final detection result. This makes it possible to suppress a decrease in the object detection rate for images taken under predetermined conditions.

[0050] Figure 7 shows an example of the evaluation results. Here, we will explain the case where the evaluation targets are the conventional detection model (detection model 30) and the detection models 30 to 33 of the detection device 100. Each detection model is designated as YOLO.

[0051] Based on the confidence scores obtained by inputting a test dataset into a conventional detection model and detection device 100, Average Confidence, Min Confidence, Max Confidence, and Range were calculated. The test dataset includes multiple frames captured under predetermined conditions. Average Confidence is the average value of the confidence scores. Min Confidence is the minimum value of the confidence scores. Max Confidence is the maximum value of the confidence scores. Range is the range from Min Confidence to Max Confidence.

[0052] For example, with the conventional technology, the Average Confidence was "0.4696", Min Confidence was "0.1690", Max Confidence was "0.8398", and Range was "0.6708". On the other hand, with the detection device 100, the Average Confidence was "0.7968", Min Confidence was "0.6799", Max Confidence was "0.9339", and Range was "0.2851". In other words, compared to the conventional technology, the detection device 100 is able to suppress the decrease in the object detection rate for images taken under predetermined conditions.

[0053] Although the details are omitted here, the inventors employed and evaluated Faster R-CNN, SSD, and other detection models. Similar to the case using YOLO, the detection device 100 can suppress the decrease in the object detection rate for images captured under predetermined conditions compared to conventional technology.

[0054] The above explains the processing of the detection phase.

[0055] Next, we will explain the processing of the training phase (1). For example, the second training dataset mentioned above includes nighttime images from which a pseudo-flash has been synthesized using a trained CycleGAN generator.

[0056] In the training phase (1), the training device for training CycleGAN is described. The training device for training CycleGAN in this embodiment is referred to as "training device 200". For example, training device 200 executes preprocessing, training, and generation in order.

[0057] The preprocessing performed by the training device 200 is described below. The training device 200 extracts nighttime images (hereinafter referred to as nighttime images) from images included in a pre-prepared dataset. The pre-prepared dataset is the BDD100K (Berkeley) dataset, etc. For example, the training device 200 acquires images from the dataset and calculates the average brightness of the images. The training device 200 extracts multiple nighttime images by repeatedly performing the process of extracting images with an average brightness less than a threshold (for example, 60) as nighttime images.

[0058] Next, the training process performed by the training device 200 will be described. The training device 200 acquires multiple images without flash from a pre-prepared dataset and sets them as the first group. The training device 200 acquires multiple images with flash from a pre-prepared dataset and sets them as the second group.

[0059] For example, CycleGAN consists of two generators and two discriminators. The two generators are designated as the first generator and the second generator. The two discriminators are designated as the first discriminator and the second discriminator.

[0060] The first generator generates a non-flash image from a flash image (night image of the second group). The second generator generates a flash image from a non-flash image (night image of the first group). The first discriminator is a discriminator that distinguishes between a non-flash image (night image of the first group) and a non-flash image generated by the first generator. The second discriminator is a discriminator that distinguishes between a flash image (night image of the second group) and a flash image generated by the second generator.

[0061] The training device 200 trains the first generator, the second generator, the first discriminator, and the second discriminator using the first group of images and the second group of images. An example of the training performed by the training device 200 is shown below, but is not limited to this. For example, training can be performed using the technique described in the document "J.-Y. Zhu, T. Park, P. Isola, and AA Efros, “Unpaired image-to-image translation using cycle-consistent ad versarial networks,” in Computer Vision (ICCV), 2017 IEEE International Conference on, 2017."

[0062] For example, the training device 200 inputs images with flash from the first group into the first generator to generate images without flash. The training device 200 inputs images without flash from the second group into the second generator to generate images with flash. The training device 200 evaluates each generation result using the first and second classifiers and calculates various losses (opposite loss, cycle consistency loss, identity loss, etc.). The training device 200 updates the parameters of the first generator, second generator, first classifier, and second classifier to minimize these losses.

[0063] The training device 200 trains the first generator, second generator, first discriminator, and second discriminator by repeatedly executing the above process. The trained second generator becomes the generator that produces GAN images from the second training dataset. On the other hand, the trained first generator becomes the generator corresponding to the removal model 22a explained in Figure 3.

[0064] Next, the generation process performed by the training device 200 will be described. The training device 200 generates multiple GAN images by inputting multiple nighttime images extracted in the preprocessing step into a trained second generator. The training device 200 then generates a second training dataset by associating the generated multiple GAN images with the correct ground truth data. For example, the training device 200 may accept the correct ground truth data corresponding to the GAN images from an external source.

[0065] As described above, the training device 200 trains the first and second generators using multiple images with and without flash. The training device 200 can generate multiple GAN images by inputting multiple night images extracted from a pre-prepared dataset into the second generator, and can train the detection model 33 using the second training dataset. Furthermore, by performing the above training, the first generator (removal model 22a) can be generated.

[0066] The above explains the processing (1) of the training phase.

[0067] Next, we will explain the training phase process (2). The training device that performs the training phase process (2) will be referred to as "training device 300". In training phase (2), we will explain the case in which training device 300 trains the detection models 30, 31, 32, and 33 described in Figure 3.

[0068] Figure 8 is a diagram to supplement the explanation of the training phase process (2). The training device 300 trains the detection model 30 using a pre-prepared training dataset 40. The training dataset 40 has multiple training data. The training data of the training dataset 40 is a combination of input data and ground truth data. The input data is an image containing an object. The ground truth data is the coordinates of the bounding box and the class label (category).

[0069] The training device 300 trains the detection model 30 by repeatedly performing a process to update the parameters of the detection model 30 using the training dataset 40, so that the output data when input data is input to the detection model 30 approaches the correct data.

[0070] The training device 300 fine-tunes the detection model 31 (or detection model 32) using a pre-prepared first training dataset 41. The first training dataset 41 contains multiple training data. The training data of the first training dataset 41 is a combination of input data and ground truth data. The input data is an MFA image. The ground truth data consists of bounding box coordinates and class labels (categories).

[0071] For example, the training device 300 sets the parameters of the trained detection model 30 as the initial values ​​for the parameters of the detection model 31. The training device 300 fine-tunes the detection model 31 by repeatedly performing a process to update the parameters of the detection model 31 using the first training dataset 41 so that the output data when input data is input to the detection model 31 approaches the correct data.

[0072] The training device 300 fine-tunes the detection model 33 using the second training dataset 42 generated in the training phase (1) described above. The second training dataset 42 has multiple training data sets. The training data in the second training dataset 42 is a combination of input data and ground truth data. The input data is a GAN image. The ground truth data consists of bounding box coordinates and class labels (categories).

[0073] For example, the training device 300 sets the parameters of the trained detection model 30 as the initial values ​​for the parameters of the detection model 33. The training device 300 fine-tunes the detection model 33 by repeatedly performing a process to update the parameters of the detection model 33 using the second training dataset 42 so that the output data when input data is input to the detection model 33 approaches the correct data.

[0074] The detection models 30-33 trained in training phase (2) are used by the detection device 100 in the detection phase described in Figure 3.

[0075] The above describes training phase (2). In training phase (1), the training device 200 performed the training, and in training phase (2), the training device 300 performed the training. However, training devices 200 and 300 may be the same training device.

[0076] Next, an example of the configuration of the detection device 100 described above will be explained. Figure 9 is a functional block diagram showing the configuration of the detection device according to this embodiment. As shown in Figure 9, this detection device 100 is connected to the camera 50 via the network 55. The detection device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0077] The communication unit 110 performs data communication with the camera 50 via the network 55. The communication unit 110 may also receive training datasets 60, etc., from external devices. For example, the communication unit 110 receives video data from the camera 50.

[0078] The input unit 120 inputs various types of information to the control unit 150.

[0079] The display unit 130 displays the information output from the control unit 150.

[0080] The memory unit 140 includes a Caracetamol framework 20 and a video buffer 141. The memory unit 140 is a memory, etc.

[0081] The Caracetamol framework 20 is the data for the Caracetamol framework 20 described in Figure 3. The Caracetamol framework 20 has a detection model 30, a classification layer 21, a detection layer 22, and an aggregation layer 23. The classification layer 21 has a classification model 21a. The detection layer 22 has a removal model 22a and detection models 31, 32, and 33. Other details regarding the Caracetamol framework 20 are the same as those described in Figure 3. The Caracetamol framework 20 is read by the control unit 150, which will be described later, and executed.

[0082] The video buffer 141 is a buffer that holds video data captured by the camera 50. The video data consists of time-series frames.

[0083] The control unit 150 includes an acquisition unit 151 and a detection unit 152. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0084] The acquisition unit 151 acquires video data from the camera 50 via the communication unit 110. The acquisition unit 151 stores the acquired video data in the video buffer 141.

[0085] The detection unit 152 reads the Caracetamol framework 20 and performs object detection using the Caracetamol framework 20. The detection unit 152 acquires frames from the video buffer 141 and inputs the acquired frames to the classification layer 21, thereby obtaining the aggregated detection result from the aggregation layer 23. The detection unit 152 repeatedly executes the above process to obtain a time-series detection result.

[0086] Next, an example configuration of the training device 200 that performs the training phase (1) will be described. Figure 10 is a functional block diagram (1) showing the configuration of the training device according to this embodiment. As shown in Figure 10, this training device 200 has a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0087] The communication unit 210 performs data communication with an external device. The communication unit 110 may receive data set 241, first group data 243, second group data 244, etc., from the external device.

[0088] The input unit 220 inputs various types of information to the control unit 250.

[0089] The display unit 230 displays the information output from the control unit 250.

[0090] The memory unit 240 contains the second training dataset 42, dataset 241, night image table 242, first group data 243, second group data 244, and CycleGAN 245. The memory unit 240 is a memory, etc.

[0091] The second training dataset 42 is a training dataset generated by the control unit 250.

[0092] Dataset 241 contains multiple images. Each image in Dataset 241 includes a vehicle, etc. Dataset 241 is the BDD100K (Berkeley) dataset, etc.

[0093] The night image table 242 has multiple night images extracted from the dataset 241 by the control unit 250.

[0094] The first group data 243 contains multiple images without flash. For example, the flash-free images of the first group data 243 are obtained in advance from the BDD100K (Berkeley) dataset, etc.

[0095] The second group data 244 contains multiple images with flash. For example, the multiple images with flash in the second group data 244 are obtained in advance from the YouTube® dataset, etc.

[0096] CycleGAN245 has a first generator, a second generator, a first discriminator, and a second discriminator. Further details regarding CycleGAN245 are the same as described above.

[0097] The control unit 250 includes a preprocessing unit 251, a training unit 252, and a generation unit 253. The control unit 250 is a CPU, GPU, etc.

[0098] The preprocessing unit 251 extracts nighttime images from the images included in the dataset 241 and registers the extracted nighttime images in the nighttime image table 242. For example, the preprocessing unit 251 extracts images with an average brightness less than a threshold (e.g., 60) as nighttime images. Further details regarding the preprocessing unit 251 correspond to the preprocessing described in training phase (1).

[0099] The training unit 252 trains CycleGAN245 using the first group data 243 and the second group data 244. CycleGAN245 has a first generator, a second generator, a first discriminator, and a second discriminator.

[0100] For example, the training unit 252 inputs a flashed image from the first group data 243 into the first generator to generate an image without flash. The training unit 252 inputs an image without flash from the second group data 244 into the second generator to generate an image with flash. The training unit 252 evaluates each generation result using the first and second classifiers and calculates various losses (opposite loss, cycle consistency loss, identity loss, etc.). The training unit 252 updates the parameters of the first and second generators, the first classifier, and the second classifier to minimize these losses.

[0101] Further explanation regarding training unit 252 corresponds to the training process described in training phase (1).

[0102] The generation unit 253 inputs each night image registered in the night image table 242 into the trained CycleGAN245 (second generator) to generate images with flashes. The generated images with flashes are used as input data for the second training dataset. For example, the generation unit 253 may display the generated images on the display unit 230 to receive correct answer data from the user, and register the pairs of input data and correct answer data in the second training dataset 42.

[0103] Further details regarding the generation unit 253 correspond to the generation process described in training phase (2).

[0104] Next, an example of the configuration of the training device 300 that performs the training phase (2) will be described. Figure 11 is a functional block diagram (2) showing the configuration of the training device according to this embodiment. As shown in Figure 11, this training device 300 has a communication unit 310, an input unit 320, a display unit 330, a storage unit 340, and a control unit 350.

[0105] The communication unit 310 performs data communication with the detection device 100, the training device 200, and other external devices. The communication unit 310 may receive the training dataset 40 and the first training dataset 41 from external devices. The communication unit 310 receives the second training dataset 42 from the training device 200.

[0106] The input unit 320 inputs various types of information to the control unit 350.

[0107] The display unit 330 displays the information output from the control unit 350.

[0108] The memory unit 340 contains detection models 30, 31(32), 33, a training dataset 40, a first training dataset 41, and a second training dataset 42. The memory unit 340 is a memory, etc.

[0109] The detection model 30 is a model trained by the control unit 350 using the same training dataset 40 as detection models used in conventional detection devices.

[0110] Detection models 31 and 32 are models that are fine-tuned by the control unit 350 using the first training dataset 41, with the parameters of the trained detection model 30 as initial values.

[0111] The detection model 33 is a model that is fine-tuned by the control unit 350 using the second training dataset 42, with the parameters of the trained detection model 30 as initial values.

[0112] The explanations for training dataset 40, the first training dataset 41, and the second training dataset 42 are the same as those given in Figure 8.

[0113] The control unit 350 includes an acquisition unit 351 and a training unit 352. The control unit 350 is a CPU, GPU, etc.

[0114] The acquisition unit 351 acquires the learning dataset 40 and the first learning dataset from an external device via the communication unit 310 and stores them in the storage unit 340. The acquisition unit 351 also acquires the second learning dataset 42 from the training device 200 via the communication unit 310 and stores it in the storage unit 340.

[0115] The training unit 352 trains the detection models 30 to 33. First, the training unit 352 trains the detection models 30 by repeatedly updating the parameters of the detection models 30 using the training dataset 40, so that the output data when input data is input to the detection models 30 approaches the correct data.

[0116] Alternatively, a pre-trained detection model 30 may be stored in the memory unit 340.

[0117] The training unit 352 sets the parameters of the trained detection model 30 as initial values ​​for the parameters of the detection model 31 (32). Using the first training dataset 41, the training unit 352 fine-tunes the detection model 31 by repeatedly updating the parameters of the detection model 31 so that the output data when input data is input to the detection model 31 approaches the correct data.

[0118] The training unit 352 sets the parameters of the trained detection model 30 as initial values ​​for the parameters of the detection model 33. Using the second training dataset 42, the training unit 352 fine-tunes the detection model 33 by repeatedly updating the parameters of the detection model 33 so that the output data when input data is input to the detection model 33 approaches the correct data.

[0119] The training device 300 outputs the trained detection models 30 to 33 to the detection device 100.

[0120] Next, an example of the processing procedure of the detection device 100 according to this embodiment will be described. Figure 12 is a flowchart of the processing procedure of the detection device according to this embodiment. As shown in Figure 12, the acquisition unit 151 of the detection device 100 acquires video data from the camera 50 and stores it in the video buffer 141 (step S101).

[0121] The detection unit 152 of the detection device 100 acquires frames from the video buffer 141 and inputs the frames to the detection model 30 and the classification layer 21 of the Caracetamol framework 20 (step S102). The detection model 30 outputs the detection results to the aggregation layer 23 (step S103).

[0122] The classification model 21a of the classification layer 21 outputs the frame to the detection layer 22 (step S105) if the frame contains a flash (step S104, Yes).

[0123] Each detection model 31-33 in the detection layer 22 outputs its detection result to the aggregation layer 23 (step S106). The aggregation layer 23 aggregates the detection results from each detection model 30-33 (step S107) and proceeds to step 110.

[0124] On the other hand, in step S104, if the frame does not contain a flash (step S104, No), the classification model 21a suppresses outputting the frame to the detection layer 22 (step S108). The aggregation layer 23 outputs the detection result of the detection model 30 as the aggregation result (step S109) and proceeds to step S110.

[0125] The detection unit 152 obtains the aggregated results from the Caracetamol framework 20 (step S110). The detection unit 152 outputs the aggregated results as the detection result (step S111). If the detection unit 152 wishes to continue processing (step S112, Yes), it proceeds to step S102. On the other hand, if the detection unit 152 does not wish to continue processing (step S112, No), it terminates the detection process.

[0126] Next, an example of the processing procedure of the training device 200 according to this embodiment will be described. Figure 13 is a flowchart (1) showing the processing procedure of the training device according to this embodiment.

[0127] As shown in Figure 13, the preprocessing unit 251 of the training device 200 extracts nighttime images from the dataset 241 and registers them in the nighttime image table (step S201).

[0128] The training unit 252 of the training device 200 trains CycleGAN245 based on the first group data 243 and the second group data 244 (step S202).

[0129] The generation unit 253 of the training device 200 generates night images with flash by generating night images from the night image table using the second generator of the trained CycleGAN245 (step S203). The generation unit 253 generates a second training dataset 42 using the generated night images with flash (step S204).

[0130] Next, an example of the processing procedure of the training device 300 according to this embodiment will be described. Figure 14 is a flowchart (2) showing the processing procedure of the training device according to this embodiment.

[0131] As shown in Figure 14, the acquisition unit 351 of the training device 300 acquires the learning dataset 40, the first learning dataset 41, and the second learning dataset 42 from the external device and the training device 200, and stores them in the storage unit (step S301).

[0132] The training unit 352 of the training device 300 trains the detection model 30 based on the training dataset 40 (step S302). The training unit 352 sets the initial values ​​of the trained detection model 30 as the initial values ​​for each detection model 31 to 33 (step S303).

[0133] The training unit 352 trains the detection model 31(32) based on the first training dataset 41 (step S304). The training unit 352 then trains the detection model 33 based on the second training dataset 42 (step S305).

[0134] Next, the effects of the detection device and training devices 200 and 300 according to this embodiment will be described. In the detection device 100, if the classification layer 21 contains a flash in the frame, it inputs the frame to the detection layer 22, and the aggregation layer 23 aggregates the detection results of the detection models 30 to 31 to obtain the final detection result. This makes it possible to suppress a decrease in the detection rate of objects for images taken under predetermined conditions, as explained in Figure 7, for example.

[0135] The training device 200 trains the first and second generators using multiple images with and without flash. The training device 200 can generate multiple GAN images by inputting multiple night images extracted from a pre-prepared dataset into the second generator, and can train the detection model 33 using the second training dataset. Furthermore, by performing the above training, the first generator (removal model 22a) can be generated.

[0136] The training device 300 uses the parameters of the trained detection model 30 as initial values ​​and fine-tunes the detection model 31 (32) using the first training dataset 41. The training device 300 uses the parameters of the trained detection model 30 as initial values ​​and fine-tunes the detection model 33 using the second training dataset 42. By utilizing each detection model trained by the training device 300, the detection rate of emergency vehicles for images taken under predetermined conditions can be improved.

[0137] Incidentally, the detection device 100 described in the above-mentioned embodiment can also be mounted and used in an automatic driving system. Figure 15 shows an example of the configuration of an automatic driving system.

[0138] The autonomous driving system 400 shown in Figure 15 is mounted on a vehicle such as an automobile. For example, the autonomous driving system 400 includes an external sensor 401, a position acquisition unit 402, a GPS (Global Positioning System) receiver 403, a map database 404, an actuator 405, a detection unit 410, and an ECU (Electronic Control Unit) 420.

[0139] External sensors 401 are sensors that detect external conditions, which are information about the vehicle's surroundings. External sensors 401 include cameras, radar, and LiDAR (Laser Imaging Detection and Ranging). The camera is a device that captures images of the vehicle's external conditions. External sensors 401 output the video (time-series frames) captured by the camera to the detection unit 410. External sensors 401 also output the detection results of the external conditions to the ECU 420.

[0140] The position acquisition unit 402 communicates with an information center located outside the vehicle to acquire position data of other autonomous vehicles and other vehicles not operating autonomously, and outputs the acquired position data to the ECU 420.

[0141] The GPS receiver 403 measures the vehicle's position (for example, the vehicle's latitude and longitude) by receiving signals from three or more GPS satellites, and outputs the measured result to the ECU 420.

[0142] Map database 404 is a database containing map data. This map data includes road location information, road shape data, and intersection and junction location information.

[0143] The actuator 405 is a device that performs vehicle driving control based on control signals output from the ECU 420. For example, the actuator 405 may include a throttle actuator, a brake actuator, a steering actuator, and so on.

[0144] The detection unit 410 performs the processing corresponding to the detection device 100 (detection unit 152) described in Figure 9. For example, the detection unit 410 acquires frames from the external sensor 401 and inputs the acquired frames to the classification layer 21, thereby obtaining the detection result, which is the aggregated result, from the aggregation layer 23. The detection unit 410 outputs the detection result to the ECU 420.

[0145] The ECU420 controls the vehicle's autonomous driving. Based on information acquired from the external sensor 401, position acquisition unit 402, GPS receiver 403, map database 404, and detection unit 410, the ECU420 calculates an appropriate and safe vehicle route and outputs a control signal to the actuator 405 according to the calculated route.

[0146] By utilizing the detection results obtained from the detection unit 410, the ECU420 can safely support autonomous driving even if a vehicle with flashing warning lights is captured in the camera's view.

[0147] Next, an example of a computer hardware configuration that achieves the same functions as the detection device 100 described above will be explained. Figure 16 is a diagram showing an example of a computer hardware configuration that achieves the same functions as the detection device according to the embodiment.

[0148] As shown in Figure 16, the computer 500 includes a CPU 501 that performs various calculations, an input device 502 that receives data input from the user, and a display 503. The computer 500 also includes a communication device 504 and an interface device 505 that exchange data with cameras, external devices, etc., via a wired or wireless network. Furthermore, the computer 500 includes a RAM 506 for temporarily storing various information and a hard disk drive 507. Each of the devices 501 to 507 is connected to a bus 508.

[0149] The hard disk drive 507 has an acquisition program 507a and a detection program 507b. The CPU 501 reads each program 507a and 507b and loads them into the RAM 506.

[0150] The acquisition program 507a functions as the acquisition process 506a. The detection program 507b functions as the detection process 506b.

[0151] The processing of acquisition process 506a corresponds to the processing of acquisition unit 151. The processing of detection process 506b corresponds to the processing of detection unit 152.

[0152] Furthermore, programs 507a and 506b do not necessarily have to be stored in the hard disk drive 507 from the beginning. For example, each program could be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 500. Then, the computer 500 could read and execute each program 507a and 507b.

[0153] Next, an example of a computer hardware configuration that achieves the same functions as the training device 200 described above will be explained. Figure 17 is Figure (1) showing an example of a computer hardware configuration that achieves the same functions as the training device according to the embodiment.

[0154] As shown in Figure 17, the computer 600 includes a CPU 601 that performs various calculations, an input device 602 that receives data input from the user, and a display 603. The computer 600 also includes a communication device 604 and an interface device 605 that exchange data with external devices via a wired or wireless network. Furthermore, the computer 600 includes a RAM 606 for temporarily storing various information and a hard disk drive 607. Each of these devices 601 to 607 is connected to a bus 608.

[0155] The hard disk drive 607 has a preprocessing program 607a, a training program 607b, and a generation program 607c. The CPU 601 reads each of the programs 607a to 607c and loads them into the RAM 606.

[0156] The preprocessing program 607a functions as the preprocessing process 606a. The training program 607b functions as the training process 606b. The generation program 607c functions as the generation process 606c.

[0157] The processing in the preprocessing process 606a corresponds to the processing in the preprocessing unit 251. The processing in the training process 606b corresponds to the processing in the training unit 252. The processing in the generation process 606c corresponds to the processing in the generation unit 253.

[0158] Furthermore, programs 607a to 606c do not necessarily have to be stored on the hard disk drive 607 from the beginning. For example, each program could be stored on a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 600. Then, the computer 600 could read and execute each program 607a to 607c.

[0159] As shown in Figure 18, the computer 700 includes a CPU 701 that performs various calculations, an input device 702 that receives data input from the user, and a display 703. The computer 700 also includes a communication device 704 and an interface device 705 that exchange data with external devices via a wired or wireless network. Furthermore, the computer 700 includes a RAM 706 for temporarily storing various information and a hard disk drive 707. Each of these devices 701 to 707 is connected to a bus 708.

[0160] The hard disk drive 707 has an acquisition program 707a and a training program 707b. The CPU 701 reads each of the programs 707a to 707c and loads them into the RAM 706.

[0161] The acquisition program 707a functions as the acquisition process 706a. The training program 707b functions as the training process 706b.

[0162] The processing of acquisition process 706a corresponds to the processing of acquisition unit 351. The processing of training process 706b corresponds to the processing of training unit 352.

[0163] Furthermore, programs 707a and 707c do not necessarily have to be stored in the hard disk drive 707 from the beginning. For example, each program could be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 700. Then, the computer 700 could read and execute each program 707a and 707c. [Explanation of Symbols]

[0164] 20 Caracetamol Framework 30, 31, 32, 33 Detection Models 40 training datasets 41. First training dataset 42. Second training dataset 50 Cameras 100 detection device 110,210,310 Communications Department 120,220,320 Input section 130, 230,330 display section 140,240,340 storage section 141 Video buffer 150, 250, 350 Control Unit 151, 351 Acquisition Department 152 Detection unit 241 datasets 242 Nighttime Image Table 243 Group 1 Data 244 Group 2 Data 245 CycleGAN 251 Preprocessing Unit 252,352 Training Department 253 Generation part

Claims

1. The camera acquires an input image, By inputting the input image to the first detection model that performs object detection, the first detection result is obtained. By using the parameters of the first detection model as initial values ​​and inputting the input image into multiple detection models trained on different training datasets, multiple detection results are obtained. The first detection result and a detection result that aggregates the multiple detection results are generated. A detection program characterized by having a computer perform the processing.

2. Determine whether the input image contains a flash, The detection program according to claim 1, characterized in that, if the input image includes a flash, the computer is further instructed to perform a process of inputting the input image to the first detection model and the plurality of detection models in order to obtain the first detection result and the plurality of detection results.

3. The detection program according to claim 2, characterized in that, if the input image does not contain a flash, the computer is further instructed to input the input image to the first detection model to obtain a first detection result, and to output the first detection result as the aggregated detection result.

4. The aforementioned plurality of detection models include a second detection model, a third detection model, and a fourth detection model. The second and third detection models are models trained using a first training dataset in which the parameters of the first detection model are used as initial values ​​and images including flashes generated by manual operation are set. The fourth detection model is a model trained using a second training dataset in which the parameters of the first detection model are used as initial values, and images including flashes are generated by inputting images into a generator included in a trained generative network. By inputting the aforementioned input image into the second detection model, a second detection result is obtained. By inputting the image obtained by removing the flash from the aforementioned input image into the third detection model, a third detection result is obtained. By inputting the aforementioned input image into the fourth detection model, a fourth detection result is obtained. The detection program according to claim 1, characterized in that it causes a computer to perform further processing.

5. The camera acquires an input image, By inputting the input image to the first detection model that performs object detection, the first detection result is obtained. By using the parameters of the first detection model as initial values ​​and inputting the input image into multiple detection models trained on different training datasets, multiple detection results are obtained. The first detection result and a detection result that aggregates the multiple detection results are generated. A detection method characterized by having a computer perform the processing.

6. The camera acquires an input image, By inputting the input image to the first detection model that performs object detection, the first detection result is obtained. By using the parameters of the first detection model as initial values ​​and inputting the input image into multiple detection models trained on different training datasets, multiple detection results are obtained. The first detection result and a detection result that aggregates the multiple detection results are generated. A detection device having a control unit that performs processing.

7. Multiple first images including the vehicle's flash and multiple second images not including the flash are obtained. A generative network having a generator and a discriminator is trained based on the plurality of first images and the plurality of second images. A training program characterized by having a computer perform a process.

8. The training program according to claim 7, characterized in that it extracts a third image whose brightness is below a threshold from a database storing multiple images, and inputs the third image into a generator included in the trained generation network, thereby causing the computer to further perform the process of generating an image including a flash.

9. The training program according to claim 7, wherein the generative network is a GAN (Generative Adversarial Network), and the training process is characterized by training the GAN based on the plurality of first images and the plurality of second images.

Images