Object detection device, learning device, object detection method, learning method, object detection program, and learning program

Abstract

In order to realize highly accurate object detection by concurrently using a background image or other image depending on situation, this object detection device (1) comprises an image acquisition unit (11) that acquires a first image, a calculation unit (12) that calculates a first map from the first image by using a first model, and a detection unit (13) that detects an object with reference to at least the first map. When the image acquisition unit (11) has also acquired a second image in addition to the first image, the calculation unit (12) calculates a second map from the second image, or from the first and second images, by using a second model, and the detection unit (13) detects the object with reference to the second map in addition to the first map.

Description

Technical Field

[0005] ,

[0001] The present invention relates to a technique for detecting an object from an image.

Background Art

[0002] Techniques for detecting an object from an image are known. In object detection, when a background image (such as when there is no target object) can be used in addition to the main image, an improvement in detection accuracy by difference information can be expected. For example, Patent Document 1 describes detecting the position of an object using an input image including the object and a background image. In addition, Non-Patent Document 1 and Non-Patent Document 2 propose a learning method (privileged learning) that uses a depth image as additional information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technology described in Patent Document 1 always requires a background image to perform inference, and has the problem that inference cannot be performed in situations where a background image is not available, such as when performing object detection at a new shooting location. On the other hand, the technologies described in Non-Patent Documents 1 and 2 have the problem that, even if a background image is available during inference, it cannot be utilized.

[0006] One aspect of the present invention has been made in view of the above-mentioned problems, and one example of its objective is to achieve accurate object detection by using images such as background images in combination depending on the situation. [Means for solving the problem]

[0007] An object detection device according to one aspect of the present invention comprises an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by referring at least to the first map. When the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0008] A learning device according to one aspect of the present invention includes: training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; first learning means for training a first model that calculates a first map from the first images by referring to the first images and the label information included in the training data; and second learning means for training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information included in the training data.

[0009] An object detection method according to one aspect of the present invention includes acquiring a first image, calculating a first map from the first image using a first model, and performing object detection by referring at least to the first map, wherein if a second image is also acquired in addition to the first image, in the calculation step, a second map is calculated from the second image, or from the first image and the second image, using a second model, and in the object detection step, object detection is performed by referring to the second map in addition to the first map.

[0010] A learning method relating to one aspect of the present invention includes: acquiring training data which includes one or more first images, one or more second images, and label information which indicates an object contained in the first image; training a first model which calculates a first map from the first image by referring to the first image and the label information contained in the training data; and training the first model and a second model which calculates a second map from the second image by referring to the first image, the second image and the label information contained in the training data.

[0011] An object detection program according to one aspect of the present invention is an object detection program that causes a computer to function as an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by referring at least to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0012] A learning program according to one aspect of the present invention causes a computer to function as: a training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects contained in the first images; a first learning means for training a first model that calculates a first map from the first images by referring to the first images and the label information contained in the training data; and a second learning means for training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information contained in the training data. [Effects of the Invention]

[0013] According to one aspect of the present invention, accurate object detection can be achieved by using images such as background images in combination depending on the situation. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing the configuration of an object detection device according to exemplary embodiment 1. [Figure 2] This is a flowchart showing the flow of the object detection method according to exemplary embodiment 1. [Figure 3] This is a block diagram showing the configuration of a learning device according to exemplary embodiment 1. [Figure 4] This is a flowchart showing the flow of the learning method according to exemplary embodiment 1. [Figure 5] This is a block diagram showing the configuration of an information processing device according to exemplary embodiment 2. [Figure 6] This figure shows an overview of the object detection process according to exemplary embodiment 2. [Figure 7] This figure shows a specific example of the object detection process according to Exemplary Embodiment 2. [Figure 8] This is a flowchart showing the flow of the object detection method according to exemplary embodiment 2. [Figure 9] This is a block diagram showing the configuration of an information processing device according to exemplary embodiment 3. [Figure 10] It is a block diagram showing an example of the hardware configuration of an object detection device, a learning device, and an information processing device in each exemplary embodiment.

Embodiments for Carrying Out the Invention

[0015] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0016] (Configuration of Object Detection Device) The configuration of the object detection device 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the object detection device 1. The object detection device 1 includes an image acquisition unit 11, a calculation unit 12, and a detection unit 13.

[0017] The image acquisition unit 11 acquires a first image. The calculation unit 12 calculates a first map from the first image using a first model. The detection unit 13 performs object detection by at least referring to the first map.

[0018] When the image acquisition unit 11 acquires a second image in addition to the first image, the calculation unit 12 calculates a second map from the second image or from the first image and the second image using a second model, and the detection unit 13 performs object detection by referring to the second map in addition to the first map.

[0019] As described above, the object detection device 1 according to this exemplary embodiment includes an image acquisition unit 11 that acquires a first image, a calculation unit 12 that calculates a first map from the first image using a first model, and a detection unit 13 that performs object detection by referring to at least the first map. When the image acquisition unit 11 acquires a second image in addition to the first image, the calculation unit 12 calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection unit 13 performs object detection by referring to the second map in addition to the first map. Therefore, the object detection device 1 according to this exemplary embodiment has the effect of achieving highly accurate object detection by using images such as background images in combination depending on the situation.

[0020] (Object detection method flow) The flow of the object detection method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the object detection method S1. The entity executing each step in the object detection method S1 may be a processor provided in the object detection device 1, a processor provided in another device, or the entity executing each step may be a processor provided in different devices.

[0021] In step S11, at least one processor acquires a first image. In step S12, at least one processor calculates a first map from the first image using a first model. In step S13, at least one processor performs object detection by referring to the first map.

[0022] Furthermore, if a second image is also acquired in addition to the first image, at least one processor calculates a second map from the second image, or from the first image and the second image, using the second model in the calculation step, and at least one processor performs object detection by referring to the second map in addition to the first map in the object detection step.

[0023] As described above, the object detection method S1 according to this exemplary embodiment includes acquiring a first image, calculating a first map from the first image using a first model, and performing object detection by referring at least to the first map. When a second image is also acquired in addition to the first image, in the calculation step, a second map is calculated from the second image, or from the first image and the second image, using a second model, and in the object detection step, object detection is performed by referring to the second map in addition to the first map. Therefore, the object detection method S1 according to this exemplary embodiment has the effect of achieving highly accurate object detection by using images such as background images in combination depending on the situation.

[0024] (Configuration of the learning device) The configuration of the learning device 2 according to this exemplary embodiment will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the learning device 2. The learning device 2 includes a teacher data acquisition unit 21, a first learning unit 22, and a second learning unit 23.

[0025] The training data acquisition unit 21 acquires training data that includes one or more first images, one or more second images, and label information indicating objects contained in the first images. The first learning unit 22 trains a first model that calculates a first map from the first images by referring to the first images and label information contained in the training data. The second learning unit 23 trains the first model and a second model that calculates a second map from the second images by referring to the first images, second images and label information contained in the training data.

[0026] As described above, the learning device 2 according to this exemplary embodiment employs a configuration comprising: a training data acquisition unit 21 that acquires training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning unit 22 that trains a first model for calculating a first map from the first images by referring to the first images and the label information included in the training data; and a second learning unit 23 that trains the first model and a second model for calculating a second map from the second images by referring to the first images, the second images, and the label information included in the training data. Therefore, the learning device 2 according to this exemplary embodiment has the effect of providing a model that can achieve accurate object detection by using images such as background images in combination depending on the situation.

[0027] (Learning process flow) The flow of the learning method S2 according to this exemplary embodiment will be explained with reference to Figure 4. Figure 4 is a flowchart showing the flow of the learning method S2. The entity executing each step in the learning method S2 may be a processor provided in the learning device 2, a processor provided in another device, or the entity executing each step may be a processor provided in a different device.

[0028] In step S21, at least one processor acquires training data which includes one or more first images, one or more second images, and label information indicating objects contained in the first images. In step S22, at least one processor trains a first model which calculates a first map from the first images by referring to the first images and the label information contained in the training data. In step S23, at least one processor trains the first model and a second model which calculates a second map from the second images by referring to the first images, the second images and the label information contained in the training data.

[0029] As described above, the learning method S2 according to this exemplary embodiment includes: acquiring training data which includes one or more first images, one or more second images, and label information which indicates objects contained in the first images; training a first model which calculates a first map from the first images by referring to the first images and the label information contained in the training data; and training the first model and a second model which calculates a second map from the second images by referring to the first images, the second images and the label information contained in the training data. Therefore, according to the learning method S2 according to this exemplary embodiment, it is possible to provide a model which can achieve accurate object detection by using images such as background images in combination depending on the situation.

[0030] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0031] <Configuration of the information processing device> Figure 5 is a block diagram showing the configuration of the information processing device 1A according to exemplary embodiment 2. The information processing device 1A is a device for detecting objects from an image. Here, the object is, for example, a moving object such as a vehicle or a person included in a satellite image. However, the object is not limited to the example described above.

[0032] The information processing device 1A comprises a control unit 10A, a storage unit 20A, an input / output unit 30A, and a communication unit 40A.

[0033] (input / output section) Input / output devices such as a keyboard, mouse, display, printer, and touch panel are connected to the input / output unit 30A. The input / output unit 30A receives various types of information from the connected input devices to the information processing device 1A. The input / output unit 30A also outputs various types of information to the connected output devices under the control of the control unit 10A. An interface such as USB (Universal Serial Bus) can be used as the input / output unit 30A. The input / output unit 30A may also include a display panel, speaker, keyboard, mouse, touch panel, etc.

[0034] (Communications Department) The communication unit 40A communicates with external devices of the information processing device 1A via a communication line. The specific configuration of the communication line is not limited to this exemplary embodiment, but examples of communication lines include wireless LAN (Local Area Network), wired LAN, WAN (Wide Area Network), public telephone network, mobile data communication network, or a combination thereof. The communication unit 40A transmits data supplied from the control unit 10A to other devices and supplies data received from other devices to the control unit 10A.

[0035] (Control Unit) The control unit 10A includes an image acquisition unit 11, a calculation unit 12, a detection unit 13, a determination unit 14, and a presentation unit 15.

[0036] (Image acquisition unit) The image acquisition unit 11 acquires a first image IMG1, or the first image IMG1 and the second image IMG2. The first image IMG1 is the target of object detection processing, and is, for example, an image of an object. The object is, for example, a moving object such as a vehicle or a person, but is not limited to these. The first image IMG1 includes, for example, an image of the R, G, and B channels. However, the first image IMG1 is not limited to the above example and may be any other image.

[0037] The second image, IMG2, is used for object detection. Examples include a background image corresponding to the first image, IMG1, a depth image sensed by a depth sensor, or an infrared image captured by an infrared camera. However, the second image, IMG2, is not limited to the examples described above and may be any other image.

[0038] (Calculation section) The calculation unit 12 calculates the first map MAP1 from the first image IMG1 using the first model MD1. Here, the first model MD1 is a model that takes the first image IMG1 as input and outputs the first map MAP1, and is one example of a convolutional neural network. The first map MAP1 is a map calculated from the first image IMG1, and is one example of a feature map obtained by processing such as a convolution operation on the first image IMG1. The first map calculated by the calculation unit 12 is referenced in the object detection process.

[0039] Furthermore, if the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the calculation unit 12 calculates the second map MAP2 using the second model MD2 from the second image IMG2, or from the first image IMG1 and the second image IMG2. The second model MD2 is a model that outputs the second map MAP2, and one example is a convolutional neural network. Here, the input to the second model MD2 is, for example, the second image IMG2, or the first image IMG 1This includes the second image IMG2. The second map MAP2 is either the second image IMG2, or a map calculated from the first and second images. The second map MAP2 is, for example, a feature map representing the features of the second image, or a weight map representing the difference between the second image IMG2 and the first image IMG1.

[0040] (Detection unit) The detection unit 13 performs object detection by referring to at least the first map MAP1. As an example, the detection unit 13 performs object detection using object detection methods such as Faster R-CNN (Regions with CNN features), SSD (Single Shot MultiBox Detector), and YOLO (You Only Look Once). Here, the detection unit 13 may be a post-stage of Faster R-CNN (R-CNN), or the detection unit 13 connected to the calculation unit 12 may be a pre-stage of Faster R-CNN (RPN (Region Proposal Networks)), SSD, YOLO, or other models. However, the method used by the detection unit 13 for object detection is not limited to the examples described above, and the detection unit 13 may perform object detection using other methods.

[0041] Furthermore, if the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the detection unit 13 performs object detection by referring to the second map MAP2 in addition to the first map MAP1. For example, the detection unit 13 performs object detection by referring to a third map obtained by calculations using the first map MAP1 and the second map MAP2.

[0042] The third map is obtained by an operation using the first map MAP1 and the second map MAP2. For example, it is obtained by multiplying the first map MAP1 by the second map MAP2. In other words, when the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the detection unit 13 performs object detection by referring to the third map obtained by multiplying the first map MAP1 by the second map MAP2. However, the third map is not limited to the example described above and may be obtained by other operations. For example, the third map may be obtained by adding the second map MAP2 to the first map MAP1.

[0043] (Judgment Department) The determination unit 14 performs a determination process to determine whether the image acquisition unit 11 acquires the first image IMG1, or acquires the first image IMG1 and the second image IMG2. For example, the determination unit 14 performs the determination process by referring to a flag that indicates whether to acquire the first image IMG1 or the first image IMG1 and the second image IMG2. However, the determination process of the determination unit 14 is not limited to the example described above, and the determination unit 14 may perform the determination process by other methods.

[0044] (Presentation part) The display unit 15 displays the results of object detection performed by the detection unit 13. The display unit 15 may display the results by outputting them to an output device (display, speaker, printer, etc.) connected to the input / output unit 30A, or it may transmit them to another device connected via the communication unit 40A. As an example, the display unit 15 displays an image representing the object detection results on the display panel provided by the input / output unit 30A.

[0045] (Storage part) The memory unit 20A stores the first image IMG1, the second image IMG2, the first map MAP1, the second map MAP2, the first model MD1, the second model MD2, and the detection result DR.

[0046] <Overview of Object Detection Process> Figure 6 shows an example of an overview of the object detection process performed by the information processing device 1A. In the example in Figure 6, the calculation unit 12 comprises a first calculation unit 12-1 and a second calculation unit 12-2. The first calculation unit 12-1 calculates a first map MAP1 from a first image IMG1 using a first model MD1. The second calculation unit 12-2 calculates a second map MAP2 from a second image IMG2, or from both the first image IMG1 and the second image IMG2, using a second model MD2. The second map MAP2 is, for example, a weight map representing the difference between the first image IMG1 and the second image IMG2. If the second image IMG2 has not been acquired, the calculation unit 12 does not perform the calculation of the second map MAP2.

[0047] Furthermore, the detection unit 13 includes a multiplication unit 13-1 and a detection execution unit 13-2. The multiplication unit 13-1 calculates a third map by multiplying the first map MAP1 by the second map MAP2. The multiplication unit 13-1 may apply the multiplication process to the entire first map MAP1, or it may apply the multiplication process to a part of the first map MAP1.

[0048] If the image acquisition unit 11 acquires the second image IMG2, the detection execution unit 13-2 performs object detection by referring to the third map. On the other hand, if the image acquisition unit 11 has not acquired the second image IMG2, the detection execution unit 13-2 performs object detection by referring to the first map MAP1.

[0049] As an example, the detection execution unit 13-2 detects objects based on the output obtained by inputting a feature map (the first map MAP1 or the third map) into a trained model. Here, the trained model is, as an example, a model constructed by supervised machine learning, such as a convolutional neural network. As an example, the input to the trained model includes a feature map of a candidate region, and as an example, the output of the trained model includes information indicating the object type and the bounding rectangle of the object. As an example of a method used by the detection execution unit 13-2 to detect objects from a feature map, examples include the Faster R-CNN and SSD methods mentioned above.

[0050] <Specific examples of object detection processing> Figure 7 shows a specific example of the object detection process according to Exemplary Embodiment 2. In the example in Figure 7, the main image IMG1_1 is an example of the first image IMG1, and the additional image IMG2_1 is an example of the second image IMG2. In the example in Figure 7, the image acquisition unit 11 acquires the main image IMG1_1, which is an image of the candidate region extracted by the RPN described above, and the additional image IMG2_1, which is the background image of the candidate region. The main image IMG1_1 is a part of the image in which the object was photographed, and the additional image IMG2_1 is a part of the photographed image corresponding to the main image IMG1_1 image, but does not contain the object.

[0051] The main image IMG1_1 contains object o1 and object o2. Object o1 is the object to be detected. On the other hand, object o2 is also included in the supplemental image IMG2_1 and does not need to be detected. Thus, feature map MAP1_1 contains object o2, which is a false focus and is different from object o1, the object to be detected.

[0052] The calculation unit 12 calculates the feature map MAP1_1 by inputting the main image IMG1_1 into the first model MD1. Feature map MAP1_1 is an example of the first map MAP1. The calculation unit 12 also calculates the weight map MAP2_1 by inputting the main image IMG1_1 and the additional image IMG2_1 into the second model MD2. Weight map MAP2_1 is an example of the second map MAP2. Here, since object o2 is included in both the main image IMG1_1 and the additional image IMG2_1, object o2 will not appear, or will be unlikely to appear, in the weight map MAP2_1, which represents the difference between the two.

[0053] The detection unit 13 calculates feature map MAP3_1 by multiplying feature map MAP1_1 by weight map MAP2_1. Feature map MAP3_1 is an example of a third map. By multiplying feature map MAP1_1 by weight map MAP2_1, object o2 that was included in feature map MAP1_1 will either not appear in feature map MAP3_1, or will appear less frequently.

[0054] The detection unit 13 calculates the object detection result DR_1 (the re-estimation result of the object type and the object's bounding rectangle) by referring to the feature map MAP3_1. The detection result DR_1 is presented by the presentation unit 15 as an example.

[0055] <Object detection method flow> Figure 8 is a flowchart showing an example of the object detection method according to exemplary embodiment 2.

[0056] (Step S201) In step S201, the calculation unit 12 calculates the feature map MAP1_1 from the main image IMG1_1.

[0057] (Step S202) In step S202, the determination unit 14 determines whether there is an additional image IMG2_1. For example, the determination unit 14 determines whether there is an additional image IMG2_1 by referring to a predetermined flag (for example, a flag attached to the main image IMG1_1). If there is an additional image IMG2_1 (YES in step S202), the determination unit 14 proceeds to the process in step S203. On the other hand, if there is no additional image IMG2_1 (NO in step S202), the determination unit 14 proceeds to the process in step S204.

[0058] (Step S203) In step S203, the detection unit 13 multiplies the weight map MAP2_1 calculated from the additional image IMG2_1 by the feature map MAP1_1 to calculate the feature map MAP3_1.

[0059] (Step S204) In step S204, the detection unit 13 calculates the object detection result from the feature map MAP3_1 calculated in step S203.

[0060] <Effects of Information Processing Devices> As described above, in the information processing apparatus 1A according to this exemplary embodiment, when the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the detection unit 13 is configured to perform object detection by referring to a third map obtained by multiplying the first map MAP1 by the second map MAP2. Therefore, according to the information processing apparatus 1A according to this exemplary embodiment, by performing object detection by referring to a third map obtained by multiplying the second map MAP2 by the first map MAP1, the effect of detecting objects with greater accuracy can be obtained.

[0061] Furthermore, the information processing apparatus 1A according to this exemplary embodiment is configured to include a determination unit 14 that performs a determination process to determine whether the image acquisition unit 11 acquires a first image IMG1, or whether it acquires both the first image IMG1 and the second image IMG2. Therefore, according to the information processing apparatus 1A according to this exemplary embodiment, it is possible to detect objects in both cases, with and without acquiring a second image, and to detect objects with greater accuracy when a second image is present. More specifically, for example, in situations where a background image may be obtained in addition to the main image, the background image can be utilized to improve the accuracy during inference.

[0062] Furthermore, in the information processing apparatus 1A according to this exemplary embodiment, the determination unit 14 performs the determination process by referring to a flag indicating whether to acquire the first image IMG1 or to acquire the first image IMG1 and the second image IMG2. Therefore, according to the information processing apparatus 1A according to this exemplary embodiment, by determining whether to acquire the second image by referring to the flag, it is possible to detect an object in both cases, with and without acquiring the second image, and it is possible to detect an object with greater accuracy when the second image is present.

[0063] [Exemplary Embodiment 3] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0064] <Configuration of the information processing device> Figure 9 is a block diagram showing the configuration of an information processing device 1B according to exemplary embodiment 3. The control unit 10A of the information processing device 1B includes an image acquisition unit 11, a calculation unit 12, a detection unit 13, a determination unit 14, and a presentation unit 15, as well as a training data acquisition unit 16, a first learning unit 17, and a second learning unit 18. The training data acquisition unit 16, the first learning unit 17, and the second learning unit 18 constitute the learning device according to this specification.

[0065] (Training Data Acquisition Department) The training data acquisition unit 16 acquires training data that includes one or more first images, one or more second images, and label information indicating objects contained in the first images. Here, the first images and second images are as described in the exemplary embodiment 2 above. The label information includes, as an example, information indicating the type of object.

[0066] (First Learning Section) The first learning unit 17 trains the first model MD1 by machine learning by referring to the first image and label information included in the training data. As described above, the first model MD1 is the model used by the calculation unit 12 when calculating the first map MAP1, and is a convolutional neural network as an example. In this exemplary embodiment, even if the training data includes a second image, the first learning unit 17 may, as an example, train the first model MD1 by supervised machine learning using the set of the first image and label information without using the second image.

[0067] (Second Learning Section) The second learning unit 18 trains the first model MD1 and the second model MD2 by machine learning, referring to the first image, the second image, and the label information included in the training data. The second model MD2 is the model used by the calculation unit 12 when calculating the second map MAP2, as described above, and is a convolutional neural network as an example. At this time, the second learning unit 18 may also use a loss function that minimizes the difference between the first map MAP1 before applying the weight map and the third map MAP3 after applying the weight map.

[0068] <Effects of Information Processing Devices> Furthermore, the information processing device 1B according to this exemplary embodiment includes a training data acquisition unit 16 that acquires training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning unit 17 that trains a first model MD1 by machine learning by referring to the first images and label information included in the training data; and a second learning unit 18 that trains the first model MD1 and the second model MD2 by machine learning by referring to the first images, second images, and label information included in the training data. Therefore, according to the information processing device 1B according to this exemplary embodiment, in addition to the effects of the object detection device 1 according to exemplary embodiment 1, it is possible to provide a model that can achieve highly accurate object detection by using images such as background images in combination depending on the situation.

[0069] [Examples] An embodiment relating to this disclosure is described below. This embodiment is an example in which the information processing devices 1A and 1B according to the above exemplary embodiment are applied to the medical and healthcare field. In this embodiment, the first image IMG1 is an image taken by an endoscopic examination of the subject. The second image IMG2 is an image taken by a past endoscopic examination of the same subject. The second image IMG2 is an image taken when no lesion was detected, and it is an image of the same location as the first image IMG1.

[0070] Furthermore, in this embodiment, the object detected by the detection unit 13 is a lesion detected from an image taken during an endoscopic examination of the subject. If there are past endoscopic examination images of the subject (second image IMG2), the detection unit 13 uses the past endoscopic images to detect the lesion. The presentation unit 15 presents the results of the lesion detection to the medical professional.

[0071] Healthcare professionals refer to the presented lesion detection results and, for example, decide on a course of action for the patient. In other words, the presentation unit 15 outputs lesion detection results to support the decision-making of healthcare professionals. That is, according to this embodiment, the information processing devices 1A and 1B can support the decision-making of healthcare professionals.

[0072] Furthermore, for example, the presentation unit 15 may present to healthcare professionals a model generated by machine learning the correspondence between lesion detection results and treatment methods, and a treatment method determined based on the lesion detection results of the subject. The method for determining the treatment method is not limited to the method described above. In this way, the information processing device can support the user's decision-making.

[0073] Furthermore, according to this embodiment, objects (lesions) can be detected both when past endoscopic examination images of the subject are available and when they are not, and the lesions can be detected with greater accuracy when past endoscopic examination images of the subject are available.

[0074] [Examples of implementation using software] Some or all of the functions of the object detection device 1, the information processing devices 1A and 1B, and the learning device 2 (hereinafter referred to as "object detection device 1, etc.") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0075] In the latter case, the object detection device 1, etc., is implemented by a computer that executes instructions for a program, which is software that implements each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as the object detection device 1, etc. In computer C, the processor C1 reads the program P from the memory C2 and executes it, thereby implementing each function of the object detection device 1, etc.

[0076] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0077] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0078] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0079] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0080] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0081] (Note 1) An object detection device comprising: an image acquisition means for acquiring a first image; a calculation means for calculating a first map from the first image using a first model; and a detection means for performing object detection by referring at least to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0082] (Note 2) The object detection device according to Appendix 1, wherein, when the image acquisition means acquires a second image in addition to the first image, the detection means performs object detection by referring to a third map obtained by multiplying the first map by the second map.

[0083] (Note 3) The object detection device according to Appendix 1 or 2, further comprising determination means for performing a determination process to determine whether the image acquisition means acquires the first image, or acquires the first image and the second image.

[0084] (Note 4) The object detection device according to Appendix 3, wherein the determination means performs the determination process by referring to a flag indicating whether to acquire the first image or to acquire the first image and the second image.

[0085] (Note 5) An object detection device according to Appendix 1 or 2, comprising: training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects contained in the first images; first learning means for training the first model by machine learning by referring to the first images and the label information contained in the training data; and second learning means for training the first model and the second model by machine learning by referring to the first images, the second images, and the label information contained in the training data.

[0086] (Note 6) The system further comprises a presentation means for outputting the detection result from the detection means, The detection means detects The object is a lesion that can be detected from images taken by an endoscopic examination of a subject, and the presentation means outputs the detection result of the lesion to support the decision-making of a medical professional, as described in Appendix 1 or 2 of the object detection device.

[0087] (Note 7) A learning device comprising: training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; first learning means for training a first model that calculates a first map from the first images by referring to the first images and the label information included in the training data; and second learning means for training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information included in the training data.

[0088] (Note 8) An object detection method comprising: acquiring a first image; calculating a first map from the first image using a first model; and performing object detection by referring at least to the first map, wherein if a second image is also acquired in addition to the first image, in the calculation step, a second map is calculated from the second image, or from the first image and the second image, using a second model; and in the object detection step, object detection is performed by referring to the second map in addition to the first map.

[0089] (Note 9) A learning method comprising: obtaining training data including one or more first images, one or more second images, and label information indicating objects contained in the first images; training a first model that calculates a first map from the first images by referring to the first images and the label information contained in the training data; and training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information contained in the training data.

[0090] (Note 10) An object detection program that causes a computer to function as an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by referring to at least the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0091] (Note 11) A learning program that causes a computer to function as: a training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects contained in the first images; a first learning means for training a first model that calculates a first map from the first images by referring to the first images and the label information contained in the training data; and a second learning means for training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information contained in the training data.

[0092] [Additional Note 3] Some or all of the embodiments described above can also be expressed as follows: An object detection device comprising at least one processor, wherein the processor performs an image acquisition process for acquiring a first image, a calculation process for calculating a first map from the first image using a first model, and a detection process for performing object detection by referring at least to the first map, wherein, in the image acquisition process, a second image is acquired in addition to the first image, the calculation process calculates a second map from the second image, or from the first image and the second image, using a second model, and the detection process performs object detection by referring to the second map in addition to the first map.

[0093] Furthermore, this object detection device may also be equipped with memory, which may store a program that causes the processor to execute the image acquisition process, the calculation process, and the detection process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium.

[0094] A learning device comprising at least one processor, the processor performing: a training data acquisition process for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning process for training a first model that calculates a first map from the first images by referring to the first images and the label information included in the training data; and a second learning process for training the first model and a second model that calculates a second map from the second images by referring to the first images, the second images and the label information included in the training data.

[0095] Furthermore, this learning device may also be equipped with memory, and this memory may store a program that causes the processor to execute the training data acquisition process, the first learning process, and the second learning process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of Symbols]

[0096] 1. Object detection device 1A, 1B Information Processing Devices 2 Learning device 11 Image acquisition unit 12 Calculation Section 13 Detection unit 14 Judgment section 15 Presentation section 16, 21 Training Data Acquisition Unit 17, 22 First Learning Section 18, 23 Second Learning Section

Claims

1. An image acquisition means for acquiring a first image, A calculation means for calculating a first map from the first image using the first model, A detection means that performs object detection by referring to at least the first map, Equipped with, When the image acquisition means acquires a second image in addition to the first image, The calculation means uses a second model to calculate a second map, which is a weight map representing the difference between the first image and the second image, from the first image and the second image. The detection means performs object detection by referring to a third map obtained by multiplying the first map by the second map. Object detection device.

2. The system further includes determination means that performs a determination process to determine whether the image acquisition means acquires the first image, or whether it acquires the first image and the second image. The object detection device according to claim 1.

3. A means for acquiring training data that includes one or more first images, one or more second images, and label information indicating objects contained in the first images, A first learning means for training the first model by machine learning, by referring to the first image and the label information included in the training data, A second learning means for training the first model and the second model by machine learning, by referring to the first image, the second image, and the label information included in the training data. The object detection device according to claim 1 or 2, comprising:

4. The system further includes a presentation means for outputting the detection results obtained by the detection means, The object detected by the aforementioned detection means is a lesion that can be detected from images taken by an endoscopic examination of the subject. The aforementioned presentation means outputs the lesion detection results to support the decision-making of healthcare professionals. The object detection device according to claim 1 or 2.

5. A means for acquiring training data that includes one or more first images, one or more second images, and label information indicating objects contained in the first images, A first learning means for training a first model that calculates a first map from a first image by referring to the first image and label information included in the training data, A second learning means for training the first model and the second model which calculates a second map from the second image by referring to the first image, the second image and the label information included in the training data. A learning device equipped with the following features.

6. To obtain the first image, Using the first model, calculate the first map from the first image, The object detection is performed by at least referring to the first map described above. Includes, If a second image is also acquired in addition to the first image, In the calculation process described above, a second map is calculated using a second model, which is a weight map representing the difference between the first image and the second image, from the first image and the second image. In the object detection step, object detection is performed by referring to a third map obtained by multiplying the first map by the second map. Object detection methods.

7. To obtain training data that includes one or more first images, one or more second images, and label information indicating objects contained in the first images, A first model that calculates a first map from a first image is trained by referring to the first image and the label information included in the training data. The first model and the second model that calculates a second map from the second image are trained by referring to the first image, the second image, and the label information included in the training data. Learning methods that include this.

8. Computers, Image acquisition means for acquiring a first image, A calculation means for calculating a first map from the first image using the first model, and An object detection program that functions as a detection means for performing object detection by referring to at least the first map, When the image acquisition means acquires a second image in addition to the first image, The calculation means uses a second model to calculate a second map, which is a weight map representing the difference between the first image and the second image, from the first image and the second image. The detection means is an object detection program that performs object detection by referring to a third map obtained by multiplying the first map by the second map.

9. Computers, Training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects contained in the first images. A first learning means for training a first model that calculates a first map from a first image by referring to the first image and label information included in the training data, and A learning program that functions as a second learning means for training the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the training data.

Images