Object detection device, learning device, object detection method, learning method, object detection program, and learning program

Abstract

In order to realize highly accurate object detection by concurrently using a background image or other image depending on situation, this object detection device (1) comprises an image acquisition unit (11) that acquires a first image, a calculation unit (12) that calculates a first map from the first image by using a first model, and a detection unit (13) that detects an object with reference to at least the first map. When the image acquisition unit (11) has also acquired a second image in addition to the first image, the calculation unit (12) calculates a second map from the second image, or from the first and second images, by using a second model, and the detection unit (13) detects the object with reference to the second map in addition to the first map.

Description

[Technical field]

[0001] The present invention relates to a technique for detecting an object from an image. [Background technology]

[0002] Techniques for detecting objects from images are known. In object detection, when a background image (when there is no target object, etc.) can be used in addition to a main image, it is expected that the detection accuracy can be improved by using differential information. For example, Patent Document 1 describes detecting the position of an object using an input image including the object and a background image. In addition, Non-Patent Documents 1 and 2 propose a learning method (privileged learning) that uses a depth image as additional information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-191501 [Non-patent literature]

[0004] [Non-Patent Document 1] Judy Hoffman et al., “Learning with Side Information through Modality Hallucination” in CVPR 2016 [Non-Patent Document 2] Shanxin Yuan et al., “3D Hand Pose Estimation from RGB Using Privileged Learning with Depth Data” in ICCV 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 has a problem in that a background image is always required to execute inference, and inference cannot be executed in a situation where a background image cannot be obtained, such as when detecting an object at a new shooting location. On the other hand, the technologies described in Non-Patent Documents 1 and 2 have a problem in that a background image cannot be utilized even if it is available during inference.

[0006] One aspect of the present invention has been made in consideration of the above problems, and one example of a purpose of the present invention is to achieve accurate object detection by using images such as background images in addition depending on the situation. [Means for solving the problem]

[0007] An object detection device according to one aspect of the present invention comprises an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by at least referring to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0008] A learning device according to one aspect of the present invention includes a teacher data acquisition means for acquiring teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images, a first learning means for learning a first model for calculating a first map from a first image by referring to the first image and the label information included in the teacher data, and a second learning means for learning the first model and a second model for calculating a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data.

[0009] An object detection method according to one aspect of the present invention includes acquiring a first image, calculating a first map from the first image using a first model, and performing object detection by at least referring to the first map. When a second image is acquired in addition to the first image, in the calculating step, a second map is calculated from the second image or from the first image and the second image using a second model, and in the object detection step, object detection is performed by referring to the second map in addition to the first map.

[0010] A learning method according to one aspect of the present invention includes acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images, training a first model that calculates a first map from a first image by referring to the first images and the label information included in the training data, and training the first model and a second model that calculates a second map from a second image by referring to the first images, the second images, and the label information included in the training data.

[0011] An object detection program according to one aspect of the present invention is an object detection program that causes a computer to function as an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by at least referring to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0012] A learning program according to one aspect of the present invention causes a computer to function as: a teacher data acquisition means for acquiring teacher data including one or more first images, one or more second images, and label information indicating objects contained in the first images; a first learning means for learning a first model for calculating a first map from a first image by referring to the first image and the label information contained in the teacher data; and a second learning means for learning the first model and a second model for calculating a second map from a second image by referring to the first image, the second image, and the label information contained in the teacher data. Effect of the Invention

[0013] According to one aspect of the present invention, accurate object detection can be achieved by using images such as a background image in addition depending on the situation. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of an object detection device according to a first exemplary embodiment. [Diagram 2] 1 is a flow diagram showing the flow of an object detection method according to exemplary embodiment 1. FIG. [Diagram 3] 1 is a block diagram showing a configuration of a learning device according to an exemplary embodiment 1. FIG. [Figure 4] 1 is a flow chart showing the flow of a learning method according to the first exemplary embodiment. [Diagram 5] FIG. 11 is a block diagram showing a configuration of an information processing device according to a second exemplary embodiment. [Figure 6] FIG. 11 is a diagram showing an overview of an object detection process according to the second exemplary embodiment. [Figure 7] FIG. 11 is a diagram showing a specific example of an object detection process according to the second exemplary embodiment. [Figure 8] 11 is a flow diagram showing the flow of an object detection method according to the second exemplary embodiment. FIG. [Figure 9] FIG. 11 is a block diagram showing a configuration of an information processing device according to an exemplary embodiment 3. [Figure 10] FIG. 2 is a block diagram showing an example of a hardware configuration of an object detection device, a learning device, and an information processing device in each exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Example Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0016] (Configuration of Object Detection Device) The configuration of an object detection device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the object detection device 1. The object detection device 1 includes an image acquisition unit 11, a calculation unit 12, and a detection unit 13.

[0017] The image acquisition unit 11 acquires a first image. The calculation unit 12 calculates a first map from the first image using a first model. The detection unit 13 performs object detection by at least referring to the first map.

[0018] Furthermore, when the image acquisition unit 11 acquires a second image in addition to the first image, the calculation unit 12 calculates a second map from the second image or from the first image and the second image using a second model, and the detection unit 13 performs object detection by referring to the second map in addition to the first map.

[0019] As described above, the object detection device 1 according to this exemplary embodiment includes an image acquisition unit 11 that acquires a first image, a calculation unit 12 that calculates a first map from the first image using a first model, and a detection unit 13 that performs object detection by at least referring to the first map, and when the image acquisition unit 11 acquires a second image in addition to the first image, the calculation unit 12 calculates a second map from the second image or from the first image and the second image using the second model, and the detection unit 13 performs object detection by referring to the second map in addition to the first map. Therefore, according to the object detection device 1 according to this exemplary embodiment, an effect is obtained in which accurate object detection can be realized by using an image such as a background image in combination depending on the situation.

[0020] (Object detection method flow) The flow of the object detection method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the object detection method S1. The execution entity of each step in the object detection method S1 may be a processor included in the object detection device 1, or may be a processor included in another device, or each step may be a processor provided in a different device.

[0021] In step S11, at least one processor acquires a first image. In step S12, at least one processor calculates a first map from the first image using a first model. In step S13, at least one processor performs object detection with at least reference to the first map.

[0022] Furthermore, when a second image is acquired in addition to the first image, at least one processor, in the calculating step, calculates a second map from the second image or from the first image and the second image using a second model, and at least one processor, in the object detection step, performs object detection by referring to the second map in addition to the first map.

[0023] As described above, the object detection method S1 according to the present exemplary embodiment includes acquiring a first image, calculating a first map from the first image using a first model, and performing object detection by at least referring to the first map. When a second image is acquired in addition to the first image, the calculating step uses a second model to calculate a second map from the second image or from the first image and the second image, and the object detection step performs object detection by referring to the second map in addition to the first map. Therefore, according to the object detection method S1 according to the present exemplary embodiment, an effect is obtained in which accurate object detection can be realized by using an image such as a background image in combination depending on the situation.

[0024] (Learning device configuration) The configuration of the learning device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the learning device 2. The learning device 2 includes a teacher data acquisition unit 21, a first learning unit 22, and a second learning unit 23.

[0025] The teacher data acquisition unit 21 acquires teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images. The first learning unit 22 trains a first model that calculates a first map from a first image by referring to the first image and the label information included in the teacher data. The second learning unit 23 trains the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data.

[0026] As described above, the learning device 2 according to the present exemplary embodiment includes a teacher data acquisition unit 21 that acquires teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images, a first learning unit 22 that learns a first model that calculates a first map from a first image by referring to the first image and the label information included in the teacher data, and a second learning unit 23 that learns the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data. Therefore, the learning device 2 according to the present exemplary embodiment has an effect of providing a model that realizes accurate object detection by using an image such as a background image in combination depending on the situation.

[0027] (Learning method flow) The flow of the learning method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the learning method S2. The execution entity of each step in the learning method S2 may be a processor included in the learning device 2, or a processor included in another device, or each step may be a processor provided in a different device.

[0028] In step S21, at least one processor acquires training data including one or more first images, one or more second images, and label information indicating objects included in the first images. In step S22, at least one processor trains a first model that calculates a first map from a first image by referring to the first image and the label information included in the training data. In step S23, at least one processor trains the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the training data.

[0029] As described above, the learning method S2 according to the present exemplary embodiment includes acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images, learning a first model that calculates a first map from a first image by referring to the first image and the label information included in the training data, and learning the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the training data. Therefore, according to the learning method S2 according to the present exemplary embodiment, it is possible to provide a model that can realize accurate object detection by using an image such as a background image in combination depending on the situation.

[0030] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0031] <Configuration of information processing device> 5 is a block diagram showing a configuration of an information processing device 1A according to an exemplary embodiment 2. The information processing device 1A is a device that detects an object from an image. Here, an example of the object is a moving body such as a vehicle or a person included in a satellite image. However, the object is not limited to the above-mentioned examples.

[0032] The information processing device 1A includes a control unit 10A, a storage unit 20A, an input / output unit 30A, and a communication unit 40A.

[0033] (input / output section) Input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel are connected to the input / output unit 30A. The input / output unit 30A accepts various types of information input from the connected input devices to the information processing device 1A. The input / output unit 30A also outputs various types of information to the connected output devices under the control of the control unit 10A. An example of the input / output unit 30A is an interface such as a Universal Serial Bus (USB). The input / output unit 30A may also include a display panel, a speaker, a keyboard, a mouse, a touch panel, and the like.

[0034] (Communications Department) The communication unit 40A communicates with devices external to the information processing device 1A via a communication line. Although the specific configuration of the communication line does not limit this exemplary embodiment, the communication line is, for example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these. The communication unit 40A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0035] (Control unit) The control unit 10A includes an image acquisition unit 11, a calculation unit 12, a detection unit 13, a determination unit 14, and a presentation unit 15.

[0036] (Image acquisition section) The image acquisition unit 11 acquires a first image IMG1, or the first image IMG1 and the second image IMG2. The first image IMG1 is a target of an object detection process, and is, for example, an image of an object. For example, the object is a moving body such as a vehicle or a person, but the object is not limited to these. For example, the first image IMG1 includes images of R, G, and B channels. However, the first image IMG1 is not limited to the above example, and may be another image.

[0037] The second image IMG2 is an image used in the object detection process, and is, for example, a background image corresponding to the first image IMG1, a depth image sensed by a depth sensor, or an infrared image captured by an infrared camera. However, the second image IMG2 is not limited to the above examples and may be other images.

[0038] (Calculation section) The calculation unit 12 calculates a first map MAP1 from the first image IMG1 using a first model MD1. Here, the first model MD1 is a model that receives the first image IMG1 as an input and outputs the first map MAP1, and is, for example, a convolutional neural network. The first map MAP1 is a map calculated from the first image IMG1, and is, for example, a feature map obtained by processing such as a convolution operation on the first image IMG1. The first map calculated by the calculation unit 12 is referred to in the object detection processing.

[0039] Furthermore, when the image acquisition unit 11 acquires a second image IMG2 in addition to the first image IMG1, the calculation unit 12 uses a second model MD2 to calculate a second map MAP2 from the second image IMG2 or from the first image IMG1 and the second image IMG2. The second model MD2 is a model that outputs the second map MAP2, and is, for example, a convolutional neural network. Here, the input of the second model MD2 is, for example, the second image IMG2 or the first image IMG 1and a second image IMG2. The second map MAP2 is a map calculated from the second image IMG2 or the first image and the second image. The second map MAP2 is, for example, a feature map representing the features of the second image, or a weight map representing the difference between the second image IMG2 and the first image IMG1.

[0040] (Detection unit) The detection unit 13 performs object detection by at least referring to the first map MAP1. As an example, the detection unit 13 performs object detection by an object detection method such as Faster R-CNN (Regions with CNN features), SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), etc. Here, the detection unit 13 may be a model of the latter stage of Faster R-CNN (R-CNN), or the detection unit 13 connected to the calculation unit 12 may be a model of the former stage of Faster R-CNN (RPN (Region Proposal Networks)), SSD, YOLO, etc. However, the method of the detection unit 13 performing object detection is not limited to the above-mentioned example, and the detection unit 13 may perform object detection by other methods.

[0041] Furthermore, when the image acquisition unit 11 acquires a second image IMG2 in addition to the first image IMG1, the detection unit 13 performs object detection by referring to the second map MAP2 in addition to the first map MAP1. As an example, the detection unit 13 performs object detection by referring to a third map obtained by a calculation using the first map MAP1 and the second map MAP2.

[0042] The third map is a map obtained by a calculation using the first map MAP1 and the second map MAP2, and as an example, is a map obtained by multiplying the first map MAP1 by the second map MAP2. In this case, in other words, when the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the detection unit 13 performs object detection by referring to the third map obtained by multiplying the first map MAP1 by the second map MAP2. However, the third map is not limited to the above example, and may be a map obtained by other calculations. The third map may be, for example, a map obtained by adding the second map MAP2 to the first map MAP1.

[0043] (Judgment Department) The determination unit 14 performs a determination process to determine whether the image acquisition unit 11 acquires the first image IMG1, or acquires the first image IMG1 and the second image IMG2. As an example, the determination unit 14 performs the above determination process by referring to a flag indicating whether the first image IMG1 is to be acquired, or the first image IMG1 and the second image IMG2 are to be acquired. However, the determination process of the determination unit 14 is not limited to the above example, and the determination unit 14 may perform the above determination process by other methods.

[0044] (Presentation part) The presentation unit 15 presents the result of the object detection by the detection unit 13. The presentation unit 15 may present the result by outputting it to an output device (display, speaker, printer, etc.) connected to the input / output unit 30A, or may transmit it to another device connected via the communication unit 40A. As an example, the presentation unit 15 displays an image showing the result of the object detection on a display panel provided in the input / output unit 30A.

[0045] (Storage part) The storage unit 20A stores a first image IMG1, a second image IMG2, a first map MAP1, a second map MAP2, a first model MD1, a second model MD2, and a detection result DR.

[0046] <Object detection process overview> FIG. 6 is a diagram showing an example of an outline of the object detection process executed by the information processing device 1A. In the example of FIG. 6, the calculation unit 12 includes a first calculation unit 12-1 and a second calculation unit 12-2. The first calculation unit 12-1 calculates a first map MAP1 from a first image IMG1 using a first model MD1. The second calculation unit 12-2 calculates a second map MAP2 from a second image IMG2 or from the first image IMG1 and the second image IMG2 using a second model MD2. The second map MAP2 is, for example, a weight map representing the difference between the first image IMG1 and the second image IMG2. Note that if the second image IMG2 has not been acquired, the calculation unit 12 does not perform the calculation process of the second map MAP2.

[0047] The detection unit 13 also includes a multiplication unit 13-1 and a detection execution unit 13-2. The multiplication unit 13-1 multiplies the first map MAP1 by the second map MAP2 to calculate a third map. The multiplication unit 13-1 may apply the multiplication process to the entire first map MAP1, or may apply the multiplication process to a part of the first map MAP1.

[0048] When the image acquisition unit 11 acquires the second image IMG2, the detection execution unit 13-2 performs object detection by referring to the third map. On the other hand, when the image acquisition unit 11 does not acquire the second image IMG2, the detection execution unit 13-2 performs object detection by referring to the first map MAP1.

[0049] As an example, the detection execution unit 13-2 detects an object based on an output obtained by inputting a feature map (the first map MAP1 or the third map) into a trained model. Here, as an example, the trained model is a model constructed by supervised machine learning, such as a convolutional neural network. As an example, the input of the trained model includes a feature map of a candidate region, and as an example, the output of the trained model includes information indicating an object type and a circumscribing rectangle of the object. As a method for the detection execution unit 13-2 to detect an object from a feature map, for example, the above-mentioned Faster R-CNN, SSD, etc. can be mentioned.

[0050] <Example of object detection processing> Fig. 7 is a diagram showing a specific example of the object detection process according to the exemplary embodiment 2. In the example of Fig. 7, the main image IMG1_1 is an example of the first image IMG1, and the additional image IMG2_1 is an example of the second image IMG2. In the example of Fig. 7, the image acquisition unit 11 acquires the main image IMG1_1, which is an image of a candidate area extracted by the above-mentioned RPN, and the additional image IMG2_1, which is a background image of the candidate area. The main image IMG1_1 is a part of an image in which an object is photographed, and the additional image IMG2_1 is a part of a photographed image corresponding to the main image IMG1_1 image and does not include an object.

[0051] The main image IMG1_1 includes an object o1 and an object o2. The object o1 is an object to be detected. On the other hand, the object o2 is an object that is also included in the additional image IMG2_1 and does not need to be detected. In this way, the feature map MAP1_1 includes an object o2 that is a false focus, different from the object o1 that is the object to be detected.

[0052] The calculation unit 12 calculates a feature map MAP1_1 by inputting the main image IMG1_1 to a first model MD1. The feature map MAP1_1 is an example of the first map MAP1. The calculation unit 12 also calculates a weight map MAP2_1 by inputting the main image IMG1_1 and the added image IMG2_1 to a second model MD2. The weight map MAP2_1 is an example of the second map MAP2. Here, since the object o2 is included in both the main image IMG1_1 and the added image IMG2_1, the object o2 does not appear or is unlikely to appear in the weight map MAP2_1 that represents the difference between the two.

[0053] The detection unit 13 calculates a feature map MAP3_1 by multiplying the feature map MAP1_1 by the weight map MAP2_1. The feature map MAP3_1 is an example of a third map. By multiplying the feature map MAP1_1 by the weight map MAP2_1, the object o2 included in the feature map MAP1_1 does not appear or is less likely to appear in the feature map MAP3_1.

[0054] The detection unit 13 calculates a detection result DR_1 of the object (a re-estimation result of the type of the object and the bounding rectangle of the object) by referring to the feature map MAP3_1. The detection result DR_1 is presented by the presentation unit 15, for example.

[0055] <Object detection method flow> FIG. 8 is a flow diagram showing an example of a flow of an object detection method according to the second exemplary embodiment.

[0056] (Step S201) In step S201, the calculation unit 12 calculates a feature map MAP1_1 from the main image IMG1_1.

[0057] (Step S202) In step S202, the determination unit 14 determines whether or not an additional image IMG2_1 is present. As an example, the determination unit 14 determines whether or not an additional image IMG2_1 is present by referring to a predetermined flag (for example, a flag attached to the main image IMG1_1). If an additional image IMG2_1 is present ("YES" in step S202), the determination unit 14 proceeds to the process of step S203. On the other hand, if an additional image IMG2_1 is not present ("NO" in step S202), the determination unit 14 proceeds to the process of step S204.

[0058] (Step S203) In step S203, the detection unit 13 multiplies the weighting map MAP2_1 calculated from the added image IMG2_1 by the feature map MAP1_1 to calculate a feature map MAP3_1.

[0059] (Step S204) In step S204, the detection unit 13 calculates a detection result of an object from the feature map MAP3_1 calculated in step S203.

[0060] <Effects of information processing devices> As described above, in the information processing device 1A according to this exemplary embodiment, when the image acquisition unit 11 acquires the second image IMG2 in addition to the first image IMG1, the detection unit 13 performs object detection by referring to the third map obtained by multiplying the first map MAP1 by the second map MAP2. Therefore, according to the information processing device 1A according to this exemplary embodiment, by performing object detection by referring to the third map obtained by multiplying the first map MAP1 by the second map MAP2, an effect is obtained in which an object can be detected with higher accuracy.

[0061] Moreover, the information processing device 1A according to the present exemplary embodiment is configured to further include a determination unit 14 that performs a determination process to determine whether the image acquisition unit 11 acquires the first image IMG1 or acquires the first image IMG1 and the second image IMG2. Therefore, according to the information processing device 1A according to the present exemplary embodiment, it is possible to obtain an effect that an object can be detected in both cases where a second image is acquired and where it is not acquired, and that an object can be detected more accurately when a second image is present. More specifically, for example, in a situation where a background image may be acquired in addition to a main image, the background image can be utilized to improve the accuracy during inference.

[0062] Furthermore, in the information processing device 1A according to the present exemplary embodiment, a configuration is adopted in which the determination unit 14 performs the above determination process by referring to a flag indicating whether to acquire the first image IMG1 or to acquire the first image IMG1 and the second image IMG2. Therefore, according to the information processing device 1A according to the present exemplary embodiment, by determining whether to acquire the second image by referring to the flag, it is possible to obtain an effect that an object can be detected in both cases where the second image is acquired and where it is not acquired, and that an object can be detected more accurately when the second image is present.

[0063] Exemplary embodiment 3 The third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and the description thereof will not be repeated.

[0064] <Configuration of information processing device> 9 is a block diagram showing the configuration of an information processing device 1B according to exemplary embodiment 3. A control unit 10A of the information processing device 1B includes a teacher data acquisition unit 16, a first learning unit 17, and a second learning unit 18 in addition to an image acquisition unit 11, a calculation unit 12, a detection unit 13, a determination unit 14, and a presentation unit 15. The teacher data acquisition unit 16, the first learning unit 17, and the second learning unit 18 configure the learning device according to the present specification.

[0065] (Teacher Data Acquisition Department) The teacher data acquisition unit 16 acquires teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images. Here, the first image and the second image are as described in the above-mentioned exemplary embodiment 2. As an example, the label information includes information indicating the type of the object.

[0066] (First Learning Section) The first learning unit 17 refers to the first image and the label information included in the teacher data to learn a first model MD1 by machine learning. As described above, the first model MD1 is a model used when the calculation unit 12 calculates the first map MAP1, and is, for example, a convolutional neural network. In the present exemplary embodiment, as an example, even if the teacher data includes a second image, the first learning unit 17 may learn the first model MD1 by supervised machine learning using a set of the first image and the label information without using the second image.

[0067] (Second Learning Section) The second learning unit 18 refers to the first image, the second image, and the label information included in the teacher data, and learns the first model MD1 and the second model MD2 by machine learning. As described above, the second model MD2 is a model used when the calculation unit 12 calculates the second map MAP2, and is, for example, a convolutional neural network. At this time, the second learning unit 18 may also use a loss function that reduces the difference between the first map MAP1 before the weight map is applied and the third map MAP3 after the weight map is applied.

[0068] <Effects of information processing devices> Further, the information processing device 1B according to the present exemplary embodiment includes a teacher data acquisition unit 16 that acquires teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images, a first learning unit 17 that learns a first model MD1 by machine learning with reference to the first image and the label information included in the teacher data, and a second learning unit 18 that learns the first model MD1 and the second model MD2 by machine learning with reference to the first image, the second image, and the label information included in the teacher data. Therefore, according to the information processing device 1B according to the present exemplary embodiment, in addition to the effect of the object detection device 1 according to the exemplary embodiment 1, an effect of providing a model that can realize accurate object detection by using an image such as a background image in combination depending on the situation is obtained.

[0069] [Example] An example according to the present disclosure will be described below. In this example, the information processing devices 1A and 1B according to the above-described exemplary embodiments are applied to the medical and healthcare fields. In this example, a first image IMG1 is an image taken by an endoscopic examination of a subject. A second image IMG2 is an image taken by a previous endoscopic examination of the same subject. The second image IMG2 is an image taken when no lesion is detected, and is an image of the same location as the first image IMG1.

[0070] In this embodiment, the object detected by the detection unit 13 is a lesion detected from an image captured by an endoscopic examination of the subject. If a past endoscopic examination image (second image IMG2) of the subject is available, the detection unit 13 performs lesion detection using the past endoscopic image. The presentation unit 15 presents the lesion detection result to a medical professional.

[0071] The medical staff refers to the presented lesion detection result and, for example, decides how to deal with the subject. In other words, the presentation unit 15 outputs the lesion detection result to support the medical staff's decision making. That is, according to this embodiment, the information processing devices 1A and 1B can support the medical staff's decision making.

[0072] Also, for example, the presentation unit 15 may present to a medical professional a model generated by machine learning the correspondence between the lesion detection result and the countermeasure, and the lesion detection result of the subject. The method of determining the countermeasure is not limited to the above-mentioned method. In this way, the information processing device can support the user's decision-making.

[0073] Furthermore, according to this embodiment, it is possible to detect objects (lesions) both when there are and when there are not past endoscopic examination images of the subject, and it is possible to detect lesions with greater accuracy when there are past endoscopic examination images of the subject.

[0074] [Software implementation example] Some or all of the functions of the object detection device 1, information processing devices 1A, 1B, and learning device 2 (hereinafter referred to as "object detection device 1, etc.") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0075] In the latter case, the object detection device 1 etc. is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 10. The computer C has at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the object detection device 1 etc. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the object detection device 1 etc.

[0076] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0077] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0078] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0079] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0080] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0081] (Appendix 1) An object detection device comprising: an image acquisition means for acquiring a first image; a calculation means for calculating a first map from the first image using a first model; and a detection means for performing object detection by at least referring to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0082] (Appendix 2) An object detection device as described in Appendix 1, wherein when the image acquisition means acquires a second image in addition to the first image, the detection means performs object detection by referring to a third map obtained by multiplying the first map by the second map.

[0083] (Appendix 3) 3. The object detection device according to claim 1 or 2, further comprising a determination means for performing a determination process to determine whether the image acquisition means acquires the first image, or acquires the first image and the second image.

[0084] (Appendix 4) The object detection device according to claim 3, wherein the determination means performs the determination process by referring to a flag indicating whether to acquire the first image or to acquire the first image and the second image.

[0085] (Appendix 5) 3. The object detection device according to claim 1 or 2, further comprising: a training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means for training the first model through machine learning by referring to the first images and the label information included in the training data; and a second learning means for training the first model and the second model through machine learning by referring to the first image, the second image, and the label information included in the training data.

[0086] (Appendix 6) The apparatus further includes a presentation unit that outputs a detection result by the detection unit, The detection means detects 3. The object detection device according to claim 1 or 2, wherein the object is a lesion that can be detected from an image taken by an endoscopic examination of a subject, and the presentation means outputs a detection result of the lesion to assist a medical professional in making decisions.

[0087] (Appendix 7) A learning device comprising: a teacher data acquisition means for acquiring teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means for training a first model that calculates a first map from a first image by referring to the first image and the label information included in the teacher data; and a second learning means for training the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data.

[0088] (Appendix 8) An object detection method including: acquiring a first image; calculating a first map from the first image using a first model; and performing object detection by at least referring to the first map; when a second image is acquired in addition to the first image, in the calculating step, a second map is calculated from the second image or from the first image and the second image using a second model; and in the object detection step, object detection is performed by referring to the second map in addition to the first map.

[0089] (Appendix 9) A learning method including: acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; training a first model that calculates a first map from a first image by referring to the first images and the label information included in the training data; and training the first model and a second model that calculates a second map from a second image by referring to the first images, the second images, and the label information included in the training data.

[0090] (Appendix 10) An object detection program that causes a computer to function as an image acquisition means for acquiring a first image, a calculation means for calculating a first map from the first image using a first model, and a detection means for performing object detection by at least referring to the first map, wherein when the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model, and the detection means performs object detection by referring to the second map in addition to the first map.

[0091] (Appendix 11) A learning program that causes a computer to function as: a teacher data acquisition means that acquires teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means that learns a first model that calculates a first map from a first image by referring to the first image and the label information included in the teacher data; and a second learning means that learns the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data.

[0092] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows. An object detection device having at least one processor, the processor executing an image acquisition process to acquire a first image, a calculation process to calculate a first map from the first image using a first model, and a detection process to perform object detection by at least referring to the first map, wherein when a second image is acquired in addition to the first image in the image acquisition process, in the calculation process, a second map is calculated from the second image or from the first image and the second image using a second model, and in the detection process, object detection is performed by referring to the second map in addition to the first map.

[0093] The object detection device may further include a memory that stores a program for causing the processor to execute the image acquisition process, the calculation process, and the detection process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium.

[0094] A learning device comprising at least one processor, the processor executing a teacher data acquisition process for acquiring teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images, a first learning process for training a first model for calculating a first map from a first image by referring to the first image and the label information included in the teacher data, and a second learning process for training the first model and a second model for calculating a second map from a second image by referring to the first image, the second image, and the label information included in the teacher data.

[0095] The learning device may further include a memory, and the memory may store a program for causing the processor to execute the teacher data acquisition process, the first learning process, and the second learning process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0096] 1. Object detection device 1A, 1B Information processing device 2 Learning device 11 Image acquisition section 12 Calculation section 13 Detection section 14 Judgment section 15 Presentation section 16, 21 Teacher data acquisition section 17, 22 First Study Section 18, 23 Second Study Section

Claims

1. An image acquisition means for acquiring a first image; a calculation means for calculating a first map from the first image using a first model; A detection means for performing object detection by at least referring to the first map; Equipped with When the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model; The detection means performs object detection by referring to the second map in addition to the first map. Object detection device.

2. When the image acquisition means acquires a second image in addition to the first image, The detection means performs object detection by referring to a third map obtained by multiplying the first map by the second map. The object detection device according to claim 1 .

3. The image capturing device further includes a determining unit that performs a determining process to determine whether the image capturing device captures the first image or the first image and the second image. The object detection device according to claim 1 or 2.

4. A training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means for learning the first model by machine learning with reference to the first image and the label information included in the training data; a second learning means for learning the first model and the second model by machine learning with reference to the first image, the second image, and the label information included in the training data; The object detection device according to claim 1 or 2, further comprising:

5. Further comprising a presentation means for outputting a detection result by the detection means, the object detected by the detection means is a lesion that can be detected from an image captured by an endoscopic examination of a subject; The presentation means outputs the lesion detection result to support decision-making by a medical professional. The object detection device according to claim 1 or 2.

6. A training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means for learning a first model that calculates a first map from a first image by referring to the first image and the label information included in the training data; a second learning means for learning the first model and a second model for calculating a second map from a second image by referring to the first image, the second image, and the label information included in the training data; A learning device comprising:

7. Obtaining a first image; calculating a first map from the first image using a first model; performing object detection with at least reference to the first map; Including, When a second image is acquired in addition to the first image, In the calculating step, a second map is calculated from the second image or from the first image and the second image using a second model; In the step of detecting an object, the object detection is performed by referring to the second map in addition to the first map. Object detection methods.

8. Obtaining teacher data including one or more first images, one or more second images, and label information indicating objects included in the first images; A first model for calculating a first map from a first image is trained by referring to the first image and the label information included in the training data; learning the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information included in the training data; Learning methods including:

9. A computer comprising: An image capture means for capturing a first image; a calculation means for calculating a first map from the first image using a first model; and An object detection program that functions as a detection means that performs object detection by at least referring to the first map, When the image acquisition means acquires a second image in addition to the first image, the calculation means calculates a second map from the second image or from the first image and the second image using a second model; The object detection program, wherein the detection means performs object detection by referring to the second map in addition to the first map.

10. A computer comprising: A training data acquisition means for acquiring training data including one or more first images, one or more second images, and label information indicating objects included in the first images; a first learning means for learning a first model that calculates a first map from a first image by referring to the first image and the label information included in the training data; and A learning program that functions as a second learning means that learns the first model and a second model that calculates a second map from a second image by referring to the first image, the second image, and the label information contained in the training data.

Images