Information processing device, information processing method, and program

Abstract

An information processing device comprising: a rejected object image acquisition unit that acquires an image of a rejected object predicted not to be a prescribed object by a first machine learning model; an uncertainty index calculation unit that references the image of the rejected object and calculates a first index indicating the uncertainty of the prediction for the rejected object; and a determination unit that references the first index and determines whether or not the rejected object was overlooked by the first machine learning model.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] There is a known technology that uses a machine learning model to detect a specific object contained in an image. However, in this technology, the machine learning model may miss the specific object. Therefore, there is a need for a technology that reduces the possibility of missing the object.

[0003] For example, Patent Document 1 describes a method for determining whether the track of an object missed by an object detector matches a reference track, and if so, specifying that there is a possibility of a false negative object detection (missed object detection).

[0004] Japanese Patent Application Publication No. 2022-151740

[0005] In the method described in Patent Literature 1, if the track of a missed object does not match the reference track, it is not defined as having a false negative object detection. In other words, the method described in Patent Literature 1 misses an object whose movement does not match the reference track. Therefore, the method described in Patent Literature 1 has the problem that it may not be possible to detect missed objects with high accuracy.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology for detecting with high accuracy oversights by a machine learning model that detects a specified object included in an image.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes rejected object image acquisition means that detects an object included in an input image and acquires an image of a rejected object that is predicted to not be the specified object by a first machine learning model that predicts whether the detected object is a specified object; uncertainty index calculation means that refers to the image of the rejected object and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and determination means that refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0008] An information processing device according to an exemplary aspect of the present disclosure includes rejected object image acquisition means that detects an object included in an input image and acquires an image of a rejected object that is predicted to not be the predetermined object by a first machine learning model that predicts whether the detected object is a predetermined object; moving object index calculation means that refers to the image of the rejected object and calculates an index indicating the likelihood of the rejected object being a moving object; and determination means that refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

[0009] An information processing method according to an exemplary aspect of the present disclosure includes: a rejected object image acquisition process in which at least one processor detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; an uncertainty index calculation process in which the at least one processor refers to the image of the rejected object and calculates a first index that indicates uncertainty of the prediction regarding the rejected object; and a determination process in which the at least one processor refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0010] An information processing method according to an exemplary aspect of the present disclosure includes: a rejected object image acquisition process in which at least one processor detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; a moving object index calculation process in which the at least one processor refers to the image of the rejected object and calculates an index indicating the likelihood of the rejected object being a moving object; and a determination process in which the at least one processor refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

[0011] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, and causes the computer to function as: rejected object image acquisition means that detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; uncertainty index calculation means that refers to the image of the rejected object and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and determination means that refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0012] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, and causes the computer to function as: rejected object image acquisition means that detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; moving object index calculation means that refers to the image of the rejected object and calculates an index that indicates the likelihood of the rejected object being a moving object; and determination processing that refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

[0013] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology can be provided that can detect with high accuracy oversights caused by machine learning models that detect predetermined objects included in images.

[0014] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 4 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 5 is an example of an image according to the present disclosure. FIG. 6 is another example of an image according to the present disclosure. FIG. 7 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 8 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 9 is a diagram showing an object detected by a first machine learning model according to the present disclosure. FIG. 10 is a diagram showing an example of an image of a detected object and an image of a rejected object according to the present disclosure. FIG. 11 is a diagram showing an example of an image output by an output unit according to the present disclosure. FIG. 12 is a diagram showing another example of an image output by an output unit according to the present disclosure. FIG. 13 is a block diagram showing a configuration of a computer that functions as an information processing device according to the present disclosure.

[0015] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0016] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0017] (Configuration of Information Processing Device 1) The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes a rejected object image acquisition unit 11, an uncertainty index calculation unit 12, and a determination unit 13. In this exemplary embodiment, the rejected object image acquisition unit 11, the uncertainty index calculation unit 12, and the determination unit 13 respectively realize a rejected object image acquisition means, an uncertainty index calculation means, and a determination means.

[0018] The rejected object image acquisition unit 11 detects objects included in an input image, and acquires images of rejected objects that a first machine learning model that predicts whether the detected object is a predetermined object predicts are not the predetermined object. The rejected object image acquisition unit 11 supplies the acquired images of the rejected objects to the uncertainty index calculation unit 12.

[0019] The uncertainty index calculation unit 12 calculates a first index indicating the uncertainty of the prediction regarding the rejected object by referring to the image of the rejected object acquired by the rejected object image acquisition unit 11. The uncertainty index calculation unit 12 supplies the calculated first index to the determination unit 13.

[0020] The determination unit 13 refers to the first index calculated by the uncertainty index calculation unit 12 and determines whether the rejected object is an object overlooked by the first machine learning model.

[0021] (Effects of Information Processing Device 1) As described above, the information processing device 1 employs a configuration including: a rejected object image acquisition unit 11 that acquires images of rejected objects that are predicted not to be the predetermined object by a first machine learning model that detects an object included in an input image and predicts whether the detected object is a predetermined object; an uncertainty index calculation unit 12 that refers to the images of the rejected objects acquired by the rejected object image acquisition unit 11 and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and a determination unit 13 that refers to the first index calculated by the uncertainty index calculation unit 12 and determines whether the rejected object has been overlooked by the first machine learning model.

[0022] Therefore, the information processing device 1 has the effect of being able to detect with high accuracy any oversights caused by a machine learning model that detects a specified object contained in an image.

[0023] (Flow of Information Processing Method S1) The flow of the information processing method S1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes a rejected object image acquisition process S11, an uncertainty index calculation process S12, and a determination process S13.

[0024] In the rejected object image acquisition process S11, the rejected object image acquisition unit 11 detects objects included in an input image, and acquires images of rejected objects that are predicted to be not the predetermined object by a first machine learning model that predicts whether the detected object is a predetermined object. The rejected object image acquisition unit 11 supplies the acquired images of the rejected objects to the uncertainty index calculation unit 12.

[0025] (Uncertainty index calculation process S12) In the uncertainty index calculation process S12, the uncertainty index calculation unit 12 refers to the image of the rejected object acquired in the rejected object image acquisition process S11, and calculates a first index indicating the uncertainty of the prediction regarding the rejected object. The uncertainty index calculation unit 12 supplies the calculated first index to the determination unit 13.

[0026] (Determination Process S13) In the determination process S13, the determination unit 13 refers to the first index calculated in the uncertainty index calculation process S12, and determines whether or not the rejected object has been overlooked by the first machine learning model.

[0027] (Effects of Information Processing Method S1) As described above, the information processing method S1 employs a configuration including: a rejected object image acquisition process S11 in which the rejected object image acquisition unit 11 detects an object included in an input image and acquires an image of a rejected object that is predicted to be not the specified object by a first machine learning model that predicts whether or not the detected object is a predetermined object; an uncertainty index calculation process S12 in which the uncertainty index calculation unit 12 refers to the image of the rejected object acquired in the rejected object image acquisition process S11 and calculates a first index indicating the uncertainty of the prediction regarding the rejected object; and a determination process S13 in which the determination unit 13 refers to the first index calculated in the uncertainty index calculation process S12 and determines whether or not the rejected object has been overlooked by the first machine learning model. Therefore, the information processing method S1 can achieve effects similar to those of the information processing device 1 described above.

[0028] (Configuration of Information Processing Device 2) The configuration of the information processing device 2 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2. As shown in Fig. 3, the information processing device 2 includes a rejected object image acquisition unit 21, a moving object index calculation unit 22, and a determination unit 23. In this exemplary embodiment, the rejected object image acquisition unit 21, the moving object index calculation unit 22, and the determination unit 23 respectively realize rejected object image acquisition means, moving object index calculation means, and determination means.

[0029] The rejected object image acquisition unit 21 detects objects included in an input image, and acquires images of rejected objects that are predicted to be not the predetermined object by a first machine learning model that predicts whether the detected object is a predetermined object. The rejected object image acquisition unit 11 supplies the acquired images of the rejected objects to the uncertainty index calculation unit 12.

[0030] The moving object index calculation unit 22 calculates an index indicating the likelihood that the rejected object is a moving object, with reference to the image of the rejected object acquired by the rejected object image acquisition unit 21. The moving object index calculation unit 22 supplies the calculated index to the determination unit 23.

[0031] The determination unit 23 refers to the index calculated by the moving object index calculation unit 22 and determines whether or not the rejected object is an object that has been overlooked by the machine learning model.

[0032] (Effects of Information Processing Device 2) As described above, the information processing device 2 employs a configuration including: a rejected object image acquisition unit 21 that acquires images of rejected objects predicted by a first machine learning model that detects objects included in an input image and predicts whether the detected object is a predetermined object; a moving object index calculation unit 22 that refers to the images of the rejected objects acquired by the rejected object image acquisition unit 21 and calculates an index indicating the likelihood of the rejected objects being moving objects; and a determination unit 23 that refers to the index calculated by the moving object index calculation unit 22 and determines whether the rejected objects have been overlooked by the machine learning model. Therefore, the information processing device 2 can achieve the same effects as the information processing device 1 described above.

[0033] (Flow of Information Processing Method S2) The flow of information processing method S2 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of information processing method S2. As shown in Fig. 4, information processing method S2 includes rejected object image acquisition processing S21, moving object index calculation processing S22, and determination processing S23.

[0034] (Rejected object image acquisition process S21) In the rejected object image acquisition process S21, the rejected object image acquisition unit 21 detects objects included in the input image, and acquires images of rejected objects that the first machine learning model, which predicts whether the detected object is a predetermined object or not, predicts are not the predetermined object. The rejected object image acquisition unit 21 supplies the acquired images of the rejected objects to the moving object index calculation unit 22.

[0035] (Moving object index calculation process S22) In the moving object index calculation process S22, the moving object index calculation unit 22 refers to the image of the rejected object acquired in the rejected object image acquisition process S21, and calculates an index indicating the likelihood of the rejected object being a moving object. The moving object index calculation unit 22 supplies the calculated index to the determination unit 23.

[0036] (Determination Process S23) In the determination process S23, the determination unit 23 refers to the index calculated in the moving object index calculation process S22 and determines whether or not the rejected object is an object overlooked by the machine learning model.

[0037] (Effects of Information Processing Method S2) As described above, the information processing method S2 employs a configuration including: a rejected object image acquisition process S21 in which the rejected object image acquisition unit 21 detects an object included in an input image, and acquires an image of a rejected object that is predicted to not be the specified object by a first machine learning model that predicts whether the detected object is a predetermined object; a moving object index calculation process S22 in which the moving object index calculation unit 22 refers to the image of the rejected object acquired in the rejected object image acquisition process S21, and calculates an index indicating the likelihood of the rejected object being a moving object; and a determination process S23 in which the determination unit 23 refers to the index calculated in the moving object index calculation process S22, and determines whether the rejected object has been overlooked by the machine learning model. Therefore, the information processing method S2 can achieve effects similar to those of the information processing device 2 described above.

[0038] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0039] (Overview of Information Processing Device 3) The information processing device 3 detects objects included in an image PI and predicts whether the detected objects are predetermined objects. Furthermore, the information processing device 3 determines whether, among the detected objects, rejected objects that are predicted not to be predetermined objects are objects that were overlooked despite being predetermined objects.

[0040] Although the image PI is not particularly limited, in this exemplary embodiment, a case will be described in which the image PI is a satellite image taken by a synthetic aperture radar (SAR).

[0041] Furthermore, the predetermined object detected by the information processing device 3 is not particularly limited, but in this exemplary embodiment, a case will be described in which the predetermined object is a moving object whose position changes over time.

[0042] Furthermore, the moving object is not particularly limited, but examples include movable equipment such as aircraft, ships, vehicles, and cranes. Another example is an object that has moved from outside the angle of view into the angle of view. Yet another example is an object that was not included in the angle of view at a certain time, but is included in the angle of view after that time (for example, several days later). Examples of such objects include newly installed facilities and equipment.

[0043] Examples of the image PI are shown in Figures 5 and 6. Figure 5 shows one example of the image PI (image PI1), and Figure 6 shows another example of the image PI (image PI2).

[0044] Images PI1 and PI2 are satellite images taken at different times. Image PI1 includes objects OB1 to OB6. Image PI2 includes objects OB1 to OB7. Objects OB5 and OB6 are moving objects because their positions change between images PI1 and PI2. Object OB7 is also a moving object because it is outside the angle of view in image PI1 but within the angle of view in image PI2.

[0045] (Configuration of information processing device 3) The configuration of the information processing device 3 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the information processing device 3. As shown in Fig. 7, the information processing device 3 includes a control unit 30, a storage unit 50, a communication unit 60, and an input / output unit 70.

[0046] The storage unit 50 stores data referenced by the control unit 30. Examples of the storage unit 50 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0047] Examples of data stored in the storage unit 50 include an image PI, an image DP of a detected object predicted to be a predetermined object, an image NDP of a rejected object predicted not to be a predetermined object, and a machine learning model LM. The machine learning model LM being stored in the storage unit 50 means that parameters defining the machine learning model LM are stored in the storage unit 50. The machine learning model LM may also be stored in another device capable of transmitting and receiving data to and from the information processing device 3. Examples of the image DP of a detected object and the image NDP of a rejected object will be described later.

[0048] As shown in FIG. 7, the machine learning model LM includes a first machine learning model LM1, a second machine learning model LM2, and a third machine learning model LM3.

[0049] The first machine learning model LM1 is a machine learning model trained to detect objects included in input images and predict whether the detected objects are predetermined objects. Specifically, the first machine learning model LM1 is a machine learning model trained by machine learning using pairs of images including the predetermined objects and correct labels as training data. The first machine learning model LM1 outputs, among the detected objects, images DP of detected objects predicted to be the predetermined objects and images NDP of rejected objects predicted not to be the predetermined objects.

[0050] The first machine learning model LM1 may receive as input an image containing an object extracted from an image PI by image processing such as CFAR (Constant False Alarm Rate) or background subtraction. In this case, if the first machine learning model LM1 predicts that an object contained in the input image is a predetermined object, it outputs the input image as an image DP of a detected object. On the other hand, if the first machine learning model LM1 predicts that the object contained in the input image is not the predetermined object, it outputs the input image as an image NDP of a rejected object. In other words, the first machine learning model LM1 may have a two-stage configuration. Alternatively, the first machine learning model LM1 may be an end-to-end deep learning model with multiple layers that receives an image PI as input and outputs an image DP of a detected object or an image NDP of a rejected object.

[0051] The second machine learning model LM2 is a machine learning model that has been trained to use an image as input and output an index indicating the uncertainty of a prediction regarding a rejected object. The "index indicating uncertainty" can be rephrased as an index indicating the degree of spread of the probability distribution of the prediction result, or an index indicating the magnitude of deviation of the prediction result. Furthermore, the "index indicating the uncertainty of a prediction regarding a rejected object" can be rephrased as an index indicating the uncertainty of a prediction that the rejected object is not a specified object.

[0052] One example of a method for the second machine learning model LM2 to output an index indicating the uncertainty of a prediction regarding a rejected object contained in an image is a method using a natural posterior network (NatPN). For example, the second machine learning model LM2 is trained by referring to the training data used to train the first machine learning model LM1, so that when an image dissimilar to the image used to train the first machine learning model LM1 for a predetermined object is input to the second machine learning model LM2, the second machine learning model LM2 outputs an index indicating the uncertainty used to identify "objects that the first machine learning model LM1 has not been able to learn."

[0053] The first machine learning model LM1 and the second machine learning model LM2 are not limited to being independent as shown in Fig. 7. For example, the feature extraction unit that executes the process of extracting features of an input image may be configured to be shared by the first machine learning model LM1 and the second machine learning model LM2.

[0054] The third machine learning model LM3 is a machine learning model that has been trained to take an image as input and output an index indicating the likelihood that an object included in the image is a moving object. Specifically, the third machine learning model is a machine learning model that has been trained using pairs of images including moving objects and correct labels as training data. The third machine learning model LM3 may be configured to receive, in addition to a target image, an image captured at the same location as the target image but at a different time. Furthermore, the third machine learning model LM3 may be configured to receive, in addition to the target image, a reference image (e.g., a background image not including a moving object).

[0055] The communication unit 60 is an interface for transmitting and receiving data via a network. Examples of the communication unit 60 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and a USB-compliant connector.

[0056] For example, the communication unit 60 receives an image PI from another device, and outputs a detection result to the other device.

[0057] The input / output unit 70 is an interface that receives input of data and outputs data. Examples of the input / output unit 70 include, but are not limited to, a microphone, a camera, a gaze input device, a keyboard, a touchpad, a speaker, and a liquid crystal display.

[0058] As one example, the input / output unit 70 acquires data input by a user and supplies the acquired data to the control unit 30. As another example, the input / output unit 70 displays an image indicated by an image signal supplied from the control unit 30.

[0059] (Control Unit 30) The control unit 30 controls each component included in the information processing device 3. Furthermore, as shown in Fig. 7, the control unit 30 includes a rejected object image acquisition unit 31, an uncertainty index calculation unit 32, a determination unit 33, a moving object index calculation unit 34, an output unit 35, and an object detection unit 36. In this exemplary embodiment, the rejected object image acquisition unit 31, the uncertainty index calculation unit 32, the determination unit 33, the moving object index calculation unit 34, and the output unit 35 respectively realize rejected object image acquisition means, uncertainty index calculation means, determination means, moving object index calculation means, and output means.

[0060] The rejected object image acquisition unit 31 acquires an image NDP of a rejected object that the first machine learning model LM1 predicts is not a predetermined object. The rejected object image acquisition unit 31 stores the acquired rejected object image NDP in the storage unit 50.

[0061] The uncertainty index calculation unit 32 refers to the rejected object image NDP and calculates a first index indicating the uncertainty of the prediction regarding the rejected object. As an example, the uncertainty index calculation unit 32 calculates the first index by inputting the rejected object image NDP to the second machine learning model LM2. With this configuration, the uncertainty index calculation unit 32 can calculate an appropriate first index. The uncertainty index calculation unit 32 supplies the calculated first index to the determination unit 33.

[0062] The determination unit 33 refers to at least one of the first index and a second index indicating the likelihood that the rejected object is a moving object (the second index will be described in detail later), and determines whether the rejected object has been overlooked by the first machine learning model LM1. The determination unit 33 supplies the determination result to the output unit 35.

[0063] The moving object index calculation unit 34 refers to the rejected object image NDP and calculates a second index indicating the moving object-likelihood of the rejected object (in other words, the certainty that the rejected object is a moving object). As an example, the moving object index calculation unit 34 calculates the second index by inputting the rejected object image NDP to the third machine learning model LM3. With this configuration, the moving object index calculation unit 34 can calculate an appropriate second index. The moving object index calculation unit 34 supplies the calculated second index to the determination unit 33.

[0064] Furthermore, as described above, the determination unit 33 may refer to the first index and the second index to determine whether the rejected object was overlooked by the first machine learning model LM1. With this configuration, when the predetermined object is a moving object, the information processing device 3 can detect the oversight if the first machine learning model LM1 overlooks the moving object.

[0065] The output unit 35 outputs data via the communication unit 60 or to the input / output unit 70. As an example, the output unit 35 refers to the determination result by the determination unit 33, and outputs an image related to a rejected object that the determination unit 33 has determined to be a missed detection by the first machine learning model LM1. With this configuration, the information processing device 3 can present information related to the rejected object to the user. Examples of images output by the output unit 35 will be described later.

[0066] The object detection unit 36 ​​inputs the image PI to the first machine learning model LM1, and acquires an image DP of the detected object predicted to be a predetermined object. The object detection unit 36 ​​stores the acquired image DP of the detected object in the memory unit 50.

[0067] (Processing flow executed by information processing device 3) The processing flow (information processing method S3) executed by information processing device 3 will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the flow of information processing method S3. As an example, the following description will be given taking as an example a case where the first machine learning model LM1 detects a ship as a predetermined object.

[0068] (Step S31) In step S31, the object detection unit 36 ​​inputs the image PI to the first machine learning model LM1.

[0069] (Step S32) In step S32, the object detection unit 36 ​​acquires an image DP of the detected object predicted to be a predetermined object (a ship) by the first machine learning model LM1. The object detection unit 36 ​​stores the acquired image DP of the detected object in the storage unit 50.

[0070] (Step S33) In step S33, the rejected object image acquisition unit 31 acquires an image NDP of a rejected object that is predicted not to be a predetermined object (ship) by the first machine learning model LM1. The rejected object image acquisition unit 31 stores the acquired rejected object image NDP in the storage unit 50.

[0071] An example of the processing in steps S31 to S33 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram showing an object detected by the first machine learning model LM1. Fig. 10 is a diagram showing an example of an image DP of a detected object and an image NDP of a rejected object.

[0072] First, in step S31, when the object detection unit 36 ​​inputs the image PI2 shown in Fig. 7 to the first machine learning model LM1, the first machine learning model LM1 detects the objects OB1 to OB7 shown in Fig. 7. As an example, the first machine learning model LM1 indicates that the objects DO1 to DO7 have been detected by surrounding the detected objects DO1 to DO7 with a rectangle, as shown in Fig. 9.

[0073] Furthermore, the first machine learning model LM1 outputs images of objects that are predicted to be ships among the detected objects DO1 to DO7 as detected object images DP. On the other hand, the first machine learning model LM1 outputs images of objects that are predicted not to be ships among the detected objects DO1 to DO7 as rejected object images NDP. As an example, the following describes a case where the first machine learning model LM1 predicts that objects DO1 and DO2 are ships and predicts that objects DO3 to DO7 are not ships.

[0074] Next, the first machine learning model LM1 outputs a detected object image DP1 including an object DO1 predicted to be a ship, and a detected object image DP2 including an object DO2 predicted to be a ship, as shown in Fig. 10. In step S32, the object detection unit 36 ​​acquires the detected object image DP1 and the detected object image DP2.

[0075] Furthermore, the first machine learning model LM1 outputs rejected object images NDP1 to NDP5, each including objects DO3 to DO7 that are predicted not to be ships, as shown in Fig. 10. The rejected object image acquisition unit 31 acquires the rejected object images NDP1 to NDP5 in step S33.

[0076] (Step S34) In step S34, the uncertainty index calculation unit 32 calculates a first index by inputting the rejected object image NDP to the second machine learning model LM2. The uncertainty index calculation unit 32 supplies the calculated first index to the determination unit 33.

[0077] (Step S35) In step S35, the moving object index calculation unit 34 calculates a second index by inputting the rejected object image NDP to the third machine learning model LM3. The moving object index calculation unit 34 supplies the calculated second index to the determination unit 33.

[0078] (Step S36) In step S36, the determination unit 33 refers to at least one of the first index and the second index to determine whether the rejected object has been overlooked by the first machine learning model LM1. The determination unit 33 supplies the determination result to the output unit 35.

[0079] As an example, the determination unit 33 refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model LM1.

[0080] For example, if the value of the first index is higher than the first threshold, the determination unit 33 determines that the rejected object is an oversight by the first machine learning model LM1. On the other hand, if the value of the first index is equal to or lower than the first threshold, the determination unit 33 determines that the rejected object is not an oversight by the first machine learning model LM1. In other words, if the prediction by the first machine learning model LM1 that the rejected object is not a predetermined object is uncertain, the determination unit 33 determines that the rejected object is an oversight by the first machine learning model LM1.

[0081] As another example, the determination unit 33 refers to the second index and determines whether the rejected object has been overlooked by the first machine learning model LM1.

[0082] For example, if the value of the second index is higher than the second threshold, the determination unit 33 determines that the rejected object was overlooked by the first machine learning model LM1. On the other hand, if the value of the second index is equal to or lower than the second threshold, the determination unit 33 determines that the rejected object was not overlooked by the first machine learning model LM1. In other words, if it is certain that the rejected object is a moving object, the determination unit 33 determines that the rejected object was overlooked by the first machine learning model LM1.

[0083] As yet another example, the determination unit 33 refers to the first index and the second index to determine whether the rejected object has been overlooked by the first machine learning model LM1.

[0084] For example, if the value of the first index is higher than the first threshold and the value of the second index is higher than the second threshold (or if the value of the first index is higher than the first threshold, or if the value of the second index is higher than the second threshold), the determination unit 33 determines that the rejected object is an oversight by the first machine learning model LM1. On the other hand, if the value of the first index is equal to or lower than the first threshold and the value of the second index is equal to or lower than the second threshold, the determination unit 33 determines that the rejected object is not an oversight by the first machine learning model LM1.

[0085] Alternatively, if the sum of the first index value and the second index value is higher than the third threshold, the determination unit 33 determines that the rejected object was overlooked by the first machine learning model LM1. On the other hand, if the sum of the first index value and the second index value is equal to or lower than the third threshold, the determination unit 33 determines that the rejected object was not overlooked by the first machine learning model LM1.

[0086] As yet another example, the determination unit 33 may further refer to the confidence level output from the first machine learning model LM1.

[0087] As an example, if the confidence level output from the first machine learning model LM1 is equal to or lower than the fourth threshold, the determination unit 33 makes a determination using the above determination method. On the other hand, if the confidence level output from the first machine learning model LM1 is higher than the fourth threshold, the determination unit 33 determines that the rejected object is not an oversight by the first machine learning model LM1. In other words, if the confidence level of the prediction by the first machine learning model LM1 that the rejected object is not a specified object is high, the determination unit 33 determines that the rejected object is not an oversight by the first machine learning model LM1.

[0088] (Step S37) In step S37, the output unit 35 refers to the determination result by the determination unit 33, and outputs an image related to the rejected object that the determination unit 33 has determined to be an oversight by the first machine learning model LM1.

[0089] 11 and 12 show examples of images output by the output unit 35. Fig. 11 is a diagram showing one example of an image output by the output unit 35. Fig. 12 is a diagram showing another example of an image output by the output unit 35. As an example, the following describes a case where the determination unit 33 determines that the objects DO5, DO6, and DO7 shown in Fig. 9 are overlooked.

[0090] As an example, the output unit 35 outputs an image OP in which the detected objects DO1 and DO2 are surrounded by a solid-line rectangle in the image PI2 input to the first machine learning model LM1, as shown in Fig. 11. Furthermore, the output unit 35 surrounds the rejected objects DO5, DO6, and DO7 determined to be missed objects in the image OP with a dotted-line rectangle indicating that they are missed objects.

[0091] In this way, the output unit 35 outputs an image OP in which the positions of the rejected objects DO5, DO6, and DO7 determined to be missed by the first machine learning model LM1 can be distinguished from other positions. With this configuration, the information processing device 3 can present to the user which objects have been determined to be missed.

[0092] Furthermore, the information processing device 3 may be configured to allow the user to select at least one of the rejected objects determined to be missed. For example, the information processing device 3 may be configured to allow the user to input that the rejected object is not a predetermined object or that the rejected object is a predetermined object.

[0093] As an example, when the user selects the rejected object DO5 and inputs that the rejected object DO5 is not a predetermined object, the information processing device 3 stores information indicating that the rejected object DO5 is not a predetermined object in the storage unit 50. As another example, when the user selects the rejected object DO6 and inputs that the rejected object DO6 is a predetermined object, the information processing device 3 stores information in the storage unit 50 that indicates that the rejected object DO6 is a predetermined object.

[0094] Furthermore, the information processing device 3 may be configured to train the first machine learning model LM1 based on a user's input. For example, when the user inputs that the rejected object DO6 is a predetermined object, the information processing device 3 may train the first machine learning model LM1 using training data that is a set of an image of the rejected object DO6 and a correct label.

[0095] With this configuration, the information processing device 3 can continuously improve the accuracy of the first machine learning model LM1.

[0096] As another example, the output unit 35 displays text indicating the detected objects and the objects determined to be missed, as shown in Fig. 12, in addition to image PI2 of Fig. 9, which is an image showing the objects detected by the first machine learning model LM1. Fig. 12 shows that the detected objects detected as ships are objects DO1 and DO2. Fig. 12 also shows that the rejected objects determined to be missed are objects DO5, DO6, and DO7.

[0097] 12, the rejected objects DO5, DO6, and DO7 determined to be missed are associated with a first index value and a second index value, respectively. Furthermore, in FIG. 12, the display positions of the text for the objects DO5, DO6, and DO7 are arranged in descending order of the value of the first index. Note that the output unit 35 may set the display positions of the text for each rejected object in descending order of the value of the second index for a plurality of rejected objects.

[0098] As yet another example, the output unit 35 may be configured to enlarge and display the rejected objects. For example, after displaying the image of FIG. 11 , the output unit 35 may enlarge (for example, enlarge by several tens of times) and display the rejected objects in descending order of the value of the first index. With this configuration, the information processing device 3 can easily allow the user to visually confirm whether or not the rejected objects are the predetermined objects.

[0099] In this way, when there are multiple rejected objects that the determination unit 33 has determined to be missed by the first machine learning model LM1, the output unit 35 outputs an image that enables the user to recognize the order of at least any of the first index values ​​and second index values ​​of each of the multiple rejected objects. With this configuration, the information processing device 3 can present to the user which objects have been determined to be highly likely to have been missed.

[0100] (Effects of information processing device 3) As described above, the information processing device 3 refers to at least one of the first index indicating the uncertainty of the prediction that the rejected object is not a specified object and the second index indicating the likelihood of the rejected object being a moving object, and determines whether the rejected object has been overlooked by the first machine learning model LM1.

[0101] Therefore, in the information processing device 3, by referring to the first index, it is possible to detect the oversight of rejected objects that were not predicted to be specified objects due to insufficient learning of the first machine learning model LM1.

[0102] In addition, in the information processing device 3, by referring to the first index, it is possible to detect oversights with higher accuracy than when referring to the confidence level of the first machine learning model LM1, which is not controlled by what could not be learned.

[0103] Furthermore, in the information processing device 3, by referring to the first index and the second index, if the predetermined object is a moving object, it is possible to more accurately detect an oversight.

[0104] [Software Implementation Example] Some or all of the functions of the information processing devices 1, 2, and 3 (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0105] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 13. Figure 13 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0106] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0107] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0108] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0109] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0110] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0111] (Supplementary Note A1) An information processing device comprising: rejected object image acquisition means for acquiring an image of a rejected object that is predicted to not be the predetermined object by a first machine learning model that detects an object included in an input image and predicts whether the detected object is a predetermined object; uncertainty index calculation means for calculating a first index indicating uncertainty of the prediction regarding the rejected object by referring to the image of the rejected object; and determination means for determining whether the rejected object has been overlooked by the first machine learning model by referring to the first index.

[0112] (Supplementary Note A2) The information processing device according to Supplementary Note A1, wherein the uncertainty index calculation means calculates the first index by inputting an image of the rejected object into a second machine learning model that receives an image as input and outputs an index indicating uncertainty of a prediction regarding the rejected object included in the image.

[0113] (Supplementary Note A3) The information processing device according to Supplementary Note A1 or A2, wherein the predetermined object is a moving object whose position changes over time, and further comprising: moving object index calculation means for calculating a second index indicating a likelihood that the rejected object is a moving object by referring to an image of the rejected object; and the determination means for determining whether the rejected object has been overlooked by the first machine learning model by referring to the first index and the second index.

[0114] (Supplementary Note A4) The information processing device according to Supplementary Note A3, wherein the moving object index calculation means calculates the second index by inputting an image of the rejected object into a third machine learning model that receives an image as input and outputs an index indicating a likelihood of an object included in the image being a moving object.

[0115] (Supplementary Note A5) The information processing device according to any one of Supplementary Notes A1 to A4, further comprising: an output unit configured to output an image related to a rejected object determined by the determination unit to have been overlooked by the first machine learning model.

[0116] (Supplementary Note A6) The information processing device according to Supplementary Note A5, wherein the output means outputs the image in which a position of a rejected object determined to be a missed object by the first machine learning model can be distinguished from other positions.

[0117] (Supplementary Note A7) The information processing device according to Supplementary Note A5 or A6, wherein, when there are a plurality of rejected objects that the determination means has determined to be overlooked by the first machine learning model, the output means outputs the image in which an order of values ​​of the first index of each of the plurality of rejected objects can be recognized.

[0118] (Appendix A8) An information processing device comprising: rejected object image acquisition means for acquiring an image of a rejected object that is predicted to not be the specified object by a first machine learning model that detects an object included in an input image and predicts whether the detected object is a specified object; moving object index calculation means for calculating an index indicating the likelihood of the rejected object being a moving object by referring to the image of the rejected object; and determination means for determining whether the rejected object has been overlooked by the machine learning model by referring to the index.

[0119] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0120] (Supplementary Note B1) An information processing method including: a rejected object image acquisition process in which at least one processor detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; an uncertainty index calculation process in which the at least one processor refers to the image of the rejected object and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and a determination process in which the at least one processor refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0121] (Supplementary Note B2) The information processing method according to Supplementary Note B1, wherein in the uncertainty index calculation process, the at least one processor calculates the first index by inputting an image of the rejected object to a second machine learning model that uses the image as input and outputs an index indicating uncertainty of a prediction regarding the rejected object included in the image.

[0122] (Supplementary Note B3) The information processing method according to Supplementary Note B1 or B2, wherein the predetermined object is a moving object whose position changes over time, and further includes a moving object index calculation process in which the at least one processor refers to an image of the rejected object and calculates a second index indicating the likelihood that the rejected object is a moving object, and in the determination process, the at least one processor refers to the first index and the second index and determines whether the rejected object has been overlooked by the first machine learning model.

[0123] (Supplementary Note B4) The information processing method according to Supplementary Note B3, wherein in the moving object index calculation process, the at least one processor calculates the second index by inputting an image of the rejected object to a third machine learning model that takes an image as input and outputs an index indicating the likelihood that an object included in the image is a moving object.

[0124] (Supplementary Note B5) The information processing method according to any one of Supplementary Notes B1 to B4, further comprising an output process in which the at least one processor outputs an image related to a rejected object determined to be a missed object by the first machine learning model in the determination process.

[0125] (Supplementary Note B6) The information processing method according to Supplementary Note B5, wherein in the output process, the at least one processor outputs the image in which a position of a rejected object determined to be a missed object by the first machine learning model can be distinguished from other positions.

[0126] (Supplementary Note B7) The information processing method according to Supplementary Note B5 or B6, wherein, when there are a plurality of rejected objects determined to be overlooked by the first machine learning model in the determination process, in the output process, the at least one processor outputs the image in which the order of values ​​of the first index of each of the plurality of rejected objects can be recognized.

[0127] (Supplementary Note B8) An information processing method including: a rejected object image acquisition process in which at least one processor detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; a moving object index calculation process in which the at least one processor refers to the image of the rejected object and calculates an index indicating the likelihood of the rejected object being a moving object; and a determination process in which the at least one processor refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

[0128] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0129] (Supplementary Note C1) A program that causes a computer to function as an information processing device, the program causing the computer to function as: rejected object image acquisition means that detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; uncertainty index calculation means that refers to the image of the rejected object, and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and determination means that refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0130] (Supplementary Note C2) The program according to Supplementary Note C1, wherein the uncertainty index calculation means calculates the first index by inputting an image of the rejected object into a second machine learning model that receives an image as input and outputs an index indicating uncertainty of a prediction regarding the rejected object included in the image.

[0131] (Supplementary Note C3) The program according to Supplementary Note C1 or C2, wherein the predetermined object is a moving object whose position changes over time, and the computer is further caused to function as moving object index calculation means that refers to an image of the rejected object and calculates a second index that indicates the likelihood of the rejected object being a moving object, and the determination means refers to the first index and the second index and determines whether the rejected object has been overlooked by the first machine learning model.

[0132] (Supplementary Note C4) The program according to Supplementary Note C3, wherein the moving object index calculation means calculates the second index by inputting an image of the rejected object into a third machine learning model that receives an image as input and outputs an index indicating a likelihood of an object included in the image being a moving object.

[0133] (Supplementary Note C5) The program according to any one of Supplementary Notes C1 to C4, further causing the computer to function as an output means for outputting an image related to a rejected object that the determination means has determined to be a missed object by the first machine learning model.

[0134] (Supplementary Note C6) The program according to Supplementary Note C5, wherein the output means outputs the image in which a position of a rejected object determined to be a missed object by the first machine learning model can be distinguished from other positions.

[0135] (Supplementary Note C7) The program according to Supplementary Note C5 or C6, wherein, when there are a plurality of rejected objects that the determination means has determined to have been overlooked by the first machine learning model, the output means outputs the image in which the order of values ​​of the first index of each of the plurality of rejected objects can be recognized.

[0136] (Appendix C8) A program that causes a computer to function as an information processing device, the program causing the computer to function as: rejected object image acquisition means that detects an object included in an input image, and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; moving object index calculation means that refers to the image of the rejected object, and calculates an index that indicates the likelihood of the rejected object being a moving object; and determination processing that refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

[0137] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0138] (Supplementary Note D1) An information processing device comprising at least one processor, the at least one processor executing: a rejected object image acquisition process that detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; an uncertainty index calculation process that refers to the image of the rejected object and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and a determination process that refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

[0139] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0140] (Supplementary Note D2) The information processing device according to Supplementary Note D1, wherein the at least one processor calculates the first index by inputting an image of the rejected object to a second machine learning model that uses an image as input and outputs an index indicating uncertainty of a prediction regarding the rejected object included in the image, in the uncertainty index calculation process.

[0141] (Supplementary Note D3) The information processing device according to Supplementary Note D1 or D2, wherein the predetermined object is a moving object whose position changes over time, and the at least one processor further executes a moving object index calculation process that refers to an image of the rejected object and calculates a second index that indicates the likelihood of the rejected object being a moving object, and in the determination process, the at least one processor refers to the first index and the second index and determines whether the rejected object has been overlooked by the first machine learning model.

[0142] (Appendix D4) The information processing device according to Appendix D3, wherein in the moving object index calculation process, the at least one processor calculates the second index by inputting an image of the rejected object to a third machine learning model that takes an image as input and outputs an index indicating the likelihood that an object included in the image is a moving object.

[0143] (Supplementary Note D5) The information processing device according to any one of Supplementary Notes D1 to D4, wherein the at least one processor further executes an output process of outputting an image related to a rejected object determined by the determination process to be a missed object by the first machine learning model.

[0144] (Supplementary Note D6) The information processing device according to Supplementary Note D5, wherein in the output process, the at least one processor outputs the image in which a position of a rejected object determined to be a missed object by the first machine learning model can be distinguished from other positions.

[0145] (Supplementary Note D7) The information processing device according to Supplementary Note D5 or D6, wherein, when there are a plurality of rejected objects determined to be overlooked by the first machine learning model in the determination process, in the output process, the at least one processor outputs the image in which the value order of the first index of each of the plurality of rejected objects can be recognized.

[0146] (Appendix D8) An information processing device including at least one processor, the at least one processor executing: a rejected object image acquisition process that detects an object included in an input image and acquires an image of a rejected object that is predicted to not be the predetermined object by a first machine learning model that detects whether or not the detected object is a predetermined object; a moving object index calculation process that refers to the image of the rejected object and calculates an index that indicates the likelihood of the rejected object being a moving object; and a determination process that refers to the index and determines whether or not the rejected object has been overlooked by the machine learning model.

[0147] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0148] (Appendix E1) A non-transient recording medium having recorded thereon a program that causes a computer to function as an information processing device, the program causing the computer to execute: a rejected object image acquisition process that detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object or not predicts is not the predetermined object; an uncertainty index calculation process that refers to the image of the rejected object and calculates a first index that indicates the uncertainty of the prediction regarding the rejected object; and a determination process that refers to the first index and determines whether the rejected object is an object that was overlooked by the first machine learning model.

[0149] (Appendix E2) A non-transient recording medium having recorded thereon a program that causes a computer to function as an information processing device, the program causing the computer to execute: a rejected object image acquisition process that detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object or not predicts is not the predetermined object; a moving object index calculation process that refers to the image of the rejected object and calculates an index that indicates the likelihood of the rejected object being a moving object; and a determination process that refers to the index and determines whether the rejected object is an object that was overlooked by the machine learning model.

[0150] 1, 2, 3 Information processing device 11, 21, 31 Rejected object image acquisition unit 12, 32 Index calculation unit 13, 23, 33 Determination unit 22, 34 Moving object index calculation unit 35 Output unit 36 ​​Object detection unit DP Image of detected object LM Machine learning model NDP Image of rejected object PI Image

Claims

1. An information processing device comprising: rejected object image acquisition means for acquiring images of rejected objects that a first machine learning model that detects objects included in an input image and predicts whether the detected object is a predetermined object predicts is not the predetermined object; uncertainty index calculation means for calculating a first index indicating the uncertainty of the prediction regarding the rejected object by referring to the image of the rejected object; and determination means for determining whether the rejected object has been overlooked by the first machine learning model by referring to the first index.

2. The information processing device according to claim 1, wherein the uncertainty index calculation means calculates the first index by inputting an image of the rejected object into a second machine learning model that receives an image as input and outputs an index indicating the uncertainty of a prediction regarding the rejected object contained in the image.

3. The information processing device according to claim 1 or 2, wherein the predetermined object is a moving object whose position changes over time, and further comprising: moving object index calculation means for calculating a second index indicating the likelihood that the rejected object is a moving object by referring to an image of the rejected object; and the determination means for determining whether the rejected object has been overlooked by the first machine learning model by referring to the first index and the second index.

4. The information processing device according to claim 3, wherein the moving object index calculation means calculates the second index by inputting an image of the rejected object into a third machine learning model that takes an image as input and outputs an index indicating the likelihood that an object contained in the image is a moving object.

5. The information processing device according to any one of claims 1 to 4, further comprising: output means for outputting an image related to a rejected object determined by the determination means to be a missed object by the first machine learning model.

6. The information processing device according to claim 5, wherein the output means outputs the image in which the position of a rejected object determined to be a missed object by the first machine learning model can be distinguished from other positions.

7. The information processing device according to claim 5 or 6, wherein, when there are a plurality of rejected objects that the determination means has determined to be overlooked by the first machine learning model, the output means outputs the image in which the order of the values ​​of the first index of each of the plurality of rejected objects can be recognized.

8. An information processing device comprising: rejected object image acquisition means for acquiring an image of a rejected object that is predicted to not be the specified object by a first machine learning model that detects an object included in an input image and predicts whether the detected object is a specified object; moving object index calculation means for calculating an index indicating the likelihood of the rejected object being a moving object by referring to the image of the rejected object; and determination means for determining whether the rejected object has been overlooked by the machine learning model by referring to the index.

9. An information processing method comprising: a rejected object image acquisition process in which at least one processor detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; an uncertainty index calculation process in which the at least one processor refers to the image of the rejected object and calculates a first index indicating the uncertainty of the prediction regarding the rejected object; and a determination process in which the at least one processor refers to the first index and determines whether the rejected object has been overlooked by the first machine learning model.

10. An information processing method comprising: a rejected object image acquisition process in which at least one processor detects an object included in an input image and acquires an image of a rejected object that a first machine learning model that predicts whether the detected object is a predetermined object predicts is not the predetermined object; a moving object index calculation process in which the at least one processor refers to the image of the rejected object and calculates an index indicating the likelihood of the rejected object being a moving object; and a determination process in which the at least one processor refers to the index and determines whether the rejected object has been overlooked by the machine learning model.

11. A program that causes a computer to function as an information processing device, the program causing the computer to function as: rejected object image acquisition means that detects objects included in an input image and acquires images of rejected objects that a first machine learning model that predicts whether the detected object is a predetermined object predicts are not the predetermined object; uncertainty index calculation means that refers to the images of the rejected objects and calculates a first index that indicates the uncertainty of the prediction regarding the rejected objects; and determination means that refers to the first index and determines whether the rejected objects have been overlooked by the first machine learning model.

12. A program that causes a computer to function as an information processing device, the program causing the computer to function as: rejected object image acquisition means that detects objects included in an input image and acquires images of rejected objects that a first machine learning model that predicts whether the detected object is a specified object predicts are not the specified object; moving object index calculation means that refers to the images of the rejected objects and calculates an index that indicates the likelihood of the rejected objects being moving objects; and determination processing that refers to the index and determines whether the rejected objects have been overlooked by the machine learning model.

Images