Information processing apparatus and information processing method

Abstract

To provide an information processor and an information processing method capable of outputting a more accurate object detection result.SOLUTION: In the information processing apparatus, in the processing unit, the image acquisition unit 201 acquires the designated image from the image storage unit 111, the object detection unit 202 acquires, for each pre-trained model, the position, size, and likelihood of the image region of the object detection from the image performed by the pre-trained model, and the selection unit 203 selects, for each pre-trained model, a result to be output among the results of the object detection from the image performed by the pre-trained model, and outputs the result selected for each pre-trained model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to information processing technology. [Background technology]

[0002] In recent years, with the rapid advancement of digital technology, the use of AI in various fields has been attracting attention. One such example is supervised learning, in which AI automatically extracts features from training data containing correct answer data to generate an inference model.

[0003] In supervised learning, to obtain a trained AI model with high generalization performance, training data is used, which consists of a set of various types of images determined by the task to be solved and annotation information for objects contained in those images. Therefore, as a preliminary step before executing AI training, an annotation process is required, in which images suitable for learning are collected and annotation information is manually added by humans to create training data. However, because a huge amount of training data is required to create a machine learning model with high generalization performance, manual annotation takes a long time, which is an issue. As a method to support manual annotation work, research is being conducted on using a model that has been pre-trained with the same type of images as the images to be annotated.

[0004] For example, Patent Document 1 discloses a method for presenting the results of object detection using a pre-trained model and annotation information that serves as a reference when assigning a correct answer. Patent Document 2 also discloses a method for scoring the results of object detection using multiple pre-trained models, automatically selecting the result with the highest score, and presenting the annotation information of the selected result by superimposing it on an image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-105923 [Patent Document 2] Patent Application No. 2021-504744 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Documents 1 and 2, all detection results of the pre-trained model are presented to the user, which poses a problem that results of erroneous detection by the pre-trained model in the input image may also be presented at the same time. The present invention provides a technology that enables the output of more accurate object detection results. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising an acquisition means for acquiring, for each pre-trained model, the results of object detection from images performed by the pre-trained model; a selection means for selecting, for each pre-trained model, a result to output from the results of object detection from images performed by the pre-trained model; and an output means for outputting the result selected by the selection means for each pre-trained model. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technique that enables more accurate object detection results to be output. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device. [Figure 3] FIG. 10 is a diagram showing a display example of an object detection result. [Figure 4] FIG. 10 is a diagram showing an example of a screen display for correction. [Figure 5] 10 is a flowchart of the operation of the information processing device. [Figure 6] FIG. 2 is a diagram showing an example of the configuration of annotation information. [Figure 7] (a) and (b) are diagrams showing an example of the configuration of a group of pre-trained models, and (c) is a diagram showing an example of the display of object detection results. [Figure 8] 10 is a flowchart of the operation of the information processing device. [Figure 9] A diagram showing an example of the process until the results of the pre-trained model that has been selected the most times are adopted. [Figure 10] FIG. 10 is a diagram showing a display example of an object detection result. [Figure 11] FIG. 10 is a diagram showing an example of an object detection result. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] First, a hardware configuration example of an information processing device according to this embodiment will be described with reference to the block diagram in Fig. 1(a). The information processing device according to this embodiment can be a computer device such as a PC, a smartphone, or a tablet terminal.

[0012] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the operation of the entire information processing device, and executes or controls various processes described as processes performed by the information processing device.

[0013] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103 or the storage unit 110, and an area for storing computer programs and data received from an external device via the communication unit 104. The RAM 102 also has a work area used by the CPU 101 when executing or controlling various processes. In this way, the RAM 102 can provide various areas as needed.

[0014] The ROM 103 stores setting data for the information processing device, computer programs and data relating to the startup of the information processing device, computer programs and data relating to the basic operation of the information processing device, and the like.

[0015] The communication unit 104 performs data communication with external devices via a network such as a LAN or the Internet. The input unit 105 is a user interface such as a keyboard, mouse, or touch panel, which can be operated by a user to input various instructions and information to the information processing device.

[0016] The display unit 106 has a liquid crystal screen or a touch panel screen, and can display the processing results by the CPU 101 as images, text, etc. The display unit 106 may be a projection device such as a projector that projects images and text.

[0017] The storage unit 110 is a non-volatile large-capacity information storage device such as a hard disk drive, etc. Note that the storage unit 110 may be a flash memory, various optical media, etc., in addition to a hard disk drive.

[0018] The storage unit 110 stores an OS, computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the information processing device.

[0019] In this embodiment, as shown in FIG. 1(b), the storage unit 110 includes an image storage unit 111, a model storage unit 112, and an annotation storage unit 113. The image storage unit 111 stores a group of images that may include various images such as captured images and CG images. The images stored in the image storage unit 111 are images to which annotation information may be added in response to a user operation. The model storage unit 112 stores a plurality of pre-trained models that are models that have been trained in advance to output the position and size of an object to be detected in an input image and a likelihood that indicates the likelihood of the object. Various types of models can be applied to such pre-trained models, and for example, an object detection model constructed using a convolutional neural network (CNN) can be applied. The model storage unit 112 also stores thresholds corresponding to each pre-trained model. Note that the pre-trained model may be a single model or a group of models. The annotation storage unit 113 stores annotation information that has been assigned to an image in response to a user operation.

[0020] The image storage unit 111, the model storage unit 112, and the annotation storage unit 113 may be separate storage devices, or may be separate storage areas in the storage unit 110.

[0021] Furthermore, the above information described as information stored in the memory unit 110 may be information generated by the user operating the input unit 105, or may be information received from an external device via the communication unit 104.

[0022] The CPU 101, RAM 102, ROM 103, communication unit 104, input unit 105, display unit 106, and storage unit 110 are all connected to a system bus 190. Note that the configuration shown in Fig. 1 is merely an example of a configuration applicable to the information processing device according to this embodiment, and can be modified / altered as appropriate.

[0023] An example of the functional configuration of the information processing device according to this embodiment is shown in the block diagram of FIG. 2. The processing unit 200 includes an image acquisition unit 201, an object detection unit 202, a selection unit 203, a presentation unit 204, and a correction unit 205. In this embodiment, a case will be described in which each functional unit of the processing unit 200 is implemented by a computer program. In the following, each functional unit of the processing unit 200 will be described as the subject of processing, but in reality, the function of each functional unit is realized by the CPU 101 executing a computer program corresponding to that functional unit. Note that one or more of the functional units of the processing unit 200 may be implemented by hardware.

[0024] The operation of the information processing device will be described with reference to the flowchart in Fig. 5. In step S501, the image acquisition unit 201 acquires an image to which annotation information is to be added from a group of images stored in the image storage unit 111. The method by which the image acquisition unit 201 acquires an image from the image storage unit 111 is not limited to a specific method.

[0025] For example, the image acquisition unit 201 may acquire from the image storage unit 111 an image specified by the user operating the input unit 105, or may acquire from the image storage unit 111 an image that has been specified in advance.

[0026] The subsequent processes of steps S503 to S514 are performed for each image acquired in step S501. In step S503, the object detection unit 202 selects one of the images acquired in step S501 that has not been selected yet as a selected image. The object detection unit 202 also reads out a plurality of pre-trained models stored in the model storage unit 112. For the sake of specific explanation, the following describes a case in which the model storage unit 112 stores four pre-trained models that have been trained in advance to detect dogs from images. In this case, the object detection unit 202 reads out these four pre-trained models from the model storage unit 112. Note that the following explanation is equally applicable to cases in which objects of categories other than dogs (people, animals, vehicles, etc.) are to be detected.

[0027] Then, for each of the four read pre-trained models, the object detection unit 202 inputs the selected image into the pre-trained model and performs calculations on the pre-trained model to acquire (infer) the position (e.g., the center position of the image area) and size (e.g., the vertical and horizontal sizes of the image area) of the image area of ​​the object detected as a dog from the selected image, and the likelihood (e.g., a real value between 0 and 1) representing the dog-likeness of the object.The object detection unit 202 then calculates the following formula (1) for the likelihood acquired for each of the four pre-trained models, thereby acquiring a detection score corresponding to the likelihood.

[0028] Detection score = 255 x (likelihood) … (1) The object detection unit 202 then outputs, as object detection results, the position and size of the image area of ​​the object detected as a dog, obtained for each of the four pre-trained models, and a detection score corresponding to the likelihood that the object is a dog.

[0029] In step S504, for each of the four pre-trained models, the selection unit 203 selects the object detection result with the highest detection score among the object detection results by the pre-trained model as the selection target. Note that if the highest detection score among the object detection results by the pre-trained models is less than the threshold value corresponding to the pre-trained model, the selection unit 203 does not select the object detection result with the highest detection score as the selection target.

[0030] In addition, if there is no selection target for all pre-trained models, for example, the selection unit 203 selects the object detection result with the highest detection score among the object detection results by all pre-trained models as the selection target.

[0031] In step S505, CPU 101 determines whether the process of adding annotation information (annotation) has not yet been performed on the first selected image. If the result of this determination is that the process of adding annotation information has not yet been performed on the first selected image, the process proceeds to step S506. On the other hand, if the process of adding annotation information has already been performed on the first selected image, the process proceeds to step S507.

[0032] In step S506, the presentation unit 204 displays the object detection results selected as selection targets in step S504 on the display unit 106. An example of the display of the object detection results in step S506 is shown in FIG. 3(a). Hereinafter, unless otherwise specified, it is assumed that the screen display on the display unit 106 is performed by the presentation unit 204.

[0033] Within the display area 302 provided on the display screen 301 of the display unit 106, there are provided areas 302a corresponding to model 1, 302b corresponding to model 2, 302c corresponding to model 3, and 302d corresponding to model 4 out of four pre-trained models (model 1, model 2, model 3, and model 4 in Figure 3(a)).

[0034] The selected image is displayed in area 302a, and a frame 307 having the "size included in the object detection result by model 1" is displayed at the "position included in the object detection result by model 1" on the selected image.

[0035] The selected image is displayed in area 302b, and a frame 308 having the "size included in the object detection result by model 2" is displayed at the "position included in the object detection result by model 2" on the selected image.

[0036] The selected image is displayed in area 302c, and a frame 309 having "the size included in the object detection result by model 3" is displayed on the selected image at "the position included in the object detection result by model 3." The selected image is displayed in area 302d, and because the object detection result by model 4 has not been selected as a selection target, no frame is displayed.

[0037] The user checks the object detection results (frame position and size) of each model displayed on the display screen shown in Figure 3(a) and identifies the model that provides the preferred object detection result. Frame 307 surrounds the dog exactly (it is detected at the correct size and in the center of the area where the dog exists), so the object detection result by model 1 is the preferred object detection result. Frame 308 surrounds the bird because model 2 misidentified it as a dog. Frame 309 surrounds the dog, but is significantly larger than the size of the dog (the center of the area where the dog exists is correct, but the size is slightly larger).

[0038] Area 303 is provided with check boxes 304 corresponding to each of models 1 to 4, and the user can check the check box 304 corresponding to the model that provides the preferred object detection result by operating input unit 105. In the above example, model 1 is the model that provides the preferred object detection result, so the user operates input unit 105 to check the check box 304 that corresponds to model 1.

[0039] When the user operates the input unit 105 to check the check box 304 corresponding to model 1 and then presses the decision button 305, the presentation unit 204 sets model 1 as the "edit screen display model" and the process proceeds to step S509.

[0040] When only the object detection results by model 3 are selected as the selection target, no frames are displayed for models 1, 2, and 4, and a frame 310 is displayed for model 3, as shown in FIG. 3(b).

[0041] In step S507, the presentation unit 204 does not display the screen as shown in Fig. 3(a), but sets the pre-trained model selected for the previous annotation information assignment work as the "edit screen display model." Then, the process proceeds to step S509.

[0042] In the following, a case where model 1 is set as the edit screen display model will be described, but the following description is equally applicable even if another model is set as the edit screen display model.

[0043] The processes of steps S509 to S512 are repeatedly executed until annotation information is assigned to the selected image and the selected image is stored in the storage unit 110. In step S509, the correction unit 205 causes the display screen 301 to display the screen shown in Fig. 4. In Fig. 4, the object detection result by model 1 set as the editing screen display model is displayed in area 401, that is, the same content as in area 302a. The user can change the position and size of frame 307 (that is, change the object detection result by model 1) by operating the input unit 105.

[0044] In step S510, correction unit 205 determines whether or not the user has operated input unit 105 to instruct result list display button 403 because the user has determined that the object detection result by model 1 is inappropriate as a standard for annotation, etc. If the result of this determination is that the user has operated input unit 105 to instruct result list display button 403, the process proceeds to step S511, and if the user has not operated input unit 105 to instruct result list display button 403, the process proceeds to step S512.

[0045] In step S511, similar to step S506 above, the presentation unit 204 causes the display unit 106 to display the object detection results selected as selection targets in step S504. Then, when the user operates the input unit 105 to check the checkbox 304 corresponding to one of the models and then presses the decision button 305, the presentation unit 204 sets the checked model as the "edit screen display model." Then, the process proceeds to step S509.

[0046] In step S512, correction unit 205 determines whether or not the user has operated input unit 105 to instruct decision button 404. If the result of this determination is that the user has operated input unit 105 to instruct decision button 404, the process proceeds to step S514, and if the user has not operated input unit 105 to instruct decision button 404, the process proceeds to step S509.

[0047] In step S514, the modifying unit 205 generates annotation information that may include information about the selected image and the object detection result based on the editing screen display model, and stores the annotation information in the annotation storage unit 113. An example of the configuration of annotation information is shown in FIG.

[0048] In the annotation information of Figure 6, for each file name of the selected image (001,jpg, 002,jpg, 002,jpg), the x coordinate of the center of the image area of ​​the object detected as a dog from the selected image of that file name (center coordinate x), the y coordinate of that center (center coordinate y), the width of that image area, and the height of that image area are registered in correspondence with each other.

[0049] The above-described configuration of annotation information is an example and is not limited to a specific configuration. Furthermore, the files may be divided and saved for each image, or may be saved all at once without being divided. Additionally, the number of times a model selected as the "editing screen display model" is selected may be recorded.

[0050] Thus, according to this embodiment, for each pre-trained model, the object detection result with the highest detection score (and equal to or greater than the threshold corresponding to the pre-trained model) among the object detection results using that pre-trained model is displayed, so it is expected that erroneous detection results will not be displayed.

[0051] The output destination of the object detection result is not limited to the display unit 106. For example, the object detection result may be transmitted to an external device via the communication unit 104 and displayed on a display screen of the external device.

[0052] <Modification> In the first embodiment, an image is input to each pre-trained model to obtain an object detection result for each pre-trained model, but an image may be input to each pre-trained model group to obtain an object detection result for each pre-trained model group. The following describes the differences between this modification and the first embodiment.

[0053] The pre-trained model group may be, for example, a model group 750 including dog detection models 751 and 752 that detect dogs from images, as shown in FIG. 7(a). Alternatively, the pre-trained model group may be, for example, a model group 760 including a dog detection model 761 that detects dogs from images and a bird detection model 762 that detects birds from images. In other words, the pre-trained model group may be a model group that detects the same object from images, or a model group that detects different objects from images. The number of pre-trained models included in the pre-trained model group is not limited to a specific number.

[0054] For example, assume that four model groups, each including a dog detection model and a bird detection model, are stored in the model storage unit 112. In this case, in step S503, the object detection unit 202 inputs the selected image into each of the dog detection model and bird detection model in the four model groups read out from the model storage unit 112, and performs calculations on the dog detection model and the bird detection model. As a result, the object detection unit 202 acquires (infers) for each of the four model groups "the position and size of the image area of ​​an object detected as a dog from the selected image, and the likelihood representing the dog-likeness of the object" (dog detection result) and "the position and size of the image area of ​​an object detected as a bird from the selected image, and the likelihood representing the bird-likeness of the object" (bird detection result). Then, as in the first embodiment, the object detection unit 202 acquires a detection score from the likelihood.

[0055] In step S504, the selection unit 203 selects the dog detection result with the highest detection score and the bird detection result with the highest detection score for each of the four model groups as selection candidates. As in the first embodiment, if the highest detection score is less than the threshold, the selection unit 203 does not select the dog detection result / bird detection result with the highest detection score as selection candidates. In such a case, in step S506, the presentation unit 204 displays the dog detection result and / or the bird detection result for each model group, as shown in FIG. 7(c).

[0056] Within a display area 402 provided on a display screen 401 of the display unit 106, there are provided areas 402a corresponding to model group 1, 402b corresponding to model group 2, 402c corresponding to model group 3, and 402d corresponding to model group 4 out of four model groups (model group 1, model group 2, model group 3, and model group 4 in FIG. 7(c)).

[0057] A selected image is displayed in area 402a, and a frame 702 having "a size included in the dog detection result by the dog detection model in model group 1" is displayed at "a position included in the dog detection result by the dog detection model in model group 1" on the selected image. Also, a frame 701 having "a size included in the bird detection result by the bird detection model in model group 1" is displayed at "a position included in the bird detection result by the bird detection model in model group 1" on the selected image.

[0058] A selected image is displayed in area 402b, and a frame 703 having "a size that is included in the dog detection result by the dog detection model in model group 2" is displayed at "a position that is included in the dog detection result by the dog detection model in model group 2" on the selected image. Note that the bird detection result by the bird detection model in model group 2 has not been selected as a selection target, so no frame is displayed.

[0059] A selected image is displayed in area 402c, and a frame 705 having "a size included in the dog detection result by the dog detection model in model group 3" is displayed at "a position included in the dog detection result by the dog detection model in model group 3" on the selected image. Also, a frame 704 having "a size included in the bird detection result by the bird detection model in model group 3" is displayed at "a position included in the bird detection result by the bird detection model in model group 3" on the selected image.

[0060] A selected image is displayed in area 402d, and a frame 707 having a "size included in the dog detection result by the dog detection model in model group 4" is displayed at a "position included in the dog detection result by the dog detection model in model group 4" on the selected image. Also, a frame 706 having a "size included in the bird detection result by the bird detection model in model group 4" is displayed at a "position included in the bird detection result by the bird detection model in model group 4" on the selected image. In this way, this modified example can handle images that include multiple objects of the same type or multiple objects of different types.

[0061] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. The operation of the information processing device will be described with reference to the flowchart in Fig. 8. The processes of steps S801 to S806, S808, and S811 to S816 in Fig. 8 are respectively the same as steps S501 to S506, S507, and S509 to S514 in Fig. 5, and therefore descriptions of steps S801 to S806, S808, and S811 to S816 will be omitted.

[0062] In step S807, CPU 101 determines whether the number of annotated selected images is equal to or greater than a preset branch number threshold. The branch number threshold may be set in advance by a user operating input unit 105, or may be obtained from an external device via communication unit 104.

[0063] If the result of this determination is that the number of selected annotated images is equal to or greater than the preset branch number threshold, the process proceeds to step S809. On the other hand, if the number of selected annotated images is less than the preset branch number threshold, the process proceeds to step S808.

[0064] In step S809, the presentation unit 204 sets the pre-trained model that has been selected the most frequently in the work performed to add annotation information as the editing screen display model.

[0065] As an example of the process until the results of the pre-trained model that has been selected the most times are adopted, the details of the process will be explained using Figure 9, which shows the process of annotating the 20th image and then transitioning to annotating the 21st image under the conditions that there are 30 images to be annotated and the branch number threshold is 20.

[0066] (i) When annotating the 20th image, annotation work has been completed up to the 19th image, so the number of annotated images is less than the branch number threshold. Therefore, as shown in FIG. 9(a), the object detection result using the pre-trained model selected for the 19th image is displayed in area 401. Furthermore, when saving the annotation information for the 19th image, the number of times each of models 1 to 4 was selected (model 1: 10 times, model 2: 4 times, model 3: 5 times, and model 4: 0 times) is also saved, as shown in FIG. 9(d). As shown in FIG. 9(a), a frame 901 is displayed around the dog's face, which is inappropriate for the results desired by the user, so the user selects button 403.

[0067] (ii) When button 403 in Fig. 9(a) is pressed, the object detection results for each of models 1 to 4, as exemplified in Fig. 3(a), are displayed as shown in the upper part of Fig. 9(b). In Fig. 9(b), annotation information for the 20th image edited by the user with model 2 selected is saved, and therefore the number of times model 2 has been selected is updated to 5, as shown in Fig. 9(e).

[0068] (iii) Next, the 21st image is processed in the order of steps S803, S804, S805, and S807. At this time, since the number of images that has been annotated is equal to or exceeds the branch number threshold, a frame 902 corresponding to the style detection result by model 1, which has been selected most frequently by users, is displayed, as shown in FIG. 9(c).

[0069] If there are multiple models that have been selected the most times by users, the object detection results of the model with the smallest model number are displayed. Specifically, if models 2 and 3 are the models that have been selected the most times by users, the object detection results of model 2 are displayed in the next image. However, which model's object detection results are to be displayed is not limited to the size of the model number as described above, and other methods such as random determination may also be used.

[0070] In this embodiment, in step S816, the modification unit 205 generates annotation information and stores it in the annotation storage unit 113, as in the first embodiment, and also stores the number of times each model is selected in a memory such as the RAM 102 or the storage unit 110.

[0071] In this way, in this embodiment, focusing on the fact that the model that can correctly detect the object annotated by the user can be determined based on the number of times the model was selected, the object detection result of the model that was selected the most is presented, thereby making it possible to present the most suitable object detection result to the user.

[0072] [Third embodiment] In this embodiment, the operation of the information processing device will be described in a case where four dog detection models are used as pre-trained models. In this embodiment, in step S506 or step S806, the display screen shown in FIG. 10(a) is displayed on the display unit 106.

[0073] An area 1000 on the display screen of the display unit 106 is provided with an operation unit 1001 that can be moved left and right by the user operating the input unit 105. Moving the operation unit 1001 further to the left sets a larger threshold value, and moving the operation unit 1001 further to the right sets a smaller threshold value. The initial position of the operation unit 1001 is the rightmost position "recommended." In other words, it is set so that the object detection result with the maximum detection score for each model is displayed.

[0074] Therefore, a selected image is displayed in area 302a, and a frame 1004 corresponding to the object detection result with the highest detection score for model 1 is displayed in the selected image. A selected image is displayed in area 302b, and a frame 1005 corresponding to the object detection result with the highest detection score for model 2 is displayed in the selected image. A selected image is displayed in area 302c, and a frame 1006 corresponding to the object detection result with the highest detection score for model 3 is displayed in the selected image. A selected image is displayed in area 302d, and a frame 1007 corresponding to the object detection result with the highest detection score for model 4 is displayed in the selected image.

[0075] The following describes a case where the result shown in FIG. 11(a) is obtained as the object detection result by Model 1, and the result shown in FIG. 11(b) is obtained as the object detection result by Model 2.

[0076] When displaying multiple object detection results for each model, the following processing is performed in step S504 and step S804 to avoid the possibility of multiple object detection results for the same object being displayed.

[0077] (i) The IoU (Intersection over Union), which indicates the degree of overlap between the area within the frame to be displayed in the display area 302 and the areas within each frame already displayed in the display area 302, is calculated using the following equation (2).

[0078] IoU=(A∩B) / (A∪B) … (2) In equation (2), IoU is calculated from the union AUB and intersection A∩B of a rectangle having area A and a rectangle having area B. IoU is calculated as a value within the range of 0.0 to 1.0.

[0079] (ii) If at least one value of a judgment value of 0.5 or more is included in the IoU values ​​calculated for a frame to be displayed in the display area 302, the object detection result corresponding to that frame is deleted, and that frame is not displayed in the display area 302. Note that the judgment value is not limited to 0.5, but is not limited to a specific value as long as it is in the range of 0.0 to 1.0. The judgment value can be set by the user using the input unit 105, for example.

[0080] Next, a method for determining the object detection results to be displayed according to the position of the operation unit 1001 will be described using Fig. 11 as an example. Taking Model 1 and Model 2 as an example, as shown in Fig. 11(c), the maximum value (210) and minimum value (130) of the detection scores (200, 150, 130) of Model 1 and the detection scores (210, 160, 140) of Model 2 are assigned to the rightmost and leftmost positions of the operation unit 1001, respectively, and the other detection scores are assigned to the corresponding positions within the movement range of the operation unit 1001.

[0081] Position 1104 is assigned a detection score of "160" for object 2 of model 2, and position 1103 is assigned a detection score of "150" for object 1 of model 1. Position 1102 is assigned a detection score of "140" for object 3 of model 2, and position 1101 is assigned a detection score of "130" for object 2 of model 1.

[0082] Therefore, when the user operates the input unit 105 to move the operation unit 1001 to position 1104, a frame representing the object detection result for object 2 in model 2 is additionally displayed. Also, when the user operates the input unit 105 to move the operation unit 1001 to position 1103, a frame representing the object detection result for object 1 in model 1 is additionally displayed.

[0083] Furthermore, when the user operates the input unit 105 to move the operation unit 1001 to the leftmost position "All," frames 1008 to 1010 and 1012 to 1014 corresponding to all of the object detection results held are additionally displayed, as shown in Fig. 10(b). Note that the method for additionally displaying frames using the operation unit 1001 is not limited to a specific method.

[0084] The user selects a pre-trained model that displays the object detection results to be annotated from the display in Figure 10(b), and the results of the selected pre-trained model are displayed.

[0085] In this way, in this embodiment, when multiple objects of the same category as the object that the user wants to detect are captured, it is possible to display the detection results of the pre-trained model for all captured objects. In the explanation of this embodiment, an example in which multiple objects of the same category are captured is used, but even when a single object is captured, it is possible to display multiple detection results by adjusting the threshold. This is expected to enable more efficient annotation.

[0086] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0087] Furthermore, some or all of the embodiments and modifications described above may be used in appropriate combination, and some or all of the embodiments and modifications described above may be used selectively.

[0088] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0089] The invention of this specification includes the following information processing device, information processing method, and computer program. (Item 1) an acquisition means for acquiring, for each pre-trained model, a result of object detection from an image performed by the pre-trained model; a selection means for selecting, for each pre-trained model, a result to be output from among the results of object detection from images performed by the pre-trained model; an output means for outputting the result selected by the selection means for each pre-trained model; An information processing device comprising: (Item 2) Item 1. The information processing device according to item 1, wherein the acquisition means acquires, for each pre-trained model, the position and size of an object detected from an image by the pre-trained model and a score based on the likelihood calculated for the object by the pre-trained model. (Item 3) Item 3. The information processing device according to item 2, wherein the selection means selects, for each pre-trained model, the result with the highest score from the results of object detection from images performed by the pre-trained model. (Item 4) The information processing device described in item 2 is characterized in that the selection means selects, for each pre-trained model, a result of the score corresponding to a threshold set in accordance with a user operation from the results of object detection from an image performed by the pre-trained model. (Item 5) 5. The information processing device according to any one of items 1 to 4, wherein the output means outputs a result of object detection using a previously selected pre-trained model in response to a user operation. (Item 6) The output means 6. The information processing device according to any one of items 1 to 5, characterized in that the selection means displays the results selected by the selection means for a pre-trained model selected in response to a user operation, and generates, as annotation information, information including the results of modifying the displayed results in response to a user operation. (Item 7) 5. The information processing device according to any one of items 1 to 4, wherein the output means outputs a result of object detection using a pre-trained model that has been selected the most frequently in response to user operations. (Item 8) the acquiring means acquires, for each pre-trained model group, a result of object detection of each pre-trained model included in the pre-trained model group; The selection means selects, for each pre-trained model group, a result to be output from among the object detection results of each pre-trained model included in the pre-trained model group. 8. The information processing device according to any one of items 1 to 7, (Item 9) An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires, for each pre-trained model, a result of object detection from an image performed by the pre-trained model; a selection step in which a selection means of the information processing device selects, for each pre-trained model, a result to be output from among results of object detection from images performed by the pre-trained model; an output step in which an output means of the information processing device outputs the results selected for each pre-trained model in the selection step; An information processing method comprising: (Item 10) A computer program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 8.

[0090] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0091] 111: Image storage unit 112: Model storage unit 113: Annotation storage unit 200: Processing unit 201: Acquisition unit 202: Object detection unit 203: Selection unit 204: Presentation unit 205: Correction unit

Claims

1. an acquisition means for acquiring, for each pre-trained model, a result of object detection from an image performed by the pre-trained model; a selection means for selecting, for each pre-trained model, a result to be output from among the results of object detection from images performed by the pre-trained model; an output means for outputting the result selected by the selection means for each pre-trained model; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the acquisition means acquires, for each pre-trained model, the position and size of an object detected from an image by the pre-trained model and a score based on the likelihood calculated for the object by the pre-trained model.

3. The information processing device according to claim 2 , wherein the selection means selects, for each pre-trained model, the result with the highest score from among the results of object detection from an image performed by the pre-trained model.

4. The information processing device according to claim 2, characterized in that the selection means selects, for each pre-trained model, a result of object detection from an image performed by the pre-trained model, a result of a score corresponding to a threshold set in accordance with a user operation.

5. The information processing apparatus according to claim 1 , wherein the output means outputs a result of object detection using a previously selected pre-trained model in response to a user operation.

6. The output means The information processing device according to claim 1, characterized in that the selection means displays the results selected by the selection means for the pre-trained model selected in response to user operation, and generates information including the results of modifying the displayed results in response to user operation as annotation information.

7. The information processing apparatus according to claim 1 , wherein the output unit outputs a result of object detection using the pre-trained model that has been selected most frequently in response to user operations.

8. the acquiring means acquires, for each pre-trained model group, a result of object detection of each pre-trained model included in the pre-trained model group; The selection means selects, for each pre-trained model group, a result to be output from among the object detection results of each pre-trained model included in the pre-trained model group.

2. The information processing apparatus according to claim 1, wherein:

9. An information processing method performed by an information processing device, an acquisition step in which an acquisition means of the information processing device acquires, for each pre-trained model, a result of object detection from an image performed by the pre-trained model; a selection step in which a selection means of the information processing device selects, for each pre-trained model, a result to be output from among results of object detection from images performed by the pre-trained model; an output step in which an output means of the information processing device outputs the results selected for each pre-trained model in the selection step; An information processing method comprising:

10. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 8.

Images