Image processing system

Abstract

To provide a technology for enabling an operator to easily acquire a desired recognition result from an instruction possible image recognition model.SOLUTION: An image processing system of the present disclosure includes an indicatable image recognition model that receives an image and instruction information as input and outputs a recognition result, and a determination unit that acquires the recognition result equal to or greater than the number of objects from the indicatable image recognition model to which one image and instruction information equal to or greater than the number of objects set independently of the image are input, and determines and outputs adoption / non-adoption information of the acquired recognition result.SELECTED DRAWING: Figure 3B

Description

[Technical Field]

[0001] The present disclosure relates to image processing systems. [Background technology]

[0002] In recent years, a model capable of controlling the recognition target using instruction information has been developed for image recognition, as disclosed in Non-Patent Document 1. In this disclosure, such a model is called a "pointable image recognition model." A pointable image recognition model outputs a mask of an object that appears in an area specified by instruction information such as a point, box, or mask on an input image, making it possible to control the recognition target from outside.

[0003] If appropriate pointing information can be generated for each input image, it will be possible to recognize various objects in the input image with high accuracy. For example, in Non-Patent Document 2, pointing information for a box is generated using an object detector and used as input for a pointable image recognition model. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Dollar, Piotr and Girshick, Ross., Segment Anything., Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2023 [Non-patent document 2] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, Lei Zhang, Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks., https: / / arxiv.org / abs / 2401.14159 Summary of the Invention [Problem to be solved by the invention]

[0005] There are two issues when applying a pointable image recognition model to an arbitrary dataset using the method described in Non-Patent Document 2. The first issue is the need for performance verification of the object detector. In Non-Patent Document 2, the pointing information used for recognition by the pointable image recognition model is generated by an object detector. Therefore, performance verification of the object detector is necessary to confirm the performance obtained when using the pointable image recognition model. If recognition is not possible without verifying the performance of the object detector, it is impossible to determine whether the pointing information could not be generated or whether the pointable image recognition model failed to recognize.

[0006] The second point is that if the object detector generates instruction information and the pointable image recognition model fails to recognize it, it is difficult to provide feedback to the object detector to improve its performance. In the method of Non-Patent Document 2, the object detector and the pointable image recognition model operate independently, making it difficult to adjust the object detector so that the operator can obtain the desired results from the pointable image recognition model.

[0007] Therefore, the present disclosure provides a technology that enables an operator to easily obtain a desired recognition result from an instruction-enabled image recognition model for any data set. [Means for solving the problem]

[0008] In order to solve the above problem, the image processing system disclosed herein is characterized by comprising an instructable image recognition model that receives an image and instruction information as input and outputs a recognition result, and a judgment unit that acquires recognition results equal to or greater than the number of objects from the instructable image recognition model to which one image and instruction information equal to or greater than the number of objects set independently of the image are input, and judges and outputs information indicating whether the acquired recognition results are accepted or rejected.

[0009] Further features related to the present disclosure will become apparent from the description of this specification and the accompanying drawings. Also, aspects of the present disclosure are achieved and realized by the elements and combinations of various elements and the aspects of the following detailed description and the appended claims. The description of this specification is merely exemplary and does not limit the scope or application of the claims of the present disclosure in any way. [Effects of the Invention]

[0010] According to the technology of the present disclosure, an operator can easily obtain a desired recognition result from a designable image recognition model for an arbitrary data set. Other problems, configurations, and effects will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 10A and 10B are diagrams for explaining the behavior of a pointable image recognition model. [Figure 2] FIG. 10 is a diagram illustrating an overview of a recognition result acceptance determination model. [Figure 3A] 1 is a hardware configuration diagram of an image processing system according to a first embodiment. [Figure 3B] 1 is a functional block diagram of an image processing system according to a first embodiment. [Figure 4] 10 is a flowchart showing a learning flow of a recognition result acceptance determination model. [Figure 5]FIG. 10 is a diagram illustrating an example of an interface screen. [Figure 6] 10 is a flowchart showing an inference flow of the recognition result acceptance determination unit 303 using a trained recognition result acceptance determination model. [Figure 7] FIG. 10 is a diagram illustrating an example of an interface screen. [Figure 8] FIG. 10 is a functional block diagram of an image processing system according to a second embodiment. [Figure 9] FIG. 10 is a diagram for explaining an outline of the operation of an instruction information generating unit. [Figure 10] 10 is a flowchart showing a learning flow of the instruction information generating unit. [Figure 11] 10 is a flowchart showing an inference flow when an instruction information generating unit is used. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] <Directible Image Recognition Model> FIG. 1 is a diagram illustrating the behavior of a pointable image recognition model 103. As shown in FIG. 1, when an image 101 and pointing information 102 (102a, 102b, or 102c) are provided to the pointable image recognition model 103, a recognition result 104 (104a, 104b, or 104c) can be obtained. For ease of viewing, the pointing information 102 is illustrated together with the image 101. The pointing information 102 can include information regarding the shape, size, and position on the image, such as a box, dot, or circle. In the case of Non-Patent Document 1, the recognition result 104 is a predicted mask.

[0013] Generally, the class and shape of the object that the operator wants to recognize will change depending on the analysis content. For example, human detection requires prediction of a mask that includes the entire human outline, while human face detection requires prediction of a mask that only includes the head. There are also cases where the operator wants to recognize classes other than people, such as cars.

[0014] In the instruction-enabled image recognition model 103, the recognition target can be controlled by controlling the instruction information 102. In this disclosure, an example is used in which a relatively bright oval object and a dark triangular object are recognized one by one against a dark background, as in image 101. For example, when box instruction information 102a is used, a predicted mask of the single object in the upper left corner is obtained as recognition result 104a. In the example of this disclosure, the target is a task of recognizing objects one by one, so recognition result 104a is preferable to the operator.

[0015] On the other hand, the recognition results 104b and 104c based on point-like instruction information 102b and box-like instruction information 102c recognize two objects using a single predictive mask. In the example of instruction information 102a, the point that is the instruction information is near the boundary between two elliptical objects, so a recognition result that includes both elliptical objects is output. In the example of instruction information 102c, the box that is the instruction information includes two elliptical objects, so a recognition result that includes both elliptical objects is output. In this way, even for the same image 101, the instruction-enabled image recognition model outputs different recognition results 104 depending on the instruction information 102. In this case, appropriate instruction information 102 is required to obtain a recognition result 104 that is preferable to the operator.

[0016] In Non-Patent Document 2, the pointing information of the box is predicted using an object detector, but the object detector needs to be adjusted to obtain a recognition result 104 that is desirable to the operator. Therefore, annotations for adjusting the object detector and evaluating its performance may be required. Furthermore, if the recognition result 104 output by the pointable image recognition model 103 is not desirable to the operator, it is not obvious how to adjust the object detector, and trial and error is required. The present disclosure provides an image processing system for solving such problems.

[0017] <Recognition result acceptance / rejection judgment model> FIG. 2 is a diagram illustrating an overview of a recognition result acceptance determination model 201 used in the image processing system of the present disclosure. The image processing system of the present disclosure configures a recognition result acceptance determination model 201 that determines whether to accept or reject the recognition result 104 output by the instruction-enabled image recognition model 103. The recognition result acceptance determination model 201 receives an image 101 and a recognition result 104 as input, and outputs an acceptance determination result 202. When a recognition result 104a that is favorable to the operator is input, the recognition result acceptance determination model 201 outputs a signal (acceptance determination result 202a) indicating that the recognition result 104a is to be adopted. Conversely, when a recognition result 104b that is unfavorable to the operator is input, the recognition result acceptance determination model 201 outputs a signal (acceptance determination result 202b) indicating that the recognition result 104b is to be rejected. Note that the instruction-enabled image recognition model 103 of Non-Patent Document 1 outputs only a prediction mask and does not output class information. Therefore, when the recognition result 104 is adopted, the recognition result acceptance determination model 201 may output an acceptance determination result 202a including class information. With this configuration, the elliptical object and the triangular object in this embodiment can be recognized as different classes.

[0018] Non-Patent Document 1 describes a method for obtaining multiple recognition results 104 from a pointable image recognition model 103 by using multiple point indication information 102. This method involves, for example, dividing an image 101 into a 32 x 32 grid and using the center coordinates of each grid as point indication information. This method makes it possible to obtain multiple recognition results from the image 101. On the other hand, by applying a recognition result acceptance determination model 201 to extract recognition results 104 that are determined to be acceptable, it becomes possible to easily obtain recognition results 104 that are preferable to the operator. The present disclosure provides a method for configuring such a recognition result acceptance determination model 201.

[0019] Furthermore, while the example of the present disclosure targets a task of recognizing objects one by one, a mask related to a portion of an object or a mask that groups multiple objects may be the recognition result 104 preferred by the operator. Furthermore, while the task targets an object recognition, a background region may be the recognition result 104 preferred by the operator. Furthermore, a task may be set to acquire recognition results for objects of each class from an image 101 that includes multiple classes, or a task may be set to acquire recognition results for objects of some classes from an image 101 that includes multiple classes. Furthermore, a task may be set to recognize a region with a specific texture. In this way, the class, shape, number, texture, etc., of the recognition result 104 preferred by the operator may change depending on the application target or the task to be used. The present disclosure provides an image processing system for easily acquiring the recognition result 104 preferred by the operator from an instructionable image recognition model 103 when the recognition result 104 preferred by the operator changes.

[0020] <Image processing system configuration example> FIG. 3A is a hardware configuration diagram of an image processing system 10 according to a first embodiment. The image processing system 10 is a computer device such as a general-purpose computer, a smartphone, a tablet, or a server device. The image processing system 10 includes a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage device 14, a display device 15, and an input device 16. The CPU 11 executes a program to realize functions provided by the image processing system 10. The ROM 12 and RAM 13 hold data used by the CPU 11. The storage device 14 stores a program, an image 101, instruction information 102, a designable image recognition model 103, a recognition result acceptance / rejection determination model 201, a recognition result 104 of the designable image recognition model 103, instruction information, etc. The storage device 14 is, for example, a storage device such as a hard disk or a USB memory. The display device 15 is, for example, a display or a touch panel. The display device 15 displays various information, a user interface, etc. on a screen. The input device 16 is a mouse, a keyboard, a touch panel, a microphone, etc. The image processing system 10 can be configured by, for example, one computer device, but can also be configured by multiple computer devices.

[0021] FIG. 3B is a functional block diagram of the image processing system 10. The image processing system 10 includes an instruction-enabled image recognition model 103, an instruction unit 301, an instruction information storage unit 302, and a recognition result acceptance / rejection determination unit 303. As described above, the instruction-enabled image recognition model 103 receives an image 101 and instruction information 102 as input, and outputs a recognition result 104. The instruction unit 301 receives input from an operator of acceptance / rejection information and class information (instruction information) for the recognition result 104 output from the instruction-enabled image recognition model 103. The instruction information storage unit 302 stores the acceptance / rejection information and class information input by the operator in association with the image 101 and the recognition result 104. The recognition result acceptance / rejection determination unit 303 predicts the acceptance / rejection information and class information of the image 101 and the recognition result 104 using the recognition result acceptance / rejection determination model 201, which has been trained using the information stored in the instruction information storage unit 302. The recognition result acceptance / rejection determination unit 303 outputs acceptance / rejection information and class information as an output 304 .

[0022] When the image processing system 10 is configured with multiple computer devices, for example, the instructionable image recognition model 103 of the image processing system 10 can be configured to be executable on another computer device. Alternatively, the instructionable image recognition model 103 may be stored in cloud storage, and the image processing system 10 can access the cloud via a network to send and receive information.

[0023] <Learning flow for recognition result acceptance / rejection model> 4 is a flowchart showing the learning flow of the recognition result acceptance determination model 201 used by the recognition result acceptance determination unit 303. The learning flow shown in FIG. 4 starts when the operator determines that learning of the recognition result acceptance determination model 201 is necessary. At this time, the operator determines that it is necessary when building the recognition result acceptance determination model 201 for a new data set or task. The learning flow in FIG. 4 may also be started when the operator determines that the performance of the trained recognition result acceptance determination model 201 is insufficient. The learning flow may also be started when the performance of the instructionable image recognition model 103 does not reach a predetermined value in an evaluation data set on the image processing system side.

[0024] In step S401, the image processing system 10 accepts the selection of an image to be annotated from the operator. At this time, the operator specifies an area in the image processing system 10 where one or more images 101 are stored, and the image processing system 10 presents the images as annotation candidates to the operator, and the operator selects the image to be annotated from the candidates. Alternatively, the operator may specify the image to be annotated directly from a screen of the image processing system 10.

[0025] In step S402, the image processing system 10 selects one piece of instruction information 102 to be used from a plurality of pieces of preset instruction information. Here, the plurality of pieces of preset instruction information may be the grid center coordinates described in Non-Patent Document 1, or may be a predetermined point, box, or other such instruction information. The pointable image recognition model 103 cannot recognize all objects when the image 101 contains more objects than the plurality of pieces of preset instruction information 102. Therefore, a number of pieces of instruction information 102 greater than the expected number of objects per image 101 is preset. Alternatively, a plurality of pieces of instruction information 102 may be set independently of the image 101, or a plurality of pieces of instruction information 102 may be acquired and used by a predetermined process for the input image. That is, a plurality of pieces of instruction information 102 may be set and used using an object detector as described in Non-Patent Document 2. However, the object detector is configured to acquire a large number of detection windows to prevent missed detections.

[0026] In step S403, the image processing system 10 inputs the image selected in step S401 and the instruction information 102 selected in step S402 to the instruction-enabled image recognition model 103, and obtains the recognition result 104.

[0027] In step S404, the image processing system 10 checks whether the recognition results 104 have been acquired for all of the preset multiple pieces of instruction information 102. If the recognition results 104 have been acquired for all of the instruction information 102, the system proceeds to step S405. If acquisition has not been completed, the system returns to step S402 and continues processing.

[0028] In step S405, the image processing system 10 removes duplicated recognition results from the multiple recognition results 104. As described in Non-Patent Document 1, a box that covers the entire prediction mask is calculated, and the degree of overlap between the prediction masks is evaluated using an index called IoU (Intersection over Union) between the boxes, and those with low scores can be removed from among the recognition results 104 that have a certain degree of overlap. In addition to removing duplicates using IoU between boxes, duplicates may also be removed based on a similarity evaluation such as IoU between prediction masks or Dice Similarity Coefficient (Dice Similarity Coefficient).

[0029] In step S406, the image processing system 10 presents to the operator the image selected in step S401 and one or more recognition results 104 that were not removed in step S405.

[0030] In step S407, the image processing system 10 receives from the operator input of acceptance / rejection information (instruction information) for each of the one or more recognition results presented in step S406.

[0031] 5 is a diagram showing an example of an interface screen 500 for performing steps S401 to S407. The interface screen 500 includes an image 101, an image selection unit 501, a recognition execution instruction unit 502, a teaching information storage instruction unit 503, a recognition result display unit 504, and a teaching information selection unit 505.

[0032] In the image selection unit 501, the operator can select an image 101 for which to display the acceptance / rejection information (step S401). This selection can be made by directly inputting the image file, or by selecting from image files stored in a predetermined storage area. The selected image 101 can also be displayed on the screen 500.

[0033] When selected by the operator, the recognition execution instruction unit 502 executes recognition processing using the instruction-enabled image recognition model 103. This corresponds to the processing from steps S402 to S405.

[0034] The recognition result display unit 504 displays the recognition result 104 from which overlaps have not been removed (step S406). In this example screen, only the recognition result 104 is displayed, but it may be displayed overlapping the image 101 or side by side with the image 101. Furthermore, the instruction information 102 used in the displayed recognition result 104 may also be displayed together.

[0035] The operator checks the recognition result display unit 504 and selects whether the recognition result is desirable or undesirable in the instruction information selection unit 505. In this screen example, the instruction information selection unit 505 selects between a reject class, which indicates that the recognition result is undesirable to the operator, and one or more pre-set classes. The operator can use the instruction information selection unit 505 to select whether the recognition result 104 is desirable or, if desirable, which class to annotate it as. The instruction information selection unit 505 also displays information corresponding to one or more recognition results 104. In this screen example, there are two recognition result display units 504a and 504b, and corresponding instruction information selection units 505a and 505b. The number of displays in the recognition result display unit 504 and the instruction information selection unit 505 is the same, and a predetermined number may be displayed each, or the number of recognition results 104 that were not removed in step S405 may be used as the display number.

[0036] In this way, by selectively assigning acceptance / rejection information and class information to the recognition result 104 of the instructable image recognition model 103, it is possible to obtain a recognition result 104 that is preferable to the operator from the recognition result 104 of the instructable image recognition model 103 and assign class information.

[0037] Furthermore, if the recognition result acceptance determination model 201 has been trained in advance, the display may be changed based on the prediction result of the recognition result acceptance determination model 201. That is, the prediction result of the recognition result acceptance determination model 201 may be specified in advance as the initial value for selection by the instruction information selection unit 505, so that the operator only needs to confirm it and no selection is required. Also, it may be set so that the recognition results 104 for which the rejection score in the prediction result of the recognition result acceptance determination model 201 is higher than a predetermined threshold are not displayed, or the display order may be determined by the rejection score in the prediction result of the recognition result acceptance determination model 201. That is, by displaying the results with the lowest rejection score first, the operator may be able to efficiently teach the desired class, or may be able to preferentially check the recognition results 104 that are not desirable for the operator and should be rejected.

[0038] The teaching unit 301 may be realized by the recognition result display unit 504 and teaching information selection unit 505.

[0039] When selected by the operator, the instruction information storage instruction unit 503 associates the image 101, the recognition result 104, and the selection result of the instruction information selection unit 505 as instruction information and stores them in the instruction information storage unit 302. Here, the instruction information storage unit 302 is, for example, the storage device 14. The above series of operations by the operator constitutes the processing up to step S407.

[0040] In step S408, the image processing system 10 determines whether the addition of teaching information to the predetermined learning images has been completed. This process may be achieved by the image processing system 10 determining whether the number of images 101 to which teaching information has been added exceeds a predetermined number, or the operator may make the determination and input it to the image processing system 10. If the addition of teaching information to the predetermined learning images has been completed, the process proceeds to step S409. If not, the process returns to step S401, where the selection of an image is accepted and teaching information is added to the next image.

[0041] In step S409, the image processing system 10 trains the recognition result acceptance determination model 201 using the instruction information corresponding to the image 101 and the recognition result 104 stored in the instruction information storage unit 302. The recognition result acceptance determination model 201 is a model that receives the image 101 and one recognition result 104 as input, as shown in Fig. 2, and predicts the instruction information selected by the instruction information selection unit 505. This is realized by, for example, a convolutional neural network or the like, and trains using the instruction information selected by the instruction information selection unit 505 with cross entropy.

[0042] Values ​​related to learning, such as the number of updates, learning rate, batch size, and model structure, are set in advance by an administrator of the image processing system 10 and registered in the storage device 14 of the image processing system 10. These values ​​may also be changed by the operator from a learning screen.

[0043] This method of selectively instructing instruction information, including whether or not to accept the recognition results, makes it possible to very easily control the behavior of the directional image recognition model 103. In contrast, a method of separately configuring an object detector requires annotation of numerous box information for adjusting and verifying the object detector. While annotation of box information requires a large amount of work, selecting instruction information can be performed very easily with little work. Furthermore, if adjustment of the object detector is required, training of the object detector is required. However, training of the image classifier used in this disclosure can generally be performed more stably and quickly than training of the object detector. Furthermore, the number of images with instruction information required for training may be smaller. Furthermore, since this directly affects the behavior of the directional image recognition model 103, a method for modifying the output of the directional image recognition model 103 becomes clear.

[0044] <Inference flow of the recognition result acceptance / rejection determination unit> 6 is a flowchart showing an inference flow of the recognition result acceptance determination unit 303 using the trained recognition result acceptance determination model 201. This flow is started, for example, when an instruction to execute the recognition result acceptance determination unit 303 is issued. This flow describes an example in which an operator performs annotation using the instructable image recognition model 103, but the image processing system 10 may automatically execute the flow without requiring the operator to confirm the results.

[0045] In step S601, the image processing system 10 receives a selection of an image to be inferred from the operator.

[0046] The processing from step S602 to step S605 is the same as step S402 to step S405 in Fig. 4. That is, the image processing system 10 processes a plurality of preset instruction information 102 one by one to obtain one or more recognition results 104 from the instruction-enabled image recognition model 103, and removes duplicated recognition results 104 using the degree of overlap of the recognition results 104.

[0047] In step S606, the image processing system 10 uses the recognition result acceptance / rejection determination model 201 learned using the learning flow shown in FIG. 4 and the image 101 selected in step S601 to predict the acceptance / rejection information and class information for each of the one or more recognition results 104 obtained in step S605.

[0048] In step S607, the recognition result acceptance determination unit 303 outputs the recognition result 104 determined to be adopted in step S606. The image processing system 10 displays the predicted recognition result 104 on the display device 15. The recognition result 104 determined to be adopted here is the recognition result 104 that predicted a class other than the rejected class in the output of the recognition result acceptance determination model 201.

[0049] In step S608, the image processing system 10 receives confirmation from the operator of the recognition result output in step S607. The operator can correct the recognition result as necessary.

[0050] 7 is a diagram showing an example of an interface screen 700 for performing steps S601 to S608. The interface screen 700 includes an image 101, an image selection unit 701, a recognition and acceptance / rejection determination execution instruction unit 702, a recognition result correction unit 703, a recognition result confirmation unit 704, an instance display unit 705, and a predicted class display unit 706.

[0051] In the image selection unit 701, the operator selects an image 101 to be recognized (step S601). The selected image 101 may be displayed on the screen.

[0052] When selected by the operator, the recognition and acceptance / rejection determination execution instruction unit 702 executes steps S602 to S607. In this example screen, the recognition result 104 output in step S607 is displayed.

[0053] The instance display unit 705 displays the recognition results 104 output in step S607 in a form that allows each to be identified. For example, this is a method of displaying each recognition result 104 using a different color or texture. The instance display unit 705 allows the operator to check the predicted mask shape for each adopted recognition result 104.

[0054] The predicted class display unit 706 displays the class predicted by the recognition result acceptance determination model 201 in step S606 in an identifiable format. This is, for example, a method of displaying each predicted class using a different color or texture. The predicted class display unit 706 allows the operator to check the predicted class information for each adopted recognition result 104.

[0055] In the recognition result correction unit 703, the operator can remove the selected recognition result 104 or add a recognition result 104 by adding additional instruction information such as a point or a box. This allows the correction of the result in step S608 to be performed.

[0056] When selected by the operator, the recognition result confirmation unit 704 stores the adopted recognition result in a storage area.

[0057] <Summary of the First Embodiment> As described above, the image processing system 10 according to the first embodiment includes an instruction-enabled image recognition model 103 that receives an image 101 and instruction information 102 as input and outputs a recognition result 104, and a recognition result acceptance / rejection determination unit 303 that acquires recognition results 104 equal to or greater than the number of objects set independently of the image from the instruction-enabled image recognition model 103 to which one image 101 and instruction information 102 equal to or greater than the number of objects set independently of the image are input, and determines and outputs an acceptance / rejection determination result 202 (acceptance / rejection information) of the acquired recognition results 104. This configuration allows the operator to easily acquire the desired recognition result 104 from the instruction-enabled image recognition model 103.

[0058] [Second embodiment] In the first embodiment described above, a method for obtaining a recognition result 104 that is preferable to the operator by using a plurality of pieces of instruction information 102 that have been set in advance has been described. However, with this method, it is difficult to output all of the recognition results 104 that are preferable to the operator when a recognition error occurs with the plurality of pieces of instruction information 102 that have been set in advance. Therefore, in the second embodiment, a configuration for outputting instruction information 102 that matches an image 101 by using the configuration of the first embodiment will be described.

[0059] 8 is a functional block diagram of an image processing system according to the second embodiment. The image processing system according to the second embodiment includes an instruction information generation unit 801 that generates instruction information 102 to be used in an instruction-enabled image recognition model 103. The output 802 of the instruction information generation unit 801 includes the instruction information 102 and a determination as to whether the instruction generation has been completed.

[0060] <Operation of the instruction information generation unit> 9 is a diagram for explaining an outline of the operation of the instruction information generation unit 801 of the second embodiment. The instruction information generation unit 801 is a machine learning model that receives an image 101 and a set of adopted recognition results 901 as input, and outputs instruction information candidates 802a and an instruction generation termination determination 802c. Here, the initial value of the set of adopted recognition results 901 may be generated using the plurality of preset instruction information 102 described in the first embodiment.

[0061] The instruction information generation unit 801 includes an instruction information prediction unit 801a and an end prediction unit 801b. The instruction information prediction unit 801a receives the image 101 and the adopted recognition result set 901 as input and outputs instruction information candidate 802a. The instruction information candidate 802a is, for example, box information, as indicated by instruction information 802b. Here, the instruction information candidate 802a may be something other than a box, and may be instruction information usable by the instruction-enabled image recognition model 103, such as a point or a mask. The end prediction unit 801b outputs an instruction generation end determination 802c.

[0062] The instruction information candidate 802a generated by the instruction information prediction unit 801a is input together with the image 101 to the instruction image recognition model 103, which outputs a recognition result 104. The recognition result 104 is input together with the image 101 to the recognition result adoption determination unit 303. The recognition result determined to be adopted here is added to the adopted recognition result set 901.

[0063] The above operation is repeated until the instruction generation end determination 802c output by the end prediction unit 801b determines that the operation is ended. The instruction generation end determination 802c is, for example, a determination as to whether or not all masks for objects appearing in the image 101 have been created.

[0064] With this configuration, even if there is an object that cannot be recognized by the plurality of pieces of instruction information 102 set in advance, it is possible to generate appropriate instruction information.

[0065] <Learning flow of the instruction information generating unit 801> FIG. 10 is a flowchart showing the learning flow of the instruction information generating unit 801.

[0066] In step S1001, the operator prepares a dataset for which the recognition result has been confirmed. At this time, instruction information and the recognition result confirmed by the operator may be prepared using the method described in the first embodiment.

[0067] In step S1002, the image processing system 10 starts learning when the operator instructs the instruction information generating unit 801 to learn.

[0068] In step S1003, the image processing system 10 acquires one image 101 and all recognition results 104 associated with the image 101 from the dataset stored in storage. Typically, machine learning models such as neural networks are trained using mini-batch learning, but for simplicity, an example of training one image at a time will be described here. Mini-batch learning may also be used during implementation.

[0069] In step S1004, the image processing system 10 determines whether or not to remove a recognition result from all of the recognition results 104 associated with the image 101 acquired in step S1003. This determination is made randomly with a predetermined probability. Furthermore, when the image processing system 10 determines to remove a recognition result, it removes a predetermined or random number of recognition results 104 from all of the recognition results 104 associated with the image 101. The recognition results 104 associated with the image 101 from which one or more recognition results have been removed correspond to a simulation of the adopted recognition result set 901, and will hereinafter be referred to as the adopted recognition result set 901.

[0070] In step S1005, the image processing system 10 determines whether or not a determination was made in step S1004 to remove the recognition result, and changes subsequent behavior depending on the determination result. If the recognition result was removed, the process proceeds to step S1006. If the recognition result was not removed, the process proceeds to step S1008.

[0071] In step S1006, when the image 101 and the adopted recognition result set 901 are input, the instruction information generation unit 801 is trained to determine whether to continue generating instruction information in the instruction generation termination determination. For example, when the instruction information generation unit 801 is configured as a neural network, a neuron that determines whether to terminate instruction generation is prepared, and the unit is trained to predict a class that will make a continuation determination using cross entropy.

[0072] In step S1007, the image processing system 10 learns to generate instruction information candidates so that the instruction information generating unit 801 can predict any one of the recognition results removed in step S1004 when predicting the instruction information candidates.

[0073] This is performed, for example, by reinforcement learning, in which recognition is performed on the instruction-enabled image recognition model 103 using instruction information candidates 802a output by the instruction information generation unit 801 when the image 101 and the adopted recognition result set 901 are input, and a high reward is given when the obtained recognition result 104 has a high degree of overlap with one or more of the recognition results removed in step S1004. Furthermore, instead of using reinforcement learning, the teaching information may be generated using the recognition result 104 obtained by the instruction information candidates 802a predicted by the instruction information generation unit 801 and the recognition result removed in step S1004 that has a high degree of overlap. For example, a box covering the recognition result 104 may be calculated as the teaching information, or mask information of the recognition result 104 may be used as the teaching information, or point information may be randomly generated from the recognition result 104 and used as the teaching information. Furthermore, if the degree of overlap between the recognition result 104 obtained using the instruction information generated by these methods and the recognition result 104 obtained using the instructable image recognition model 103 is low with either of the removed recognition results 104, the instruction information may not be able to provide a recognition result 104 that is favorable to the operator, and therefore the instruction information may be controlled not to be used.

[0074] Additionally, although the description here is of the case where there is a data set that has been confirmed by the operator, the data set may include images 101 whose recognition results 104 have not been confirmed by the operator. In this case, learning may be performed using the determination result of the recognition result acceptance determination unit 303 for the recognition result 104 obtained from the instruction information candidates 802a output by the instruction information generation unit 801. In other words, when reinforcement learning is used, learning may be performed so that a high reward is obtained when the recognition result 104 is adopted by the recognition result acceptance determination unit 303. Furthermore, when the recognition result 104 is adopted by the recognition result acceptance determination unit 303, information such as a box, mask, and points calculated from the recognition result 104 may be learned as training information.

[0075] In step S1008, the image processing system 10 learns so that the instruction generation end decision 802c makes an end decision, using cross entropy in the same way as in step S1006.

[0076] In step S1009, the image processing system 10 determines whether the learning termination condition is met. If the learning termination condition is met, the learning flow ends. If the learning termination condition is not met, the process returns to step S1003, and the learning flow continues from sampling of learning data.

[0077] According to this embodiment, it is possible to train the instruction information generation unit 801 that predicts the instruction information 102 so that the operator can obtain a recognition result 104 that is preferable to the operator from the instruction-enabled image recognition model 103. At this time, it is possible to control the teaching information in consideration of the behavior of the instruction-enabled image recognition model 103. Furthermore, by using the recognition result acceptance determination unit 303, it is possible to exploratory train the instruction information generation unit 801 even for images 101 whose recognition results 104 are incomplete.

[0078] <Inference flow of the instruction information generation unit> FIG. 11 is a flowchart showing an inference flow when the instruction information generating unit 801 is used.

[0079] In step S1101, an image to be inferred is selected, similarly to step S601. At this time, the image to be inferred may be selected by the operator, or may be automatically selected by the image processing system 10.

[0080] In step S1102, the image processing system 10 collects recognition results that are determined to be adopted using a plurality of preset instruction information 102 and the recognition result adoption determination unit 303. This is performed using the same procedure as the method described in steps S602 to S607. The recognition results 104 collected here are defined as an adopted recognition result set 901.

[0081] In step S1103, the instruction information generation unit 801 uses the image 101 selected in step S1101 and the adopted recognition result set 901 output in step S1102 to output instruction information candidates 802a and an instruction generation termination decision 802c. If the instruction generation termination decision 802c determines that the instruction generation is terminated, the process proceeds to step S1106, and if the determination determines that the instruction generation is continued, the process proceeds to step S1104. The termination decision in step S1103 may be performed a predetermined number of times before proceeding to step S1106.

[0082] In step S1104, the image processing system 10 obtains the recognition result 104 using the instruction information candidate 802a generated in step S1103, the image 101, and the instruction-capable image recognition model 103.

[0083] In step S1105, the image processing system 10 obtains the result of the acceptance / rejection determination of the recognition result 104 obtained in step S1104 using the recognition result acceptance determination unit 303. If the recognition result 104 is accepted, the recognition result 104 is added to the accepted recognition result set 901. Thereafter, steps S1103 to S1105 are repeated until an end determination is made in step S1103.

[0084] In step S1106, the image processing system 10 outputs the recognition results adopted in steps S1102 and S1105.

[0085] In step S1107, similar to step S608, the image processing system 10 receives confirmation from the operator of the recognition result output in step S607. The operator can correct the recognition result as necessary.

[0086] <Summary of the second embodiment> According to the above configuration, it is possible to generate only instruction information 102 that outputs a recognition result 104 that is preferable to the operator, thereby reducing recognition omissions.

[0087] Furthermore, the methods described in the first and second embodiments may be used not only for image recognition but also for annotation. By using the recognition results collected in these embodiments as training information to train a lighter model than the configurable image recognition model, it is expected that a lighter and faster inference engine can be constructed.

[0088] [Variations] The present disclosure is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present disclosure, and it is not necessary to include all of the described configurations. Furthermore, a part of one embodiment can be replaced with a configuration of another embodiment. Furthermore, a configuration of another embodiment can be added to a configuration of one embodiment. Furthermore, a part of the configuration of each embodiment can be added to, deleted from, or substituted for a part of the configuration of another embodiment. [Explanation of symbols]

[0089] 10...Image processing system 101...Image 102…Instruction information 103...Directible image recognition model 104…Recognition result 201…Recognition result acceptance / rejection judgment model 301...Instruction Department 302...Teaching information storage unit 303...Recognition result acceptance / rejection determination unit 801…Instruction information generation unit

Claims

1. 1. An image processing system, comprising: an instruction-enabled image recognition model that receives an image and instruction information as input and outputs a recognition result; An image processing system comprising: a judgment unit that acquires recognition results for more than the number of objects from the instructable image recognition model to which one image and instruction information for more than the number of objects set independently of the image are input, and judges and outputs information on whether the acquired recognition results are accepted or rejected.

2. 2. The image processing system according to claim 1, a display unit that displays the recognition result output by the pointable image recognition model in association with the image, using the one image and pointing information equal to or greater than the number of objects as input; an input unit that receives input of the acceptance / rejection information regarding the displayed recognition result; a storage unit that stores the image, the displayed recognition result, and the acceptance / rejection information input to the input unit in association with each other; Furthermore, The determination unit An image processing system characterized in that it is trained to predict the acceptance / rejection information stored in the storage unit using the image stored in the storage unit and the recognition result as input.

3. 3. The image processing system according to claim 2, further comprising an instruction information generation unit that is trained to generate the instruction information using the image and the adopted recognition result as input; An image processing system characterized by using the instruction information generated by the instruction information generation unit when the image is input, outputting the recognition result using the instruction-enabled image recognition model, inputting the output recognition result into the judgment unit, and outputting the judgment result of the acceptance / rejection information.

4. 4. The image processing system according to claim 3, The instruction information generation unit An image processing system characterized by inputting the image and the adopted recognition result, outputting new instruction information and a determination of completion of instruction information generation, and repeating generation of the instruction information until the completion determination indicates that generation of the instruction information has been completed.

5. 5. The image processing system according to claim 4, an image processing system that outputs the recognition result that is determined to be adopted by the determination unit;

6. 3. The image processing system according to claim 2, The image processing system is characterized in that the input unit has an interface for inputting acceptance / rejection information in a format of selecting a class representing rejection of the recognition result from preset class information.

7. 5. The image processing system according to claim 4, An image processing system characterized in that the instruction information generation unit is trained using either reinforcement learning using the output of the judgment unit, or supervised learning using teaching information calculated from recognition results that are preferable to the operator.

Images