Region extraction method, region extraction system

Abstract

To enable a cost reduction for updating a neural network model.SOLUTION: A region extracting method that is executed by a computer includes: an image obtaining step for obtaining a picked-up image; a feature extracting step for generating a feature map by using the picked-up image; a region extracting step for extracting a candidate frame that is a frame which represents a region where an object is present in the picked-up image by using the feature map; a segmentation step for generating a mask that is a frame which represents the region where the object is present in the picked-up image by using the feature map and the candidate frame; a user inputting step for obtaining input information which is input by a user and which is information relating to the object in the picked-up image; a region estimating step for extracting the candidate frame by using the feature map and the input information; and an interaction encode step for updating the feature map by using the candidate frame extracted in the region estimating step and the input information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a region extraction method and a region extraction system.

Background Art

[0002] In recent years, with the emergence of Deep Neural Network (DNN), image segmentation for predicting the region of an object in pixel units has advanced by leaps and bounds and is expected to be applied in the real world. However, since the performance of DNN depends on a large amount of learning data, manual annotation is required. Therefore, in recent years, an interactive segmentation method that can semi-automate the annotation of images has been studied. Furthermore, DNN has also been proposed for interactive segmentation work in order to further simplify the manual steps and reduce the time cost. In Non-Patent Document 1, first, an object mask is automatically predicted using a segmentation DNN model, and then, using another interactive segmentation DNN model, user interactions such as clicking or drawing a region box are input to provide guidance to the model on where to correct the mask.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In real-world applications, when using a well-trained DNN model, the accuracy gradually decreases as the number of untrained target categories increases. Similarly, annotation systems using deep learning face the same challenge; to maintain accuracy, the model needs to be frequently retrained on newly acquired data. The method disclosed in Non-Patent Document 1 uses two DNN models for annotation, resulting in significant time and effort required for model updates. A method is needed to reduce the cost of frequent retraining of DNN models. [Means for solving the problem]

[0005] A first aspect of the present invention is a region extraction method performed by a computer, comprising: an image acquisition step of acquiring a captured image; a feature extraction step of generating a feature map using the captured image; a region extraction step of extracting candidate frames, which are frames indicating regions in the captured image where objects exist, using the feature map; a segmentation step of generating a mask, which is a frame indicating regions in the captured image where objects exist, using the feature map and the candidate frames; a user input step of acquiring input information, which is information about objects in the captured image, input by a user; a region estimation step of extracting candidate frames using the feature map and the input information; and an interaction encoding step of updating the feature map using the candidate frames extracted in the region estimation step and the input information, wherein the region estimation step can further extract candidate frames using the mask generated in the segmentation step, and the interaction encoding step can further update the feature map using the mask generated in the segmentation step. A region extraction system according to a second aspect of the present invention includes: an image acquisition unit that acquires a captured image; a feature extraction unit that generates a feature map using the captured image; a region extraction unit that extracts candidate frames, which are rectangular frames indicating regions in the captured image where objects exist, using the feature map; a segmentation head that generates a mask, which is a closed region following the outline of the object in the captured image, using the feature map and the candidate frames; a user input unit that acquires input information, which is information about the object in the captured image, input by the user; and the feature map, oh and a region estimation unit that extracts the candidate frame using the input information, and the candidate frame extracted by the region estimation unit, oh The region estimation unit further extracts candidate frames using the mask generated by the segmentation head, and the interaction encoding unit further updates the feature map using the mask generated by the segmentation head. [Effects of the Invention]

[0006] According to the present invention, the cost of updating neural network models can be reduced. [Brief explanation of the drawing]

[0007] [Figure 1] Hardware configuration diagram of the computing unit [Figure 2] Functional block diagram of an image processing application [Figure 3] Flowchart showing the processing of the region estimation unit [Figure 4] Flowchart showing the processing of the interaction encoding unit. [Figure 5] Flowchart showing the processing of the interaction encoding unit. [Figure 6] This figure shows an example of applying an image processing device to an X-ray inspection system. [Figure 7]Flowchart showing the operation of the X-ray inspection system [Modes for carrying out the invention]

[0008] —Embodiment— Hereinafter, embodiments of the computing device and region extraction method according to the present invention will be described with reference to Figures 1 to 7.

[0009] (Hardware configuration) Many aspects disclosed herein are described as sequences of operations performed by computer devices. Furthermore, the various operations described herein may be performed by specific circuits, such as application-specific integrated circuits (ASICs), program instructions executed by one or more processors, or a combination of both. In addition, the sequences of operations described herein may be stored as sets of computer instructions on any computer-readable storage medium, causing a processor to execute the functions corresponding to those sequences at runtime. Thus, the various aspects of this disclosure may be embodied in numerous different forms, all of which fall within the technical scope of the present invention.

[0010] The embodiments of this disclosure relate to neural networks. Generally, a "neural network" is a computational model that has functions such as machine learning and pattern recognition. Neural networks may be applied to various operating environments, such as local devices, server / client configurations, and distributed networks.

[0011] Figure 1 is a hardware diagram of the arithmetic unit 300. Mechanisms and apparatus of various embodiments disclosed herein may be applied to any suitable computing system. The main components of the arithmetic unit 300 include one or more processors 302, memory 304, terminal interface 306, and storage interface 308. These components are connected via a communication bus.

[0012] The processor 302 is a central processing unit and includes one or more arithmetic cores. The memory 304 may include a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. The memory 304 may store all or part of the programs, modules, and data structures for implementing the functions described herein. For example, the memory 304 may store an image processing application 350. In certain embodiments, the image processing application 350 may include instructions or descriptions for executing functions on the processor 302 as described hereinafter.

[0013] The image processing application 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. The image processing application 350 may include data other than instructions or descriptions.

[0014] The terminal interface 306 can be attached with user output devices such as video display devices, speaker TVs, and user input devices such as keyboards, mice, keypads, touch pads, trackballs, buttons, light pens, or other pointing devices. The user may input input data and instructions to the arithmetic unit 300 and receive output data from the arithmetic unit 300 by operating the user input device using the user interface.

[0015] The storage interface 308 can be attached with one or more disk drives or storage devices (not shown). The content of the memory 304 can be stored in the storage device and read from the storage device as needed. The storage device may be a magnetic disk drive storage device, or may be an array of disk drives configured to appear as a single disk drive or other storage device. The network interface 318 may provide a communication path so that the arithmetic unit 300 and other devices can communicate with each other.

[0016] The arithmetic unit 300 may be a device that receives requests from other computer systems (clients) without a user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. The arithmetic unit 300 may also be a desktop computer, a portable computer, a notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other appropriate electronic device.

[0017] (Functional Configuration) FIG. 2 is a functional block diagram showing the functions of the image processing application 350 as functional blocks. The image processing application 350 includes a region automatic segmentation module 10 and an interaction module 20. Although not shown in FIG. 2, the image processing application 350 also has a loss calculation unit 30. The region automatic segmentation module 10 includes an image acquisition unit 11, a feature extraction unit 12, a region extraction unit 13, and a segmentation head 14. The interaction module 20 includes a user input unit 21, a region estimation unit 22, and an interaction encoding unit 23. The region estimation unit 22 includes a first convolutional layer 22A that is a learned neural network. The interaction encoding unit 23 includes a second convolutional layer 23A that is a learned neural network.

[0018] The image acquisition unit 11 acquires captured images of objects and outputs them to the feature extraction unit 12. The image acquisition unit 11 utilizes the terminal interface 306. The feature extraction unit 12 generates a feature map using the captured images acquired by the image acquisition unit 11. The feature map is output to the region extraction unit 13 and the region estimation unit 22. The feature extraction unit 12 can be implemented, for example, by a convolutional neural network (CNN), a Vision Transformer, or a combination of a CNN and a Vision Transformer.

[0019] The region extraction unit 13 uses the feature map output by the feature extraction unit 12 to extract candidate frames, which are rectangular frames indicating the regions where objects exist in the captured image, using known methods. The region extraction unit 13 extracts zero or one or more objects. The region extraction unit 13 can, for example, use a known Region Proposal Network (RPN). The segmentation head 14 uses the output of the region extraction unit 13 or the interaction module 20 to generate and output a mask, which is a closed region that follows the outline of the objects in the captured image. In other words, the segmentation head 14 may process using the output of the region extraction unit 13 or using the output of the interaction module 20. The mask calculated by the segmentation head 14 may be output to the image processing application 350 as a calculation result, or the image processing application 350 may perform further processing using the mask.

[0020] The user input unit 21 acquires input information, which is information about objects in the captured image, input by the user, and outputs it to the region estimation unit 22 and the interaction encoding unit 23. The user input unit 21 is, for example, an input interface that receives signals from a pointing device operated by the user. The user inputs information about the presence of objects in the captured image using the pointing device. Information about the presence of objects includes information about the location or region where the object exists, and information about the location or region where the object does not exist.

[0021] An example of the operation of the user input unit 21 is as follows. For example, when the user moves the mouse pointer and presses the left mouse button on the captured image, i.e., performs a left click, the user input unit 21 outputs the coordinate information of the clicked location on the captured image to the region estimation unit 22 as the location where an object exists. Hereafter, the user pointing to the location where an object exists will also be referred to as a "positive click". Also, when the user presses the right mouse button on the captured image, i.e., performs a right click, the user input unit 21 outputs the coordinate information of the clicked location on the captured image to the region estimation unit 22 and the interaction encoding unit 23 as the location where no object exists.

[0022] The region estimation unit 22 calculates candidate frames using the feature map output by the feature extraction unit 12 and the input information output by the user input unit 21. However, the region estimation unit 22 may also use the mask information output by the automatic region segmentation module 10 when calculating candidate frames. Specifically, when the user input unit 21 receives an operation from the user, if the segmentation head 14 has already output mask information, the region estimation unit 22 uses that mask information as an initial mask to calculate candidate frames. Also, when the user input unit 21 receives an operation from the user, if the segmentation head 14 has not yet output mask information, the region estimation unit 22 calculates candidate frames without using an initial mask.

[0023] The interaction encoding unit 23 rewrites a portion of the feature map using the input information output by the user input unit 21, the candidate frame output by the region estimation unit 22, and the feature map output by the feature extraction unit 12, and outputs it to the segmentation head 14. However, the interaction encoding unit 23 may also receive the input information output by the user input unit 21 and the feature map cropped by the candidate frame output by the region estimation unit 22.

[0024] Figure 3 is a flowchart showing the processing of the region estimation unit 22. In step S221, the region estimation unit 22 generates multiple candidate frames based on the user's input. For example, the region estimation unit 22 generates candidate rectangular frames with different widths and heights, centered at the position specified by the user. The aspect ratio of each rectangle may be constant or there may be multiple different aspect ratios. For example, a total of nine candidate frames are generated with short sides of 100 pixels, 200 pixels, and 300 pixels, and aspect ratios of 1:2, 1:1, and 2:1.

[0025] In the following step S222, the region estimation unit 22 determines whether an initial mask exists. If the region estimation unit 22 determines that an initial mask exists, it proceeds to step S223; otherwise, it proceeds to step S224. An initial mask exists when the region estimation unit 22 operates after the segmentation head 14 has operated. In step S223, the region estimation unit 22 modifies the candidate frames generated in step S221 based on the initial mask. For example, the region estimation unit 22 expands the candidate frames so that the entire region of the initial mask is included in the candidate frames. In step S224, the region estimation unit 22 selects one of the unprocessed candidate frames. Hereinafter, the candidate frame selected in this step will be referred to as the "candidate frame to be processed".

[0026] In the following step S225, the region estimation unit 22 inputs the feature map generated by the feature extraction unit 12 and the candidate frame to be processed into the first convolutional layer 22A and calculates an evaluation value. The first convolutional layer 22A is a pre-trained neural network having at least convolutional layers. The first convolutional layer 22A has been pre-trained to output a numerical value indicating the likelihood that the candidate frame to be processed encloses an object in the feature map.

[0027] In the following step S226, the region estimation unit 22 determines whether or not there are any unprocessed candidate frames. If the region estimation unit 22 determines that there are any unprocessed candidate frames, it returns to step S224; otherwise, it proceeds to step S227. In step S227, the region estimation unit 22 identifies the candidate frame with the highest evaluation value among the evaluation values ​​calculated in step S225. In step S228, the region estimation unit 22 cuts out the feature map using the candidate frame identified in step S227 and outputs it to the interaction encoding unit 23.

[0028] Figure 4 is a flowchart showing the processing of the interaction encoding unit 23. First, in step S241, the interaction encoding unit 23 generates an interaction map based on the input information obtained by the user input unit 21. The interaction map is, for example, a map that assigns the value "1" to the location indicated by the user as having an object and sets all others to "0", or a map that has a Gaussian distribution from "0" to "1" around the location indicated by the user. The interaction map has one or more channels to include one or more types of interactions.

[0029] In the following step 242, the interaction encoding unit 23 inputs the trimmed feature map, the interaction map generated in step S241, and the initial mask to the second convolutional layer 23A. The trimmed feature map is obtained by cutting the feature map output by the feature extraction unit 12 using the candidate frame output by the region estimation unit 22. If an initial mask does not exist, a blank map filled with zeros is input instead. The second convolutional layer 23A is a trained neural network that rewrites, i.e., updates the feature map based on the interaction map.

[0030] When a positive click is input as a user interaction, these are concatenated into an (N+a) channel map. "N" is the number of channels in the feature map, and "a" in this case is one interaction map and one initial mask. "a" can also be greater than 2 for other interaction types such as negative clicks and strokes. In the subsequent step S243, the interaction encoding unit 23 outputs the updated feature map output by the second convolutional layer 23A to the segmentation head 14, completing the process shown in Figure 4.

[0031] (Learning Methods) Figure 5 is a schematic diagram showing the learning method in the computing unit 300. Figure 5 simplifies the process to illustrate the overview of learning. First, the automatic region segmentation module 10 creates a mask using the captured image. This initially created mask is called the "initial mask". Here, we assume that the user is not satisfied with the accuracy of this initial mask and inputs information indicating the position of objects in the captured image. In this case, the interaction module updates the feature map using the feature map previously generated by the feature extraction unit 12 of the automatic region segmentation module 10, the input information entered by the user, and the initial mask, and inputs this to the segmentation head 14 to generate a new mask, the "improved mask".

[0032] The user judges whether the improved mask is successful or not. If the user judges it negatively, i.e., judges that further improvement is needed, it is input to the interaction module 20 as the initial mask. If the user judges the improved mask positively, i.e., judges that it has sufficient accuracy, it is input to the loss calculation unit 30 as the ground truth. The user's judgment of whether the improved mask is successful or not can be made by inputting an instruction to that effect to the calculation unit 300, by indirectly indicating a negative judgment by inputting new input information, or by indirectly indicating a positive judgment by not making any input for a predetermined period of time.

[0033] The loss calculation unit 30 calculates the learning loss L using the ground truth and initial mask as shown in Equation 1 below, and updates the parameters of the automatic region division module 10 and the interaction module 20. In other words, the loss calculation unit 30 calculates the learning loss L using the masks generated first and last by the automatic region division module 10.

[0034] TIFF0007846558000001.tif22170

[0035] However, in equation 1, N represents the total number of regions extracted by the region extraction unit 13, and M n p This indicates the initial mask in the nth region included in the captured image, M n g This indicates the ground truth in the nth region included in the captured image. Also, M' p M' indicates the initial mask in the area where the user interacted. g This indicates the ground truth in the area where the user interacted.

[0036] The first part of Equation 1 represents the difference between the initial mask and ground truth for all objects included in the captured image, and is the loss for updating the parameters of the automatic region segmentation module 10. The second part of the equation represents the difference between the incompletely updated post-update mask and ground truth, and is the loss for updating the parameters of the interaction module and the shared parts of the automatic region segmentation module 10, such as the feature extraction unit 12 and the segmentation head 14.

[0037] The parameters of the interaction module 20 are updated only after two or more interactive segmentation processes have been performed, allowing for optimization to reduce the number of iterations required for mask refinement. The entire network, including the domain auto-segmentation module 10 and the interaction module 20, can be learned end-to-end, allowing their parameters to be updated at once without requiring any other computations outside the network. This is because the domain auto-segmentation module 10 and the interaction module 20 are designed to share the same feature extraction unit 12 and mask output.

[0038] Furthermore, if multi-class classification heads are also considered, a classification loss must be added that calculates the difference between the ground truth of the manually corrected classes and the predicted results. In addition to improving the classification accuracy of learned classes, there are also methods for learning unlearned classes. When labeling objects of unlearned classes, the system cannot automatically predict the class at first, so the user needs to add class labels. This can be done by performing online training or by collecting images using a trial-and-error approach before training. For example, in the latent space, new class clusters can be learned by shortening the distance between features of the same class and lengthening the distance between features of different classes. As a result, the system gradually learns to predict new classes, becomes more automated, and the time cost required for annotation can be reduced.

[0039] The loss calculation unit 30 uses the terminal interface 306 to obtain the user's pass / fail judgment on the update mask in order to determine whether or not to treat the update mask as ground truth. However, the user may also indicate a rejection of the update mask by inputting new information, in which case the loss calculation unit 30 may use the output of the user input unit 21.

[0040] (Application to X-ray inspection equipment) Figure 6 shows an example of applying the computing device 300 to an X-ray inspection system 500. The X-ray inspection system 500 includes an X-ray device 501, a communication network 502, a user interface 503, and an image processing device 504. The X-ray device 501, the user interface 503, and the image processing device 504 are connected via the communication network 502.

[0041] The communication network 502 may include, for example, a local area network (LAN), a wide area network (WAN), a satellite network, a cable network, a wireless LAN network, or any combination thereof. Furthermore, the connection between the X-ray apparatus 501, the user interface 503, and the image processing apparatus 504 may be wired or wireless.

[0042] The X-ray apparatus 501 is a device for taking X-ray images. The X-ray apparatus 501 consists of, for example, an X-ray generator that emits X-rays and an X-ray detector that detects and analyzes reflected X-rays. The type of X-ray apparatus 501 is not particularly limited and may be an X-ray apparatus for luggage, a backscatter X-ray inspection apparatus, or a medical X-ray apparatus. The X-ray apparatus 501 takes an X-ray image of a predetermined subject and transmits it to the user interface 503 and the image processing apparatus 201 via the communication network 502. In other words, in this example, the captured image processed by the image processing apparatus 504 is an X-ray image.

[0043] In Figure 6, an example configuration for processing X-ray images is shown, which includes an X-ray apparatus 501. However, the present invention is not limited to this, and the X-ray apparatus 501 may be, for example, any camera, sensor, or other device that provides an input image to be processed. The user interface 503 includes a display that presents the captured image to the user, and a mouse and keyboard that accept user input.

[0044] The image processing device 504 includes an automatic region division module 10, an interaction module 20, a loss calculation unit 30, and a training data storage 40. The operation of the automatic region division module 10, the interaction module 20, and the loss calculation unit 30 is as described above. The training data storage 40 stores training data, and the image processing device 504 can perform pre-training using the data stored in the training data storage 40.

[0045] Figure 7 is a flowchart showing the operation of the X-ray inspection system 500. However, in Figure 7, the user's actions are represented by a different frame shape. First, in step S550, the X-ray device 501 takes an X-ray image, and the image acquisition unit 11 of the image processing device 504 acquires it. In the following step S551, the area automatic division module 10 of the image processing device 504 calculates an initial mask. In the following step S552, the user interface 503 presents the X-ray image taken in step S550 and the initial mask calculated in step S551 to the user.

[0046] In the following step S553, the image processing device 504 obtains the user's judgment regarding the initial mask via the user interface 503 and determines whether the judgment is OK or NG. If the image processing device 504 determines that the user has made an OK judgment, that is, that the initial mask has sufficient accuracy and is acceptable, it proceeds to step S557. If the image processing device 504 determines that the user has made an NG judgment, that is, that the initial mask has insufficient accuracy and should be improved, it proceeds to step S554. For example, if the user inputs information about an object into the captured image using the user interface 503, such as the position information of an object through a click operation, the image processing device 504 determines that the user has made an NG judgment, and if there is no user operation on the user interface 503 within a predetermined time, it determines that the user has made an OK judgment.

[0047] In the following step S554, the image processing device 504 regenerates the mask using the interaction module 20 and the segmentation head 14. In the following step S555, the image processing device 504 determines the user's judgment on the mask created in step S554 in the same way as in step S553. If it determines that the user has made a positive judgment, it proceeds to step S556; if it determines that the user has made a negative judgment, it returns to step S554. In other words, the process in step S554 is repeated until the user makes a positive judgment.

[0048] In step S556, the image processing device 504 calculates the learning loss L shown in Equation 1 above, using the initial mask calculated in step S551 and the ground truth, which is the mask created in step S554, i.e., the mask that the user has affirmed, and updates the parameters of the neural network included in the image processing device 504. In step S557, the user who has viewed the X-ray image performs a baggage check as necessary. This concludes the explanation of Figure 7.

[0049] By using this system for X-ray baggage inspection, the network accuracy can be improved and new objects can be continuously learned without the need for periodic offline learning, thus maintaining high efficiency in the inspection process. Furthermore, compared to the conventional two-step method, the cost required for updating the network in actual applications can be reduced. It is particularly suitable for application scenarios that require human cooperation. Note that each functional unit included in the image processing device 201 may be a software module constituting the image processing application 350 in the computing device 300 shown in Figure 1, or it may be an independent dedicated hardware device. In addition, the above functional units may be implemented in the same computing environment or in a distributed computing environment.

[0050] With the configuration described above, the image processing device 504 processes input images and user interactions received from the X-ray apparatus 501 and external devices such as the user interface 503 via the communication network 502, thereby enabling the prediction of the region of each object.

[0051] According to the first embodiment described above, the following effects and advantages can be obtained. (1) The region extraction system 1 includes an image acquisition unit 11 that acquires captured images, a feature extraction unit 12 that generates a feature map using the captured images, a region extraction unit 13 that extracts candidate frames, which are rectangular frames indicating the regions in the captured images where objects exist, using the feature map, a segmentation head 14 that generates a mask, which is a closed region that follows the outline of the objects in the captured images, using the feature map and candidate frames, a user input unit 21 that acquires input information, which is information about objects in the captured images, input by the user, a region estimation unit 22 that extracts candidate frames using the feature map and input information, and an interaction encoding unit 23 that updates the feature map using the candidate frames extracted by the region estimation unit 22 and the input information. The region estimation unit 22 can further extract candidate frames using the mask generated by the segmentation head. The interaction encoding unit 23 can further update the feature map using the mask generated by the segmentation head. Therefore, the cost of updating the DNN model can be reduced.

[0052] (2) The region estimation unit 22 includes a first convolutional layer which is a trained neural network. The region estimation unit 22 generates multiple frames using the input information and inputs each frame and feature map into the first convolutional layer 22A to extract the frame with the highest evaluation value as a candidate frame. If the region estimation unit 22 further uses a mask calculated by the segmentation head 14, it generates multiple frames so that the region indicated by the mask is included.

[0053] (3) The interaction encoding unit 23 includes a second convolutional layer 23A, which is a trained neural network. The interaction encoding unit 23 creates an interaction map, which is information on the probability of an object's existence at each location, based on the input information. The interaction encoding unit 23 inputs the interaction map and the feature map cropped by the candidate frame into the second convolutional layer 23A to calculate an updated feature map. If a mask is to be used further, the interaction encoding unit 23 inputs the mask further into the second convolutional layer 23A.

[0054] (4) The region extraction system 1 includes a loss calculation unit 30 that updates the parameters of the neural network included in the region estimation unit and the interaction encoding unit. The loss calculation unit 30 obtains the user's pass / fail judgment on the mask output by the segmentation head. If the user rejects the mask, the loss calculation unit 30 operates the region estimation unit 22, the interaction encoding unit 23, and the segmentation head 14 using new input information to update the mask and obtains the user's pass / fail judgment on the updated mask. If the user accepts the mask, the loss calculation unit 30 updates the parameters using the loss function shown in Equation 1. The loss function includes the initial mask, which is the mask initially generated by the segmentation head, and the ground truth, which is the mask that the user has accepted.

[0055] In the embodiment described above, the image processing application 350 was executed by a single computing device 300. However, the image processing application 350 may also be realized by the coordinated operation of multiple computing devices, in which case it can be said that the image processing application 350 is realized by a region extraction system.

[0056] In the embodiments described above, the configuration of the functional blocks is merely an example. Several functional configurations shown as separate functional blocks may be integrated, or a configuration represented in one functional block diagram may be divided into two or more functions. Furthermore, a configuration in which some of the functions of one functional block are provided by other functional blocks may also be used.

[0057] The embodiments and modifications described above may be combined in any way. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that can be conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of Symbols]

[0058] 10...Automatic area partitioning module 11…Image acquisition unit 12...Feature extraction section 13…Region extraction part 14…Segmentation head 20…Interaction Module 21...User input section 22...Region estimation section 22A...First convolutional layer 23…Interaction Encoding Unit 23A...Second convolutional layer 30…Loss calculation section 300...Arithmetic device 306…Terminal Interface 350…Image processing applications

Claims

1. A method for extracting regions that is performed by a computer, The image acquisition step involves obtaining the captured image, A feature extraction step that generates a feature map using the aforementioned captured image, A region extraction step in which candidate frames, which are frames indicating the region in the captured image in which an object exists, are extracted using the feature map; A segmentation step in which a mask is a frame indicating the region in the captured image in which an object exists, using the feature map and the candidate frame, A user input step in which input information, which is information about an object in the captured image, is entered by the user, A region estimation step in which candidate frames are extracted using the feature map and the input information, The interaction encoding step includes updating the feature map using the candidate frames extracted in the region estimation step and the input information, In the region estimation step, the candidate frames can be further extracted using the mask generated in the segmentation step. The interaction encoding step can further update the feature map using the mask generated by the segmentation step, a region extraction method.

2. In the region extraction method described in claim 1, The computer has a first convolutional layer which is a trained neural network, In the region estimation step, multiple frames are generated using the input information, and the frame with the highest evaluation value obtained by inputting each of the frames and the feature map into the first convolutional layer is extracted as a candidate frame. A region extraction method comprising, if the region estimation step further uses the mask, generating the plurality of frames such that they include the region indicated by the mask.

3. In the region extraction method described in claim 1, The aforementioned computer has a second convolutional layer which is a trained neural network, In the interaction encoding step, an interaction map, which is information on the probability of an object's existence at each location, is created based on the input information, and the interaction map and the feature map cropped by the candidate frame are input to the second convolutional layer to calculate the updated feature map. A region extraction method comprising, if the mask is further used in the interaction encoding step, further inputting the mask to the second convolutional layer.

4. In the region extraction method described in claim 1, The method further includes a loss calculation step of updating the parameters of the neural network possessed by the computer, In the loss calculation step, the user's pass / fail judgment for the mask output by the segmentation step is obtained. In the loss calculation step, if the user rejects the mask, the region estimation step, the interaction encoding step, and the segmentation step are performed using the new input information to update the mask, and the user's pass / fail judgment on the updated mask is obtained. The loss calculation unit updates the parameters using the loss function when the user determines that the mask is acceptable. A region extraction method wherein the loss function includes an initial mask, which is the mask initially generated by the segmentation step, and a ground truth, which is the mask that the user has determined to be acceptable.

5. An image acquisition unit that acquires captured images, A feature extraction unit that generates a feature map using the aforementioned captured image, A region extraction unit extracts candidate frames, which are rectangular frames indicating the regions in the captured image where objects exist, using the feature map. A segmentation head that generates a mask which is a closed region following the outline of an object in the captured image, using the feature map and the candidate frame, A user input unit that acquires input information, which is information about an object in the captured image, input by the user, A region estimation unit that extracts candidate frames using the feature map and the input information, The system includes an interaction encoding unit that updates the feature map using the candidate frames extracted by the region estimation unit and the input information, The region estimation unit can further extract candidate frames using the mask generated by the segmentation head. The interaction encoding unit further includes a region extraction system that can update the feature map using the mask generated by the segmentation head.

6. In the region extraction system according to claim 5, The region estimation unit includes a first convolutional layer which is a trained neural network, The region estimation unit generates a plurality of frames using the input information, inputs each of the frames and the feature map into the first convolutional layer, and extracts the frame with the highest evaluation value as the candidate frame. The region estimation unit, when using the mask further, generates the plurality of frames so that the region indicated by the mask is included, a region extraction system.

7. In the region extraction system according to claim 5, The interaction encoding unit includes a second convolutional layer which is a trained neural network. The interaction encoding unit creates an interaction map, which is information on the probability of an object's existence at each location, based on the input information, inputs the interaction map and the feature map cropped by the candidate frame into the second convolutional layer, and calculates an updated feature map. The interaction encoding unit further inputs the mask to the second convolutional layer if the mask is to be used, thereby providing a region extraction system.

8. In the region extraction system according to claim 5, The system further includes a loss calculation unit that updates the parameters of the neural network included in the region estimation unit and the interaction encoding unit, The loss calculation unit obtains the user's pass / fail judgment for the mask output by the segmentation head, If the user rejects the mask, the loss calculation unit operates the region estimation unit, the interaction encoding unit, and the segmentation head using the new input information to update the mask, and obtains the user's pass / fail judgment on the updated mask. The loss calculation unit updates the parameters using the loss function when the user determines that the mask is acceptable. A region extraction system in which the loss function includes an initial mask, which is the mask initially generated by the segmentation head, and a ground truth, which is the mask that the user has determined to be acceptable.

Images