Nuclei classification by avoiding artifact regions

Abstract

A method and system for training a neural network model includes augmenting an original training dataset by applying image artifacts to portions of original images of the original dataset to generate artifact images (306). A target image corresponding to the artifact image is generated by removing labels from the target image at the locations of the artifacts (308). A neural network model is trained (204) using the augmented training dataset and the corresponding target images, the neural network model having a first output identifying an artifact region and another output identifying an object.

Description

[Technical field]

[0001] Related application information This application claims priority to U.S. Non-provisional Patent Application No. 17 / 711,546, filed April 1, 2022, and U.S. Patent Application No. 63 / 170,677, filed April 5, 2021, both of which are incorporated by reference in their entireties. [Background technology]

[0002] Technical Field The present invention relates to the detection and classification of tumor cells, and more particularly to training machine learning models to identify regions of tissue images that contain artifacts.

[0003] 2. Description of Related Art To obtain the tumor cell ratio, an index of cancer aggressiveness, it is necessary to count the number of cells in a tumor area. Depending on the tissue sample, the number of cells can be in the millions. However, image artifacts such as ink spots, blurred areas, tissue folds, and overstained areas can impair cell classification. Summary of the Invention

[0004] A method for training a neural network model includes augmenting an original training dataset to generate an augmented training dataset, and applying image artifacts to portions of original images of the original dataset to generate artifact images. A target image corresponding to the artifact image is generated by removing labels from the target image at locations of the artifacts. A neural network model is trained using the augmented training dataset and the corresponding target image, the neural network model including a first output identifying an artifact region and another output identifying an object.

[0005] A method for analyzing a tissue sample includes analyzing an image of the tissue sample with a neural network model having a first output identifying cells, a second output identifying tumor cells, and a third output identifying artifact regions. In response to determining that a percentage of the tissue sample image within one or more detected artifact regions exceeds a threshold, a correction action is performed to generate a corrected tissue sample image. Analysis of the corrected tissue sample image is performed.

[0006] A system for training a neural network model includes a hardware processor and a memory storing a computer program, which, when executed by the hardware processor, causes the hardware processor to: augment an original training dataset by applying image artifacts to portions of original images of the original dataset to generate artifact images to generate an augmented training dataset, generate target images corresponding to the artifact images by removing labels from the target images at locations of the artifacts, and train a neural network model using the augmented training dataset and the corresponding target images, the neural network model having a first output identifying an artifact region and another output identifying an object.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0008] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] FIG. 1 is a block diagram of a slide analysis system capable of identifying and removing artifact regions in a scanned slide image in accordance with one embodiment of the present invention.

[0010] [Diagram 2] FIG. 1 is a block / flow diagram of a method for diagnosis and treatment based on scanned slide images using analysis capable of identifying and excluding artifact regions in scanned slide images in accordance with one embodiment of the present invention.

[0011] [Diagram 3] FIG. 2 is a block / flow diagram of a method for augmenting a training dataset to include training samples that include artifacts, according to one embodiment of the present invention.

[0012] [Figure 4] FIG. 2 is a block diagram of a neural network model with a different output layer for each function, according to one embodiment of the present invention.

[0013] [Diagram 5] FIG. 2 illustrates analysis of a slide image containing an artifact region according to one embodiment of the present invention.

[0014] [Figure 6] FIG. 1 is a block diagram of a computing device including software for training models and performing diagnosis and treatment using artifact detection in slide images, according to one embodiment of the present invention.

[0015] [Figure 7] FIG. 2 is a diagram of a neural network architecture that can be used to implement portions of the model, according to one embodiment of the present invention.

[0016] [Figure 8] FIG. 1 is a diagram of a deep neural network architecture that can be used to implement portions of the model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Normal and tumor cells can be automatically counted in a given area of ​​a tissue sample to determine the tumor cell ratio (TCR). High magnification images are used to show details of individual cells, while lower magnification images reveal how cells are arranged in larger structures such as glands. Cancer cells can be contrasted to healthy cells by individual cell features, cell arrangement, or both.

[0018] However, there may be areas where TCR calculation should be avoided, for example, if the slide contains areas with artifacts. If such artifact areas occupy a large part of the slide area, the slide may need to be rescanned or replaced. Such areas can be detected using machine learning models trained to identify artifacts along with cell identification and classification capabilities. This is an indicator of slide quality and can be used to determine the reliability of the slide analysis results.

[0019] A machine learning model can include a convolutional neural network with multiple output layers that perform different classification tasks. For example, three different output layers can be used: one to identify the location of all nuclei, one to specifically identify tumor nuclei, and one to identify artifact regions. To train the machine learning model in a way that does not allow artifact detection to overwhelm the other output layers, a loss function can be used, for example as a weighted sum of the losses of the individual output layers.

[0020] The machine learning model can be trained with a dataset that has been augmented to include artifact information. For example, starting with an original training dataset that includes images with labeled cells and tumor cells, one can introduce augmented training samples that add artifacts to the original training samples. This can be done, for example, by adding blurring or color shifting to the original training samples. To ensure that the machine learning model correctly learns the boundaries between artifact and normal regions, the corrections can be based on a randomly generated mask rather than being performed on the entire original training samples. Nuclear labels can be removed from artifact regions of the augmented samples to prevent nuclei from being detected in those regions.

[0021] Referring now in detail to the figures, where like numerals represent the same or similar elements, and initially to Figure 1, there is shown a system for analyzing tissue samples and providing a TCR report. A tissue sample 102 is obtained and placed on a slide. The slide is scanned with a whole slide scanner 104 and a slide image is generated and stored in a slide database 106.

[0022] Slide analysis 108 may divide each image into processing tiles, which may be performed according to a regular grid and / or may be determined according to user instructions of interesting sections of the slide image. Slide analysis 108 may generate a report 110 that characterizes information obtained from slide 102, including, for example, TCR, location of cancer cells, etc. This report 110 may be used by medical professionals to aid in patient diagnosis, identify the type and extent of cancer, identify areas of tissue that can be genetically tested, identify treatment options, etc.

[0023] As described herein, slide analysis 108 may include identifying portions of the slide image that contain artifacts such as ink areas, blurry areas, tissue folds, or areas of over-staining. These areas should be excluded from the report 110 determination and may prompt a rescan or replacement of the tissue sample 102 if the artifact areas occupy a threshold amount or more of the slide image.

[0024] Referring now to FIG. 2, a method for performing analysis of tissue samples is shown. Two phases are shown, including creation of a multi-classification model in block 201, and diagnosis and treatment in block 205. The two phases may occur at different times and in different locations. For example, creation of a multi-classification model in block 201 may be performed far in advance, with a trained model distributed for use in hospitals and research centers. Diagnosis and treatment in block 205 may be performed using a pre-trained model based on a recently obtained tissue sample from a patient. It is worth noting that each of these phases may be performed independently by different entities.

[0025] The creation of the multi-classification model in block 201 begins with extending the training dataset in block 202. This extension, which is described in more detail below, includes processing images from the existing training dataset to include versions of the images that contain artificially generated artifacts. Block 204 then uses the extended dataset to train a multi-output slide classifier. Images from the training dataset may be divided into image tiles since an entire slide may be too large to process at once. Each tile is then input to the model, which performs this processing on multiple tiles in parallel. The classifier outputs for different tiles may be compiled into a report for the entire slide. The different outputs of the classifier may generate, for example, cell identification information, tumor cell classification information, and artifact region information. The cell and tumor cell identification information may include the location of each cell and each tumor cell, respectively, within a given slide image. The artifact region information may include, for example, a boundary identifying one or more regions of the slide image that contain artifacts, and may further include statistical information such as the percentage of the slide image occupied by artifact regions.

[0026] Diagnosis and treatment 205 includes scanning a tissue sample to generate a new slide image 206. The tissue sample may be obtained by any suitable means and scanned with a whole slide scanner 104. Block 208 uses the slide image as input to a trained multi-classification model to generate information about any cells, any tumor cells, and any artifact regions depicted in the slide image. The artifact region information may include, for example, for each pixel of the slide image, a value indicating the likelihood that the pixel belongs to an artifact. The artifact region is defined by applying a threshold, and pixels with a sufficiently high value are added to the artifact region.

[0027] Block 209 performs an analysis on the classified image, for example to determine a TCR value. Analysis 209 may exclude artifact regions from consideration so that information about cells or tumor cells in those regions does not influence the analysis. Block 210 performs treatment, for example, requesting further testing, such as genetic testing, administering chemotherapy, or performing surgery, based on the output of the multi-classification model.

[0028] 3, a method for augmenting a training dataset is shown. Block 302 copies a training image from the original training dataset. Block 304 generates a mask that defines artifact regions in the copied image. The mask can identify random portions of the image in one or more discrete regions.

[0029] Block 306 applies artifacts to the masked regions of the copy image. For example, applying the artifacts may include convolving a Gaussian kernel with the image to add a synthetic blur, and ink may be simulated by applying a combination of blur and color shift. The target map for each artifact may be understood to be equivalent to a label in a fully convolutional model and may be used to calculate a loss. The loss may be, for example, a mean squared error obtained by averaging the square of the difference for each pixel between the output and the target. Block 308 then removes the labels of nuclei in the masked regions, so that nuclei are not detected in the artifacted regions.

[0030] Referring now to Figure 4, a block diagram of a neural network model is shown. The model may be based on any suitable fully convolutional network that receives an input of any suitable size and produces one or more output maps of the same size. Depending on the information to be extracted from the input image, the same fully convolutional network 402 can be used for different output layers. Thus, the model can include a whole cell output layer 404, a tumor cell output layer 406, and an artifact region output layer 408.

[0031] When training a model, we can optimize a loss function that weights each output. For example, the loss function can be expressed as: L totai = aL(output 1, target 1) + bL(output 2, target 2) + cL(output 3, target 3) where the function L(·,·) is a loss function such as mean squared error or binary cross entropy, and a, b, and c are weights given to the whole-cell task, the tumor cell task, and the artifact segmentation task. In each case, the output and target values ​​reflect the output of each task and their respective expectations.

[0032] 5, an exemplary slide image 500 is shown after identifying artifact regions in block 208. Slide image 500 includes a scanned tissue sample 502. In this example, a number of tiles 504 are identified within the image by applying a grid over the scanned tissue image 502, with pixels that fall within each grid making up each tile. Also shown is a set of markers 506 provided by a human operator to constrain the tiles such that tiles 504 are only determined within the boundaries established by the markers 506. In some examples, TCR analysis and other analyses may be limited to the marked regions.

[0033] Additionally, an artifact region 508 is shown. No cellular information is considered in the artifact region independent of the region marked by the human operator. Thus, if the marked region includes part or all of the artifact region, the cells present within the artifact region are not counted in the analysis.

[0034] 6, an exemplary computing device 600 is shown, in accordance with an embodiment of the present invention. The computing device 600 is configured to perform classifier enhancement.

[0035] Computing device 600 may be embodied as any type of computing or computer device capable of performing the functions described herein, such as, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 600 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically separated computing device.

[0036] As shown in FIG. 6, computing device 600 illustratively includes a processor 610, an input / output subsystem 620, a memory 630, a data storage device 640, and a communication subsystem 650, and / or other components and devices typically found in a server or similar computing device. Computing device 600 may, in other embodiments, include other or additional components such as typically found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the illustrative components may be incorporated in or otherwise form part of another component. For example, memory 630, or portions thereof, may be incorporated in processor 610 in some embodiments.

[0037] The processor 610 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0038] Memory 630 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, memory 630 may store various data and software used during operation of computing device 600, such as an operating system, applications, programs, libraries, and drivers. Memory 630 is communicatively coupled to processor 610 via I / O subsystem 620 and may be embodied as circuits and / or components to facilitate input / output operations with processor 610, memory 630, and other components of computing device 600. For example, I / O subsystem 620 may be embodied as or may otherwise include a memory controller hub, an input / output control hub, a platform controller hub, an integrated control circuit, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 620 may form part of a system on a chip (SOC) and be integrated with the processor 610, memory 630, and other components of the computing device 600 on a single integrated circuit chip.

[0039] The data storage device 640 can be embodied as any type of device or device configured for short-term or long-term storage of data, such as, for example, a memory device and circuit, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 640 can store program code 640A for training a mutation policy model and program code 640B for mutating a peptide sequence according to the mutation policy model. The communication subsystem 650 of the computing device 600 can be embodied as any network interface controller or other communication circuit, device, or collection thereof that can enable communication between the computing device 600 and other remote devices over a network. The communication subsystem 650 can be configured to achieve such communication using any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0040] As shown, computing device 600 may also include one or more peripheral devices 660. Peripheral devices 660 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 660 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0041] Of course, computing device 600 may include other elements (not shown) or omit certain elements, as would be readily appreciated by one of ordinary skill in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 600 depending on the particular implementation of the same, as would be readily appreciated by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Additionally, additional processors, controllers, memories, and the like may be utilized in various configurations. These and other variations of processing system 600 will be readily contemplated by one of ordinary skill in the art in view of the teachings of the present invention provided herein.

[0042] Reference is now made to Figures 7 and 8. Referring to Figures 7 and 8, exemplary neural network architectures are shown, which can be used to implement portions of the present model. A neural network is a generalized system whose capabilities and accuracy improve with exposure to additional empirical data. A neural network learns by exposure to empirical data. During training, the neural network stores and adjusts a number of weights that are applied to the input empirical data. By applying the adjusted weights to the data, the neural network can identify that the data belongs to a particular predefined class from a set of classes, and output the probability that the input data belongs to each class.

[0043] Empirical data (also called training data) from a set of examples is formatted as a string of values ​​and fed to the input of the neural network. Each example is associated with a known outcome or output. Each column is represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple distinct values. The network can have one input node for each value that makes up the input data of the examples, and separate weights can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0044] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​of examples and adjusting stored weights to minimize the difference between the output values ​​and the known values. Adjustments can be made to the stored weights through backpropagation, where the influence of the weights on the output values ​​is determined by calculating the mathematical gradient and adjusting the weights in a way that shifts the output to the minimum difference. This optimization, called gradient descent, is a non-limiting example of how training is done. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of the neural network.

[0045] In operation, the trained neural network can be used on new data that was not previously used for training or validation, through generalization. The adjusted neural network weights can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0046] In a layered neural network, the nodes are arranged in layers. An exemplary simple neural network has an input layer 720 of source nodes 722 and a single computational layer 730 with one or more computational nodes 732 that also function as output nodes, with a single computational node 732 for each possible category into which an input example may be classified. The input layer 720 may have a number of source nodes 722 equal to the number of data values ​​712 in the input data 710. The data values ​​712 in the input data 710 may be represented as column vectors. Each computational node 732 in the computational layer 730 generates a linear combination of weighted values ​​from the input data 710 fed to the input node 720 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network may perform classification on linearly separable examples (e.g., patterns).

[0047] A deep neural network, such as a multi-layer perceptron, may have an input layer 720 of source nodes 722, one or more computational layers 730 with one or more computational nodes 732, and an output layer 740 with one output node 742 for each category into which an input example may be classified. The input layer 720 may have a number of source nodes 722 equal to the number of data values ​​712 in the input data 710. The computational nodes 732 in the computational layer 730 are also referred to as hidden layers because they are between the source nodes 722 and the output nodes 742 and are not directly observed. Each node 732, 742 in the computational layer generates a weighted linear combination of values ​​from the values ​​output from the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-i ,w n The output layer provides the network's overall response to the input data. Deep neural networks may be fully connected, where each node in a computational layer is connected to every node in the previous layer, or the connections between layers may have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0048] Training a deep neural network involves two phases: a forward phase, in which the weights of each node are fixed and the input is propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0049] Computational nodes 732 in one or more computational (hidden) layers 730 perform nonlinear transformations on the input data 712 that generate a feature space. Classes or categories may be more easily separated in the feature space than in the original data space.

[0050] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements, in a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0051] The embodiments may include a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may include a computer readable storage medium such as a semiconductor or solid state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk.

[0052] Each computer program may be tangibly stored on a machine-readable storage medium or device (e.g., a program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention may also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0053] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0054] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0055] As employed herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, an image processing unit, and / or a separate processor or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In some embodiments, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0056] In one embodiment, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code to achieve a particular result.

[0057] In other embodiments, the hardware processor subsystem may include dedicated, dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0058] These and other variations of hardware processor subsystems are also contemplated in accordance with embodiments of the present invention.

[0059] In the specification, references to "one embodiment" or "one embodiment" of the invention, as well as other variations, mean that the particular features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment of the invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment" in various places in this specification, as well as any other variations, do not necessarily all refer to the same embodiment. However, it will be understood that features of one or more embodiments can be combined in view of the teachings of the invention provided herein.

[0060] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one of," such as "A and / or B," "at least one of A and B," will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as there are listed items.

[0061] The foregoing is understood in all respects to be illustrative and illustrative, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Those skilled in the art could make various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is as set forth in the appended claims.

Claims

1. 1. A computer-implemented method for training a neural network model, comprising: augmenting 306 the original training dataset by applying image artifacts to portions of original images of the original dataset to generate images of the artifacts to generate an augmented training dataset; generating a target image corresponding to an image of the artifact by removing labels of objects in the mask from a target image representing the image artifact at the location of the artifact (308); and training (204) a neural network model using the augmented training data set and the corresponding target images, the neural network model including a first output that identifies an artifact region and another output that identifies an object.

2. The method of claim 1 , wherein applying the image artifact comprises determining a mask that identifies an area of ​​the original image to which the image artifact is applied.

3. The method of claim 2 , wherein applying the image artifact further comprises convolving the image of the identified region with a Gaussian kernel to blur it.

4. The method of claim 3 , wherein applying the image artifact further comprises applying a color shift in the identified region to simulate ink.

5. The method of claim 2 , wherein determining the mask comprises generating a random number of regions, each of the random number of regions having a randomly generated shape.

6. 2. The method of claim 1 , wherein training the neural network model comprises minimizing a loss function comprising a weighted sum of a loss term from the first output and each loss term from the other outputs that identify an object.

7. The method of claim 1 , wherein the neural network model is a fully convolutional neural network model having an output map.

8. 8. The method of claim 7, wherein the first output of a fully convolutional neural network model is a map.

9. 1. A system for training a neural network model, comprising: A hardware processor (610); A memory (630) for storing a computer program; The computer program causes the hardware processor to: augmenting the original training dataset by applying image artifacts to portions of original images of the original dataset to generate images of the artifacts (306) to generate an augmented training dataset; generating a target image corresponding to an image of the image artifact by removing labels of objects in the mask from a target image representing the image artifact at the location of the artifact (308); and training (204) a neural network model using the augmented training data set and the corresponding target images, the neural network model including a first output that identifies an artifact region and another output that identifies an object.

10. 10. The system of claim 9, wherein the computer program further causes the hardware processor to perform the step of determining a mask that identifies an area of ​​the original image to which the image artifact is applied.

11. The system of claim 10 , wherein the computer program further causes the hardware processor to execute a procedure of convolving the image of the identified region with a Gaussian kernel to blur the image.

12. The system of claim 11 , wherein the computer program further causes the hardware processor to perform the steps of: applying a color shift in the identified region to simulate ink.

13. 11. The system of claim 10, wherein the computer program further causes the hardware processor to execute a procedure for generating a random number of regions for the mask, each of the random number of regions having a randomly generated shape.

14. 10. The system of claim 9, wherein the computer program further causes the hardware processor to execute a procedure of minimizing a loss function comprising a weighted sum of a loss term from the first output and each loss term from the other outputs that identify an object.

15. 10. The system of claim 9, wherein the neural network model is a fully convolutional neural network model having an output map.

16. 10. The system of claim 9, wherein the first output of a fully convolutional neural network model is a map.

Images