Systems, devices, and methods for recognizing defects in medical implant procedures

Abstract

A system and method for identifying material components of an implant tissue product includes an image capture device for acquiring image data of the implant tissue product and a processor for processing the image data with an artificial neural network that identifies and classifies materials of the implant tissue product from the image data. The image capture device can include an optical filter and an ultraviolet light source for ultraviolet fluorescent imaging of the implant tissue product. Using the captured image data, the artificial neural network can identify unwanted materials of the implant tissue product, such as fascia or flesh on the skin of a fish, for subsequent removal.

Description

[Technical Field]

[0001] The present disclosure relates to systems and methods for identifying material components in the processing of scaffolding materials or implant tissue products, particularly using fish skin, for wound care and / or other tissue healing applications. [Background technology]

[0002] A variety of human, animal, and synthetic materials are currently described or used in medical procedures to augment, repair, or correct tissue defects. The extracellular matrix (ECM) of vertebrates is a complex structural entity that surrounds and supports cells and has been found to provide a particularly advantageous scaffolding material for skin graft tissue products. The ECM is composed of a complex mixture of structural proteins, of which collagen is the most abundant, as well as other specialized proteins and proteoglycans.

[0003] Fish skin can be processed into scaffold materials, including acellular scaffolds, that retain most of the natural biological ECM components. To be most effective, the native three-dimensional structure, composition, and function of the dermal ECM remain essentially unchanged, providing a scaffold to support cell migration, adhesion, proliferation, and differentiation, thus facilitating tissue repair and / or replacement. These products must be properly fabricated as medical-grade skin substitutes through various processing steps before use to prevent adverse reactions, including infection and allergic reactions.

[0004] A key part of this process is the thorough removal of certain unwanted materials from the inside of the dried skin, i.e., the fascia and any remaining flesh. While the fascia may be removed primarily for aesthetic reasons, it is important to remove all remaining flesh due to the potential for allergic reactions to fish proteins in patients. The removal process is typically performed manually, with skilled technicians using sharp tools to carefully scrape the fascia and remaining flesh from the skin. This process must be performed precisely to ensure that the unwanted material is removed without damaging the ECM of the fish skin, requiring technicians to rely on visual inspection of the skin to estimate the appropriate degree of scraping required. Such work is time-consuming, expensive, physically demanding, tedious, and prone to human error in locating and identifying the unwanted material on the fish skin. Furthermore, when skilled technicians remove unwanted material, the physicality and length of the removal process often fatigue the technician, which can significantly reduce the accuracy and consistency of the removal process.

[0005] The inventors of the present disclosure have determined that the significant need to properly identify and classify unwanted material on fish skin remains a significant obstacle to improving quality control, automating processing, and scaling up manufacturing processes. The high cost of manual evaluation in the manufacturing process can limit the availability of implant tissue products formed from fish skin or otherwise make such products unaffordable to a larger population. Furthermore, the manual nature of such processing inevitably results in inaccuracies. Due to stringent medical safety requirements that increasingly demand a level of precision and consistency achievable only by automated systems rather than human evaluation, the inventors have determined that there is an increasingly important need for systems and methods for automatically and accurately identifying and distinguishing components of fish skin.

[0006] Unfortunately, the difficulty of this task is compounded by the fact that the waste material, while having a different texture, often looks similar to the skin of the accompanying fish and shares the same milky white color. These obstacles make any assessment by existing image recognition approaches highly questionable. Thus, using visual analysis to distinguish thin sections of white fascia and thin sections of white meat residue from a thin layer of white clean skin remains a complex and difficult challenge. Currently, no improved systems and methods exist to prevent errors in this process, and quality control remains an unsolved problem and must rely on human visual inspection.

[0007] Thus, the identification, classification, and removal of unwanted material from fish skin remains a matter of subjective and sometimes arbitrary intuition, preference, and user experience, rather than being performed with quantitative precision. Thus, the creation of implant tissue products using fish skin can be subject to many errors and inefficiencies. Summary of the Invention [Means for solving the problem]

[0008] A system for identifying and differentiating material components of an implant tissue product is provided, comprising an image capture device configured to acquire image data of the implant tissue product, and a processor configured to process the image data using an artificial neural network, the artificial neural network configured to localize and classify the material of the implant tissue product from the image data.

[0009] Also provided is a method for identifying and distinguishing material components of an implant tissue product, the method including capturing an image of the implant tissue product using an image capture device and using a processor to process the image data using an artificial neural network to localize and classify the material of the implant tissue product from the image of the implant tissue product.

[0010] Also provided is a non-transitory hardware storage device having stored thereon computer-executable instructions that, when executed by one or more processors of a computer, configure the computer to capture an image of the implant tissue product with an image capture device and to use the processor to process the image data using an artificial neural network to localize and classify the material of the implant tissue product from the image of the implant tissue product.

[0011] Preferred embodiments of systems and methods for identifying material components of implant tissue products advantageously utilize a novel architecture that leverages ultraviolet fluorescence imaging in embodiments utilizing material fluorescence in the visible spectrum for image capture, in combination with a convolutional neural network specifically adapted for identifying and classifying material components of implant tissue products. The systems and methods of these and other disclosed embodiments can develop a comprehensive feature map of material components on fish skin to provide quality control in the removal of unwanted materials from implant tissue products. Using the comprehensive feature map in conjunction with automated processing tools, the systems and methods can automatically remove unwanted materials from implant tissue products. [Brief explanation of the drawings]

[0012] These and other features, aspects, and advantages of the present disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings.

[0013] [Figure 1A] 1 is a diagram of a system for identifying material components of an implant tissue product according to an embodiment of the present disclosure. [Figure 1B] FIG. 10 is a diagram of a system for identifying material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 1C]FIG. 10 is a diagram of a system for identifying material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 2A] 10 is an image tile of an implant tissue product captured by an image capture device according to an embodiment of the present disclosure. [Figure 2B] 10 is another image tile of an implant tissue product captured by an image capture device according to another embodiment of the present disclosure. [Figure 3] 1 is a flow diagram of a method for identifying material components of an implant tissue product according to an embodiment of the present disclosure. [Figure 4] 4A-4C are diagrams of contracting paths of an artificial neural network according to the embodiment of FIG. 3. [Figure 5] FIG. 4 is a diagram of the final output step of the artificial neural network according to the embodiment of FIG. 3. [Figure 6] FIG. 4 is a diagram of an expanding path of an artificial neural network according to the embodiment of FIG. 3. [Figure 7A] FIG. 7A includes one of a plurality of image tiles, an overlay image, and a classified image according to the embodiment of FIG. [Figure 7B] FIG. 7B includes another one of a plurality of image tiles, an overlay image, and a classified image according to the embodiment of FIG. [Figure 8] FIG. 1 is a diagram of a system for identifying material components of an implant tissue product according to an embodiment of the present disclosure, including a computing device. [Figure 9A] FIG. 2 is a diagram of a convolution operation according to an embodiment of the present disclosure. [Figure 9B] FIG. 10 is a diagram of another convolution operation according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a diagram of symmetric pathways of an artificial neural network according to an embodiment of the present disclosure. [Figure 11]FIG. 10 is a diagram of a system for identifying and removing material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 12] 1 is a flow diagram of a method for identifying and removing material components of an implant tissue product according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] overview

[0015] Various embodiments of the present disclosure can be better understood from the following description read in conjunction with the accompanying drawings, in which like reference characters refer to like elements and in which:

[0016] While the disclosure is susceptible to various modifications and alternative constructions, specific illustrative embodiments have been shown in the drawings and are described below. It should be understood, however, that it is not intended to limit the disclosure to the particular embodiments disclosed, but rather, the intention is to cover all modifications, alternative constructions, combinations, and equivalents falling within the spirit and scope of the disclosure.

[0017] Unless a term is expressly defined in this application to have an explained meaning, it is understood that no intention is made to explicitly or implicitly limit the meaning of such term beyond its ordinary or usual meaning.

[0018] For purposes of this application, in a preferred embodiment, the term "graft tissue product" includes "fish skin," "fish skin," "acellular fish skin," or similar, including Kerecis™ Omega3 Wound by Kerecis, Kerecis™ Omega3 acellular fish skin from Atlantic cod (Gadus morhua), and any other or similar fish skin graft tissues. These graft tissue products undergo processing that maintains biological structure and bioactive compounds, including omega-3 polyunsaturated fatty acids (PUFAs), but removes allergenic and other unwanted components, and specifically removes tissues that may provoke an immune response in the recipient. The thickness of the fish skin used to make the graft tissue product can vary, as can the dimensions of the overlying unwanted material, such as fascia, flesh, or scales. The fish skin can have a thickness of approximately 0.35 mm to 2.25 mm, while the thickness of the overlying unwanted material can range from approximately 0.15 mm to 1.75 mm. The creation of graft tissue products from fish skin may require the removal of substantially all fascia, flesh, or other undesirable material from the skin or tissue to be used as the graft tissue product. While described as a preferred embodiment, the graft tissue product can include any other captive fish species used for skin graft or skin substitute products, such as, but not limited to, Atlantic cod (Gadus morhua), tilapia, including Nile tilapia (Oreochromis niloticus), or other fish species. In other embodiments, the "graft tissue product" can be made from harvested mammalian skin, including, but not limited to, porcine skin graft tissue products, or from sources other than fish, including human allografts or autografts, or cadaver skin graft tissue products. The graft tissue product can ultimately be made as an acellular or decellularized graft tissue product, or it can be a cellular skin graft tissue product (i.e., a graft tissue product in which skin cells remain viable or intact).Additionally, transplant tissue products can include non-skin tissues, such as placenta transplant tissues, used as skin graft tissue products, where the tissue to be used as the transplant tissue product is to have unwanted secondary tissues or portions removed. Additionally, transplant tissue products can include biological, synthetic, or hybrid skin substitutes, where the transplant tissue product is inspected for unwanted tissues, portions, or materials to be removed before use as a transplant tissue product. Finally, transplant tissue products can include autograft, allograft, or xenograft products, and are not limited to skin graft tissue products used in human patients, but also include autograft, allograft, or xenograft products used or made to be used in other non-human species, including horses, cows, monkeys, rabbits, mice, rats, guinea pigs, or other mammalian or non-mammalian species to which the skin graft tissue product is applied.

[0019] As described herein, a "convolutional neural network" refers to an artificial neural network for performing analysis of images, etc., based on a weight-sharing architecture of convolutional kernels configured for pixel-by-pixel evaluation of the image. A convolutional neural network is capable of adjusting and improving the kernel weights and biases through automated learning or training, for example, using a training dataset. In embodiments, a convolutional neural network can include a contracting path or encoder followed by a symmetric expanding path or decoder, making the network an end-to-end fully convolutional neural network that includes convolutional layers and no dense layers.

[0020] Various embodiments and components used therewith

[0021] As discussed above, it has been found that, in general, when preparing transplant tissue products from, for example, fish skin, certain materials, such as fascia and flesh, can cause undesirable reactions or results, such as immunological reactions and results, or otherwise adversely affect the aesthetics or quality of the product. For example, in the case of fish skin, portions or layers of fascia and flesh on fish skin can be extremely difficult to reliably distinguish from the acellular matrix of fish skin due to the thin thicknesses involved and the milky white color these components share. Due to the challenges of accurately classifying and localizing these different material components using existing imaging approaches, evaluation of the components of the transplant tissue product, such as during fish skin scraping or after preparation of the transplant tissue product, is often manual, expensive, and inaccurate due to the poor suitability of fish skin for accurately distinguishing between visually similar material components.

[0022] Furthermore, manual assessment of fish skin does not provide usable information regarding the type, amount, and arrangement of material on fish skin in an accurate, reproducible, and rapid manner.

[0023] In view of the above, there is a need for a system and method for identifying material components of implant tissue products that addresses the problems and shortcomings of existing approaches to identifying, assessing, and determining the location, quantity, and arrangement of material components of implant tissue products, including the limitations of costly and time-consuming manual material component identification, as well as existing 2D image recognition approaches. The inventors have recognized the need for a material identification system that provides improved accuracy, speed, and consistency in distinguishing between material components of implant tissue products to provide insight that is accessible and quantifiable to experts or automated processing systems.

[0024] Embodiments of systems and methods for identifying material components of implant tissue products according to the present disclosure advantageously overcome deficiencies in existing approaches to distinguishing, identifying, and localizing material components of implant tissue products, such as fascia and flesh on fish skin, which have limited accuracy and require significant time and effort to implement.

[0025] In embodiments of the present systems and methods, ultraviolet fluorescence imaging is synergistically combined with an improved neural network architecture to classify and localize material components of an implant tissue product, preferably fish skin. The ultraviolet fluorescence imaging can be provided using an ultraviolet light source configured to illuminate the implant tissue product and an image capture device configured to capture, store, and process RGB (i.e., red, green, blue) images of the fluorescence induced from the material components in the visible spectrum. In various embodiments, the ultraviolet light source can be configured to irradiate the implant tissue product with light having a wavelength of approximately 365 nm to 395 nm. In one aspect, the ultraviolet light source can have a fixed center band of 395 nm, e.g., a light-emitting diode with a fixed center band of 395 nm. Preferably, the wavelength of the ultraviolet light source is configured to irradiate the implant tissue product without substantially heating the implant tissue product. The image capture device can be integrated with a sensor device such as a camera or similar device, and the image data can be processed using an application on a computing device configured to receive, store, process, and / or transmit the captured images, which can then be evaluated to classify and localize material components of the implant tissue product.

[0026] A system and method for identifying material components of a transplant tissue product leverages RGB images of induced fluorescence from fish skin in combination with an artificial neural network to segment, map, and identify material components within spaces, such as fascia and flesh on the skin of the fish. The system and method include an image capture device configured to capture an image, e.g., an RGB image, and a processor configured to process the image data using an artificial neural network, preferably a convolutional neural network, to localize and classify the material of the transplant tissue product from the image data. The captured image may be an RGB image or a true color image, or any other suitable type of color image capturing ultraviolet-induced fluorescence of the fish skin, as will be understood by those skilled in the art from this disclosure. The system and method alleviate the need for manual visual assessment of the material components of the fish skin.

[0027] The image capture device can incorporate an optical filter that filters incident light to form a captured image. Alternatively, the optical filter is separate from (not integrated with) the image capture device but positioned so that incident light passes through the optical filter before entering the image capture device. In various embodiments, the optical filter can include a long-pass filter that reflects short wavelengths while transmitting long wavelengths. In one aspect, the cut-on wavelength of the long-pass filter can be in the range of approximately 375 nm to 675 nm, 400 nm to 600 nm, 400 nm to 500 nm, 425 nm to 625 nm, or for some embodiments, approximately 435 nm. The optical filter can have a transmittance of approximately 85% at the selected wavelength. Because images captured without a physical optical filter would be saturated with light to the point that differentiation between material components of the implant tissue product would be impractical, the optical filter may necessarily be present as a hardware component or a physical filter on the image capture device. Additionally, optical filtering or additional optical filtering may be provided through digital processing of image data obtained from the image capture device.

[0028] Embodiments of the present systems and methods can provide pixel-by-pixel evaluation of the captured images to mark and classify the physical material at each location of the implant tissue product. In various embodiments, each pixel is approximately 200-800 μm. 2 The size is about 300 to 600 μm. 2 or for some embodiments, about 400 μm 2 Alternatively, the imaging system may be configured to have at least 200 pixels / mm 2 pixel resolutions of at least 400 pixels / mm are available, more particularly at least 400 pixels / mm 2 , at least 600 pixels / mm 2, at least 800 pixels / mm 2 , at least 1000 pixels / mm 2 , at least 1500 pixels / mm 2 , or at least 2000 pixels / mm 2 Advantageously, pixel-by-pixel evaluation of captured images in accordance with the disclosed systems and embodiments allows for the detection of even very small amounts of unwanted material, ensuring higher quality and greater accuracy of the resulting implant tissue product.

[0029] The systems and methods can include dividing a captured image into multiple image tiles for input to an artificial neural network. The image tiles can be of the same size, e.g., 512 x 512 pixels, and the captured image can be resized as needed to allow for the division of the image tiles. Preferably, the multiple image tiles include at least 20 distinct tiles, at least 25 distinct tiles, at least 30 distinct tiles, at least 35 distinct tiles, or for some embodiments, 35 distinct tiles. Image tiles can also be used to train and retrain a learning artificial neural network, such as by enabling the convolutional neural network to adjust the kernel, kernel bias, or kernel weights in each convolutional layer based on feedback from a training dataset.

[0030] In a first aspect, the present system and method provide a stepped contracting path or a first phase of an encoder for contextualizing each image tile, the stepped contracting path including multiple contracting steps. Each step of the stepped contracting path may include a first convolutional layer configured to filter each pixel of the captured image to form a feature map for input to additional layers, followed by a first rectification layer that determines whether a feature is present for each pixel on the feature map from the first convolutional layer. Each step may further include a second convolutional layer followed by a second rectification layer, the first rectification layer providing input to the second convolutional layer. Each of the multiple steps may also include storing a contracted output or contracted feature map resulting from the second rectification layer, such as a feature map resulting from a material component of an implant tissue product. A pooling layer can be provided at each step of the stepwise shrinkage path to select salient features of the stored shrinkage feature map as input for subsequent layers and / or steps.

[0031] According to various embodiments, the aforementioned rectification layer of the artificial neural network can comprise a nonlinear rectification layer. The nonlinear rectification layer can, for example, comprise a threshold operation that returns zero for values ​​less than zero and returns the input value directly for values ​​greater than zero. Thus, the nonlinear rectification layer can comprise an activation function that takes the value calculated from the immediately preceding convolutional layer and transforms it into an output. Preferably, the rectification layer can comprise a nonlinear rectification (ReLu) type activation.

[0032] In a second aspect, the present system and method provide a symmetric stepped expanding path or second decoder phase for localizing features in a captured image, the stepped expanding path including multiple expanding steps. Each step of the symmetric stepped expanding path includes a first convolutional layer followed by a first rectification layer and a second convolutional layer followed by a second rectification layer, where the first rectification layer provides input to the second convolutional layer. Each step may further include an upsampling layer following the second rectification layer, where the upsampling layer forms an upsampled feature map. Each step of the symmetric stepped expanding path may include a concatenation or stacking operation between the upsampled feature map and a stored contracted feature map. The upsampled feature map and the stored contracted feature map selected for the concatenation operation may be selected based on having a common or the same size.

[0033] In a third aspect, the present systems and methods provide a third phase of an output operation or step that includes a first convolutional layer followed by a first rectification layer and a second convolutional layer followed by a second rectification layer, the output operation further including a sigmoid layer following the second rectification layer.

[0034] The image capture device and / or computing device may be configured to locally perform the above steps and other steps described herein. The computing device may include storage, a processor, a power source, and an interface. The instructions in the storage may be executed by the processor to capture images and utilize one or more neural networks as described herein to classify and identify material components of the implant tissue product. While in embodiments the above steps are performed locally, it will be appreciated that the captured images are transmitted to a remote server and one or more steps described herein are performed via cloud computing at a processor located on the remote server that is configured to classify and identify material components of the implant tissue product.

[0035] 1A is a schematic diagram of a system 100 for identifying material components of an implant tissue product 110 in what may be considered its most basic form, according to an embodiment of the present disclosure. The system 100 includes an ultraviolet light source 120 configured to illuminate the implant tissue product 110 and an image capture device 130 configured to capture an image of the implant tissue product 110. The captured image includes induced fluorescence of the implant tissue product 110. The image capture device 130 can include an optical filter 132 configured to filter incident light below a predetermined wavelength from entering the image capture device.

[0036] 1B is a schematic diagram of another embodiment of a system 150 for identifying material components of an implant tissue product 110. The system 150 includes a first ultraviolet light source 121, a second ultraviolet light source 123, and an image capture device 130 configured to capture an image of the implant tissue product 110, with each ultraviolet light source 121, 123 configured to illuminate the implant tissue product 110. In one embodiment, the image capture device 130 may include a Sony IMX219 8-megapixel sensor or equivalent imaging device. Although an 8-megapixel sensor can be used, higher resolution can be obtained using a larger megapixel sensor, such as a 12-megapixel sensor, a 20-megapixel sensor, a 30-megapixel sensor, or a 40-megapixel sensor. The image capture device 130 may be configured to capture still and / or video images of the implant tissue product 110 as it is transported by the conveyor 160 on which it is placed directly transverse to the imaging direction of the image capture device 130. In certain embodiments, the implant tissue product 110 may be secured in place or on the conveyor 160 by restraining elements such as clips, rods, suction devices, and / or transparent coverings such as plexiglass sheets 111.

[0037] The conveyor 160 may be a conveyor belt and is controlled by a conveyor controller 165, which may include a conveyor drive system. The conveyor controller 165 can change the speed and direction of the conveyor 160. In addition, the conveyor controller is controlled by a computing device 190. The captured image includes the induced fluorescence of the implant tissue product 110. The image capture device 130 or system 150 may include an optical filter 132 configured to filter incident light below a predetermined wavelength from entering the image capture device. The system 150 includes a computer or computing device 140, which is described in further detail herein (e.g., in the following embodiment shown in FIG. 8).

[0038] The computing device 190 receives image data, e.g., in the form of still image data or video image data, from the image capture device. The computing device may also control the first ultraviolet light source 121 and the second ultraviolet light source 123. Although hardwired connections between the computing device 190, the first and second ultraviolet light sources 121, 123, the image capture device 130, and the conveyor control device 165 may be included in various embodiments, data between these components may be transmitted via wireless communication, such as Bluetooth®. Furthermore, while FIG. 1B illustrates an embodiment with data connections (wired or wireless) between the computing device 190, the first and second ultraviolet light sources 121, 123, the image capture device 130, and the conveyor control device 165, the computing device 190 does not necessarily control the first and second ultraviolet light sources 121, 123 or the conveyor control device 165. Of particular importance is that the computing device 190 receives image data from the image capture device 130.

[0039] As described further herein, computing device 190 includes input and output, one or more processors, and memory storage. Additionally, in another embodiment, computing device 190 is physically coupled to image capture device 130 such that computing device 190 and image capture device 130 are provided as an integrated unit.

[0040] In the embodiment of FIG. 1B, the first ultraviolet light source 121 is positioned to emit ultraviolet light at an angle (α1) relative to the vertical from the plane of the conveyor 160. The second ultraviolet light source 123 can be positioned to emit ultraviolet light at an angle similar to angle (α1) but on the opposite side of the conveyor 160. Alternatively, the second ultraviolet light source 123 can be positioned at an angle (b1) different from angle (a1). The angles (a1) and (b1) can be in the range of 0° to 60° or 0° to 45° relative to the vertical, in some embodiments in the range of 0° to 30° or 30° to 60°, and in certain embodiments, approximately 45°. Similarly, the second ultraviolet light source 123 is positioned at a height (hl) above the plane of the conveyor 160. The first ultraviolet light source 121 can be positioned at a similar height above the plane of the conveyor 160. Alternatively, the first ultraviolet light source 121 can be disposed at a second height (h2) different from the first height (h1) of the second ultraviolet light source 123. The heights (h1) and (h2) can be in the range of 4 cm to 12 cm, or 6 cm to 10 cm, or in certain embodiments, can be approximately 8 cm. Furthermore, the image capture device is disposed at a height (hc) above the plane of the conveyor 160. In various embodiments, the height (hc) can be less than 60 cm or less than 30 cm, more particularly less than 12 cm or less than 6 cm, or 2 cm to 8 cm, more particularly 4 cm to 6 cm, or approximately 5 cm, more particularly approximately 5.3 cm, as can be determined depending on the requirements of the image capture device. In some embodiments, components of the image capture device can be provided or enclosed within a housing or frame, such as to limit extraneous light from the environment or otherwise supporting components of the image capture device.

[0041] Upon receiving image data from image capture device 130, computing device 190 stores the image data in its memory storage. One or more processors of computing device 190 can divide the captured image into multiple image tiles for input to the artificial neural network. The image tiles can be the same size, e.g., 512 x 512 pixels, and the captured image can be resized as needed to allow for the division of the image tiles. Preferably, the multiple image tiles include at least 20 distinct tiles, at least 25 distinct tiles, at least 30 distinct tiles, at least 35 distinct tiles, or for some embodiments, 35 distinct tiles. Image tiles can also be used to train and retrain a learning artificial neural network, such as by allowing the convolutional neural network to adjust the kernel, kernel bias, or kernel weights in each convolutional layer based on feedback from a training dataset.

[0042] 1C is a schematic diagram of another embodiment of a system 170 for identifying material components of an implant tissue product 110. The system 170 includes a plurality of image capture devices, including at least a first image capture device 131 and a second image capture device 133, each configured to capture an image of the implant tissue product 110. The system 170 may be provided with an ultraviolet light source 125 in the form of a diffuse circular light or a diffuse light of other shape (e.g., rectangular, rod-shaped, etc.), which illuminates the implant tissue product 110. Each of the plurality of image capture devices 131, 133 may include or be provided with a corresponding optical filter 132, as described in detail in other embodiments of the present disclosure.

[0043] In one embodiment, the image capture devices 131, 133 can each include a 12 megapixel sensor or equivalent imaging device. The image capture devices 131, 133 can be configured to obtain still images or video images, or both, of the implant tissue product, such that the image data captured by each image capture device 131, 133 can be processed individually by a neural network or merged before processing. The use of two or more image capture devices 131, 133 can be configured to allow the implant tissue product 110 to be imaged at once without the need to move the implant tissue product 110.

[0044] 2A and 2B include example image tiles 202, 204 from captured images of an implant tissue product in accordance with the present disclosure. In the illustrated embodiment, the image tiles 202, 204 show material components including fascia, flesh, and skin. As a result of the combined effect of ultraviolet fluorescence and the optical filters of the image capture device 130, the fascia appears dark blue or purple, the flesh appears dark yellow, and the skin appears light blue. However, accurately distinguishing and localizing the fascia, flesh, and skin from the image tiles 202, 204 of the captured image remains challenging given the color similarity and lack of contrast between their individual material components. Therefore, advantageous identification and classification of materials in the image tiles in accordance with the present disclosure requires collaboration with an artificial neural network.

[0045] FIG. 3 is a flow diagram of a method 300 for pixel-by-pixel evaluation of a captured image to mark and classify physical materials at each location on an implant tissue product, according to an embodiment of the present disclosure. In a first step 302, an implant tissue product is provided to the system 100, and an image of the implant tissue product is captured under ultraviolet light. To evaluate the captured image, the captured image is divided into a plurality of equally sized image tiles (304). In the example shown in FIGS. 2A and 2B, each image tile has dimensions of 512 x 512 pixels for each of red, green, and blue, such that each image tile forms a 512 x 512 x 3 matrix. Each entry in the matrix can have an 8-bit value ranging from 0 to 255. Each image tile can be input to an artificial neural network (306) to classify and localize the constituent materials appearing in the captured image of the fish skin.

[0046] In a first aspect, the artificial neural network may comprise a convolutional neural network, and the method includes inputting each image tile to a stepped contracting path, or encoder, of the convolutional neural network (308) to contextualize each image tile in a first phase. As shown in Figure 4, the stepped contracting path 400 may include multiple contracting steps 410, 422, 424, each step including a first contracting convolutional layer 412 configured to filter each pixel of the captured image to form a feature map for input to additional layers, followed by a first contracting rectifier layer 414 that determines whether a feature is present for each pixel on the feature map from the first contracting convolutional layer 412. Each step may further include a second contracted convolution layer 416 followed by a second contracted rectification layer 418, with the first contracted rectification layer 412 providing input to the second contracted convolution layer 416. Each of the multiple steps may also include storing a contracted output or contracted feature map resulting from the second contracted rectification layer 418, such as a feature map of the material components of the implant tissue product. A pooling layer 420 may be provided at each step of the incremental contraction path to select salient features of the stored contracted feature map as input for the subsequent layer. Step 410 of the contracting path 400 may be repeated multiple times with the contracted feature map from an earlier step provided as input to the first contracted convolution layer 412 of subsequent steps 422, 424. As shown in FIG. 4, the above contraction path may include multiple steps 424, preferably six steps.

[0047] In a second aspect, the present systems and methods include a second phase of the convolutional neural network that includes inputting each image tile (310) to a symmetric stepped expanding path or decoder for accurately identifying features of the captured image, where the stepped expanding path 500 includes multiple expansion steps 510, 522, and 524. Each step 510 of the symmetric stepped expanding path 500 may include a first expanded convolutional layer 512 followed by a first expanded rectification layer 514 and a second expanded convolutional layer 516 followed by a second expanded rectification layer 518, where the first expanded rectification layer 514 provides input to the second expanded convolutional layer 516. Each step may further include an upsampling layer 520 followed by the second expanded rectification layer 518, where the upsampling layer 520 forms an upsampled feature map. Each step of the symmetrical stepwise expansion path 500 may include a concatenation or stacking operation 540 between an upsampled feature map and a stored contracted feature map from a step of the contraction path 400. The upsampled feature map and the stored contracted feature map selected for the concatenation operation 540 may be selected based on having a common or the same size. Step 510 of the expansion path 500 may be repeated multiple times with the expanded feature map from an earlier step provided as input to a first expanded convolutional layer 512 in subsequent steps 522, 524. As shown in FIG. 5, the expanding path 500 may include multiple steps 524, preferably the same number of steps as the contraction path 400, e.g., six steps.

[0048] As can be seen by comparing Figures 4 and 5, the contraction path 400 and the expansion path 500 are substantially symmetric paths. In one aspect, the convolutional neural network of some embodiments may have a U-shaped architecture as shown in Figure 10. In the diagram of Figure 10, steps 1410, 1422, 1424, 1510, 1522, and 1524 may correspond to steps 410, 422, 424, 510, 522, and 524 of Figures 4 and 5, with the contraction path 1400 and the expansion path 1500 forming the symmetry planes of the U-shaped architecture. It should be noted that the embodiment of Figure 10 shows only a four-stage deep network for ease of understanding. Preferably, the artificial neural network of the present disclosure includes a larger six-stage deep network, as detailed in other embodiments.

[0049] In a third aspect, the present systems and methods provide a third phase of the neural network that includes inputting (312) the concatenated or stacked outputs from the upsampled feature map and the stored shrinkage feature map into an output step. As shown in FIG. 6 , the output step 600 can include a first output convolutional layer 612 followed by a first output rectification layer 614 and a second output convolutional layer 616 followed by a second output rectification layer 618. The output step 600 can include a sigmoid layer 620 followed by the second output rectification layer 618. The sigmoid layer 620 can include an activation function configured to map all pixel values ​​to values ​​between 0 and 1, such that a binary decision can be made for each pixel regarding the presence or absence of unwanted material. The output step 600 provides output 314 in the form of a classification image 650 that identifies and characterizes the material components of the implant tissue product.

[0050] The classification images can include captured images with overlays that distinguish unwanted materials, such as fascia and meat, from fish skin. Figures 7A and 7B show captured images 702, 704 corresponding to the RGB images of Figures 2A and 2B, captured by the image capture device of the described embodiments. An artificial neural network of the present disclosure can be configured to create overlay images 706, 708 containing predicted classifications and identifications of unwanted materials and fish skin from the captured images 702, 704. In the overlay images 706, 708, pixels 710 classified as fish skin appear solid black, while pixels 712 classified as unwanted material remain open or transparent. As described above, the sigmoid layer 620 can be configured to map all pixel values ​​to a value between 0 and 1 for this purpose, returning a binary image of the prediction, with all values ​​below 0.5 displayed as black and all values ​​above 0.5 displayed as white or transparent. In other multi-class embodiments, such as distinguishing both flesh and fascia from skin, or flesh and fascia from each other, the final convolution operation can use as many filters as there are defect classes, and the sigmoid layer 620 can be replaced with an argmax or softmax activation function to return a score-per-class for each pixel. The classified images 714, 716 include a combination of the captured images 702, 704 and the overlay images 706, 708. As will be apparent to those skilled in the art, the classified images 714, 716 reveal each pixel 712 where unwanted material remains.

[0051] Returning to method 300 of Figure 3, the classification images 714, 716 can be used to guide the removal (316) of unwanted material from the implant tissue product. In various embodiments, the removal of unwanted material can be done manually or by an automated machine or process. Preferably, method 300 can be repeated as an iterative process to ensure the creation of a fish skin that is free of unwanted material.

[0052] Embodiments of methods and systems according to the present disclosure may further include a user interface configured to display the classified images 714, 716 to a user. The classified images may be labeled by a processor to output labeled images, for example, including bounding boxes, tags, or color changes configured to highlight the location of unwanted material to be removed. Similarly, the user interface may provide comparisons between a series of classified images separated by steps of removing unwanted material from the implant tissue product to allow a user to understand progress made over time. Various embodiments of the user interface may include a variety of input and output devices to facilitate user interaction, including display screens, touch screens, speakers, audible alarms, indicator lights, etc., including conventional control devices such as keyboards, control panels, computer mice, or similar devices.

[0053] 8 is a diagram of a system 800 including a computing device 840 for identifying material components of an implant tissue product 810 according to an embodiment of the present disclosure. The system 800 includes an ultraviolet light source 820 configured to illuminate the implant tissue product 810 and an image capture device 830 configured to capture an image of the implant tissue product 810. The image capture device 830 may include an optical filter 832 configured to filter incident light below a predetermined wavelength from entering the image capture device. The computing device 840 may include a power source 842, a processor 844, a communications module 846, and storage 848.

[0054] Storage 848 includes instructions stored in a non-transitory format for operating a system for identifying material constituents of an implant tissue product that, when executed by processor 844, cause processor 844 to perform one or more of the steps described herein, particularly receiving image data and identifying and classifying the material of the implant tissue product from the image data. Computing device 840 may include one or more AI modules 850 configured to apply the artificial neural networks described above with respect to the embodiments of Figures 1-7.

[0055] In an embodiment, the computing device 840 may be configured to operate an image capture device to capture image data, such as RGB image data, and to process the captured image data locally and in near real time using an artificial neural network stored in the AI ​​module 850 to output classified and identified images as described above.

[0056] As discussed above with respect to FIGS. 4 and 5, an artificial neural network according to embodiments of the present disclosure may include multiple convolutional layers 412, 416, 512, 516, 612, 616 configured to filter each pixel of a captured image to form a feature map for input to additional layers. FIG. 9A is a diagram of a convolution operation 900, including an input matrix 910, a kernel 920, and a result 930. The input matrix 910 according to various embodiments of the present disclosure may include image data captured from an implanted tissue product. In embodiments, the image data may include color intensity values ​​912 for each pixel location of the captured image, e.g., 8-bit values ​​ranging from 0 to 255. While shown in FIG. 9A as a simplified two-dimensional input matrix 910 having dimensions of only 4×4 for ease of understanding, the image data in the disclosed embodiments may preferably include a three-dimensional matrix including color intensity values ​​912 for each of the red, green, and blue intensities in an RGB image.

[0057] In one embodiment, the image data for the captured image may include an 8 megapixel image with a resolution of 3280 x 2190. The captured image may be resized to 3584 x 2560 to facilitate generating image tiles of the same size therefrom, for example, by cropping the resized image into exactly 35 individual image tiles of size 512 x 512 pixels. In this example, each image tile includes a three-dimensional input matrix having dimensions 512 x 512 x 3, where 512 x 512 includes the pixel location in the image and three additional values ​​in the third dimension are the red, green, and blue color intensities in the image data.

[0058] Referring to FIG. 9A , a kernel 920 can be applied to an input matrix 910 to enhance features of the input matrix 910 in an output result 930. The kernel 920 can include a plurality of predetermined weights 922 for transforming color intensity values ​​912 of the input matrix 910. As discussed with respect to the illustrated input matrix 910, although shown as a two-dimensional matrix in FIG. 9A , the kernel 920 in the disclosed embodiments preferably comprises a three-dimensional matrix corresponding to the three-dimensional input matrix. The kernel 920 is applied to the input matrix 910 with a fixed pathway and stride. For the three-dimensional matrices of the present disclosure, the kernel can move front to back through the color dimension, and can move left to right and top to bottom with a predetermined stride. According to the two-dimensional illustration of FIG. 9A , the kernel 920 can be moved from a first position 940 to a second position 942, a third position 944, and a fourth position 946.

[0059] Entry 932 in the convolution result 930 for each position 940, 942, 944, 946 is calculated based on an operation between the color intensity values ​​912 of the input matrix 910 and the predetermined weight values ​​922 of the kernel 920. According to the illustrated example of Figure 9A, entry 932 of the convolution result for the first position 940 of the kernel 920 can be calculated as follows: TIFF0007821900000001.tif13139

[0060] The entries 932 of the results 930 can cumulatively form feature maps, such as contracted or expanded feature maps, for each pathway for input to additional layers of the artificial neural network. In an example according to the illustrated embodiment of FIG. 9A, a convolution operation on a three-dimensional matrix can be understood by repeating the illustrated convolution operation three times, once for each red, green, and blue input matrix. Such an example of a convolution operation corresponds to a feature map, as the embodiment of FIG. 9A, where three convolution operations are repeated for each red, green, and blue input matrix, includes a single filter channel. Additional filter channels, if provided, result in additional feature maps, providing improved sensitivity and detail in detecting features in the image data. The feature map may then be passed through a rectified nonlinear unit layer, such as those shown in FIGS. 4-6, which determines whether a feature exists at a given location in the image based on the score from the convolution. Pooling can be used to select the maximum values ​​on the feature map and use them as input to subsequent layers, thus further enhancing the most salient features.

[0061] In the disclosed method and system, the first contraction step may include 64 filter channels in each convolutional layer, with the number of filter channels doubling with each contraction step. Thus, if there are six contraction steps, the convolutional layer of the final contraction step may include 4,096 filter channels. The first expansion step may then reduce the number of filter channels in each convolutional layer by half until the final expansion step has the same number of filter channels in each convolutional layer as the first contraction step. The output step may include a number of filter channels equal to the number of material classes to be identified, which serve to aggregate data into an image provided as the output of the neural network. For example, to aggregate data into a single image for output as a monochrome image, only one filter is needed to distinguish between unwanted material and skin of a transplant tissue product, while three filters are needed to distinguish between flesh, fascia, and skin.

[0062] According to various aspects of the present disclosure, the artificial neural network can adjust and improve kernel weights through automated learning or training, for example, using a training dataset based on user-annotated images. In this manner, a configuration of kernels 920, including predetermined weight values ​​922, pathways, and strides in convolution operations 900, can comprise the decision-making portion of the artificial neural network. In another aspect, the decision-making portion of the artificial neural network can comprise a configuration of kernels and rectified nonlinear unit layers, along with additional parameters as understood in view of the disclosed embodiments and features.

[0063] FIG. 9B is a three-dimensional diagram of another convolution operation 950, including an input matrix 960, a kernel 970, and a result 980. As in convolution operation 900, image data is shown as a simplified two-dimensional input matrix 960 containing color intensity values ​​962 for each pixel location in the captured image, and kernel 970 is shown as a simplified two-dimensional matrix containing a plurality of predetermined weight values ​​972 for transforming the color intensity values ​​962 in input matrix 960. Kernel 970 can be applied to input matrix 960 with a fixed pathway and stride. According to the two-dimensional illustration of FIG. 9B, kernel 970 is depicted at only a first location 990.

[0064] 9B, the convolution operation on a three-dimensional matrix can be understood by repeating the illustrated convolution operation 950 three times, including convolution for each position of the kernel, once for each red, green, and blue input matrix. As discussed in other embodiments, the entries 982 of the result 980 can cumulatively form a feature map, which can then be passed through additional layers, such as a layer of rectified nonlinear units, additional convolutional layers, and pooling layers.

[0065] 11 is a schematic diagram of another embodiment of a system 1150 for automatically identifying material components of and removing unwanted material from an implant tissue product 1110. The system 1150 includes a first ultraviolet light source 1121 and a second ultraviolet light source 1123, each configured to illuminate the implant tissue product 1110, and an image capture device 1130 configured to capture images of the implant tissue product 1110. The image capture device 1130 can be configured to obtain still and / or video images of the implant tissue product 1110 as the implant tissue product is transported by a conveyor 1160 on which the implant tissue product 1110 is placed, in a direction transverse to the imaging direction of the image capture device 1130.

[0066] The conveyor 1160 may be a conveyor belt and is controlled by a conveyor controller 1165, which may include a conveyor drive system. The conveyor controller 1165 may change the speed and direction of the conveyor 1160. Furthermore, the conveyor controller may be controlled by a computing device 1190. The captured image includes the induced fluorescence of the implant tissue product 1110. The image capture device 1130 or system 1150 may include an optical filter 1132 configured to filter incident light below a predetermined wavelength from entering the image capture device 1130. The system 1150 includes a computer or computing device 1190.

[0067] The computing device 1190 receives image data from the image capture device, for example, in the form of still image data or video image data. The computing device 1190 can be configured to process the image data using an artificial neural network configured to identify and classify implant product material from the image data, as described in other embodiments herein. Based on the identification and classification of the implant product 1110 material from the image data, the computing device 1190 can control a scraping or cutting device 1180 to remove unwanted material from the implant tissue product 1110. Embodiments of the scraping or cutting device 1180 can include a blade, a reciprocating saw, a cutter head, an extrusion die, a water jet, an air jet, or other similar device for separating a thin layer of material from the implant tissue product 1110. Preferably, the cutting device 1180 can remove material from only a localized location on the implant tissue product 1110.

[0068] 11 , the cutting device 1180 comprises a rotary cutterhead including a plurality of cutting elements 1182. The cutting device 1180 may have a fixed position relative to the conveyor 1160, or may be configured to be movable in a predetermined area above the conveyor 1160, either along the imaging direction, toward or away from the conveyor 1160, along a direction perpendicular to the transport direction of the implant tissue product 1110, or along both directions. Alternatively, the cutting device 1180 may comprise a plurality of cutting elements 1182 distributed across the conveyor 1160, along the imaging direction, along a direction perpendicular to the transport direction, or along both directions, to remove unwanted material at predetermined locations, or may be movable only between a plurality of fixed positions. In various arrangements, the computing device 1190 can control the cutting device 1180, such as by changing the position of the cutting device 1180 or activating the cutting elements of the cutting device 1180 only in specific locations, to remove unwanted material from the implant tissue product only in specific identified areas of the implant tissue product and prevent damage to the desired material thereon.

[0069] In some embodiments, the transplant tissue product 1110 can be secured to the conveyor 1160 by a restraining device, such as using a mechanical clamping arm, a suction device on or within the conveyor 1160, or similar device, to facilitate operation of the cutting device 1180 on the transplant tissue product 1110. Thus, the position of the transplant tissue product 1110 can be adjusted and regulated by the computing device 1190 through control of the conveyor 1160 and any restraining device and cutting device 1180 to precisely remove the fascia and flesh from the transplant tissue product without damaging the skin with excessive scraping, cutting, or pressure.

[0070] As described further herein, automatically identifying the material components of the implant tissue product 1110 and removing unwanted material therefrom can include an iterative system or method. Thus, the implant tissue product 1110 can be repeatedly input to and output from the system 1150. To this end, the implant tissue product 1110 can be transported, either manually or via a conveyor, from the cutting apparatus 1180 to the image capture device 1130. In other embodiments, the system 1150 can be replicated along a single processing path such that the implant tissue product 1110 passes through multiple image capture devices 1130 and cutting apparatuses 1180 to complete its processing. In other embodiments, the system 1150 can be provided with additional image capture devices after the cutting apparatus 1180 so that the effectiveness of the cutting or scraping operation can be evaluated before determining subsequent operations.

[0071] 12 is a flow diagram of a method 1200 for automatically identifying material components of an implant tissue product and removing unwanted material therefrom, such as may be performed using system 1150. The method may include feeding 1201 the implant tissue product onto a conveyor that transports the implant tissue product to an image capture device, illuminating 1203 the implant tissue product using at least one ultraviolet light source, capturing 1202 image data of the implant tissue product, transferring 1205 the image data to a computing device, and processing 1207 the image data by the computing device using an artificial neural network, the artificial neural network configured to identify and classify the material of the implant tissue product from the image data, as described in other embodiments herein.

[0072] The method may further include determining 1209 whether unwanted material is present on the implant tissue product. If unwanted material is not identified, the method may proceed by outputting 1218 the implant tissue product, either by conveying the implant tissue product or by indicating to the user that the implant tissue product is free of unwanted material. If unwanted material is identified on the implant tissue product, the method may proceed by conveying 1213 the implant tissue product to a cutting device and, as the implant tissue product is conveyed to or by the cutting device, controlling 1216 a scraping or cutting device based on the identification and classification of the implant tissue product's material from the artificial neural network to remove the unwanted material from the implant tissue product, such as by using position information from the computing device to position or operate the cutting device at specific areas of the implant tissue product classified as unwanted material. Following operation of the cutting device, the method may be repeated 1218 using the same system or with replication of some or all components of the same system in a processing line. In some embodiments, if all unwanted material was successfully removed in the previous step, only the steps of providing the implant tissue product to a conveyor that transports the implant tissue product to an image capture device 1201, illuminating the implant tissue product using at least one ultraviolet light source 1203, capturing image data of the implant tissue product 1202, transferring the image data to a computing device 1205, and processing the image data by the computing device using an artificial neural network 1207 may be repeated.

[0073] Embodiments of the present disclosure may comprise or utilize special-purpose or general-purpose computer systems including computer hardware such as, for example, one or more processors and system memory, as described in more detail below. Embodiments within the scope of the present disclosure also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and / or data structures are computer storage media. Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Thus, by way of example, embodiments of the present disclosure may include at least two distinctly different types of computer-readable media: computer storage media and transmission media.

[0074] A computer storage medium is a physical storage medium that stores computer-executable instructions and / or data structures. Physical storage media include computer hardware such as RAM, ROM, EEPROM, solid-state drives (“SSD”), flash memory, phase-change memory (“PCM”), optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other hardware storage device(s) that can be used to store program code in the form of computer-executable instructions or data structures that can be accessed and executed by a general-purpose or special-purpose computer system to implement the disclosed functionality of the present disclosure.

[0075] Transmission media can include networks and / or data links usable to carry program code in the form of computer-executable instructions or data structures and accessible by a general-purpose or special-purpose computer system. A "network" may be defined as one or more data links that enable electronic data to be transported between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer system over a network or another communications connection (either hardwired, wireless, or a combination of hardwired and wireless), the computer system can view the connection as a transmission medium. Combinations of the above should also be included within the scope of computer-readable media.

[0076] Furthermore, upon reaching the various computer system components, program code in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system's RAM and / or to less volatile computer storage media within the computer system. Thus, it should be understood that computer storage media may be included in computer system components that also (or primarily) utilize transmission media.

[0077] Computer-executable instructions may include, for example, instructions and data that, when executed by one or more processors, cause a general-purpose computer system, special-purpose computer system, or special-purpose processing device to perform a certain function or group of functions. Computer-executable instructions may be, for example, binary, or instructions in an intermediate format such as assembly language, or even source code.

[0078] The disclosure of this application may be implemented in networked computing environments with many types of computer system configurations, including, but not limited to, personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cellular phones, PDAs, tablets, pagers, routers, switches, etc. The disclosure may also be implemented in distributed system environments where tasks are performed by both local and remote computer systems that are linked through a network (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links). Thus, in a distributed system environment, a computer system may include multiple component computer systems. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0079] The disclosure of this application may also be implemented in a cloud computing environment. Although this is not required, the cloud computing environment may be distributed. If distributed, the cloud computing environment may have components that are distributed internationally within a single organization and / or owned across multiple organizations. For purposes of this description and the claims that follow, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the many other benefits that derive from such a model when properly deployed.

[0080] Cloud computing models can consist of a variety of characteristics, such as on-demand self-service, pervasive network access, resource pooling, rapid elasticity, and measured service. Cloud computing models can exist in the form of various service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (laaS). Cloud computing models can also be deployed using various deployment models, such as private cloud, community cloud, public cloud, and hybrid cloud.

[0081] Some embodiments, such as cloud computing environments, may include a system including one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machines emulate an operational computing system, supporting an operating system and possibly one or more other applications. In some embodiments, each host includes a hypervisor that emulates virtual resources for the virtual machine using physical resources abstracted from the virtual machine's view. The hypervisor also provides appropriate isolation between the virtual machines. Thus, the hypervisor provides the illusion that the virtual machine interfaces with physical resources, even though from the perspective of any given virtual machine, the virtual machine interfaces only with the appearance of physical resources (e.g., virtual resources). Examples of physical resources, including processing power, are memory, disk space, network bandwidth, media drives, etc.

[0082] By providing a system and method according to the disclosed embodiments for identifying material components of implant tissue products, the problem that existing 2D image recognition and manual identification approaches are expensive, time-consuming, and ill-suited to the task of distinguishing between visually similar materials, such as fascia, flesh, and skin, within implant tissue products is addressed. The disclosed embodiments advantageously provide systems and methods that demonstrate and provide improved accuracy, speed, and consistency in identifying material components of implant tissue products to provide actionable, quantifiable insight to a technician or automated processing system.

[0083] Various features of the present disclosure may be better understood by reference to a specific example of a method according to the present disclosure for identifying material components of an implant tissue product, as detailed in the attached Appendix, which is expressly incorporated herein by reference. The example provided is essentially illustrative of one application of the principles according to the present disclosure and is not intended to be limiting. In particular, the Appendix illustrates a significantly reduced scale of a neural network for processing input images in the form of a 4x4x3 frame, and many values ​​are assumed for simplicity.

[0084] Not necessarily all such objects or advantages will be achieved in accordance with all embodiments of the present disclosure, and those skilled in the art will recognize that the present disclosure can be embodied or practiced to achieve or optimize one advantage or group of advantages taught without achieving other objects or advantages taught or suggested.

[0085] A skilled artisan will recognize the interchangeability of various components from the various described embodiments. In addition to the variations described, other equivalents for each feature can be mixed and matched by those skilled in the art of building or using systems or methods for identifying material components of implant tissue products under the principles of the present disclosure. Thus, the described embodiments can be adapted for material identification and characterization for fascia, flesh, scales, hide, or any other suitable material of implant tissue products.

[0086] Possible combinations of embodiments and features

[0087] The present disclosure provides various examples, embodiments, and features that should be understood as being combinable with other examples, embodiments, or features described herein, unless expressly stated otherwise or mutually exclusive.

[0088] In addition to the above, further embodiments and examples include the following.

[0089] 1. A system for identifying material components of an implant tissue product, the system comprising: an image capture device configured to acquire image data of the implant tissue product; and a processor configured to process the image data using an artificial neural network, the artificial neural network configured to identify and classify the material of the implant tissue product from the image data.

[0090] 2. A system according to any one of 1 above or 3 to 19 below, or a combination thereof, wherein the image capture device comprises an ultraviolet light source, an optical filter, and an image sensor.

[0091] 3. A system according to any one of 1 to 2 above or 4 to 19 below, or a combination thereof, wherein the ultraviolet light source is configured to emit light having a wavelength of 365 nm to 395 nm.

[0092] 4. A system according to any one of 1 to 3 above or 5 to 19 below, or a combination thereof, wherein the optical filter comprises a long-pass filter configured with a cut-on wavelength of 435 nm.

[0093] 5. A system according to any one of 1 to 4 above or 6 to 19 below, or a combination thereof, wherein the optical filter has a transmittance of 85% for wavelengths greater than 435 nm.

[0094] 6. A system according to any one or combination of 1 to 5 above or 7 to 19 below, wherein the processor is configured to divide the image data into a plurality of image tiles.

[0095] 7. A system according to any one of 1 to 6 above or 8 to 19 below, or a combination thereof, wherein each of the multiple image tiles has the same size.

[0096] 8. A system according to any one of 1 to 7 above or 9 to 19 below, or a combination thereof, wherein the plurality of image tiles includes 35 image tiles.

[0097] 9. A system according to any one of 1-8 above or 10-19 below, or a combination thereof, wherein the graft tissue product comprises fish skin having waste material including at least one of fascia and flesh.

[0098] 10. A system according to any one of 1 to 9 above or 11 to 19 below, or a combination thereof, wherein the artificial neural network comprises a convolutional neural network.

[0099] 11. A system according to any one of 1 to 10 above or 12 to 19 below, or a combination thereof, wherein the convolutional neural network comprises a stepped contracting path, and each step of the stepped contracting path comprises a first contraction convolutional layer, a second contraction convolutional layer, a first contraction rectification layer following the first contraction convolutional layer, a second contraction rectification layer following the second contraction convolutional layer, a storage operation for storing an output following the second contraction rectification layer, and a pooling layer following the storage operation.

[0100] 12. A system according to any one of 1 to 11 above or 13 to 19 below, or a combination thereof, wherein the convolutional neural network comprises a stepped expanding path, and each step of the stepped expanding path comprises a first expanded convolutional layer, a second expanded convolutional layer, a first expanded rectification layer following the first expanded convolutional layer, a second expanded rectification layer following the second expanded convolutional layer, an upsampling layer following the second expanded rectification layer, and a concatenation operation that stacks the stored output of the stepped contraction path onto the output of the upsampling layer.

[0101] 13. A system according to any one of 1 to 12 above or 14 to 19 below, or a combination thereof, wherein the stepwise contraction path and the stepwise expansion path include the same number of steps.

[0102] 14. A system according to any one of 1 to 13 above or 15 to 19 below, or a combination thereof, wherein the stepwise contraction path and stepwise expansion path each include six steps.

[0103] 15. A system according to any one or combination of 1 to 14 above or 16 to 19 below, wherein the output step includes a first output convolutional layer, a second output convolutional layer, a first output rectification layer following the first output convolutional layer, a second output rectification layer following the second output convolutional layer, and a sigmoid layer following the second output rectification layer.

[0104] 16. A system according to any one of 1-15 above or 17-19 below, or a combination thereof, wherein the convolutional neural network is configured to output an image defining the region of each material feature of the implant tissue product.

[0105] 17. A system according to any one of 1 to 16 above or 18 to 19 below, or a combination thereof, wherein the optical filter has a cut-on wavelength of 400 nm to 600 nm.

[0106] 18. A system according to any one of 1 to 17 above or 19 below, or a combination thereof, wherein the plurality of image tiles each have equal dimensions of 512 x 512 pixels.

[0107] 19. A system according to any one or combination of 1 to 18 above, wherein the image data is resized before being divided into a plurality of image tiles.

[0108] 20. A method for identifying material components of an implant tissue product, comprising the steps of capturing image data of the implant tissue product using an image capture device, and using a processor to process the image data using an artificial neural network to identify and classify the material of the implant tissue product from the image data.

[0109] 21. The method according to any one of 20 above or 22-38 below, or a combination thereof, wherein capturing image data of the implant tissue product further comprises illuminating the implant tissue product with an ultraviolet light source, filtering the emitted light reflected by the implant tissue product with an optical filter of the image capture device, and capturing the filtered light using an image sensor of the image capture device.

[0110] 22. A method according to any one of 20 to 21 above or 23 to 38 below, or a combination thereof, wherein the ultraviolet light source is configured to emit light having a wavelength of 365 nm to 395 nm.

[0111] 23. A method according to any one of 20 to 22 above or 24 to 38 below, or a combination thereof, wherein the optical filter includes a long-pass filter configured with a cut-on wavelength of 435 nm.

[0112] 24. A method according to any one of the above 20 to 23 or the following 25 to 38, or a combination thereof, wherein the optical filter has a transmittance of 85% for wavelengths greater than 435 nm.

[0113] 25. The method according to any one of 20-24 above or 26-38 below, or a combination thereof, further comprising using a processor to divide the image data into a plurality of image tiles.

[0114] 26. A method according to any one of 20 to 25 above or 27 to 38 below, or a combination thereof, wherein each of the multiple image tiles has the same size.

[0115] 27. The method according to any one of 20-26 above or 28-38 below, or a combination thereof, wherein the plurality of image tiles includes 35 image tiles.

[0116] 28. The method according to any one of 20-27 above or 29-38 below, or a combination thereof, wherein the graft tissue product comprises fish skin having waste material including at least one of fascia and flesh.

[0117] 29. A method according to any one of 20-28 above or 30-38 below, or a combination thereof, wherein the artificial neural network includes a convolutional neural network.

[0118] 30. A method according to any one of 20 to 29 above or 31 to 38 below, or a combination thereof, further comprising inputting each of the plurality of image tiles into a stepped contracting path of an artificial neural network, each step of the contraction path comprising a first contraction convolutional layer, a second contraction convolutional layer, a first contraction rectification layer following the first contraction convolutional layer, a second contraction rectification layer following the second contraction convolutional layer, a storage operation following the second contraction rectification layer for storing an output, and a pooling layer following the storage operation.

[0119] 31. A method according to any one of 20 to 30 above or 32 to 38 below, or a combination thereof, further comprising the step of inputting each of the plurality of image tiles into a symmetric stepped expanding path of an artificial neural network, each step of the stepped expanding path comprising a first stepped convolutional layer, a second stepped convolutional layer, a first stepped rectification layer following the first stepped convolutional layer, a second stepped rectification layer following the second stepped convolutional layer, an upsampling layer following the second stepped rectification layer, and a concatenation operation that stacks the stored output of the stepped erosion path onto the output of the upsampling layer.

[0120] 32. A method according to any one of 20-31 above or 33-38 below, or a combination thereof, wherein the stepwise contraction pass and the stepwise expansion pass include the same number of steps.

[0121] 33. A method according to any one of 20-32 above or 34-38 below, or a combination thereof, wherein the stepwise contraction pass and stepwise expansion pass each include six steps.

[0122] 34. A method according to any one of 20 to 33 above or 35 to 38 below, or a combination thereof, further comprising inputting each of the plurality of image tiles to an output step of an artificial neural network, the output step including a first output convolutional layer, a second output convolutional layer, a first output rectification layer following the first output convolutional layer, a second output rectification layer following the second output convolutional layer, and a sigmoid layer following the second output rectification layer.

[0123] 35. The method according to any one of 20-34 above or 36-38 below, or a combination thereof, further comprising the step of outputting an image defining the area of ​​each material feature of the implant tissue product following the artificial neural network output step.

[0124] 36. A method according to any one of 20 to 35 above or 37 to 38 below, or a combination thereof, wherein the optical filter has a cut-on wavelength of 400 nm to 600 nm.

[0125] 37. A method according to any one of 20 to 36 above or 38 below, or a combination thereof, wherein the plurality of image tiles each have equal dimensions of 512 x 512 pixels.

[0126] 38. The method according to any one of 20 to 37 above or a combination thereof, wherein the image data is resized before being divided into a plurality of image tiles.

[0127] 39. A non-transitory hardware storage device having stored thereon computer-executable instructions, the computer-executable instructions, when executed by one or more processors of the computer, configure the computer to perform at least the following: capturing image data of an implant tissue product with an image capture device; and using the processor to process the image data using an artificial neural network to identify and classify material of the implant tissue product from the image data.

[0128] While a system or method for identifying material components of an implant tissue product has been disclosed in certain preferred embodiments and examples, it will therefore be understood by those skilled in the art that the present disclosure extends beyond the disclosed embodiments to other alternative embodiments and / or uses of the system or method for identifying material components of an implant tissue product, as well as obvious modifications and equivalents. It is intended that the scope of the disclosed system or method for identifying material components of an implant tissue product should not be limited by the above disclosed embodiments, but should be determined solely by a fair reading of the claims which follow.

Claims

1. 1. A system for identifying material components of skin for use as a tissue implant product, comprising: The system comprises: an image capture device configured to acquire image data of the skin; a processor configured to process the image data using an artificial neural network, the processor configured to classify materials on the skin from the image data using the artificial neural network and to identify locations of the classified materials on the skin; Equipped with If the skin includes unwanted material thereon, including at least fascia or flesh, the processor classifies the unwanted material as fascia or flesh on the skin from the image data and identifies the location of the fascia or flesh on the skin.

2. The system of claim 1 , wherein the image capture device comprises an ultraviolet light source, an optical filter, and an image sensor.

3. The system of claim 2 , wherein the ultraviolet light source is configured to emit light having a wavelength between 365 nm and 395 nm.

4. The system of claim 2 , wherein the optical filter comprises a long-pass filter configured with a cut-on wavelength of 435 nm.

5. The system of claim 2 , wherein the optical filter has a transmittance of 85% for wavelengths greater than 435 nm.

6. The system of claim 1 , wherein the processor is configured to divide the image data into a plurality of image tiles.

7. The system of claim 6 , wherein each of the plurality of image tiles has the same size.

8. The system of claim 1 , wherein the skin comprises fish skin.

9. The system of claim 1 , wherein the artificial neural network comprises a convolutional neural network.

10. the convolutional neural network comprises a stepwise shrinkage path; Each step of the stepwise shrinkage path comprises: a first contractive convolutional layer; and a second contractive convolutional layer; and a first contracted rectifying layer following the first contracted convolution layer; a second contracted rectifying layer following the second contracted convolution layer; a storage operation following the second contraction rectification layer to store the output; a pooling layer following the storage operation; The system of claim 9 , comprising:

11. the convolutional neural network comprises a stepwise expansion path; Each step of the gradual expansion path comprises: a first dilated convolutional layer; and a second dilated convolutional layer; and a first dilated rectifying layer following the first dilated convolutional layer; a second dilated rectifying layer following the second dilated convolutional layer; an upsampling layer following the second enhanced rectification layer; a concatenation operation that stacks the stored outputs of the stepwise erosion paths onto the outputs of the upsampling layer; The system of claim 10, comprising:

12. The system of claim 11 , wherein the stepwise contraction path and the stepwise expansion path include the same number of steps.

13. The system of claim 11 , wherein the stepwise contraction path and the stepwise expansion path each include six steps.

14. The output step is a first output convolutional layer; and a second output convolutional layer; and a first output rectification layer following the first output convolutional layer; a second output rectification layer following the second output convolutional layer; and a sigmoid layer following the second output rectifying layer; Including, The system of claim 1 .

15. 10. The system of claim 9, wherein the convolutional neural network is configured to output an image defining a region of each material feature of the skin.

16. The system of claim 2 , wherein the optical filter comprises a cut-on wavelength between 400 nm and 600 nm.

17. The system of claim 6 , wherein the processor is configured to resize the image data before dividing the image data into the plurality of image tiles.

18. 1. A method for identifying material components of skin for use as a tissue implant product, comprising: The method comprises: capturing image data of the skin using an image capture device; and using a processor to process the image data using an artificial neural network to classify materials on the skin from the image data and to identify locations of the classified materials on the skin; This includes the steps: If the skin includes unwanted material thereon, including at least fascia or flesh, the processor classifies the unwanted material as fascia or flesh on the skin from the image data and identifies the location of the fascia or flesh on the skin.

19. Capturing the image data of the skin includes: illuminating the skin with an ultraviolet light source; filtering the emitted light reflected by the skin with an optical filter of the image capture device; capturing the filtered light using an image sensor of the image capture device; 20. The method of claim 18, further comprising:

20. 20. A non-transitory hardware storage device having stored computer-executable instructions that, when executed by one or more processors of a computer, configure the computer to perform the method of claim 18.

21. The system of claim 1, wherein if the skin contains unwanted material thereon, including at least fascia, the processor classifies the unwanted material from the image data as fascia on the skin and identifies the location of the fascia on the skin.

22. The system of claim 1, wherein if the skin contains unwanted material thereon, including meat, the processor classifies the unwanted material as meat on the skin from the image data and identifies the location of the meat on the skin.

23. The system of claim 1, wherein the artificial neural network is configured to output an image defining the region of each material component by evaluating the image data pixel by pixel and classifying each pixel into each material component contained in the implant tissue product.

24. The system described in claim 1, further comprising a device for removing unnecessary material from the transplant tissue product based on the classification result of the material of the transplant tissue product by the artificial neural network.

Images