Systems, devices and methods for recognizing defects in medical implant treatments - Patents.com

Abstract

A system and method for identifying material components of an implant tissue product includes an image capture device for acquiring image data of the implant tissue product and a processor for processing the image data with an artificial neural network that identifies and classifies materials of the implant tissue product from the image data. The image capture device can include an optical filter and an ultraviolet light source for ultraviolet fluorescent imaging of the implant tissue product. Using the captured image data, the artificial neural network can identify unwanted materials of the implant tissue product, such as fascia or flesh on the skin of a fish, for subsequent removal.

Description

[Technical field]

[0001] The present disclosure relates to systems and methods for identifying material components in the processing of scaffolding materials or implant tissue products, particularly using fish skin, for wound care and / or other tissue healing applications. [Background technology]

[0002] A variety of human, animal, and synthetic materials are currently described or used in medical procedures to augment, repair, or correct tissue defects. The vertebrate extracellular matrix (ECM) is a complex structural entity that surrounds and supports cells and has been found to provide a particularly favorable scaffolding material for skin graft tissue products. The ECM is composed of a complex mixture of structural proteins, the most abundant of which are collagens, other specialized proteins, and proteoglycans.

[0003] Fish skin can be processed into scaffold materials, including acellular scaffolds, with the majority of the natural biological ECM components intact. To be most effective, the native three-dimensional structure, composition, and function of the dermal ECM must remain essentially unchanged, providing a scaffold to support cell migration, adhesion, proliferation, and differentiation, thus facilitating tissue repair and / or replacement. These products must be properly fabricated as medical-grade skin substitutes through various processing steps prior to use to prevent adverse manifestations, including infection and allergic reactions.

[0004] An important part of this process is the thorough removal of certain unwanted materials from the inside of the dried skin, i.e., the fascia and all meat residues. While the fascia may be removed primarily for aesthetic reasons, it is important to remove all meat residues due to the possibility of allergic reactions to fish proteins in patients. The removal process is generally performed manually, with skilled technicians using sharp tools to carefully scrape the fascia and meat residues from the skin. This process must be performed precisely to ensure that the unwanted material is removed without damaging the ECM of the fish skin, requiring the technician to rely on visual inspection of the skin to estimate the appropriate degree of scraping required. Such work is time-consuming, costly, physically demanding, tedious, and subject to human error in locating and identifying the unwanted material on the fish skin. Additionally, when a skilled technician removes the unwanted material, the physicality and extent of the removal process often fatigues the technician, which can significantly reduce the accuracy and consistency of the removal process.

[0005] The inventors of the present disclosure have determined that the significant need to properly identify and classify waste materials on fish skin continues to be a significant impediment to improved quality control, automation of processing, and scaling up of manufacturing processes. The high cost of manual evaluation in the manufacturing process may limit the availability of implant tissue products formed from fish skin or otherwise make such products unaffordable to a larger audience. Furthermore, such processing, by its manual nature, is necessarily inaccurate. Due to stringent medical safety requirements that increasingly demand a level of precision and consistency that can only be achieved by automated systems rather than human evaluation, the inventors have determined that there is an increasingly important need for systems and methods for automatically and accurately identifying and identifying components of fish skin.

[0006] Unfortunately, the difficulty of this task is compounded by the fact that the waste material, while having a different texture, often looks similar to the skin of the accompanying fish and shares the same milky white color. These obstacles make any assessment by existing image recognition approaches highly questionable. Thus, using visual analysis to distinguish thin sections of white fascia and thin sections of white meat residue from a thin layer of white clean skin remains a complex and difficult challenge. Currently, there are no improved systems and methods to prevent errors in this process, and quality control remains an unsolved problem and must rely on human visual inspection.

[0007] Thus, the identification, sorting, and removal of unwanted material from fish skin remains a matter of subjective and sometimes arbitrary intuition, preference, and user experience, rather than being performed with quantitative precision. Thus, the creation of implant tissue products using fish skin is subject to many errors and inefficiencies. Summary of the Invention [Means for solving the problem]

[0008] A system for identifying and differentiating material components of an implant tissue product is provided, the system comprising an image capture device configured to acquire image data of the implant tissue product, and a processor configured to process the image data using an artificial neural network, the artificial neural network configured to localize and classify materials of the implant tissue product from the image data.

[0009] A method of identifying and distinguishing material components of an implant tissue product is also provided, the method including capturing an image of the implant tissue product using an image capture device and using a processor to process the image data using an artificial neural network to localize and classify materials of the implant tissue product from the image of the implant tissue product.

[0010] Also provided is a non-transitory hardware storage device having stored thereon computer-executable instructions that, when executed by one or more processors of a computer, configure the computer to capture an image of the implant tissue product with an image capture device, and to use the processor to process the image data using an artificial neural network to localize and classify material of the implant tissue product from the image of the implant tissue product.

[0011] A preferred embodiment of the system and method for identifying material components of implant tissue products advantageously utilizes a novel architecture that leverages ultraviolet fluorescence imaging in embodiments that utilize material fluorescence in the visible spectrum for image capture, in combination with a convolutional neural network specifically adapted to identify and classify material components of implant tissue products. The systems and methods of these and other disclosed embodiments can develop a comprehensive feature map of material components on fish skin to provide quality control in the removal of unwanted materials from implant tissue products. Using the comprehensive feature map in conjunction with automated processing tools, the systems and methods can automatically remove unwanted materials from implant tissue products. [Brief description of the drawings]

[0012] These and other features, aspects, and advantages of the present disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings.

[0013] [Figure 1A] 1 is a diagram of a system for identifying material components of an implant tissue product according to an embodiment of the present disclosure. [Figure 1B] FIG. 11 is a diagram of a system for identifying material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 1C]FIG. 11 is a diagram of a system for identifying material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 2A] 1 is an image tile of an implant tissue product captured by an image capture device according to an embodiment of the present disclosure. [Figure 2B] 11 is another image tile of an implant tissue product captured by an image capture device according to another embodiment of the present disclosure. [Diagram 3] 1 is a flow diagram of a method for identifying material components of an implant tissue product according to an embodiment of the present disclosure. [Figure 4] 4A-4C are diagrams of contracting paths of an artificial neural network according to the embodiment of FIG. 3. [Diagram 5] FIG. 4 is a diagram of the final output step of the artificial neural network according to the embodiment of FIG. 3. [Figure 6] FIG. 4 is a diagram of an expanding path of an artificial neural network according to the embodiment of FIG. 3. [Figure 7A] FIG. 7A includes one of a number of image tiles, an overlay image, and a classified image according to the embodiment of FIG. [Figure 7B] FIG. 7B includes another one of a plurality of image tiles, an overlay image, and a classified image according to the embodiment of FIG. [Figure 8] FIG. 1 is a diagram of a system for identifying material components of an implant tissue product according to an embodiment of the present disclosure, including a computing device. [Figure 9A] FIG. 2 is a diagram of a convolution operation according to an embodiment of the present disclosure. [Figure 9B] FIG. 13 is a diagram of another convolution operation according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a diagram of symmetric pathways of an artificial neural network according to an embodiment of the present disclosure. [Figure 11]1 is a diagram of a system for identifying and removing material components of an implant tissue product according to another embodiment of the present disclosure. [Figure 12] 1 is a flow diagram of a method for identifying and removing material components of an implant tissue product according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] overview

[0015] Various embodiments of the present disclosure can be better understood from the following description read in conjunction with the accompanying drawings, in which like reference characters refer to like elements and in which:

[0016] While the disclosure is susceptible to various modifications and alternative constructions, specific illustrative embodiments have been shown in the drawings and are described below. It is to be understood, however, that there is no intention to limit the disclosure to the particular embodiments disclosed, but rather, the intention is to cover all modifications, alternative constructions, combinations, and equivalents falling within the spirit and scope of the disclosure.

[0017] Unless a term is expressly defined in this application to have an explained meaning, it is understood that there is no intention to explicitly or implicitly limit the meaning of such term beyond its ordinary or usual meaning.

[0018] For the purposes of this application, in a preferred embodiment, the term "graft tissue product" includes "fish skin", "fish skin", "acellular fish skin", or similar, such as Kerecis™ Omega3 Wound by Kerecis, Kerecis™ Omega3 acellular fish skin from Atlantic cod (Gadus morhua), and any other or similar fish skin graft tissues as previously described. These graft tissue products undergo processing that maintains biological structure and bioactive compounds, including omega-3 polyunsaturated fatty acids (PUFAs), but removes allergenic and other unwanted components, and specifically removes tissues that may cause an immune response from the recipient patient. The thickness of the fish skin used to make the graft tissue product may vary, as may the dimensions of the waste material thereon, such as fascia, flesh, or scales. The fish skin may have a thickness of approximately 0.35 mm to 2.25 mm, while the thickness of the waste material thereon may range from approximately 0.15 mm to 1.75 mm. The creation of graft tissue products from fish skin may require the removal of substantially all fascia, flesh, or other undesirable material from the skin or tissue to be used as the graft tissue product. Although described as a preferred embodiment, the graft tissue product can include any other caught fish species used for skin grafts or skin replacement products, such as but not limited to Atlantic cod (Gadus morhua), tilapia, including Nile tilapia (Oreochromis niloticus), or other fish species. In other embodiments, the "graft tissue product" can be made from harvested mammalian skin, including but not limited to porcine skin graft tissue products, or sources other than fish, including human allografts or autografts, or cadaveric skin graft tissue products. The graft tissue product can ultimately be made as an acellular or decellularized graft tissue product, or can be a cellular skin graft tissue product (i.e., a graft tissue product in which skin cells remain viable or intact).In addition, the transplant tissue product may include non-skin tissue, such as placental transplant tissue, used as a skin transplant tissue product, where the tissue to be used as the transplant tissue product is to have undesired second tissue or parts removed. In addition, the transplant tissue product may include biological, synthetic, or hybrid skin substitutes, where the transplant tissue product is inspected for undesired tissue, parts, or materials to be removed before use as a transplant tissue product. Finally, the transplant tissue product may include autograft, allograft, or xenograft products, and is not limited to skin transplant tissue products used on human patients, but also includes autograft, allograft, or xenograft products used or made to be used on other non-human species, including horses, cows, monkeys, rabbits, mice, rats, guinea pigs, or other mammalian or non-mammalian species to which the skin transplant tissue product is applied.

[0019] As described herein, a "convolutional neural network" refers to an artificial neural network for performing analysis of images, etc., based on a weight-sharing architecture of convolution kernels configured for pixel-by-pixel evaluation of an image. A convolutional neural network is capable of tuning and improving kernel weights and biases through automated learning or training, for example, using a training data set. In an embodiment, a convolutional neural network can include a contracting path or encoder followed by a symmetric expanding path or decoder, making the network an end-to-end fully convolutional neural network that includes convolutional layers and does not include dense layers.

[0020] Various embodiments and components used therewith

[0021] As mentioned above, it has been found that in the case of making graft tissue products, for example from fish skin, certain materials, such as fascia and flesh, can cause undesirable reactions or results, such as immunological reactions and results, or otherwise adversely affect the aesthetics or quality of the product. For example, in the case of fish skin, the fascia and flesh portions or layers on the fish skin can be extremely difficult to reliably distinguish from the acellular matrix of the fish skin due to the thin thicknesses involved and the milky white color shared by these components. Due to the challenges of accurately classifying and localizing these different material components using existing imaging approaches, evaluation of the components of the graft tissue product, such as during the scraping of the fish skin or after the creation of the graft tissue product, is often manual, expensive, and inaccurate due to the poor suitability of fish skin to accurately distinguish between the visually similar material components.

[0022] Furthermore, manual assessment of fish skin does not provide usable information regarding the type, amount, and arrangement of material on the fish skin in an accurate, reproducible, and rapid manner.

[0023] In view of the above, there is a need for a system and method for identifying material components of implant tissue products that addresses the problems and shortcomings of existing approaches to identifying, assessing, and determining the location, amount, and arrangement of material components of implant tissue products, including the limitations of existing 2D image recognition approaches as well as costly and time-consuming manual identification of material components. The inventors have recognized a need for a material identification system that provides improved accuracy, speed, and consistency in distinguishing between material components of implant tissue products to provide insight that is accessible and quantifiable to experts or automated processing systems.

[0024] Embodiments of systems and methods for identifying material components of implant tissue products according to the present disclosure advantageously overcome deficiencies in existing approaches to distinguishing, identifying, and localizing material components of implant tissue products, such as fascia and meat on fish skin, which have limited accuracy and require significant time and effort to implement.

[0025] In an embodiment of the system and method, ultraviolet fluorescence imaging is synergistically combined with an improved neural network architecture to classify and localize material components of an implant tissue product, preferably fish skin. The ultraviolet fluorescence imaging can be provided using an ultraviolet light source configured to illuminate the implant tissue product and an image capture device configured to capture, store, and process RGB (i.e., red, green, blue) images of the fluorescence induced from the material components in the visible spectrum. In various embodiments, the ultraviolet light source can be configured to irradiate the implant tissue product with light having a wavelength of about 365 nm to 395 nm. In one aspect, the ultraviolet light source can have a fixed center band of 395 nm, for example, a light emitting diode having a fixed center band of 395 nm. Preferably, the wavelength of the ultraviolet light source is configured to irradiate the implant tissue product without substantially heating the implant tissue product. The image capture device can be integrated with a sensor device, such as a camera or similar device, and the image data can be processed using an application on a computing device configured to receive, store, process, and / or transmit the captured images. The captured or transmitted images can then be evaluated to classify and localize material components of the implant tissue product.

[0026] A system and method for identifying material components of a transplant tissue product leverages RGB images of induced fluorescence from fish skin in combination with an artificial neural network to segment, map, and identify material components within spaces such as fascia and flesh on the skin of the fish. The system and method includes an image capture device configured to capture an image, e.g., an RGB image, and a processor configured to process the image data using an artificial neural network, preferably a convolutional neural network, where the artificial neural network is configured to localize and classify the material of the transplant tissue product from the image data. The captured image may be an RGB image or a true color image, or any other suitable type of color image that captures ultraviolet induced fluorescence of the skin of the fish, as will be appreciated by those skilled in the art from this disclosure. The system and method alleviates the need for manual visual assessment of material components of the skin of the fish.

[0027] The image capture device can be integrated with an optical filter that filters the incident light to form a captured image. Alternatively, the optical filter is separate (not integrated) from the image capture device, but is positioned such that the incident light passes through the optical filter before entering the image capture device. In various embodiments, the optical filter can include a long pass filter that reflects short wavelengths while transmitting long wavelengths. In one aspect, the cut-on wavelength of the long pass filter can be in the range of about 375 nm to 675 nm, 400 nm to 600 nm, 400 nm to 500 nm, 425 nm to 625 nm, or for some embodiments, about 435 nm. The optical filter can have a transmittance of about 85% at the selected wavelength. The optical filter may necessarily be present as a hardware component or a physical filter on the image capture device, since an image captured without a physical optical filter would be saturated with light to the extent that differentiation between material components of the implant tissue product would be infeasible. Furthermore, optical filtering or additional optical filtering may be provided through digital processing of image data obtained from the image capture device.

[0028] Embodiments of the present systems and methods can provide pixel-by-pixel evaluation of the captured images to mark and classify the physical material at each location of the implant tissue product. In various embodiments, each pixel is approximately 200-800 μm. 2 Size: about 300-600μm 2 or for some embodiments, about 400 μm 2 Alternatively, the imaging system may be configured to have at least 200 pixels / mm2 corresponding to an area of ​​the implant tissue product. 2 and more particularly at least 400 pixels / mm 2 , at least 600 pixels / mm 2, at least 800 pixels / mm 2 , at least 1000 pixels / mm 2 , at least 1500 pixels / mm 2 , or at least 2000 pixels / mm 2 Advantageously, pixel-by-pixel evaluation of captured images in accordance with the disclosed systems and embodiments allows for detection of even very small amounts of unwanted material, ensuring higher quality and greater accuracy of the resulting implant tissue product.

[0029] The system and method may include splitting the captured image into a plurality of image tiles for input to the artificial neural network. The image tiles may be of the same size, e.g., 512×512 pixels, and the captured image may be resized as necessary to allow for splitting of the image tiles. Preferably, the plurality of image tiles includes at least 20 separate tiles, at least 25 separate tiles, at least 30 separate tiles, at least 35 separate tiles, or for some embodiments, 35 separate tiles. Image tiles may also be used to train and retrain a learning artificial neural network, such as by allowing the convolutional neural network to adjust kernels, kernel biases, or kernel weights in each convolutional layer based on feedback from a training dataset.

[0030] In a first aspect, the system and method provide a stepped contracting path or a first phase of an encoder for contextualizing each image tile, the stepped contracting path including a plurality of contracting steps. Each step of the stepped contracting path may include a first convolutional layer configured to filter each pixel of the captured image to form a feature map for input to additional layers, the first convolutional layer being followed by a first rectification layer that determines whether a feature is present for each pixel on the feature map from the first convolutional layer. Each step may further include a second convolutional layer followed by a second rectification layer, the first rectification layer providing an input to the second convolutional layer. Each of the plurality of steps may also include storing a contracted output or contracted feature map resulting from the second rectification layer, such as a feature map resulting from a material component of an implant tissue product. A pooling layer may be provided at each step of the stepwise shrinkage path to select salient features of the stored shrinkage feature map as input for subsequent layers and / or steps.

[0031] According to various embodiments, the above-mentioned rectification layer of the artificial neural network can include a nonlinear rectification layer. The nonlinear rectification layer can, for example, include a threshold operation that returns zero for values ​​less than zero and returns the input value directly for values ​​greater than zero. The nonlinear rectification layer can thus include an activation function that takes the value calculated from the immediately preceding convolutional layer and transforms it into an output. Preferably, the rectification layer can include a nonlinear rectification (ReLu) type activation.

[0032] In a second aspect, the system and method provide a symmetric stepped expanding path or a second phase of a decoder for localizing features of a captured image, the stepped expanding path including a plurality of expanding steps. Each step of the symmetric stepped expanding path includes a first convolutional layer followed by a first rectification layer and a second convolutional layer followed by a second rectification layer, the first rectification layer providing an input to the second convolutional layer. Each step may further include an upsampling layer following the second rectification layer, the upsampling layer forming an upsampled feature map. Each step of the symmetric stepped expanding path may include a concatenation or stacking operation of the upsampled feature map and the stored eroded feature map. The upsampled feature map and the stored eroded feature map selected for the concatenation operation may be selected based on having a common or same size.

[0033] In a third aspect, the present systems and methods provide a third phase of an output operation or step that includes a first convolutional layer followed by a first rectification layer and a second convolutional layer followed by a second rectification layer, the output operation further including a sigmoid layer following the second rectification layer.

[0034] The image capture device and / or the computing device may be configured to locally perform the above steps and other steps described herein. The computing device may comprise a storage, a processor, a power source, and an interface. The instructions in the storage may be executed by the processor to capture the images and utilize one or more neural networks as described herein to classify and identify material components of the implant tissue product. It will be appreciated that while in an embodiment the above steps are performed locally, the captured images are transmitted to a remote server and one or more steps described herein are performed by cloud computing with a processor located at the remote server configured to classify and identify material components of the implant tissue product.

[0035] 1A is a schematic diagram of a system 100 for identifying material components of an implant tissue product 110 in what may be considered its most basic form, according to an embodiment of the present disclosure. The system 100 includes an ultraviolet light source 120 configured to illuminate the implant tissue product 110 and an image capture device 130 configured to capture an image of the implant tissue product 110. The captured image includes induced fluorescence of the implant tissue product 110. The image capture device 130 can include an optical filter 132 configured to filter incident light below a predetermined wavelength from entering the image capture device.

[0036] 1B is a schematic diagram of another embodiment of a system 150 for identifying material components of an implant product 110. The system 150 includes a first ultraviolet light source 121, a second ultraviolet light source 123, and an image capture device 130 configured to capture an image of the implant product 110, with each ultraviolet light source 121, 123 configured to illuminate the implant product 110. In one embodiment, the image capture device 130 can include a Sony IMX219 8-megapixel sensor, or an equivalent imaging device. Although an 8-megapixel sensor can be used, a larger megapixel sensor can be used to obtain a higher resolution, such as a 12-megapixel sensor, a 20-megapixel sensor, a 30-megapixel sensor, or a 40-megapixel sensor. The image capture device 130 may be configured to capture still and / or video images of the implant tissue product 110 as it is transported by the conveyor 160 on which it is placed, directly transverse to the imaging direction of the image capture device 130. In certain embodiments, the implant tissue product 110 may be secured in place or on the conveyor 160 by restraining elements such as clips, bars, suction devices, and / or transparent coverings such as plexiglass sheets 111.

[0037] The conveyor 160 may be a conveyor belt and is controlled by a conveyor controller 165, which may include a conveyor drive system. The conveyor controller 165 may change the speed and direction of the conveyor 160. In addition, the conveyor controller is controlled by a computing device 190. The captured image includes the induced fluorescence of the implant tissue product 110. The image capture device 130 or system 150 may include an optical filter 132 configured to filter incident light below a predetermined wavelength from entering the image capture device. The system 150 includes a computer or computing device 140, which is described in more detail herein (e.g., in the following embodiment shown in FIG. 8).

[0038] The computing device 190 receives image data, for example in the form of still image data or video image data, from the image capture device. The computing device may also control the first ultraviolet light source 121 and the second ultraviolet light source 123. Although hardwired connections between the computing device 190, the first and second ultraviolet light sources 121, 123, the image capture device 130, and the conveyor control device 165 may be included in various embodiments, data between these components may be transmitted via wireless communication, such as Bluetooth. Furthermore, while FIG. 1B illustrates an embodiment with data connections (wired or wireless) between the computing device 190, the first and second ultraviolet light sources 121, 123, the image capture device 130, and the conveyor control device 165, the computing device 190 does not necessarily control the first and second ultraviolet light sources 121, 123 or the conveyor control 165. Of particular importance is that the computing device 190 receives image data from the image capture device 130.

[0039] As described further herein, computing device 190 includes input and output, one or more processors, and memory storage. Additionally, in another embodiment, computing device 190 is physically coupled to image capture device 130 such that computing device 190 and image capture device 130 are provided as an integrated unit.

[0040] In the embodiment of FIG. 1B, the first ultraviolet light source 121 is positioned to emit ultraviolet light at an angle (α1) from the plane of the conveyor 160 to the vertical. The second ultraviolet light source 123 can be positioned to emit ultraviolet light at an angle similar to the angle (a1) but on the opposite side of the conveyor 160. Alternatively, the second ultraviolet light source 123 can be positioned at an angle (b1) different from the angle (a1). The angles (a1) and (b1) can be in the range of 0° to 60° or 0° to 45° from the vertical, in some embodiments in the range of 0° to 30° or 30° to 60°, and in certain embodiments about 45°. Similarly, the second ultraviolet light source 123 is positioned at a height (hl) above the plane of the conveyor 160. The first ultraviolet light source 121 can be positioned at a similar height above the plane of the conveyor 160. Alternatively, the first ultraviolet light source 121 can be disposed at a second height (h2) different from the first height (h1) of the second ultraviolet light source 123. The height (h1) and the height (h2) can be in the range of 4 cm to 12 cm, or 6 cm to 10 cm, or in certain embodiments can be about 8 cm. Furthermore, the image capture device is disposed at a height (hc) above the plane of the conveyor 160. In various embodiments, the height (hc) can be less than 60 cm or less than 30 cm, more particularly less than 12 cm or less than 6 cm, or 2 cm to 8 cm, more particularly 4 cm to 6 cm, or about 5 cm, more particularly about 5.3 cm, as may be determined depending on the requirements of the image capture device. In some embodiments, components of the image capture device can be provided or enclosed in a housing or frame, such as to limit extraneous light from the environment or otherwise supporting components of the image capture device.

[0041] Upon receiving image data from the image capture device 130, the computing device 190 stores the image data in the memory storage of the computing device. One or more processors of the computing device 190 can divide the captured image into multiple image tiles for input to the artificial neural network. The image tiles can be of the same size, for example, 512×512 pixels, and the captured image can be resized as necessary to allow for the division of the image tiles. Preferably, the multiple image tiles include at least 20 separate tiles, at least 25 separate tiles, at least 30 separate tiles, at least 35 separate tiles, or for some embodiments, 35 separate tiles. The image tiles can also be used to train and retrain the learning artificial neural network, such as by allowing the convolutional neural network to adjust the kernel, kernel bias, or kernel weights in each convolutional layer based on feedback from a training dataset.

[0042] 1C is a schematic diagram of another embodiment of a system 170 for identifying material components of an implant tissue product 110. The system 170 includes a plurality of image capture devices, including at least a first image capture device 131 and a second image capture device 133, each configured to capture an image of the implant tissue product 110. The system 170 may be provided with an ultraviolet light source 125 in the form of a diffuse circular light or a diffuse light of other shape (e.g., square, rod-like, etc.), which illuminates the implant tissue product 110. Each of the plurality of image capture devices 131, 133 may include or be provided with a corresponding optical filter 132, as described in detail in other embodiments of the present disclosure.

[0043] In one embodiment, the image capture devices 131, 133 can each include a 12 megapixel sensor or equivalent imaging device. The image capture devices 131, 133 can be configured to obtain still and / or video images of the implant tissue product such that image data captured by each image capture device 131, 133 can be processed individually by a neural network or merged prior to processing. The use of two or more image capture devices 131, 133 can be configured to allow the implant tissue product 110 to be imaged at once without the need to move the implant tissue product 110.

[0044] 2A and 2B include examples of image tiles 202, 204 from captured images of an implant tissue product according to the present disclosure. In the illustrated embodiment, the image tiles 202, 204 show material components including fascia, flesh, and skin. As a result of the combined effect of ultraviolet fluorescence and the optical filters of the image capture device 130, the fascia appears dark blue or purple, the flesh appears dark yellow, and the skin appears light blue. However, accurately distinguishing and localizing the fascia, flesh, and skin from the image tiles 202, 204 of the captured images remains a challenge given the color similarity and lack of contrast between their individual material components. Therefore, advantageous identification and classification of materials in the image tiles according to the present disclosure requires collaboration with an artificial neural network.

[0045] FIG. 3 is a flow diagram of a method 300 for evaluating a captured image pixel by pixel to mark and classify physical materials at each location of an implant tissue product according to an embodiment of the present disclosure. In a first step 302, an implant tissue product is provided to the system 100 and an image of the implant tissue product is captured under ultraviolet light. To evaluate the captured image, the captured image is divided into a number of image tiles of equal size (304). In the example shown in FIGS. 2A and 2B, each image tile has a size of 512×512 pixels for each of red, green, and blue, such that each image tile forms a 512×512×3 matrix. Each entry in the matrix can have an 8-bit value ranging from 0 to 255. Each image tile can be input to an artificial neural network (306) to classify and localize the constituent materials appearing in the captured image of the fish skin.

[0046] In a first aspect, the artificial neural network may comprise a convolutional neural network, and the method includes inputting each image tile to a stepped contracting path, or encoder, of the convolutional neural network (308) for contextualizing each image tile in a first phase. As shown in FIG. 4, the stepped contracting path 400 may include a number of contracting steps 410, 422, 424, each step including a first contracting convolutional layer 412 configured to filter each pixel of the captured image to form a feature map for input to additional layers, followed by a first contracting rectifier layer 414 that determines whether a feature is present for each pixel on the feature map from the first contracting convolutional layer 412. Each step may further include a second contracted convolution layer 416 followed by a second contracted rectification layer 418, with the first contracted rectification layer 412 providing input to the second contracted convolution layer 416. Each of the multiple steps may also include storing a contracted output or contracted feature map resulting from the second contracted rectification layer 418, such as a feature map of material components of the implant tissue product. A pooling layer 420 may be provided at each step of the incremental contraction path to select salient features of the stored contracted feature map as input for the subsequent layer. Step 410 of the contracting path 400 may be repeated multiple times with the contracted feature map from the previous step provided as input to the first contracted convolution layer 412 of the subsequent steps 422, 424. As shown in FIG. 4, the above contraction path may include multiple steps 424, preferably six steps.

[0047] In a second aspect, the systems and methods include a second phase of the convolutional neural network including inputting each image tile to a symmetric stepped expanding path or decoder (310) for accurately identifying features of the captured image, where the stepped expanding path 500 includes multiple expansion steps 510, 522, 524. Each step 510 of the symmetric stepped expanding path 500 may include a first expanded convolutional layer 512 followed by a first expanded rectification layer 514 and a second expanded convolutional layer 516 followed by a second expanded rectification layer 518, where the first expanded rectification layer 514 provides input to the second expanded convolutional layer 516. Each step may further include an upsampling layer 520 followed by the second expanded rectification layer 518, where the upsampling layer 520 forms an upsampled feature map. Each step of the symmetrical stepwise expanding path 500 may include a concatenation or stacking operation 540 of the upsampled feature map with the stored contracted feature map from the contraction path 400 step. The upsampled feature map and the stored contracted feature map selected for the concatenation operation 540 may be selected based on having a common or the same size. The step 510 of the expanding path 500 may be repeated multiple times with the expanded feature map from the previous step provided as input to the first expanding convolutional layer 512 of the subsequent steps 522, 524. As shown in FIG. 5, the expanding path 500 may include multiple steps 524, preferably the same number of steps as the contraction path 400, e.g., six steps.

[0048] As can be seen by comparing Figures 4 and 5, the contraction path 400 and the expansion path 500 are substantially symmetric paths. In one aspect, the convolutional neural network of some embodiments may have a U-shaped architecture as shown in Figure 10. In the diagram of Figure 10, steps 1410, 1422, 1424, 1510, 1522, 1524 may correspond to steps 410, 422, 424, 510, 522, 524 of Figures 4 and 5, and the contraction path 1400 and the expansion path 1500 form the symmetric planes of the U-shaped architecture. It should be noted that the embodiment of Figure 10 shows only a four-stage deep network for ease of understanding. Preferably, the artificial neural network of the present disclosure includes a larger six-stage deep network, as detailed in other embodiments.

[0049] In a third aspect, the system and method provide a third phase of the neural network that includes inputting (312) the concatenated or stacked output from the upsampled feature map and the stored shrinkage feature map to an output step. As shown in FIG. 6, the output step 600 can include a first output convolutional layer 612 followed by a first output rectification layer 614 and a second output convolutional layer 616 followed by a second output rectification layer 618. The output step 600 can include a sigmoid layer 620 followed by the second output rectification layer 618. The sigmoid layer 620 can include an activation function configured to map all pixel values ​​to values ​​between 0 and 1 such that a binary decision can be made for each pixel whether unwanted material is present. The output step 600 provides an output 314 in the form of a classification image 650 that identifies and identifies material components of the implant tissue product.

[0050] The classification images can include captured images with overlays that distinguish unwanted materials, such as fascia and meat, from fish skin. Figures 7A and 7B show captured images 702, 704 corresponding to the RGB images of Figures 2A and 2B, captured by the image capture device of the described embodiment. The artificial neural network of the present disclosure can be configured to create overlay images 706, 708 that include predicted classifications and identifications of unwanted materials and fish skin from the captured images 702, 704. In the overlay images 706, 708, pixels 710 classified as fish skin appear solid black, while pixels 712 classified as unwanted materials remain open or transparent areas. As mentioned above, the sigmoid layer 620 can be configured to map all pixel values ​​to a value between 0 and 1 for this purpose, with all values ​​below 0.5 displayed as black and all values ​​above 0.5 displayed as white or transparent to return a binary image of the prediction. In other multi-class embodiments, such as distinguishing both flesh and fascia from skin, or flesh and fascia from each other, the final convolution operation can use as many filters as there are classes of defects, and the sigmoid layer 620 can be replaced with an argmax or softmax activation function to return a score-per-class for each pixel. The classified images 714, 716 include a combination of the captured images 702, 704 and the overlay images 706, 708. As will be apparent to one skilled in the art, the classified images 714, 716 reveal each pixel 712 where unwanted material remains.

[0051] Returning to the method 300 of Figure 3, the classification images 714, 716 can be used to guide the removal (316) of waste material from the implant tissue product. In various embodiments, the removal of waste material can be done manually or by an automated machine or process. Preferably, the method 300 can be repeated as an iterative process to ensure the creation of a fish skin that is free of waste material.

[0052] Embodiments of methods and systems according to the present disclosure may further include a user interface configured to display the classified images 714, 716 to a user. The classified images may be labeled by a processor to output a labeled image, for example, including bounding boxes, tags, or changes to color configured to highlight the location of the unwanted material to be removed. Similarly, the user interface may provide a comparison between a series of classified images separated by steps of removing unwanted material from the implant tissue product to allow the user to understand the progress made over time. Various embodiments of the user interface may include a variety of input and output devices to facilitate interaction with the user, including display screens, touch screens, speakers, audible alarms, indicator lights, and the like, including conventional control devices such as keyboards, control panels, computer mice, or similar devices.

[0053] 8 is a diagram of a system 800 including a computing device 840 for identifying a material component of an implant tissue product 810 according to an embodiment of the present disclosure. The system 800 includes an ultraviolet light source 820 configured to illuminate the implant tissue product 810 and an image capture device 830 configured to capture an image of the implant tissue product 810. The image capture device 830 can include an optical filter 832 configured to filter incident light below a predetermined wavelength from entering the image capture device. The computing device 840 can include a power source 842, a processor 844, a communication module 846, and storage 848.

[0054] Storage 848 includes instructions stored in a non-transitory format for operating a system for identifying material components of an implant tissue product that, when executed by processor 844, cause processor 844 to perform one or more of the steps described herein, in particular receiving image data and identifying and classifying materials of the implant tissue product from the image data. Computing device 840 can include one or more AI modules 850 configured to apply an artificial neural network as described above with respect to the embodiment of Figures 1-7.

[0055] In an embodiment, the computing device 840 may be configured to operate an image capture device to capture image data, such as RGB image data, and to process the captured image data locally and in near real-time using an artificial neural network stored in the AI ​​module 850 to output images that have been classified and identified as described above.

[0056] As described above with respect to Figures 4 and 5, an artificial neural network according to an embodiment of the present disclosure may include multiple convolution layers 412, 416, 512, 516, 612, 616 configured to filter each pixel of a captured image to form a feature map for input to additional layers. Figure 9A is a diagram of a convolution operation 900 including an input matrix 910, a kernel 920, and a result 930. The input matrix 910 according to various embodiments of the present disclosure may include image data captured from an implanted tissue product. In an embodiment, the image data may include color intensity values ​​912 for each pixel location of the captured image, e.g., 8-bit values ​​ranging from 0 to 255. Although shown in Figure 9A as a simplified two-dimensional input matrix 910 having dimensions of only 4x4 for ease of understanding, the image data in the disclosed embodiment may preferably include a three-dimensional matrix including color intensity values ​​912 for each of the red, green, and blue intensities in an RGB image.

[0057] In one embodiment, the image data for the captured image may include an 8 megapixel image with a resolution of 3280 x 2190. The captured image may be resized to 3584 x 2560 to facilitate generating image tiles of the same size therefrom, for example, by cropping the resized image into exactly 35 individual image tiles of size 512 x 512 pixels. In this example, each image tile includes a three-dimensional input matrix having dimensions 512 x 512 x 3, where 512 x 512 includes pixel locations in the image, and three other values ​​in the third dimension are the intensities of red, green, and blue colors in the image data.

[0058] Referring to FIG. 9A, a kernel 920 can be applied to an input matrix 910 to enhance features of the input matrix 910 in an output result 930. The kernel 920 can include a number of predefined weight values ​​922 for transforming color intensity values ​​912 of the input matrix 910. As discussed with respect to the illustrated input matrix 910, although shown as a two-dimensional matrix in FIG. 9A, the kernel 920 in the disclosed embodiment may preferably include a three-dimensional matrix corresponding to the three-dimensional input matrix. The kernel 920 is applied to the input matrix 910 with a fixed pathway and stride. For the three-dimensional matrices of the present disclosure, the kernel can move front to back through the color dimension and can move left to right and top to bottom with a predefined stride. According to the two-dimensional illustration of FIG. 9A, the kernel 920 can be moved from a first position 940 to a second position 942, a third position 944, and a fourth position 946.

[0059] An entry 932 in the convolution result 930 for each position 940, 942, 944, 946 is calculated based on an operation between the color intensity values ​​912 of the input matrix 910 and the predefined weight values ​​922 of the kernel 920. According to the illustrated example of Figure 9A, the entry 932 of the convolution result for the first position 940 of the kernel 920 can be calculated as follows: TIFF2025510091000002.tif13139

[0060] The entries 932 of the results 930 can cumulatively form feature maps, such as contracted or expanded feature maps in each pathway, for input to additional layers of the artificial neural network. In an example according to the illustrated embodiment of FIG. 9A, the convolution operation on the three-dimensional matrix can be understood from repeating the illustrated convolution operation three times, once for each of the red, green, and blue input matrices. Such an example of the convolution operation corresponds to a feature map, where the embodiment of FIG. 9A includes a single filter channel and is repeated three times for each of the red, green, and blue input matrices. Additional feature maps result if additional filter channels are provided, resulting in improved sensitivity and detail in detecting features in the image data. The feature map may then be passed through a layer of rectified nonlinear units, such as those shown in FIGS. 4-6, which determine whether a feature exists at a given location in the image based on the score from the convolution. Pooling can be used to select the maxima on the feature map and use these as input to subsequent layers, thus further enhancing the most salient features.

[0061] In the disclosed method and system, the first contraction step may include 64 filter channels in each convolutional layer, and the number of filter channels doubles in each contraction step. Thus, if there are six contraction steps, the convolutional layer of the last contraction step may include 4,096 filter channels. Then, the first expansion step may reduce the number of filter channels in each convolutional layer by half until the last expansion step has the same number of filter channels in each convolutional layer as the first contraction step. The output step may include as many filter channels as the number of material classes to be identified, and these filter channels serve to aggregate the data into an image provided as the output of the neural network. For example, to aggregate the data into one image for output as a monochrome image, only one filter is needed to distinguish between unwanted material and skin of a transplant tissue product, while three filters are needed to distinguish between flesh, fascia, and skin.

[0062] According to various aspects of the present disclosure, the artificial neural network can adjust and improve the kernel weights through automatic learning or training, for example, using a training data set based on user-annotated images. In this manner, the configuration of kernels 920, including predefined weight values ​​922, pathways, and strides in the convolution operation 900, can comprise the decision-making portion of the artificial neural network. In another aspect, the decision-making portion of the artificial neural network can comprise the configuration of kernels and rectified nonlinear unit layers, along with additional parameters as will be understood in view of the disclosed embodiments and features.

[0063] 9B is a three-dimensional diagram of another convolution operation 950 including an input matrix 960, a kernel 970, and a result 980. As in the convolution operation 900, the image data is shown as a simplified two-dimensional input matrix 960 including color intensity values ​​962 for each pixel location of the captured image, and the kernel 970 is shown as a simplified two-dimensional matrix including a number of predefined weight values ​​972 for transforming the color intensity values ​​962 of the input matrix 960. The kernel 970 may be applied to the input matrix 960 with a fixed pathway and stride. According to the two-dimensional illustration of FIG. 9B, the kernel 970 is depicted at only a first location 990.

[0064] 9B, the convolution operation on a three-dimensional matrix can be understood by repeating the illustrated convolution operation 950 three times, including convolving each position of the kernel, once for each of the red, green, and blue input matrices. As discussed in other embodiments, the entries 982 of the result 980 can cumulatively form a feature map, which can then be passed through additional layers, such as a layer of rectified nonlinear units, additional convolutional layers, and pooling layers.

[0065] 11 is a schematic diagram of another embodiment of a system 1150 for automatically identifying material components of an implant tissue product 1110 and removing unwanted materials therefrom. The system 1150 includes a first ultraviolet light source 1121 and a second ultraviolet light source 1123, each configured to illuminate the implant tissue product 1110, and an image capture device 1130 configured to capture an image of the implant tissue product 1110. The image capture device 1130 can be configured to obtain still and / or video images of the implant tissue product as the implant tissue product is transported by a conveyor 1160 on which the implant tissue product 1110 is placed, transverse to an imaging direction of the image capture device 1130.

[0066] The conveyor 1160 may be a conveyor belt and is controlled by a conveyor controller 1165, which may include a conveyor drive system. The conveyor controller 1165 may change the speed and direction of the conveyor 1160. Furthermore, the conveyor controller may be controlled by a computing device 1190. The captured image includes the induced fluorescence of the implant tissue product 1110. The image capture device 1130 or system 1150 may include an optical filter 1132 configured to filter incident light below a predetermined wavelength from entering the image capture device 1130. The system 1150 includes a computer or computing device 1190.

[0067] The computing device 1190 receives image data from the image capture device, for example in the form of still image data or video image data. The computing device 1190 can be configured to process the image data using an artificial neural network configured to identify and classify the implant product material from the image data, as described in other embodiments herein. Based on the identification and classification of the implant product 1110 material from the image data, the computing device 1190 can control the scraping or cutting device 1180 to remove the unwanted material from the implant product 1110. Embodiments of the scraping or cutting device 1180 can include a blade, a reciprocating pin, a cutter head, an extrusion die, a water jet, an air jet, or other similar device for separating a thin layer of material from the implant product 1110. Preferably, the cutting device 1180 can remove material from only a localized location on the implant product 1110.

[0068] 11, the cutting device 1180 comprises a rotary cutter head including a plurality of cutting elements 1182. The cutting device 1180 may have a fixed position relative to the conveyor 1160 or may be configured to be movable in a predetermined area above the conveyor 1160, either toward or away from the conveyor 1160 along the imaging direction, along a direction perpendicular to the transport direction of the implant tissue product 1110, or along both directions. Alternatively, the cutting device 1180 may comprise a plurality of cutting elements 1182 distributed across the conveyor 1160, along the imaging direction, along a direction perpendicular to the transport direction, or along both directions, to remove the waste material at predetermined locations, or may be movable only between a plurality of fixed locations. In various arrangements, the computing device 1190 can control the cutting device 1180, such as by changing the position of the cutting device 1180 or activating cutting elements of the cutting device 1180 only in specific locations, to remove unwanted material from the implant tissue product only in specific identified areas of the implant tissue product and prevent damage to desired material thereon.

[0069] In some embodiments, the transplant tissue product 1110 can be secured to the conveyor 1160 by a restraining device, such as using a mechanical clamp arm, a suction device on or within the conveyor 1160, or similar device to facilitate operation of the cutting device 1180 on the transplant tissue product 1110. Thus, the position of the transplant tissue product 1110 can be adjusted and regulated by the computing device 1190 through control of the conveyor 1160 and any restraining devices as well as the cutting device 1180 to precisely remove the fascia and flesh from the transplant tissue product without damaging the skin by excessive scraping, cutting, or pressure.

[0070] As described further herein, automatically identifying the material components of the implant tissue product 1110 and removing unwanted material therefrom may include an iterative system or method. Thus, the implant tissue product 1110 may be repeatedly input to and output from the system 1150. To this end, the implant tissue product 1110 may be transferred, either manually or via a conveyor, from the cutting apparatus 1180 to the image capture device 1130. In other embodiments, the system 1150 may be replicated along a single processing path such that the implant tissue product 1110 passes through multiple image capture devices 1130 and cutting apparatus 1180 to complete its processing. In other embodiments, the system 1150 may be provided with additional image capture devices after the cutting apparatus 1180 so that the effectiveness of the cutting or scraping operation may be evaluated before determining subsequent operations.

[0071] 12 is a flow diagram of a method 1200 for automatically identifying material components of an implant tissue product and removing unwanted materials therefrom, as may be performed using system 1150. The method may include providing 1201 the implant tissue product to a conveyor that transports the implant tissue product to an image capture device, illuminating 1203 the implant tissue product using at least one ultraviolet light source, capturing 1202 image data of the implant tissue product, transferring 1205 the image data to a computing device, and processing 1207 the image data by the computing device using an artificial neural network, the artificial neural network configured to identify and classify the material of the implant tissue product from the image data, as described in other embodiments herein.

[0072] The method may further include determining 1209 whether unwanted material is present on the implant tissue product. If unwanted material is not identified, the method may proceed by outputting 1218 the implant tissue product, either by conveying the implant tissue product or by indicating to a user that the implant tissue product is free of unwanted material. If unwanted material is identified on the implant tissue product, the method may proceed by conveying 1213 the implant tissue product to a cutting device and controlling 1216 a scraping or cutting device based on the identification and classification of the material of the implant tissue product from the artificial neural network to remove the unwanted material from the implant tissue product as the implant tissue product is conveyed to or by the cutting device, such as by using position information from the computing device to position or operate the cutting device at specific areas of the implant tissue product classified as unwanted material. Following operation of the cutting device, the method may be repeated 1218 using the same system or with duplication of some or all components of the same system in a processing line. In some embodiments, if all unwanted material has been successfully removed in the previous step, only the steps of providing the implant tissue product on a conveyor that transports the implant tissue product to an image capture device 1201, illuminating the implant tissue product using at least one ultraviolet light source 1203, capturing image data of the implant tissue product 1202, transferring the image data to a computing device 1205, and processing the image data by the computing device using an artificial neural network 1207 may be repeated.

[0073] Embodiments of the present disclosure may comprise or utilize a dedicated or general-purpose computer system including computer hardware, such as one or more processors and system memory, as described in more detail below. Embodiments within the scope of the present disclosure also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and / or data structures are computer storage media. Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Thus, by way of example, embodiments of the present disclosure may include at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

[0074] A computer storage medium is a physical storage medium that stores computer-executable instructions and / or data structures. Physical storage media includes computer hardware such as RAM, ROM, EEPROM, solid-state drive ("SSD"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other hardware storage device(s) that can be used to store program code in the form of computer-executable instructions or data structures that can be accessed and executed by a general-purpose or special-purpose computer system to implement the disclosed functionality of the present disclosure.

[0075] Transmission media can include networks and / or data links that can be used to carry program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer system. A "network" can be defined as one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer system over a network or another communications connection (either hardwired, wireless, or a combination of hardwired and wireless), the computer system can view the connection as a transmission medium. Combinations of the above should also be included within the scope of computer-readable media.

[0076] Furthermore, upon reaching the various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM in a network interface module (e.g., a "NIC") and then eventually transferred to the computer system's RAM and / or to a less volatile computer storage medium in the computer system. Thus, it should be understood that computer storage media can be included in computer system components that also (or primarily) utilize transmission media.

[0077] Computer-executable instructions may include, for example, instructions and data that, when executed by one or more processors, cause a general-purpose computer system, special-purpose computer system, or special-purpose processing device to perform a certain function or group of functions. Computer-executable instructions may be, for example, binary, or intermediate format instructions such as assembly language, or even source code.

[0078] The disclosure of the present application may be implemented in a network computing environment with many types of computer system configurations including, but not limited to, personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be implemented in a distributed system environment where tasks are performed by both local and remote computer systems that are linked through a network (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links). Thus, in a distributed system environment, a computer system may include multiple constituent computer systems. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0079] The disclosure of this application may also be implemented in a cloud computing environment. Although this is not required, the cloud computing environment may be distributed. When distributed, the cloud computing environment may have components that are distributed internationally within one organization and / or owned across multiple organizations. In this description and in the claims that follow, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the numerous other advantages that derive from such a model when properly deployed.

[0080] Cloud computing models can consist of various characteristics such as on-demand self-service, pervasive network access, resource pooling, rapid elasticity, measured service, etc. Cloud computing models can exist in the form of various service models such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (laaS). Cloud computing models can also be deployed using various deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0081] Some embodiments, such as a cloud computing environment, may include a system including one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machines emulate an operational computing system, supporting an operating system, and possibly one or more other applications. In some embodiments, each host includes a hypervisor that emulates virtual resources for the virtual machines using physical resources abstracted from the virtual machine's view. The hypervisor also provides proper isolation between the virtual machines. Thus, the hypervisor provides the illusion that the virtual machines interface with physical resources, even though from the perspective of any given virtual machine, the virtual machine interfaces only with the appearance of physical resources (e.g., virtual resources). Examples of physical resources, including processing power, are memory, disk space, network bandwidth, media drives, etc.

[0082] Providing a system and method according to the disclosed embodiments for identifying material components of an implant tissue product addresses the problem that existing 2D image recognition and manual identification approaches are expensive, time-consuming, and not well-suited to the challenge of distinguishing between visually similar materials such as fascia, flesh, and skin within an implant tissue product. The disclosed embodiments advantageously provide systems and methods that demonstrate and provide improved accuracy, speed, and consistency in identifying between material components of an implant tissue product to provide actionable, quantifiable insight to a technician or automated processing system.

[0083] Various features of the present disclosure may be better understood by reference to a specific example of a method according to the present disclosure for identifying material components of an implant tissue product, as detailed in the attached Appendix, which is expressly incorporated herein by reference. The example provided is essentially illustrative of one application of the principles according to the present disclosure and is not intended to be limiting. In particular, the Appendix shows a greatly reduced scale of a neural network for processing an input image in the form of a 4x4x3 frame, with many values ​​assumed for simplicity.

[0084] Not necessarily all such objects or advantages are achieved in accordance with all embodiments of the present disclosure, and those skilled in the art will recognize that the present disclosure can be embodied or practiced to achieve or optimize one advantage or group of advantages taught without achieving other objects or advantages taught or suggested.

[0085] A skilled artisan will recognize the interchangeability of various components from the various embodiments described. In addition to the variations described, other equivalents for each feature can be mixed and matched by those skilled in the art of building or using systems or methods for identifying material components of implant tissue products under the principles of the present disclosure. Thus, the described embodiments can be adapted for material identification and identification for fascia, flesh, scales, hide, or any other suitable material of implant tissue products.

[0086] Possible combinations of embodiments and features

[0087] The present disclosure provides various examples, embodiments, and features that should be understood as being combinable with other examples, embodiments, or features described herein, unless expressly stated otherwise or mutually exclusive.

[0088] In addition to the above, further embodiments and examples include the following.

[0089] 1. A system for identifying a material component of an implant tissue product, the system comprising: an image capture device configured to acquire image data of the implant tissue product; and a processor configured to process the image data using an artificial neural network, the artificial neural network configured to identify and classify a material of the implant tissue product from the image data.

[0090] 2. A system according to 1 above or any one of 3 to 19 below or a combination thereof, wherein the image capture device comprises an ultraviolet light source, an optical filter, and an image sensor.

[0091] 3. A system according to any one of 1 to 2 above or 4 to 19 below or a combination thereof, wherein the ultraviolet light source is configured to emit light having a wavelength of 365 nm to 395 nm.

[0092] 4. A system according to any one of 1 to 3 above or 5 to 19 below or a combination thereof, wherein the optical filter comprises a long-pass filter configured with a cut-on wavelength of 435 nm.

[0093] 5. A system according to any one of 1 to 4 above or 6 to 19 below, or a combination thereof, wherein the optical filter has a transmittance of 85% for wavelengths greater than 435 nm.

[0094] 6. A system according to any one or combination of 1 to 5 above or 7 to 19 below, wherein the processor is configured to divide the image data into a plurality of image tiles.

[0095] 7. A system according to any one or combination of 1 to 6 above or 8 to 19 below, wherein each of the multiple image tiles has the same size.

[0096] 8. A system according to any one or combination of 1 to 7 above or 9 to 19 below, wherein the plurality of image tiles includes 35 image tiles.

[0097] 9. A system according to any one or combination of 1-8 above or 10-19 below, wherein the graft tissue product comprises fish skin having waste material comprising at least one of fascia and flesh.

[0098] 10. A system according to any one of 1 to 9 above or 11 to 19 below or a combination thereof, wherein the artificial neural network comprises a convolutional neural network.

[0099] 11. A system according to any one of 1 to 10 above or 12 to 19 below, or a combination thereof, wherein the convolutional neural network comprises a stepped contracting path, each step of the stepped contracting path comprising a first contracting convolutional layer, a second contracting convolutional layer, a first contracting rectification layer following the first contracting convolutional layer, a second contracting rectification layer following the second contracting convolutional layer, a storage operation following the second contracting rectification layer for storing an output, and a pooling layer following the storage operation.

[0100] 12. A system according to any one of 1 to 11 above or 13 to 19 below or a combination thereof, wherein the convolutional neural network comprises a stepped expanding path, each step of the stepped expanding path comprising a first expanded convolutional layer, a second expanded convolutional layer, a first expanded rectification layer following the first expanded convolutional layer, a second expanded rectification layer following the second expanded convolutional layer, an upsampling layer following the second expanded rectification layer, and a concatenation operation that stacks the stored output of the stepped contraction path onto the output of the upsampling layer.

[0101] 13. A system according to any one of 1-12 above or 14-19 below or a combination thereof, wherein the stepwise contraction path and the stepwise expansion path include the same number of steps.

[0102] 14. A system according to any one of 1 to 13 above or 15 to 19 below or a combination thereof, wherein the stepwise contraction path and the stepwise expansion path each include six steps.

[0103] 15. A system according to any one or combination of 1 to 14 above or 16 to 19 below, wherein the output step includes a first output convolutional layer, a second output convolutional layer, a first output rectification layer following the first output convolutional layer, a second output rectification layer following the second output convolutional layer, and a sigmoid layer following the second output rectification layer.

[0104] 16. A system according to any one or combination of 1-15 above or 17-19 below, wherein the convolutional neural network is configured to output an image defining an area of ​​each material feature of the implant tissue product.

[0105] 17. A system according to any one of 1 to 16 above or 18 to 19 below, or a combination thereof, wherein the optical filter has a cut-on wavelength of 400 nm to 600 nm.

[0106] 18. A system according to any one or combination of 1 to 17 above or 19 below, wherein each of the multiple image tiles has equal dimensions of 512 x 512 pixels.

[0107] 19. A system according to any one or combination of 1 to 18 above, wherein the image data is resized before being divided into a plurality of image tiles.

[0108] 20. A method for identifying a material component of an implant tissue product, comprising the steps of capturing image data of the implant tissue product using an image capture device, and using a processor to process the image data using an artificial neural network to identify and classify the material of the implant tissue product from the image data.

[0109] 21. The method according to any one of 20 above or 22-38 below or a combination thereof, wherein capturing image data of the implant tissue product further comprises illuminating the implant tissue product with an ultraviolet light source, filtering the emitted light reflected by the implant tissue product with an optical filter of an image capture device, and capturing the filtered light using an image sensor of the image capture device.

[0110] 22. A method according to any one of 20 to 21 above or 23 to 38 below, or a combination thereof, wherein the ultraviolet light source is configured to emit light having a wavelength of 365 nm to 395 nm.

[0111] 23. A method according to any one of 20 to 22 above or 24 to 38 below, or a combination thereof, wherein the optical filter includes a long-pass filter configured with a cut-on wavelength of 435 nm.

[0112] 24. A method according to any one of 20 to 23 above or 25 to 38 below, or a combination thereof, wherein the optical filter has a transmittance of 85% for wavelengths greater than 435 nm.

[0113] 25. The method according to any one of 20-24 above or 26-38 below or a combination thereof, further comprising using a processor to divide the image data into a plurality of image tiles.

[0114] 26. A method according to any one of 20 to 25 above or 27 to 38 below or a combination thereof, wherein each of the multiple image tiles has the same size.

[0115] 27. A method according to any one or combination of 20-26 above or 28-38 below, wherein the plurality of image tiles includes 35 image tiles.

[0116] 28. The method according to any one of 20-27 above or 29-38 below or a combination thereof, wherein the graft tissue product comprises fish skin having waste material comprising at least one of fascia and flesh.

[0117] 29. A method according to any one of 20 to 28 above or 30 to 38 below, or a combination thereof, wherein the artificial neural network includes a convolutional neural network.

[0118] 30. A method according to any one of 20 to 29 above or 31 to 38 below or a combination thereof, the method further comprising the step of inputting each of the plurality of image tiles into a stepped contracting path of an artificial neural network, each step of the contraction path comprising a first contraction convolutional layer, a second contraction convolutional layer, a first contraction rectification layer following the first contraction convolutional layer, a second contraction rectification layer following the second contraction convolutional layer, a storage operation following the second contraction rectification layer for storing an output, and a pooling layer following the storage operation.

[0119] 31. A method according to any one of 20 to 30 above or 32 to 38 below or a combination thereof, further comprising the step of inputting each of the plurality of image tiles into a symmetric stepped expanding path of an artificial neural network, each step of the stepped expanding path comprising a first stepped convolutional layer, a second stepped convolutional layer, a first stepped rectification layer following the first stepped convolutional layer, a second stepped rectification layer following the second stepped convolutional layer, an upsampling layer following the second stepped rectification layer, and a concatenation operation of stacking the stored output of the stepped contraction path onto the output of the upsampling layer.

[0120] 32. A method according to any one of 20-31 above or 33-38 below or a combination thereof, wherein the stepwise contraction pass and the stepwise expansion pass include the same number of steps.

[0121] 33. A method according to any one of 20-32 above or 34-38 below or a combination thereof, wherein the stepwise contraction pass and the stepwise expansion pass each include six steps.

[0122] 34. A method according to any one of 20 to 33 above or 35 to 38 below or a combination thereof, further comprising inputting each of the plurality of image tiles to an output step of an artificial neural network, the output step including a first output convolutional layer, a second output convolutional layer, a first output rectification layer following the first output convolutional layer, a second output rectification layer following the second output convolutional layer, and a sigmoid layer following the second output rectification layer.

[0123] 35. A method according to any one of 20-34 above or 36-38 below or a combination thereof, further comprising the step of outputting an image defining an area of ​​each material feature of the implant tissue product following the artificial neural network output step.

[0124] 36. A method according to any one of 20 to 35 above or 37 to 38 below, or a combination thereof, wherein the optical filter has a cut-on wavelength of 400 nm to 600 nm.

[0125] 37. A method according to any one of 20 to 36 above or 38 below or a combination thereof, wherein each of the multiple image tiles has equal dimensions of 512 x 512 pixels.

[0126] 38. The method according to any one of 20 to 37 above or a combination thereof, wherein the image data is resized before being divided into a plurality of image tiles.

[0127] 39. A non-transitory hardware storage device having stored thereon computer-executable instructions, which when executed by one or more processors of a computer, configure the computer to perform at least the following: capturing image data of an implant tissue product with an image capture device, and using the processor to process the image data using an artificial neural network to identify and classify material of the implant tissue product from the image data.

[0128] Although a system or method for identifying a material component of an implant tissue product has been disclosed in certain preferred embodiments and examples, it will therefore be understood by those skilled in the art that the present disclosure extends beyond the disclosed embodiments to other alternative embodiments and / or uses of the system or method for identifying a material component of an implant tissue product, as well as obvious modifications and equivalents. It is intended that the scope of the disclosed system or method for identifying a material component of an implant tissue product should not be limited by the above disclosed embodiments, but should be determined solely by a fair reading of the claims which follow.

Claims

1. 1. A system for identifying material components of an implant tissue product, comprising: The system comprises: an image capture device configured to capture image data of the implant product; a processor configured to process the image data using an artificial neural network, the artificial neural network configured to identify and classify a material of the implant product from the image data; and A system comprising:

2. The system of claim 1 , wherein the image capture device comprises an ultraviolet light source, an optical filter, and an image sensor.

3. The system of claim 2 , wherein the ultraviolet light source is configured to emit light having a wavelength between 365 nm and 395 nm.

4. The system of claim 2 , wherein the optical filter comprises a longpass filter configured with a cut-on wavelength of 435 nm.

5. The system of claim 2 , wherein the optical filter has a transmission of 85% for wavelengths greater than 435 nm.

6. The system of claim 1 , wherein the processor is configured to divide the image data into a plurality of image tiles.

7. The system of claim 6 , wherein each of the plurality of image tiles has the same size.

8. The system of claim 1 , wherein the graft tissue product comprises fish skin having waste material including at least one of fascia and flesh.

9. 10. The system of claim 1, wherein the artificial neural network comprises a convolutional neural network.

10. the convolutional neural network comprises a stepwise shrinkage path; Each step of the stepwise shrinkage path comprises: a first deflated convolution layer; and a second deflated convolution layer; and a first contraction rectification layer following the first contraction convolution layer; a second contraction rectification layer following the second contraction convolution layer; a storage operation following the second contraction rectification layer to store the output; a pooling layer following the storage operation; The system of claim 9 , comprising:

11. the convolutional neural network comprises a gradual expansion path; Each step of the gradual expansion path comprises: a first dilated convolutional layer; and a second dilated convolutional layer; and a first dilated rectification layer following the first dilated convolutional layer; a second dilated rectification layer following the second dilated convolutional layer; an upsampling layer following the second enhanced rectification layer; a concatenation operation for stacking the stored outputs of the stepwise erosion paths onto the outputs of the upsampling layer; The system of claim 10 , comprising:

12. The system of claim 11 , wherein the stepwise contraction path and the stepwise expansion path include a same number of steps.

13. The system of claim 11 , wherein the stepwise contraction path and the stepwise expansion path each include six steps.

14. The output step is a first output convolutional layer; and a second output convolutional layer; and a first output rectification layer following the first output convolution layer; a second output rectification layer following the second output convolution layer; a sigmoid layer following the second output rectifying layer; Including, The system of claim 1 .

15. 10. The system of claim 9, wherein the convolutional neural network is configured to output an image defining an area of ​​each material feature of the implant tissue product.

16. The system of claim 2 , wherein the optical filter comprises a cut-on wavelength between 400 nm and 600 nm.

17. The system of claim 6 , wherein the processor is configured to resize the image data prior to dividing the image data into the plurality of image tiles.

18. 1. A method for identifying material components of an implant tissue product, comprising: The method comprises: capturing image data of the implant product using an image capture device; and using a processor to process the image data using an artificial neural network to identify and classify the material of the implant product from the image data; A method comprising the steps of:

19. Capturing the image data of the implant tissue product includes: illuminating the implant product with an ultraviolet light source; filtering the emitted light reflected by the implant product with an optical filter of the image capture device; capturing the filtered light using an image sensor of the image capture device; The method according to any one of 20 above or 22 to 38 below, or a combination thereof, further comprising:

20. When executed by one or more processors of a computer, capturing image data of the implant product with an image capture device; processing the image data using an artificial neural network to identify and classify the implant product material from the image data; A non-transitory hardware storage device having stored computer-executable instructions that configure a computer to perform at least the steps of:

Images