Apparatus and method for sorting photographs of gel card reactions

The apparatus uses multiple neural networks to classify gel card reactions, addressing the need for frequent recalibration and improving atypical reaction recognition in image-based systems.

JP7584435B2Active Publication Date: 2024-11-15BIO RAD EURO GMBH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2021558515
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-28
Filing Date
2020-03-26
Publication Date
2024-11-15
Estimated Expiration
2040-03-26

AI Technical Summary

Technical Problem

Existing image-based recognition devices for classifying blood agglutination reactions in gel cards require frequent recalibration due to variations in camera type, gel card transparency, or light source, and struggle with atypical reactions.

Method used

An apparatus utilizing multiple neural networks to analyze photographs of gel card reactions, where a first neural network provides initial probabilities for reaction categories, and a second network classifies atypical reactions after determining a predetermined portion of the image, with a screening stage to filter unsuitable inputs.

Benefits of technology

The system reduces the need for frequent recalibration and improves the recognition of atypical reactions, maintaining accuracy despite variations in imaging conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584435000001
    Figure 0007584435000001
  • Figure 0007584435000002
    Figure 0007584435000002
  • Figure 0007584435000003
    Figure 0007584435000003
Patent Text Reader

Abstract

An apparatus for classifying photographs of reactions of reactants in predetermined containers is disclosed, the apparatus including: a first neural network arranged to receive input photographs of reactions of reactants in predetermined containers and to provide, for each reaction category of a first plurality of reaction categories, a probability that the input photograph depicts a reaction belonging to that reaction category; and a second neural network arranged to receive a predetermined portion of the input photograph and to provide, for each reaction category of a second plurality of reaction categories, a probability that the predetermined portion of the input photograph depicts a reaction belonging to that reaction category, if the first neural response provides the highest probability that the input photograph depicts a reaction belonging to a certain predetermined reaction category. TIFF2022527504000002.tif76128
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Technical Field This presentation relates to an apparatus and method that uses multiple neural networks to classify photographs, and more specifically, to classify photographs of agglutination reactions, such as agglutination reactions occurring in gel cards. [Background technology]

[0002] background One known format for testing blood, e.g., to determine the blood type of a blood sample, includes the steps of introducing the blood sample into a vessel filled with a separation matrix, e.g., a gel, such as a dextran acrylamide gel, overlaid with a fluid that may contain a reagent (a reagent capable of forming a complex with an analyte to be detected in the blood sample) and / or a reticulation agent (a reticulation agent capable of cross-linking sensitized red blood cells or sensitized latex particles), such as a secondary antibody (e.g., an anti-human globulin antibody or an anti-human IgA antibody); incubating the vessel; and then centrifuging the vessel; and the position of the agglutinated red blood cells within the gel provides information regarding the blood type of the tested blood sample.

[0003] FIG. 1 is a side view of an exemplary known "gel card" 10; the gel card 10 is made of a transparent plastic plate 11 that holds together a number of transparent plastic containers or "wells" 12, each containing a quantity of gel 14 and a fluid that may contain a reticulating agent, and optionally also a control well 16 that contains gel and a fluid that does not contain a reticulating agent. The gel card is used by introducing a drop of blood sample (and reagents, if necessary) into each well, incubating the card, and then centrifuging the card so that it is accelerated toward the bottom of the wells of the card. FIG. 1 illustrates an exemplary gel card in which red blood cells have clumped and remained at the top in the first (RH2; C), third (RH1; D), and fifth (RH5; e) wells, and have not clumped and sunk to the bottom in the second (RH4; c), fourth (RH3; E), and sixth (control) wells.

[0004] 2 is a detailed elevational view of well 12 of card 10; well 12 includes a wide incubation chamber 18 at the top for receiving the blood sample (and reagents, if necessary) to be tested; a narrow well 20 at the bottom, connected to chamber 18 by a tapered lumen and having a tapered lower end; and a bottom 22 (or reaction chamber) of well 12, filled with a separating matrix, e.g., gel 24, which may contain a reticulation agent. FIG. 2 also shows blood sample 26 reacting with reagents 28 in fluid 30 in chamber 18 before centrifugation of well 12 pushes 32 the reacted or unreacted blood sample toward bottom 22. vinegar .

[0005] In the past, a human operator was responsible for viewing the wells 12 after centrifugation and determining whether a reaction in the well was positive or negative. However, to increase efficiency, several systems have been developed to automatically determine whether a reaction in the well is positive or negative. Known systems generate a photograph of the well and then use image processing software to make the image contain recognizable features; the recognizable features are then used to determine whether a reaction in the well is positive or negative.

[0006] One problem with image-based recognition devices is that when one component of the device is changed, such as the type of camera used to generate the photograph, the transparency or color of the gel card, or the type of light used to generate the photograph, the device software must be recalibrated to account for the change in the photograph caused by the change in that component. Additionally, image-based recognition devices have difficulty recognizing atypical responses as positive or negative.

[0007] 3A-3F show photographs of wells, each with an atypical response that is generally not recognized by image-processing based recognition devices.

[0008] What is needed is an apparatus and / or method that allows for automatic recognition of atypical reaction pictures, such as blood agglutination pictures, in addition to known automatic recognition apparatus and methods; and an apparatus and / or method that does not require recalibration as frequently as known recognition apparatus and / or methods. Aspects of the present presentation address these and other needs. Summary of the Invention

[0009] overview One embodiment of the present presentation includes an apparatus for classifying photographs; the apparatus receives as separate input each pixel of each color of an input photograph, and in return classifies the pixel according to whether the photograph exhibits a first predetermined response sequence. vinegar a first neural network that outputs a probability; and receives as separate inputs each pixel of each color in a predetermined portion of an input photograph, and in return, determines if the probability returned by the first neural network for a predetermined response exceeds a predetermined level, that the photograph exhibits a second predetermined response sequence. vinegar A second neural network outputs a probability.

[0010] One aspect of the present presentation relates to an apparatus for classifying a photograph of a reaction of reactants in a predefined container; the apparatus receives an input photograph of the reaction of reactants in a predefined container, and for each reaction category of a first plurality of reaction categories: The input photo The first multiple reaction category indicates the reaction that belongs to that reaction category. vinegar a first neural network arranged to provide probabilities; The input photo The first category indicates reactions that belong to a predefined reaction category. vinegar If the first neural response provides the highest probability (i.e., the highest of the probabilities generated by the first neural network is related to a predefined response category of the first multiple response category), receive the predefined portion of the input photograph, and for each response category of the second multiple response category: The default part of the input photo The second multiple response category indicates the response that belongs to that response category. vinegarand a second neural network arranged to provide the probabilities.

[0011] In one embodiment of the present presentation, the device is arranged to issue a control message indicating that the input photograph does not belong to the first multiple response category if the highest probability issued by the first neural network is below a predefined threshold.

[0012] In one embodiment of the present presentation, the apparatus includes a decision tree arranged to classify the input photograph as depicting a response within the response category having the highest probability output by the first neural network if the second neural network is not activated; and arranged to classify the input photograph as depicting a response within the response category having the highest probability output by the second neural network if the second neural network is activated.

[0013] In one embodiment of the present presentation, the apparatus includes a screening stage having a third neural network; the third neural network is arranged to: determine whether an input photograph is suitable for analysis by the first neural network, and if so, the input photograph is provided to the first neural network; or determine whether the input photograph is not suitable for analysis by the first neural network, and if so, the input photograph is not provided to the first neural network. In one embodiment of the present presentation, an input photograph may be determined to be "unsuitable" if it is recognized by the third network as exhibiting a type of response defined as "non-suitable" for the third network, or "suitable" otherwise, or vice versa.

[0014] In one embodiment of the present disclosure, the third neural network is adapted to generate a plurality of quantifications of the reaction of reactants in a given container. Known It is a neural network that is trained in a supervised mode by classifying each input photo as either good or bad.

[0015] In one embodiment of the present presentation, the screening stage includes a filter arranged to receive an input photograph; generate an information-reduced photograph based on the input photograph; and input the information-reduced photograph to a third neural network.

[0016] In one embodiment of the present presentation, the input photograph is a color photograph, and the filters are arranged to convert the color photograph into a black-and-white photograph; the third neural network includes as many single pixel inputs as there are pixels in the black-and-white photograph, and each pixel of the black-and-white photograph is connected to a pixel input of the first neural network.

[0017] In one embodiment of the present presentation, the input photograph is a combination of a red input photograph, a green input photograph, and a blue input photograph; the first neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs, and each pixel of the red, green, and blue input photographs is connected to a pixel input of the first neural network.

[0018] In one embodiment of the present presentation, each single pixel input of the first neural network includes a predetermined number of single bit inputs.

[0019] In one embodiment of the present presentation, the second neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs in a given portion of the input photograph.

[0020] In one embodiment of the present presentation, each single pixel input of the second neural network includes a predetermined number of single bit inputs.

[0021] In one embodiment of the present disclosure, the first neural network comprises a first plurality of reaction categories, the first plurality of reaction categories being a plurality of reaction categories of reactants in a predetermined container. KnownIt is a neural network that is trained in a supervised mode by associating each of the input photographs.

[0022] In one embodiment of the present disclosure, Known the container is a well of a gel card, and Known The photograph and input photograph are side-on photographs of the reactants in the wells of the gel card.

[0023] In one embodiment of the present disclosure, the plurality of Known The photographs include a set of original photographs, each of which is a photograph of a different reaction of a reactant in a given container; and at least one of a color-modified original photograph set, a mirror image of the original photograph set, a rotated original photograph set with a given container position, a brightness-modified original photograph set, a contrast-modified original photograph set, a blurred original photograph set, a cropped original photograph set, a translated original photograph set with a given container position, a scaled original photograph set, and an original photograph set with noise added.

[0024] In one embodiment of the present presentation, the second neural network is configured to generate a second plurality of response categories for at least one of the second plurality of response categories. Known The network is trained in a supervised mode by associating a predefined portion of each of a subset of photos.

[0025] In one embodiment of the present disclosure, Known the container is a well of a gel card, and Known The photograph and input photograph are side-on photographs of the reactants in the wells of the gel card; the predetermined portion of the photograph is the portion of the photograph that contains the bottom of the well.

[0026] Another aspect of the present presentation includes a device for centrifugation and inspection of containers (which may be gel cards), comprising: a centrifugation apparatus arranged to centrifuge at least one container; an imaging device for generating images of the containers after centrifugation (e.g. images of each well of a gel card after centrifugation); and an apparatus for classifying pictures of reactions of reactants in given containers as detailed above, the apparatus arranged to receive as input each image of the container after centrifugation.

[0027] Another aspect of the present presentation relates to a method of programming an apparatus for classifying pictures of reactions of reactants in predefined containers as detailed above, the method comprising the steps of: a first Known A series of photographs and the first Known Each of the photos Known training a first neural network in a supervised mode with a sequence of response categories; grouping the at least two response categories as a default category if the at least two response categories are not distinguished from each other by the first neural network; and Known The second category has a reaction Known Select a photo in the photo series (e.g. the first Known training a second neural network in a supervised mode with selected photos from the sequence of photos, the selected photos being photos cropped to a predetermined portion of the input photos, the predetermined portion of the input photos being an area that highlights at least one difference between the at least two response categories.

[0028] Another aspect of the present presentation relates to a method for classifying a photograph of a reaction of reactants in a predefined container, the method comprising the steps of: inputting an input photograph of the reaction of reactants in a predefined container into a first neural network, the first neural network selecting, for each reaction category of a first plurality of reaction categories: The input photo The first multiple reaction category indicates the reaction that belongs to that reaction category. vinegararranged to provide a probability; and The input photo The first category indicates reactions that belong to a predefined reaction category. vinegar If the first neural response provides the highest probability, inputting the predetermined portion of the input photograph into a second neural network, the second neural network performing, for each response category of the second plurality of response categories: The default part of the input photo The second multiple response category indicates the response that belongs to that response category. vinegar Arranged to provide a probability, stages.

[0029] In one embodiment of the present presentation, the method includes issuing a control message indicating that the input photograph does not belong to the first multiple response category if the highest probability issued by the first neural network is below a predefined threshold.

[0030] In one embodiment of the present presentation, the method includes the steps of: if the second neural network is not activated, using a first decision tree to classify the input photograph as exhibiting a response within the response category output by the first neural network with the highest probability; and, if the second neural network is activated, using a second decision tree to classify the input photograph as exhibiting a response within the response category output by the second neural network with the highest probability.

[0031] In one embodiment of the present presentation, the method includes the steps of: prior to the step of inputting the input photograph into the first neural network, inputting the input photograph into a third neural network arranged to determine whether the input photograph is suitable for analysis by the first neural network; and performing the step of inputting the input photograph into the first neural network if the third neural network determines that the input photograph is suitable for analysis by the first neural network; or, alternatively, not inputting the input photograph into the first neural network if the third neural network determines that the input photograph is not suitable for analysis by the first neural network.

[0032] In one embodiment of the present disclosure, the third neural network is adapted to generate a plurality of quantifications of the reaction of reactants in a given container. Known It is a neural network that is trained in a supervised mode by classifying each input photo as either good or bad.

[0033] In one embodiment of the present presentation, the method includes filtering the photographic input in the third neural network to reduce information contained in the photographic input to the third neural network.

[0034] In one embodiment of the present presentation, the input photograph is a color photograph, and the filtering step includes converting the color photograph to a black and white photograph.

[0035] In one embodiment of the present presentation, the input photograph is a combination of a red input photograph, a green input photograph, and a blue input photograph, and the first neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs; inputting the input photograph into the first neural network includes inputting each pixel of the red, green, and blue input photographs as a different pixel input of the first neural network.

[0036] In one embodiment of the present presentation, the second neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs of a predetermined portion of the input photograph; inputting the predetermined portion of the input photograph to the second neural network includes inputting each pixel of the red, green, and blue predetermined portions of the input photograph as a different pixel input of the second neural network.

[0037] In one embodiment of the present presentation, the method comprises: adding a plurality of reaction categories of reactants in a predetermined container to at least one reaction category of a first plurality of reaction categories; Known The method includes training the first neural network in a supervised mode by associating each of the input photographs.

[0038] In one embodiment of the present disclosure, Known the container is a well of a gel card, and Known The photograph and input photograph are side-on photographs of the reactants in the wells of the gel card.

[0039] In one embodiment of the present disclosure, the method comprises: generating a plurality of images from a set of original photographs, each image being a photograph of a different reaction of a reactant in a given container; Known forming a photograph; and forming at least one of a color-altered original photo set, a mirror image of the original photo set, a rotated original photo set with a predetermined container position, a brightness-altered original photo set, a contrast-altered original photo set, a blurred original photo set, a cropped original photo set, a translated original photo set with a predetermined container position, a scaled original photo set, and an original photo set with added noise.

[0040] In one embodiment of the present presentation, the method further comprises: Known Training the second neural network in a supervised mode by associating predetermined portions of each of the subsets of photographs.

[0041] In one embodiment of the present disclosure, Known the container is a well of a gel card, and Known The photograph and input photograph are side-on photographs of the reactants in the wells of the gel card; the predetermined portion of the photograph is the portion of the photograph that contains the bottom of the well.

[0042] These and other features and advantages will become more apparent from the following detailed description and the accompanying drawings, in which numerals indicate various features and like numerals refer to like features throughout the drawings and description. [Brief description of the drawings]

[0043] [Figure 1] A known gel card is shown. [Diagram 2] 2 shows a detail of the wells of the gel card of FIG. 1. [Diagram 3] 3A-3F show wells of a gel card with atypical reactions. [Figure 4] 1 illustrates a schematic diagram of an apparatus according to aspects of the present disclosure; [Diagram 5] 1 illustrates a schematic diagram of a neural network according to aspects of the present presentation. [Figure 6] A detailed diagram of how input photos are processed to train various neural networks. [Figure 7] Illustrates an overview of how input photos are processed to train various neural networks. [Figure 8] We show photos that must be processed in this way to be recognized by known photo recognition devices, and photos that can be recognized by a device based on the present proposal. [Figure 9] 1 is a schematic diagram of a gel card centrifugation and testing device according to one embodiment of the present disclosure.

[0044] Those skilled in the art will appreciate that the accompanying drawings are not necessarily to scale, are presented for illustrative purposes only and should not be used to limit the scope of the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] Detailed Description In the following description, numerous specific details are set forth in order to clearly explain the various embodiments disclosed herein. However, it will be understood by those skilled in the art that the claimed invention may be practiced without all of the specific details described below. In other instances, well-known features have not been described so as not to obscure the claimed invention.

[0046] 4 shows a schematic diagram of a device 40 according to one embodiment of the present presentation for classifying a photograph 42 of a reaction of a reactant in a given container, for example a reaction between a blood sample and an agglutination reactant in a well of a gel card, after centrifugation. In one embodiment of the present presentation, the device 40 receives an input photograph 42 and in return, as will be described in more detail below: Input photo 42 Reactions belonging to each of the first multiple reaction categories R1 to Rn show The image sensor 40 includes a first neural network 44 arranged to provide probabilities X1-Xn. As shown in FIG. 4 and described in detail below, the image 42 may be a color image and may be composed of a combination of one red image R, one blue image B, and one green image G. In one embodiment of the present presentation, the neural network 44 is arranged such that each probability X1-Xn is between 0 and 100% and the sum of X1-Xn is 100. In one embodiment, the neural network 44 may also be arranged to output an overall confidence of the probabilities X1-Xn. In one embodiment of the present presentation, the values ​​X1-Xn may be stored in columns 46 of memory buffers R1-Rn.

[0047] In one embodiment of the present presentation, the device 40 includes a first decision circuit 48 arranged to determine whether further analysis of the photograph 42 is required. As will be described in more detail below, the neural network 44 is arranged such that at least one response category Rm among the response categories R1-Rn includes a plurality of response categories that are difficult to distinguish by the neural network 44. In one embodiment of the present presentation, the decision circuit 48 is Photo 42 Reactions in reaction category Rm are shown. vinegar The neural network 50 is arranged to receive all of the values ​​X1 to Xn, including the likelihood value Xm, and to send the photograph 42 to a second neural network 50 (e.g. via a controlled switch 49 as shown in Figure 4) if the value Xm is the maximum of the values ​​X1 to Xn.

[0048] In one embodiment of the present presentation, a segmentation circuit 52 is arranged to receive the photograph 42 sent by the decision circuit 48 to the second neural network 50, extract a predetermined portion 42' from the photograph 42, and input the predetermined portion 42' to the second neural network 50. In one embodiment of the present presentation, the segmentation circuit 52 is arranged to segment only the lower portion of the photograph 42, as illustrated in FIG.

[0049] In one embodiment of the present presentation, and as will be described in more detail below, the second neural network 50 receives a portion 42' of the input photograph 42 and, in return, Part 42' Reactions belonging to one of the second multiple reaction categories Rn+1 to Rn+k are indicated. vinegar In one embodiment of the present presentation, the values ​​Xn+1 to Xn+k may be stored in a column 51 of memory buffers Rn+1 to Rn+k. In one embodiment of the present presentation, each probability Xn+1 to Xn+k is between 0 and 100%, and the sum of Xn+1 to Xn+k is 100.

[0050] In one embodiment of the present presentation, the device 40 includes a second decision circuit 53 arranged to classify the input photograph 42 as depicting a response in one of the categories R1 to Rn having the highest probability among the probabilities X1 to Xn output by the first neural network 44, if the second neural network 50 is not activated by the first decision circuit 48; and to classify the input photograph 42 as depicting a response in one of the categories Rn+1 to Rn+k having the highest probability among the probabilities Xn+1 to Xn+k output by the second neural network 50, if the second neural network 50 is activated by the first decision circuit 48. In one embodiment of the present presentation, the control signal 55 generated by the decision circuit 48 for controlling the switch 49 may also be used to control the second decision 53 not to take into account the values ​​X1 to Xn if the second neural network 50 is not activated.

[0051] In one embodiment of the present presentation, if the highest probability that the first neural response provided is below a predefined threshold, a control message such as signal 59 may be sent to the second decision circuit 53; the second decision circuit 53 then determines, for example, that the input photo 42 does not indicate any of the responses R1 to Rn+k. difference The device may be arranged to output a message indicating that the device is likely not to be able to receive the data.

[0052] In one embodiment of the present presentation, the device 40 includes a screening stage 54 including a filter 56 arranged to receive the input photograph 42 and in return generate an information-reduced photograph 42". In one embodiment of the present presentation where the photograph 42 is a color photograph, the information-reduced photograph 42" is a black and white photograph. In one embodiment of the present presentation, and as will be described in more detail below, the screening stage 54 further includes a third neural network 58 arranged to determine whether the input photograph 42 is suitable for analysis by the first neural network 44. In one embodiment of the present presentation, if the neural network 58 determines that the input photograph 42 is suitable for analysis by the first neural network 44, the input photograph 42 is input to the first neural network 44, such as by using a controlled switch 60. On the other hand, if the neural network 58 determines that the input photograph 42 is unsuitable for analysis by the first neural network 44, then a control message, such as signal 59, may be sent to the second decision circuit 53; the second decision circuit 53 may, for example, determine that the input photograph 42 does not exhibit any of the responses R1 through Rn+k. difference The device may be arranged to output a message indicating that the device is likely not to be able to receive the data.

[0053] It is noted that FIG. 4 shows a single decision circuit 48 arranged to monitor whether the value Xm is the largest of the values ​​X1 to Xn, and finally to cause the second neural network 50 to receive and examine a predetermined portion 42' of the input photograph 42. vinegar However, the embodiments of the present presentation may also have more than one decision circuit 48, each arranged to monitor which distinct probability Xi (each associated with a different response category Ri) is the largest among the values ​​X1-Xn, and ultimately to cause different predefined portions 42' of the input photograph 42 to be received and examined by different second neural networks 50. For example, a first predefined portion 42' of the input photograph 42 may be the bottom of the photograph as detailed above, while a second predefined portion 42' of the input photograph 42 may be the middle of the center of the photograph.

[0054] 5 illustrates a schematic diagram of one embodiment of the first neural network 44 of the device 40 and how it accepts an input photograph 42. In the illustrated embodiment, the neural network 44 includes an input layer with as many pixel input neurons 62 as there are pixels in all of the red, blue, and green images R, B, and G that form the image 42. Thus, for example, if each of the images R, G, and B includes 50 x 100 = 5000 pixels, the neural network 44 includes 15.000 pixel input neurons. Throughout this presentation, the term "neuron" refers to an artificial neuron circuit implemented as either dedicated hardware, a mixture of dedicated hardware and software, or software running on standard programmable hardware such as a computer.

[0055] In one embodiment of the present presentation, each pixel input neuron 62 includes as many single-bit neurons 62' as there are coded bits in one pixel. In the example illustrated in FIG. 5, each pixel of each of the R, G, B pictures forming the picture 42 is coded on 8 bits, and each pixel input neuron 62 includes eight single-bit neurons 62' arranged to receive a binary input on their respective dendrites. Alternatively, each pixel input neuron 62 may be arranged to receive on its dendrites a single 8-bit coded pixel signal that may have a value between 0 and 255. In one embodiment of the present presentation, the axon of each pixel input neuron 62 of the neural network 44 is connected to a hidden neuron layer 64, which is itself connected to an output neuron layer made up of output neurons 66; the axon of each neuron of the output layer 66 is connected to the input of one of the memory registers R1 to Rn.

[0056] In one embodiment of the present presentation, the neural network 44 comprises a plurality of Known The photos are continuously input to the neural network 44, and each time, KnownThe neural network is trained in a supervised mode by associating the photographs with one of the reaction categories R1-Rn. In one embodiment of the present presentation, each photograph is a side view of a well of a gel card after incubation of the reaction and after centrifugation. In one embodiment of the present presentation, the step of teaching the neural network is performed by an experienced operator who selects each input Known Recognizing a photo as belonging to a response category and determining whether that category is included in the input in network 44. Known However, as will be described in more detail below, in another aspect of the present disclosure, the step of teaching the neural network requires only that a skilled operator Known It may be that only a subset of a set of pictures is recognized as belonging to one response category; e.g. Known The rest of the photo set is Known For example, they may be generated from a subset of photos by modifying the appearance of the photos (tilting them, changing their color, etc.) so that the remaining photos have a known category despite their modified appearance.

[0057] FIG. 6 illustrates a number of neural networks 44 that may be used to train the neural network 44 in one embodiment of the present disclosure. KnownIllustrated is a method of creating a photograph. An original photograph 70 is shown. Photograph 70 may be captured by a camera, such as the camera used to generate photograph 42. In one embodiment of the present presentation, a further photograph 72 is formed from photograph 70 by horizontally flipping (mirroring) photograph 70. In one embodiment of the present presentation, a further photograph 74 is formed from photograph 70 or 72 by darkening the photograph and vertically shifting the position of the well. In one embodiment of the present presentation, a further photograph 76 is formed from photograph 70 or 72 by vertically shifting the photograph of the well and rotating the photograph of the well by a few degrees. In one embodiment of the present presentation, a further photograph 78 is formed from photograph 70 or 72 by darkening the photograph and vertically and horizontally shifting the position of the well. In one embodiment of the present presentation, a further photograph 80 is formed from photograph 70 or 72 by darkening the photograph, vertically shifting the photograph of the well and rotating the photograph of the well by a few degrees. In one embodiment of this presentation, a further photograph 82 is formed from photograph 70 or 72 by making the photograph darker and by vertically moving the photograph of the well. In one embodiment of this presentation, a further photograph 84 is formed from photograph 70 or 72 by changing the color of the photograph, rotating the photograph of the well by a few degrees, and by vertically moving the photograph of the well in a manner different from the previous photograph. In one embodiment of this presentation, a further photograph 86 is formed from photograph 70 or 72 by making the photograph darker, moving the photograph of the well vertically, and by vertically rotating the photograph of the well by a few degrees in a manner different from the previous photograph.

[0058] In one embodiment of the present presentation, a plurality of neural networks 44 are used to train the neural network 44. Known The photos start with a set of original photos such as photo 70, and then move on to photos 72-86, etc. Known formed by creating additional sets of photos; KnownThe additional set of photographs includes at least one of a color-modified version of the original photo set, a mirror image of the original photo set, a rotated version of the original photo set with a predetermined container position, a brightness-modified version of the original photo set, a contrast-modified version of the original photo set, a blurred version of the original photo set, a cropped version of the original photo set, a translation version of the original photo set with a predetermined container position, a scaled version of the original photo set, and an original photo set with noise added.

[0059] In one embodiment of the present disclosure, Known The additional set of photos may be automatically created by randomly combining any of the photo modifications detailed above.

[0060] In one embodiment of the present presentation, since each original photograph exhibits the same response as each additional photograph generated from that original photograph, neural network 44 may be automatically fed all additional photographs generated from each original photograph, and the response of each additional photograph may be automatically taught to be the same as the response taught for that original photograph (the response identified by the operator). Known Reduced size, individually associated with response categories Known From the photo set: Known Large responses associated with each response category Known In one embodiment of the present disclosure, the photo set can be generated automatically. Known Large responses associated with each response category Known Once the photo set has been generated, the neural network 44 Known Photo sets and their Known The response categories may be automatically trained.

[0061] In one embodiment of the present presentation, once the neural network 44 has been trained, a number of unknown photographs are fed into the neural network 44, and a trained operator verifies that each unknown photograph has been correctly identified as exhibiting the appropriate response category (by issuing a probability value Xj associated with the appropriate response category Rj that is greater than the probability values ​​issued for the other response categories). If an unknown photograph exhibiting photographs of two distinct categories is miscategorized by the neural network 44, then the neural network 44 is re-evaluated as described in detail above. Known Network 44 may be additionally trained with additional sets of photos. However, if neural network 44 continues to mischaracterize at least two of the response categories, it may collapse the two response categories into a single response category Rm; then, neural network 44 will determine that input photo 42 belongs to response category Rm. high and issuing a probability Xm that indicates at least one difference between the at least two categories. vinegar , triggering a detailed analysis by the neural network 50 of a given detail / portion 42' of the input photo.

[0062] For example, reaction category Rm may be a grouping of multiple negative reaction categories that appear distinct to a trained operator but are not well differentiated by neural network 44. And the predetermined portion 42' of the input photograph may be the bottom portion of the input photograph, including, for example, the bottom of the well where at least one difference between the multiple negative reaction categories is found.

[0063] 5 illustrates not only the first neural network 44 but also the second neural network 50, where reference numerals 44 and 42 are replaced by reference numerals 50 and 42', and, consistent with the above description of neural network 44, neural network 50 is considered to include an input neuron layer having as many pixel input neurons as there are pixels in the three portions of the red, blue, and green photo forming given photo portion 42'. Also consistent with the above description of neural network 44, the input neuron layer of neural network 50 is connected to a hidden neuron layer, which is itself connected to an output neuron layer; the axon of each output neuron of the output neuron layer is connected to one of memory registers Rn+1 to Rn+k (instead of R1 to Rn as shown).

[0064] Neural network 50 can be taught in a similar manner to neural network 44, but using (at least) two response categories that are not adequately distinguished by neural network 44 and are therefore taught to neural network 44 as a single response category Rm. Known Selected by the operator as a photo Known It's only a part of the photo set.

[0065] Alternatively, in one embodiment of the present presentation, the neural network 44 need not identify the (at least) two indistinguishable response categories as a single response category Rm, but in that case the decision circuit 48 must be arranged to activate the neural network 50 if the output of the neural network 44 indicates that the input photograph belongs to one of the (at least) two indistinguishable response categories (the highest probability is output).

[0066] Similar to the teaching of neural network 44, the neural network 50 may be taught Known Photo set, and corresponding KnownThe set of reaction categories is Known A smaller set of photographs may be generated automatically by randomly varying their appearance. Similar to teaching neural network 44, the neural network 50 may also be trained using the Known Photo set and corresponding Known The set of reaction categories is Known Once automatically generated from the smaller set of photographs, they may be automatically fed to train the neural network 50.

[0067] FIG. 5 illustrates not only the first neural network 44 but also a third neural network 58, substituting the reference numerals 44 and 42 for the reference numerals 58 and 42", and considering neural network 58 to include an input neuron layer having as many pixel input neurons as there are pixels in black and white portion 42", consistent with the above description of neural network 44. Also consistent with the above description of neural network 44, the input neuron layer of neural network 58 is connected to a hidden neuron layer, which is itself connected to a single output neuron; the axon of the output neuron is arranged to provide a signal 59 identifying photograph 42 as suitable or unsuitable for analysis by first neural network 44.

[0068] Neural network 58 can be trained similarly to neural network 44, but using a set of neural networks each with a single "fit for analysis" or "unfit for analysis" label. Known Only a set of black and white photographs. Known Black and white photo set Known In one embodiment of the present disclosure, Known The color photograph set includes a number of original photographs that were recognized as showing identifiable responses; as well as, e.g. differenceThe set of photos may include a number of raw photos that are recognized as not showing an identifiable response, such as a photo with air bubbles as in FIG. 3B or a photo with an opaque response as in FIG. 3C. Each photo may be manually associated by an operator with a label of "suitable for analysis" or "unsuitable for analysis." Alternatively, "unsuitable for analysis" photos may be automatically or manually selected from a set of photos that are consistently misanalyzed by the first and second neural networks after training, while "suitable for analysis" photos may be automatically or manually selected from a set of photos that are consistently properly analyzed by the first and second neural networks after training.

[0069] Next, a set of original black and white photographs (suitable and unsuitable) Known The set of photographs may be generated automatically and then used to train the neural network 58, for example by randomly introducing small changes to the appearance of the original black and white photographs. Known Each has a label Known A set of black and white photos may be generated automatically.

[0070] Alternatively, each photo may have a suitable / unsuitable label, for example by randomly introducing small changes to the appearance of the original photo as detailed above. Known A set of color photographs (good and bad) Known It may be automatically generated from a set of photographs, and Known A set of color photos may be fed into a neural network 58 in a supervised mode through a filter 56 that converts each color photo to a black and white photo in real time.

[0071] Figure 7 shows the results of training the neural network 58. Known A set of 90 black and white photos for training neural networks; Known A set of 92 color photographs; for training neural networks; KnownIt illustrates that a set of partial color photographs 94 can be generated from a single set 96 of original color photographs.

[0072] FIG. 8 shows a photograph 98 of a well of a blood agglutination gel card after centrifugation as it must look in order to be classifiable by known photo recognition devices; and a photograph 100 of a well of a blood agglutination gel card after centrifugation as it may look in order to be classifiable by a recognition device based on the present teachings.

[0073] FIG. 9 illustrates a gel card centrifugation and testing device 102 according to one embodiment of the present disclosure, including a centrifugation apparatus 104 arranged to centrifuge at least one gel card 106; an imaging device, such as a digital camera 108, for generating an image 42 of each well of the gel card 106 after centrifugation; and an apparatus 110 as shown in FIG. 4 for receiving as input each image 42 of the gel card after centrifugation and classifying it as indicative of one of the reaction categories R1 to Rm-1, Rm+1 to Rn, or Rn+1 to Rn+k, or as indicative of an unidentifiable reaction.

[0074] It is noted that the apparatus 40 as shown in FIG. 4 is robust with respect to changes that may occur and that may alter the image 42. For example, changes in the optics that affect the color, sharpness, or brightness of the photograph 42 will not cause the apparatus 40 to misclassify the photograph 42. Similarly, accidental tilting of the gel card when the photograph 42 is taken will not cause the apparatus 40 to misclassify the photograph 42. Additionally, changes in the camera 108 that result in an increase in the number of pixels in the image 42 can be accommodated by cropping the new image to match the original number of pixels so that the wells remain part of the cropped photograph. Similarly, changes in the camera that result in a decrease in the number of pixels in the image 42 can be accommodated by stretching the new image to match the original number of pixels.

[0075] The present invention has been described in accordance with the requirements of the patent statutes; those skilled in the art will recognize how to make modifications and variations to the present invention to meet their particular requirements or conditions. Such modifications and variations can be made without departing from the scope and spirit of the invention disclosed herein.

[0076] The foregoing detailed description of exemplary and preferred embodiments is presented for purposes of illustration and disclosure in accordance with the requirements of the law. The foregoing detailed description is not intended to be exhaustive or to limit the invention to the precise form described, but is intended only to enable one skilled in the art to understand how the invention may be adapted to a particular application or embodiment. Possible modifications and variations will be apparent to those skilled in the art. The description of exemplary embodiments may include tolerances, feature dimensions, specific operating conditions, or engineering specifications, and the like, and may vary from embodiment to embodiment or as the state of the art changes, but no limitation is intended and should be implied therefrom. Applicant has made this presentation with respect to the current state of the art, but also anticipates advances, and anticipates that future adaptations may take those advances into account based on the state of the art. The scope of the invention is defined by the claims as written, and equivalents where appropriate. References to elements of the claims in the singular are not intended to mean "one and only one" unless expressly stated so. Furthermore, no element, component, or method or process step in this presentation is intended to be made available to the public, regardless of whether that element, component, or step is expressly recited in the claims. No element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited with the phrase "means for ...," and no method or process step herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the step or steps are expressly recited with the phrase "comprising the step(s) of ...."

Claims

1. a first neural network arranged to receive input photographs of reactions of reactants in predefined containers and provide, for each reaction category of a first plurality of reaction categories, a probability that the input photograph represents a reaction belonging to that reaction category of the first plurality of reaction categories; a second neural network arranged to receive a predetermined portion of the input photograph and provide, for each response category of a second plurality of response categories, a probability that the predetermined portion of the input photograph exhibits a response belonging to that response category of the second plurality of response categories if the first neural response provides a highest probability that the input photograph exhibits a response belonging to a predetermined response category of the first plurality of response categories; 1. An apparatus for classifying a picture of a reaction of reactants in a predetermined container, comprising:

2. 2. The apparatus of claim 1, further arranged to issue a control message indicating that the input photograph does not belong to the first multiple response category if the highest probability issued by the first neural network is below a predetermined threshold.

3. if the second neural network is not activated, arranged to classify the input photograph as depicting a response within the response category having the highest probability output by the first neural network; and When the second neural network is activated, it is arranged to classify the input photograph as depicting a response within a response category having a highest probability output by the second neural network. decision tree 2. The apparatus of claim 1, comprising:

4. arranged to determine whether an input photograph is suitable for analysis by a first neural network, and if so, providing the input photograph to the first neural network; or and determining whether the input photograph is unsuitable for analysis by the first neural network, and if unsuitable, the input photograph is not provided to the first neural network. This is the third neural network.

13. The apparatus of claim 1, further comprising a screening stage having:

5. 5. The apparatus of claim 4, wherein the third neural network is a neural network trained in a supervised mode by classifying each of a plurality of known input photographs of reactions of reactants in a given container as either favorable or unfavorable.

6. The screening stage is a filter configured to receive an input photograph; generate an information-reduced photograph based on the input photograph; and input the information-reduced photograph to a third neural network.

5. The apparatus of claim 4, comprising:

7. 7. The apparatus of claim 6, wherein the input photograph is a color photograph and the filter is arranged to convert the color photograph into a black-and-white photograph; and the third neural network includes as many single pixel inputs as there are pixels in the black-and-white photograph, and each pixel of the black-and-white photograph is connected to a pixel input of the first neural network.

8. 2. The apparatus of claim 1, wherein the input photograph is a combination of a red input photograph, a green input photograph, and a blue input photograph; and wherein the first neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs, and each pixel of the red, green, and blue input photographs is connected to a pixel input of the first neural network.

9. 9. The apparatus of claim 8, wherein the second neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs of a given portion of the input photograph.

10. 2. The apparatus of claim 1, wherein the first neural network is a neural network that is trained in a supervised mode by associating each of a plurality of known input photographs of reactions of reactants in predefined containers with at least one reaction category of the first plurality of reaction categories.

11. The apparatus of claim 10, wherein the known container is a well of a gel card, and the known photograph and the input photograph are side photographs of a reaction of a reactant in the well of the gel card.

12. the plurality of known input photographs, a set of original photographs, each of which is a photograph of a different reaction of reactants in a given container; a color-modified version of the original photo set; a mirror image of said original photograph set; a rotation of the position of the default container in the original photograph set; A brightness-modified version of the original photo set; a contrast-modified version of the original photo set; A blurred version of the original photo set; A crop of said original photo set; a translation of the position of the predetermined container in the original photograph set; a scaled version of the original photograph set; or The original photo set with added noise At least one of 11. The apparatus of claim 10, comprising:

13. 11. The apparatus of claim 10, wherein the second neural network is a network trained in a supervised mode by associating a predetermined portion of each of the subsets of the plurality of known input photographs with at least one response category of a second plurality of response categories.

14. The apparatus of claim 13, wherein the known container is a well of a gel card, and the known photograph and the input photograph are side photographs of a reaction of a reactant in the well of the gel card; and the predetermined portion of the photograph is a portion of the photograph that contains the bottom of the well.

15. a centrifuge device arranged to centrifuge the at least one container; an imaging device for generating an image of the container; 2. The apparatus of claim 1 for receiving as input each image of the container after centrifugation. A centrifugation and testing device comprising:

16. training a first neural network in a supervised mode with a first sequence of known photographs and a sequence of known response categories for each of the first known photographs; grouping at least two response categories as a default category if the at least two response categories are not distinguished from one another by the first neural network; and training a second neural network in a supervised mode with selected photographs of a second sequence of known photographs having known response categories belonging to the at least two response categories, the selected photographs being photographs cropped to a predetermined portion of the input photograph, the predetermined portion of the input photograph being an area that highlights at least one difference between the at least two response categories; 2. A method of programming the device of claim 1, comprising:

17. inputting input photographs of reactions of reactants in predefined containers into a first neural network, the first neural network being arranged to provide, for each reaction category of a first plurality of reaction categories, a probability that the input photograph represents a reaction belonging to that reaction category of the first plurality of reaction categories; and if the first neural response provides a highest probability that the input photograph exhibits a response belonging to a predefined response category of the first plurality of response categories, inputting the predefined portion of the input photograph into a second neural network, the second neural network being arranged to provide, for each response category of a second plurality of response categories, a probability that the predefined portion of the input photograph exhibits a response belonging to that response category of the second plurality of response categories. A method for classifying a photograph of a reaction of reactants in a predefined container, comprising:

18. if the second neural network is not activated, classifying the input photograph using the first decision tree as exhibiting a response within the response category having the highest probability output by the first neural network; and once the second neural network has been activated, classifying the input photograph using a second decision tree as exhibiting a response within the response category having the highest probability output by the second neural network.

20. The method of claim 17, further comprising:

19. prior to the step of inputting the input photograph into the first neural network, inputting the input photograph into a third neural network arranged to determine whether the input photograph is suitable for analysis by the first neural network; and performing said step of inputting said input photograph into said first neural network if said third neural network determines that said input photograph is suitable for analysis by said first neural network; or not inputting the input photograph to the first neural network if the third neural network determines that the input photograph is unsuitable for analysis by the first neural network.

20. The method of claim 17, further comprising:

20. 20. The method of claim 19, wherein the third neural network is a neural network trained in a supervised mode by classifying each of a plurality of known input photographs of reactions of reactants in a given container as either favorable or unfavorable.

21. filtering the photographic input to the third neural network to reduce the information contained in the photographic input to the third neural network.

20. The method of claim 19, further comprising:

22. 22. The method of claim 21, wherein the input photograph is a color photograph and the filtering step includes converting the color photograph to a black and white photograph.

23. the input photograph is a combination of a red input photograph, a green input photograph, and a blue input photograph, and the first neural network includes a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs; and The step of inputting an input photograph to a first neural network includes inputting each pixel of the red, green, and blue input photograph as a different pixel input of the first neural network.

18. The method of claim 17.

24. a second neural network including a number of single pixel inputs equal to the combined number of pixels present in the red, green, and blue input photographs of a predetermined portion of the input photograph; and The step of inputting the predetermined portion of the input photograph to the second neural network includes inputting each pixel of the red, green, and blue predetermined portions of the input photograph as a different pixel input of the second neural network.

24. The method of claim 23.

25. training the first neural network in a supervised mode by associating each of a plurality of known input photographs of reactions of reactants in a given container with at least one reaction category of the first plurality of reaction categories; 18. The method of claim 17, further comprising:

26. The method of claim 25, wherein the known container is a well of a gel card, and the known photograph and the input photograph are side photographs of a reaction of reactants in the well of the gel card.

27. forming said plurality of known input photographs from a set of original photographs, each of which is a photograph of a different reaction of reactants in a given container; and a color-modified version of the original photo set; a mirror image of said original photograph set; a rotation of the position of the default container in the original photograph set; A brightness-modified version of the original photo set; a contrast-modified version of the original photo set; A blurred version of the original photo set; A crop of said original photo set; a translation of the position of the predetermined container in the original photograph set; a scaled version of the original photo set; or The original photo set with added noise forming at least one of 26. The method of claim 25, further comprising:

28. training a second neural network in a supervised mode by associating a predetermined portion of each of the subsets of the plurality of known input photographs with at least one response category of a second plurality of response categories.

26. The method of claim 25, further comprising:

29. The method of claim 28, wherein the known container is a well of a gel card, and the known photograph and the input photograph are side photographs of a reaction of a reactant in the well of the gel card; and the predetermined portion of the photograph is a portion of the photograph that contains the bottom of the well.

Citation Information

Patent Citations

  • Immunological diagnostic test card with markings to display

    JP2011512524A