Sample state determination device, sample state determination method, and sample inspection device

Abstract

A sample state determination device includes: a processor that executes a program for performing image processing on an image of a target sample container; and a memory for storing a result of the image processing. The processor executes a process for inputting an image, a process for detecting boundary positions from the image, a process for determining the types of upper and lower regions of each boundary position and determining the boundary position of each contained object, and a process for determining whether or not the target sample container can be inspected on the basis of the boundary position of each contained object.

Description

[0001] Reference-based import

[0002] This application claims the priority of Japanese patent application No. 2022-101278 filed on June 23, 2022, the contents of which are incorporated into this application by reference. Technical Field

[0003] The present invention relates to a sample state determination device, a sample state determination method and a sample inspection device, for example, to an image processing technology using machine learning for detecting and identifying specific sample contents (for example, serum (plasma) etc.) contained in a captured image. Background Art

[0004] In recent years, the importance of blood tests has increased, and the number of tests and test items in blood analysis has also increased, requiring an increase in test efficiency. In order to improve test efficiency, the demand for detecting the accurate content of serum, etc., or determining the sample container that can be tested before the test has increased. In order to detect the content of serum, etc., there are technologies proposed in Japanese Patent Publication No. 10-232228, Japanese Patent Publication No. 9-133687, and Japanese Patent Publication No. 2005-265813, for example. In Japanese Patent Publication No. 10-232228, multiple color extraction units are used to detect the boundary position of the upper and lower parts of the serum. In Japanese Patent Publication No. 9-133687, the boundary position between the serum part and other parts is calculated using chromaticity information, and the amount of serum is calculated. In Japanese Patent Publication No. 2005-265813, the mechanism for irradiating and receiving infrared rays is moved up and down to detect the position of the separator, serum and blood clot. Summary of the invention

[0005] Problems to be solved by the invention

[0006] However, bubbles may be generated in the specimen, and even if the multiple color extraction means of Japanese Patent Laid-Open No. 10-232228, the chromaticity information of Japanese Patent Laid-Open No. 9-133687, and the infrared irradiation and light receiving mechanism of Japanese Patent Laid-Open No. 2005-265813 are used, there is a problem that the influence of bubbles cannot be suppressed and the accurate boundary position of serum, etc. cannot be obtained. In addition, Japanese Patent Laid-Open No. 10-232228, Japanese Patent Laid-Open No. 9-133687, and Japanese Patent Laid-Open No. 2005-265813 have the following problems: the content of all the contents such as serum, separation agent, and blood clot cannot be determined, and it is impossible to determine whether the specimen container containing the specimen composed of a combination of these contents can be inspected.

[0007] The present invention has been made in view of such circumstances, and provides a technology capable of detecting the boundary position of serum (plasma) or the like in an image and accurately determining an inspectable sample container.

[0008] Means for solving problems

[0009] A sample state determination device according to one embodiment of the present invention includes: a processor that executes a program for performing image processing on an image of a target sample container; a memory that stores the result of the image processing, wherein the processor executes: processing of inputting the image; processing of detecting one or more boundary positions from the image; processing of determining the properties (types) of upper and lower regions of each of the one or more boundary positions and determining the boundary position of each contained object; and processing of determining whether the target sample container can be inspected based on the determined boundary positions of each contained object.

[0010] Further features related to the present invention will become apparent from the description and drawings of this specification. In addition, the present invention is achieved and implemented by the combination of elements and various elements, the following detailed description and the scope of the attached claims.

[0011] It should be understood that the descriptions in this specification are merely typical examples and do not limit the scope of patent protection or application examples of the present invention in any sense.

[0012] Effects of the Invention

[0013] According to one embodiment of the present invention, it is possible to detect the boundary position in an image and accurately determine the inspectable specimen container. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a block diagram showing the functions of the sample state determination device according to the first embodiment.

[0015] Figure 2 It is a diagram showing a hardware configuration example of the sample state determination device according to the first embodiment.

[0016] Figure 3 This is a diagram for explaining an example of the operation of the container recognition unit.

[0017] Figure 4 This is a diagram for explaining an example of the operation of the boundary position detection unit.

[0018] Figure 5 This is a diagram for explaining an example of the operation of the content determination unit.

[0019] Figure 6 This is a diagram for explaining an example of the operation of the boundary position correction unit.

[0020] Figure 7 This is a diagram for explaining an example of the operation of the content calculation unit.

[0021] Figure 8 This is a diagram for explaining an example of the operation of the content determination unit.

[0022] Fig. 9 This is a diagram for explaining an example of upper and lower area recognition of each boundary by the content determination unit.

[0023] Fig.10 It is a diagram for explaining an example of each boundary recognition by the boundary position correction unit.

[0024] Fig.11 This is a diagram for explaining an example of bubble recognition by the boundary position correction unit.

[0025] Fig.12 This is a diagram for explaining an example of the operation of the content calculation unit.

[0026] Fig.13 This is a diagram for explaining an example of the operation of the sample state determination unit.

[0027] Fig.14 This is a diagram for explaining an example of display of determination results and the like by the output unit.

[0028] Fig.15 This is a flowchart for explaining the overall operation of the sample state determination device 1 according to the first embodiment.

[0029] Fig.16 It is a diagram showing a schematic configuration of a specimen inspection apparatus according to a second embodiment in which the specimen state determination apparatus according to the embodiment is mounted. DETAILED DESCRIPTION

[0030] The embodiments of the present specification provide a sample state determination device and a sample state determination method, which detect the boundary positions of multiple contents while determining the type (property) of the contents around the boundary positions, and further, calculate the content while correcting the boundary positions, and determine the presence or absence of each content (e.g., serum (plasma), separation agent, blood clot, urine, etc.) of the specimen in the container based on the determination results of the multiple boundary positions and the contents, thereby achieving high-precision determination of the content of each content (e.g., serum (plasma), separation agent, blood clot, urine, etc.) and determination of whether the container of the specimen can be inspected.

[0031] Hereinafter, the embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be shown with the same reference numerals. In addition, the accompanying drawings show specific embodiments and installation examples that follow the principles of the present invention, but these embodiments and installation examples are used to understand the present invention and are by no means used to limit the interpretation of the present invention.

[0032] In the present embodiment, the invention is described in full detail for those skilled in the art to implement the invention, but other installations and methods are also possible, and it is necessary to understand that the structure and construction can be changed and various elements can be replaced without departing from the scope and spirit of the technical idea of ​​the invention. Therefore, the following description is not limited to this for interpretation.

[0033] Furthermore, as described later, the embodiments of the present invention may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

[0034] In the following, each processing in the embodiment of the present invention is described with "each processing unit as a program (for example, a content determination unit, etc.)" as the subject (action subject), but while using a memory and a communication port (communication control device), the program is performed by executing the processing determined by the processor (CPU, etc.), so the processor can also be used as the subject for description.

[0035] A sample state determination device according to one embodiment of the present specification may include a function of determining the contents by detecting the boundary position in the container and a function of determining the state of the sample container. The sample container is a combination of a container and a sample contained in the container. The sample state determination device may also include a function of correcting the boundary position. In one embodiment of the present specification, even if there are bubbles or the like near the boundary position, the boundary position is detected to determine the contents and bubbles or the like around the boundary position, and the probability of the boundary around the boundary position is analyzed in detail to correct the boundary position. Thus, it is possible to achieve high-precision determination of the boundary position and the content of serum (plasma) or the like in the image. In addition, by determining whether the container contains various contents (serum (plasma), separation agent, blood clot, urine, etc.) based on the determination results of multiple boundary positions and contents, it is possible to determine whether the object specimen container can be inspected.

[0036] (1) First Embodiment

[0037] <Functional Structure of Sample State Determination Device>

[0038] Figure 11 is a block diagram showing the functional structure of a sample state determination device according to an embodiment. The sample state determination device 1 includes an input unit 10, a container identification unit 11, a boundary position detection unit 12, a content determination unit 13, a boundary position correction unit 14, a content calculation unit 15, a specimen state determination unit 16, a control unit 19, and a memory 90. However, in the sample state determination device, when the container information is provided in advance, the container identification unit 11 is not a necessary structure. The sample state determination device 1 can also be installed in a specimen inspection device as described later (second embodiment). In addition, the embodiment described below can be applied to a sample different from a specimen collected from a human.

[0039] The container recognition unit 11, boundary position detection unit 12, content determination unit 13, boundary position correction unit 14, content calculation unit 15 and specimen state determination unit 16 in the sample state determination device 1 may be realized by a program or in a modular form.

[0040] Image data is input to the input unit 10. For example, the input unit 10 may also obtain coded still image data in JPG, Jpeg2000, PNG, BMP format, etc. captured at predetermined time intervals by a camera or other imaging unit built into the image acquisition device, and use the image as an input image. In addition, the input unit 10 may also extract still image data of frames at predetermined intervals from motion image data in MotionJPEG, MPEG, H.264, HD / SDI format, etc., and use the image as an input image. In addition, the input unit 10 may also use an image acquired by the imaging unit via a bus or a network as an input image. In addition, the input unit 10 may also use an image already stored in a removable recording medium as an input image. The image input from the input unit 10 is output to the container recognition unit 11, the boundary position detection unit 12, the content determination unit 13, and the boundary position correction unit 14 via the memory 90.

[0041] The container recognition unit 11 detects and recognizes the container from the input image using a network that detects the entire container or a portion of the container (for example, a container detector such as YOLO). In addition, the container recognition unit 11 stores the calculated container information in the memory 90.

[0042] The boundary position detection unit 12 detects the boundary positions of each boundary position and the boundary of the barcode from the input image using a network that detects the boundary positions of each boundary position and the boundary of the barcode (for example, a boundary position detector such as YOLO). The boundary position is an interface between the contained objects or between the contained objects and other substances. The barcode, for example, represents information of the specimen or the patient from whom the specimen was collected. The boundary position detection unit 12 stores the calculated information such as the boundary positions of each boundary position and the boundary of the barcode in the memory 90.

[0043] The content determination unit 13 calculates the recognition result of each boundary position and the upper and lower regions of the barcode using a recognition unit composed of a network (e.g., a convolutional neural network, etc.) that recognizes each boundary position and the contents of the upper and lower regions of the barcode detected by the boundary position detection unit 12 (e.g., serum (plasma), separation agent, blood clot, urine, etc.), other than the contents (e.g., background, etc.), and the barcode. In addition, the content determination unit 13 stores information such as the calculated recognition result of each boundary position and the upper and lower regions of the barcode in the memory 90.

[0044] The boundary position correction unit 14 uses a recognizer composed of a network (e.g., Convolutional Neural Network, etc.) that recognizes each boundary position in detail, recognizes the periphery of each boundary position detected by the boundary position detection unit 12 in detail, detects the boundary position of the accurate position, and corrects the boundary position. In addition, a recognizer composed of a network (e.g., Convolutional Neural Network, etc.) that recognizes whether there are bubbles around the boundary position is used to recognize whether there are bubbles. If there are bubbles, the boundary position is corrected to the bottom of the bubble. In addition, the boundary position correction unit 14 stores information such as each corrected boundary position in the memory 90.

[0045] The content calculation unit 15 calculates the content of the target content (e.g., serum (plasma), separation agent, blood clot, urine, etc.) in the container based on each boundary position corrected by the boundary position correction unit 14 and the container information or the container information recognized by the container recognition unit 11. The content calculation unit 15 stores information such as the calculated content of each content in the container in the memory 90.

[0046] The specimen state determination unit 16 determines whether the target specimen container (container containing the specimen) can be inspected using the content calculated by the content calculation unit 15 and / or the identification result of each area obtained by the content determination unit 13. In addition, the specimen state determination unit 16 stores information such as the determination result of whether the target container can be inspected in the memory 90.

[0047] The control unit 19 is realized by a processor and is connected to each element in the sample state determination device 1. The operation of each element of the sample state determination device 1 is performed by the autonomous operation of each component described above or by the instruction of the control unit 19.

[0048] Thus, in the sample state determination device 1 of the present embodiment, the container identification unit 11 identifies the type of the container. The boundary position detection unit 12 detects each boundary position of the container and the boundary position of the barcode. The content determination unit 13 calculates the recognition results of each boundary position of the container and the upper and lower areas of the barcode. The boundary position correction unit 14 recognizes the periphery of each boundary position in detail to correct the boundary position, and the content calculation unit 15 calculates the content of the object contained in the container. The specimen state determination unit 16 is characterized in that the calculated content and the recognition results of the peripheral area of ​​each boundary position are used to determine the state of the specimen in the container, thereby detecting the content of the contents in the container from the image with high accuracy, and determining whether the object container can be inspected.

[0049] <Hardware Structure of Identifier Generator>

[0050] Figure 2 It is a diagram showing a hardware configuration example of the sample state determination device 1 according to the embodiment of the present invention.

[0051] The sample state determination device 1 has a CPU (processor) 201 for executing various programs, a memory 202 for storing various programs, a storage device (equivalent to the memory 90) 203 for storing various data, an output device 204 for outputting the boundary positions, the contents, the determination results of whether the container can be inspected, etc., and an input device 205 for inputting user instructions, images, etc., which are interconnected via a bus 207.

[0052] The CPU 201 reads and executes various programs from the memory 202 as necessary. The memory 202 stores the input unit 10, the container recognition unit 11, the boundary position detection unit 12, the content determination unit 13, the boundary position correction unit 14, the content calculation unit 15, and the sample state determination unit 16 as programs.

[0053] The storage device 203 stores an input image, information on the type of container recognized by the container recognition unit 11, information on the boundary positions of the container and the boundary position of the barcode detected by the boundary position detection unit 12, recognition results of the boundary positions of the container and the upper and lower areas of the barcode determined by the content determination unit 13, information on the boundary position corrected by the boundary position correction unit 14, information on the content of the object in the container calculated by the content calculation unit 15, and a determination result on whether the object container can be inspected determined by the specimen state determination unit 16.

[0054] The output device 204 is composed of devices such as a display, a printer, and a speaker. For example, the output device 204 displays data generated by the container recognition unit 11, the boundary position detection unit 12, the content determination unit 13, the boundary position correction unit 14, the content calculation unit 15, and the sample state determination unit 16 on the display screen.

[0055] The input device 205 is composed of devices such as a keyboard, a mouse, and a microphone. The user's instructions (including the decision to input an input image) are input to the sample state determination device 1 through the input device 205 .

[0056] The sample state determination device of the present invention identifies the container type, detects the boundary positions of the container and the boundary positions of the barcode, calculates the recognition results of the boundary positions of the container and the upper and lower areas of the barcode, identifies the periphery of each boundary position in detail to correct the boundary position, calculates the content of the contents of the object in the container, and then uses the calculated content and the recognition results of the peripheral areas of each boundary position to determine the state of the specimen in the container, thereby detecting the content of the contents in the container from the image with high precision and determining with high precision whether the object container can be inspected.

[0057] <Structure and operation of each part>

[0058] The structure and operation of each element will be described in detail below.

[0059] (i) Container identification unit 11

[0060] Figure 3 The container recognition unit 11 detects and recognizes the container from the input image using a network that detects the entire container or a portion of the container (for example, a container detector such as YOLO), and calculates the container type ct and the position of the container top 31, or calculates the container type ct based on the calculated container type ct and the container information in the memory 90. Figure 3 The container recognition unit 11 stores the information of the recognized container type, and the positions of the container top and container bottom in the image in the memory 90 .

[0061] (ii) Boundary Position Detection Unit 12

[0062] The boundary position detection unit 12 detects the boundary position (interface or boundary) of each content in the container and the boundary of the barcode. Figure 3 A barcode 32 , serum 33 , a separating agent 34 , and a blood clot 35 are shown. Figure 4 1 shows an example of boundary position detection. The boundary position detection unit 12 uses a network (for example, a boundary position detector such as YOLO) for detecting each boundary position in the container (boundary of the content, etc.) and the boundary of the barcode, such as Figure 4As shown in FIG. 1 , each boundary position and the boundary of the barcode 32 are detected from the input image. Figure 4 In the example shown, a boundary 41 between the serum (plasma) 33 and the background, a boundary 42 between the serum 33 and the separating agent 34, and a boundary 43 between the blood clot and the separating agent are detected. In addition, an upper boundary 44 and a lower boundary 45 of the barcode 32 are detected.

[0063] When the boundary position detection unit 12 cannot detect any boundary position or the like, the boundary position detection result lr is set to 0 and the boundary position detection result lr is stored in the memory. When the boundary position detection unit 12 detects more than one boundary position or the like, the boundary position detection result lr is set to the number of detected boundaries. The boundary position detection unit 12 stores the calculated boundary position detection result lr and the information of each detected boundary position in the memory 90.

[0064] (iii) Content determination unit 13

[0065] like Figure 5 As shown, the content determination unit 13 calculates the recognition results of each boundary position and the upper and lower regions of the barcode boundary, using a recognizer composed of a network (e.g., a convolutional neural network, etc.) that recognizes the contents (e.g., serum (plasma), separation agent, blood clot, urine, etc.) or other than the contents (e.g., background, barcode, etc.). In addition, when the container includes containers of various materials such as transparent type containers and opaque type containers, the content determination unit 13 selects a recognizer suitable for the container using the information of the container type recognized by the container recognition unit 11, and calculates the recognition results of each boundary position and the upper and lower regions of the barcode using the selected recognizer.

[0066] As an example, Fig. 9 An example of obtaining a recognition result is shown. An input image A1 is input to a feature extractor A91, which outputs a feature quantity FAi. A logistic regression layer 92 recognizes a region in the input image A1 based on the feature quantity FAi. Fig. 9 CNN stands for Convolutional Neural Network.

[0067] The content determination unit 13 reads the weight w, filter coefficient wj, and offset values ​​b and bi in equations (1) and (2) from the memory 90 .

[0068] For example, using feature extractor A91, formula (1) is used to calculate the feature value FAi of the contents (for example, serum (plasma), separator, blood clot, urine, etc.) and other contents (for example, background, barcode, etc.) for each boundary position in the input image A1 and the upper and lower areas of the barcode.

[0069] The filter coefficient wj shown in formula (1) is a coefficient obtained by machine learning, etc. in a manner that identifies each content as each content (for example, serum (plasma) is identified as serum (plasma), a separator is identified as a separator, a blood clot is identified as a blood clot, urine is identified as urine, etc.), and in addition, identifies other than each content as other than the content (for example, the background is identified as the background, a barcode is identified as a barcode, etc.).

[0070] In formula (1), pj represents a pixel value, bi represents an offset value, m represents the number of filter coefficients, and h represents a nonlinear function. Using formula (1), the calculation results of each filter are obtained from the upper left to the lower right of each area of ​​the target image, thereby obtaining the feature value fi of any filter i. For example, the matrix of the feature value fi obtained by the feature extractor A91 is used as the feature value FAi of the input image A1.

[0071] [Mathematical formula 1]

[0072]

[0073] like Fig. 9 As shown, using the calculated feature quantity FAi (matrix f) of the feature extractor A91, in the logistic regression layer 92 of the identifier, the value of the possibility of the content to be detected or other than the content (serum (plasma) possibility, separator possibility, blood clot possibility, background possibility, barcode possibility, etc.) is calculated by formula (2), and it is determined whether each area in the input image A1 (for example, the upper and lower areas of the boundary position) is the content to be detected (for example, serum (plasma), separator, blood clot, urine, etc.) or other than the content to be detected (for example, background, barcode, etc.).

[0074] In formula (2), w represents a weight matrix, b represents an offset value, g represents a nonlinear function, and y represents a recognition result. The weight of w and the offset value of b are calculated in advance using a learning image by using a known machine learning technique. For example, a Convolutional Neural Network can also be used as a machine learning technique. In addition, the weight of w and the offset value of b obtained during learning are used to identify each area of ​​the object image and calculate the recognition result y.

[0075] [Mathematical formula 2]

[0076] y=gg(w×f+b)…(2)

[0077] The content determination unit 13 determines what each boundary is based on the boundary positions detected by the boundary position detection unit 12 and the recognition result y of the upper area and the lower area of ​​the boundary of the barcode. Figure 8 As shown, if the recognition result of the upper region is the background and the recognition result of the lower region is the barcode 32, then the boundary is determined to be the barcode upper portion 44 (p1 is set to the determination result pr). In addition, if the recognition result of the upper region is the barcode 32 and the recognition result of the lower region is any one of the serum 33, the blood clot 35, the separating agent 34, and the background, then the boundary is determined to be the barcode lower portion 45 (p2 is set to the determination result pr).

[0078] In addition, if the recognition result of the upper region is the background and the recognition result of the lower region is the serum 33, the boundary 41 is determined to be the upper part of the serum (p3 is set to the judgment result pr). In addition, if the recognition result of the upper region is the serum 33 and the recognition result of the lower region is any one of the separating agent 34 and the blood clot 35, the boundary 42 is determined to be the lower part of the serum (p4 is set to the judgment result pr). In addition, if the recognition result of the upper region is the separating agent 34 and the recognition result of the lower region is any one of the blood clot 35 and the container bottom, the boundary 43 is determined to be the lower part of the separating agent (p5 is set to the judgment result pr).

[0079] In addition, if the recognition result of the upper region is any one of the separating agent 34, the serum 33, and the background, and the recognition result of the lower region is the blood clot 35, its boundary 43 is determined to be the upper part of the blood clot (p6 is set to the determination result pr). In addition, if the recognition result of the upper region is the blood clot 35 and the recognition result of the lower region is the separating agent 34, its boundary (not shown) is determined to be the lower part of the blood clot (p7 is set to the determination result pr). In addition, if the recognition result of the upper region of the container bottom is the blood clot 35, its boundary 44 is determined to be the lower part of the blood clot (p8 is set to the determination result pr).

[0080] In addition, if the recognition result of the upper area of ​​the container bottom is serum 33, its boundary (not shown) is determined to be the lower part of the serum (p9 is set for the judgment result pr). In addition, if the recognition result of the upper area of ​​the container bottom is the separator 34, its boundary (not shown) is determined to be the lower part of the separator (p10 is set for the judgment result pr). In addition, if the recognition result of the upper area is any one of serum 33, background, and blood clot 35, and the recognition result of the lower area is the separator 34, its boundary 42 is determined to be the upper part of the separator (p11 is set for the judgment result pr). For example, in Figure 5 In the case of , the boundary judgment result pr is set to p1, p2, p3, p4, p5, p6, p8, and p11.

[0081] The content determination unit 13 excludes boundary candidates that are not the boundary positions detected by the boundary position detection unit 12 or the boundary of the barcode.

[0082] The content determination unit 13 stores the calculated recognition result y and the determination result pr of each boundary in the memory 90 .

[0083] (iv) Boundary Position Correction Unit 14

[0084] The boundary position correction unit 14 uses a recognition unit (including a feature extractor B) composed of a network (e.g., a convolutional neural network, etc.) that recognizes the boundary of each contained substance (e.g., serum (plasma), separation agent, blood clot, urine, etc.) in detail for the boundary of each contained substance calculated by the contained substance determination unit 13, and calculates the boundary of each contained substance (e.g., serum (plasma), separation agent, blood clot, urine, etc.) by using equations (1) and (2). Figure 6 As shown, the recognition region 62 is moved in pixel units in the search range 61 while the recognition is performed, and the boundary position of each contained object is corrected at the position where the value of the recognition result y is the largest. As an example, Fig.10 An example of obtaining a recognition result is shown. An input image A1 is input to a feature extractor B101, which outputs a feature quantity FBi. The logistic regression layer 102 recognizes a boundary in the input image A1 based on the feature quantity FBi. Fig.10 CNN stands for Convolutional Neural Network.

[0085] The boundary position correction unit 14 reads the weight w, filter coefficient wj, offset value b and bi of equations (1) and (2) for identifying the boundary of each contained substance from the memory 90, and calculates the recognition result y of the boundary possibility. For example, in the search range 61 around the serum boundary, the serum boundary is identified while moving the recognition area 62 up and down in pixel units, and the boundary position of the serum is corrected to the position where the recognition result y is maximized.

[0086] In addition, the boundary position correction unit 14 may use a recognizer (including a feature extractor C) composed of a network (such as a convolutional neural network, etc.) for recognizing bubbles 63 for the boundary of the content determined by the content determination unit 13 as the upper part of the serum to recognize the presence or absence of bubbles in the recognition area 62 by using equations (1) and (2). When it is recognized that there are bubbles, the boundary position of the upper part of the serum is corrected to the position of the lowest end of the bubble. As an example, Fig.11An example of obtaining the recognition result is shown. The input image A1 is input to the feature extractor C111, and the feature quantity FCi is output. The logistic regression layer 112 determines whether there are bubbles in the input image A1 based on the feature quantity FCi. Fig.11 CNN stands for Convolutional Neural Network.

[0087] The boundary position correction unit 14 reads the weight w, filter coefficient wj, offset values ​​b and bi of equations (1) and (2) for identifying the presence or absence of bubbles from the memory 90 , and calculates the identification result y of the bubbles 63 .

[0088] The boundary position correction unit 14 stores the corrected boundary positions or the position of the boundary of the barcode and the recognition results thereof or the recognition results of the bubbles in the memory 90 .

[0089] (v) Content calculation unit 15

[0090] The content calculation unit 15 uses the upper and lower boundary positions of each contained object corrected by the boundary position correction unit 14 to calculate Figure 7 For example, based on the information of the container type identified by the container identification unit 11, the content information of the height from the bottom of the container and the area of ​​the inner side of the container at the height is read from the memory 90, and the contents from the height of the upper part of the serum to the height of the lower part of the serum are added to calculate the content lv of the serum 33. In addition, the content calculation unit 15 calculates the distance ds from the position of the upper part of the container calculated by the container identification unit 11 to the boundary of the upper part of the serum.

[0091] like Fig.12 As shown, depending on the viewpoint of the camera lens during shooting, sometimes the more the pixel information is distorted in the up and down directions. Therefore, the content calculation unit 15 can also use formulas (3) to (8), for example, to correct the pixel information (x1, y1) at the boundary position between the upper part of the serum and the lower part of the serum to (x3, y3), and then calculate the serum content lv. Among them, x and y in formula (3) represent points containing distortion on camera coordinates, x1 and y1 represent points containing distortion on image coordinates, cx and cy represent image centers, fx and fy represent focal lengths, k1 and k2 in formula (5) represent distortion coefficients in the radial direction, p1 and p2 in formula (6) represent distortion coefficients in the circumferential direction, x2 and y2 in formula (7) represent points after the distortion on camera coordinates is corrected, and x3 and y3 in formula (8) represent points after the distortion on image coordinates is corrected.

[0092] [Mathematical formula 3]

[0093]

[0094] [Formula 4]

[0095] r 2 =x 2 +y 2 …(4)

[0096] [Mathematical formula 5]

[0097] rad=1+k1r 2 +k2r 4 …(5)

[0098] [Mathematical formula 6]

[0099]

[0100] [Mathematical formula 7]

[0101]

[0102] [Mathematical formula 8]

[0103]

[0104] The content calculation unit 15 stores the calculated content lv and distance ds in the memory 90 .

[0105] (vi) Sample status determination unit 16

[0106] The specimen state determination unit 16 determines whether the inspection target container is permitted or not, using the boundary position detection result lr calculated by the boundary position detection unit 12 , the determination result pr of each boundary calculated by the content determination unit 13 , and the content lv calculated by the content calculation unit 15 . Fig.13 A flowchart of the sample state determination unit 16 is shown.

[0107] For example, if the boundary position detection result lr is 0, the specimen state determination unit 16 determines that the target container is an empty container, sets 0 (not inspectable) for the inspection determination result, and sets "empty container" for the inspection determination reason. In addition, when the content lv is less than the threshold value TH, the inspection determination result is set to 0 (not inspectable), and "insufficient content" is set for the inspection determination reason.

[0108] In addition, when the boundary determination result pr does not include p3 and p4, or p3 and p9, the inspection determination result is set to 0 (inspection is not possible), and the inspection determination reason is set to "no serum". In addition, when the boundary position detection result lr is 1 or more, the content lv is greater than the threshold value TH, and the boundary determination result pr includes p3 and p4, or p3 and p9, the inspection determination result iajr is set to 1 (inspection is possible), and the inspection determination reason is set to "serum is present".

[0109] <Processing Procedure of Sample Status Determination Unit 16>

[0110] (i) Step 1301

[0111] The specimen state determination unit 16 determines whether the boundary position detection result lr is 0. If lr is other than 0, the process proceeds to step 1302. If lr is 0, the process proceeds to step 1307.

[0112] (ii) Step 1302

[0113] The sample state determination unit 16 determines whether the content lv calculated by the content calculation unit 15 is greater than or equal to a threshold value TH (eg, 4 μl). If the content lv is greater than or equal to the threshold value TH, the process proceeds to step 1303. If the content lv is less than the threshold value TH, the process proceeds to step 1307.

[0114] (iii) Step 1303

[0115] The specimen state determination unit 16 determines whether the determination result pr of each boundary calculated by the content determination unit 13 includes p3. If the determination result pr of each boundary includes p3, the process proceeds to step 1304. If the determination result pr of each boundary does not include p3, the process proceeds to step 1307.

[0116] (iv) Step 1304

[0117] The specimen state determination unit 16 determines whether the determination result pr of each boundary calculated by the content determination unit 13 includes p4. If the determination result pr of each boundary includes p4, the process proceeds to step 1305. If the determination result pr of each boundary does not include p4, the process proceeds to step 1306.

[0118] (v) Step 1305

[0119] The sample status determination unit 16 sets 1 (inspection possible) to the inspection possibility determination result iajr.

[0120] (vi) Step 1306

[0121] The specimen state determination unit 16 determines whether the determination result pr of each boundary calculated by the content determination unit 13 includes p9. If the determination result pr of each boundary includes p9, the process proceeds to step 1305. If the determination result pr of each boundary does not include p9, the process proceeds to step 1307.

[0122] (vii) Step 1307

[0123] The sample status determination unit 16 sets 0 (inspection not possible) to the inspection possibility determination result iajr.

[0124] The sample status determination unit 16 stores the calculated inspection possibility determination result iajr and inspection possibility determination reason in the memory 90 .

[0125] (vii) Output unit 17

[0126] As an example, the output unit 17 is Fig.14 The GUI (Graphical User Interface) shown displays the container type ct calculated by the container recognition unit 11, the lines of each boundary detected by the boundary position detection unit 12, the recognition results of the upper area and the lower area of ​​each boundary and the boundary judgment result determined by the content determination unit 13, the lines of each boundary corrected by the boundary position correction unit 14 (for example, boundary (correction)), the content lv and distance ds of each content calculated by the content calculation unit 15, and the judgment result and reason for the judgment on whether the target container can be inspected obtained by the specimen state judgment unit 16.

[0127] Fig.14 1 is a diagram showing an example of a situation in which a target container contains serum, a separating agent, and a blood clot, and the boundary position of the serum is corrected. Fig.14 In the figure, whether the object container can be inspected is "yes", the reason for the inspection is "serum is present", the container type is "A", the content (serum) is 20μl, the boundary judgment results are p1, p2, p3, p4, p5, p6, p8, p11, the bubble recognition result is "yes", the distance from the top of the container is 18mm, and examples of the upper boundary line of the serum and the corrected upper boundary line of the serum, the lower boundary line of the serum (the upper boundary line of the separation agent), the lower boundary line of the separation agent (the upper boundary line of the blood clot), and the boundary line of the bottom of the container are displayed.

[0128] The output unit 17 is not an essential component of the sample state determination device 1 . When the sample inspection device includes the output unit, the sample state determination device 1 may not have the output unit 17 .

[0129] <Processing Procedure of Sample State Determination Device>

[0130] Fig.15 This is a flowchart for explaining the operation of the sample state determination device 1 according to the embodiment of the present invention. In the following, each processing unit (input unit 10, content determination unit 13, etc.) is described as the action subject, but it can also be replaced by using the CPU 201 as the action subject, and the CPU 201 executes each processing unit as a program.

[0131] (i) Step 1501

[0132] The input unit 10 receives an object image and outputs the input image to the container recognition unit 11 .

[0133] (ii) Step 1502

[0134] The container recognition unit 11 detects the container from the input image using a container detector, further recognizes the container type ct, detects the positions of the container top and the container bottom, and stores the obtained information in the memory 90 .

[0135] (iii) Step 1503

[0136] The boundary position detection unit 12 detects each boundary position in the container (boundary of the contents, etc.) and the boundary of the barcode using a boundary position detector, and stores the obtained information in the memory 90 .

[0137] (iv) Step 1504

[0138] The content determination unit 13 calculates the recognition results of each boundary position or the upper and lower regions of the barcode boundary and the determination result pr of each boundary using equations (1) and (2), and stores the obtained information in the memory 90.

[0139] (v) Step 1505

[0140] The boundary position correction unit 14 obtains the corrected boundary positions or the position of the barcode boundary and the recognition results thereof or the bubble recognition results using equations (1) and (2), and stores the obtained information in the memory 90 .

[0141] (vi) Step 1506

[0142] The content calculation unit 15 uses the information of the position of the upper part of the container and the boundary position of the upper part and the lower part of each content to calculate the content lv of each content and the distance ds from the upper part of the container to the upper part of the serum, and stores the calculated information in the memory 90. In addition, the content calculation unit 15 corrects the information of each boundary position by using equations (3) to (8), calculates the content lv of each content and the distance ds from the upper part of the container to the upper part of the serum based on the corrected information of each boundary position, and stores each content lv and distance ds in the memory 90.

[0143] (vii) Step 1507

[0144] The specimen status determination unit 16 uses the boundary position detection result lr calculated by the boundary position detection unit 12, the determination result pr of each boundary calculated by the content determination unit 13, and the content lv calculated by the content calculation unit 15 to determine whether the object container can be inspected, and stores the calculated inspection determination result iajr in the memory 90.

[0145] According to the first embodiment, the container type is identified, the boundary positions of the container and the boundary positions of the barcode are detected, the recognition results of the boundary positions of the container and the upper and lower areas of the barcode are calculated, the periphery of each boundary position is identified in detail to correct the boundary position, the content of the contents of the object in the container is calculated, and then the calculated content and the recognition results of the peripheral areas of each boundary position are used to determine the state of the specimen in the container. As a result, the content of the contents in the container can be detected from the image with high precision and it can be determined with high precision whether the object container can be inspected.

[0146] Furthermore, by accurately detecting the amount of the contents in the container from the image, the amount of the sample collected from the patient can be reduced to a minute amount.

[0147] (2) Second Embodiment

[0148] Fig.16 1 is a functional block diagram showing the structure of a specimen inspection apparatus 1600 according to a second embodiment of the present invention. The specimen inspection apparatus 1600 includes the sample state determination apparatus 1 according to the first embodiment, a specimen suction apparatus 1601 , and a display apparatus 1602 .

[0149] The specimen suction device 1601 has, for example, a device that controls a specimen sampling mechanism having a suction nozzle and calculates analysis results of test items related to the specimen (for example, HbA1c, TP, AST, etc. related to biochemistry and immunoassay) using the sucked specimen and reagent.

[0150] The specimen suction device 1601 receives the information on whether or not the inspection is possible (such as the inspection determination result iajr) from the sample state determination device 1. When the inspection determination result iajr is 1 (inspectable), the specimen sampling mechanism is controlled to suck a part of the specimen and the analysis results of each inspection item are calculated using the reagent. The display device 1602 displays the analysis results on the display screen. On the other hand, when the inspection determination result iajr is 0 (inspectable), the specimen is not sucked in the specimen sampling mechanism, and the display device 1602 displays "inspectable" on the display screen. In addition, the specimen suction device 1601 receives information on the distance ds from the upper part of the container from the sample state determination device 1, and controls the suction position of the suction nozzle of the specimen sampling mechanism.

[0151] As the specimen suction device 1601, a biochemical analyzer, an immunoassay analyzer, or the like equipped with a specimen sampling mechanism may be used.

[0152] According to the second embodiment, the following specimen inspection device can be provided: using the inspection feasibility information determined by the sample state determination device, whether the specimen can be sucked is controlled with high precision by the specimen suction device, and the sucked specimen and reagent are used to analyze each inspection item related to the specimen, and the analysis results of each inspection item are displayed by the display device, thereby enabling the inspection to be performed efficiently and further enabling the amount of specimen collected from the patient to be suppressed to a trace amount.

[0153] The above-described embodiments can be modified as follows: In the content determination unit 13 and the boundary position correction unit 14, a plurality of feature quantities are obtained by machine learning using filters, but other feature quantities such as HOG may be used to achieve the same effect.

[0154] In addition, in the container recognition unit 11 and the boundary position detection unit 12, YOLO is used as a container detector and a boundary position detector to detect the container and the boundary position, but SSD (Single Shot MultiBox Detector), R-CNN, Fast R-CNN, Faster R-CNN, etc. can also be used to achieve the same effect.

[0155] The present invention can also be implemented by the program code of the software that realizes the functions of the embodiments. In this case, a storage medium having the program code recorded thereon is provided to a system or device, and the computer (or CPU, MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code itself read out from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium having the program code stored thereon constitute the present invention. As a storage medium for supplying such a program code, for example, a floppy disk, a CD-ROM, a DVD-ROM, a hard disk, an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, etc. are used.

[0156] In addition, based on the instructions of the program code, the OS (operating system) running on the computer may perform part or all of the actual processing, and the functions of the above-mentioned embodiments may be realized through this processing. In addition, after the program code read from the storage medium is written into the memory on the computer, the CPU of the computer may perform part or all of the actual processing based on the instructions of the program code, and the functions of the above-mentioned embodiments may be realized through this processing.

[0157] Furthermore, the program code of the software that realizes the functions of the implementation method can be distributed via a network and saved in a storage unit such as a hard disk or memory of the system or device or a storage medium such as CD-RW, CD-R, etc. When in use, the computer (or CPU or MPU) of the system or device reads and executes the program code saved in the storage unit or the storage medium.

[0158] Finally, the processes and techniques described herein do not inherently involve any particular device and can be implemented by any appropriate combination of components. Moreover, various types of general-purpose devices can be used in accordance with the methods described herein. In order to perform the steps of the methods described herein, it is sometimes beneficial to build a dedicated device. In addition, various inventions can be formed by appropriate combinations of multiple structural elements disclosed in the embodiments. For example, several structural elements can also be deleted from all structural elements shown in the embodiments. In addition, the structural elements of different embodiments can also be appropriately combined. The present invention is described in association with specific examples, but this is not intended to be limited in all viewpoints, but to be used for illustration. For those skilled in the art, it is known that there are multiple combinations of hardware, software and firmware corresponding to the implementation of the present invention. For example, the described software can be implemented by a wide range of programs or scripting languages ​​such as assembly language, C / C++, perl, Shell, PHP, Java, etc.

[0159] Furthermore, in the above-mentioned embodiments, the control lines and information lines are those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. All the structures may be connected to each other.

[0160] In addition, other implementations of the present invention will be apparent to those having ordinary knowledge in the art from the description of the present invention and the embodiments disclosed herein. The various forms and / or components of the described embodiments may be used alone or in any combination.

Claims

1. A sample state determination device, characterized in that: Include: a processor that executes a program for performing image processing on an image of a target specimen container; and a memory for storing the result of the image processing, The processor performs: input processing of said image; A process for detecting a boundary position from the image; Determining the types of upper and lower regions of each of the detected boundary positions and deciding processing of the boundary positions of each contained object; as well as The process of determining whether or not the target sample container can be inspected is performed based on the determined boundary positions of the respective contents.

2. The sample state determination device according to claim 1, characterized in that: The processor performs a process of correcting each of the detected boundary positions based on the determined type of upper and lower areas of each of the detected boundary positions.

3. The sample state determination device according to claim 1, characterized in that: The processor performs: a process of identifying a container from the image and calculating container information; selecting a detector process according to the container information; and determining the type of the upper and lower regions of each boundary position detected by the selected detector and determining the boundary position of each contained object, The processor determines whether the target sample container can be inspected based on the determined boundary position of each of the contained substances.

4. The sample state determination device according to any one of claims 1 to 3, characterized in that: The processor executes a process of calculating the content based on the determined boundary position of each contained substance, Whether or not the target sample container can be inspected is determined based on the calculated content.

5. The sample state determination device according to claim 2, characterized in that: The processor performs processing for identifying a container from the image and calculating container information. The processor executes a process of calculating the content based on the container information and the determined boundary position of each contained object, The processor determines whether the target sample container can be inspected based on the calculated content.

6. A sample state determination method for determining the state of a target sample container in an image, characterized in that: A processor that executes a program for performing image processing on an input image inputs an image of a target sample container, detects boundary positions from the image, determines the types of upper and lower areas of each detected boundary position, and determines the boundary position of each contained object, and determines whether the target sample container can be inspected based on the determined boundary position of each contained object.

7. The method for determining the state of a sample according to claim 6, characterized in that: The processor corrects each of the detected boundary positions based on the determined types of upper and lower areas of each of the detected boundary positions.

8. The method for determining the state of a sample according to claim 6, characterized in that: The processor identifies the container from the image and calculates container information, selects a detector based on the container information, determines the type of upper and lower areas of each boundary position detected by the selected detector, and determines the boundary position of each contained object, and determines whether the object sample container can be inspected based on the determined boundary position of each contained object.

9. The method for determining a sample state according to any one of claims 6 to 8, characterized in that: The processor calculates the content based on the determined boundary positions of the respective contained substances, and determines whether the target sample container can be inspected based on the calculated content.

10. The method for determining the state of a sample according to claim 7, wherein: The processor recognizes the container from the image and calculates container information, calculates the content based on the container information and the determined boundary position of each contained object, and determines whether the target sample container can be inspected based on the calculated content.

11. A sample inspection device, characterized in that: Include: The sample state determination device according to any one of claims 1 to 3; a sample suction device that controls whether or not the sample can be sucked to analyze the inspection items of the sample; and a display device which displays the analysis result, The sample suction device uses the information on whether the inspection is possible determined by the sample state determination device to control whether the sample in the target sample container can be sucked, and uses the sucked sample and reagent to analyze the inspection items related to the sample. The display device displays the analysis result of the inspection item.

Images