Drug Imaging Device

Abstract

To provide a device and software for identifying medicines.SOLUTION: Medicines dispensed at a pharmacy are often dispensed, with tablets and capsules wrapped in a long packaging sheet. In identifying the packaging sheet, when there are a medicine identification device and software capable of changing over medicine package modes of identifying individual medicines placed in a regular Petri dish and identifying long packaging sheets, with their connected to each other, flexibility of operators may greatly increase. In the medicine package modes, the medicine identification device is so configured that light reaches the medicine through polarizing members provided in a plurality of prescribed arrangements, so as to appropriately capture an image by reflection of lighting, etc. inside the device.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a drug imaging device for taking images of drugs and drug identification software for identifying drugs.

Background Art

[0002] Currently, it is common for a single patient to visit multiple medical institutions. In this case, a doctor needs to know the drugs prescribed to the patient at other medical institutions. Therefore, doctors often ask patients to bring the drugs prescribed at other medical institutions. In this case, if a drug identification code is printed on the bag in which the drug is packaged, the prescribed drug can be immediately identified. However, it often happens that the code is not printed or the drug is brought in unpackaged. In that case, the medical institution has to identify the prescribed drug based only on the appearance of the drug. Such work is called identification work and is usually the job of a pharmacist or technician. Currently, pharmacists manually identify unknown drugs based on the size, shape, color, etc. of the drugs. Unfortunately, the identification work is difficult, requires a lot of time, and places a heavy burden on pharmacists.

[0003] In order to reduce such a burden, drug identification devices that automatically identify drugs have been proposed (Patent Document 1, Patent Document 2). However, such devices still remain at the literature level and have not reached practical use. In order to complete them as practically usable products, numerous developments are still required in various aspects of the drug identification system, such as devices and software.

[0004] Furthermore, medications dispensed at pharmacies may undergo verification. This verification confirms whether the dispensed medication matches the prescription. Devices for performing this verification have also been developed. However, unlike differential diagnosis, verification only requires determining whether the dispensed medication is correct against the prescription; it does not require searching for a candidate medication from an infinite number of existing medications. Therefore, verification techniques cannot be directly applied to differential diagnosis. In addition, medications dispensed at pharmacies may come in the form of tablets or capsules, wrapped in long, individual packets.

[0005] Furthermore, in recent years, it has become common practice in pharmacies to individually package medications according to dosage instructions for morning, noon, and evening, or for specific days of the week, such as Monday through Sunday. This allows patients to take multiple medications correctly, at the right time, and without making mistakes. These individual packaging sheets often have white printing on the film, such as a solid white background or a white band, which contains information about dosage instructions, such as morning, noon, and evening, or for specific days of the week. To quickly identify the medications within these individual packaging sheets, it is necessary to identify them while the long, continuous sheets remain connected. Thus, given the current trends in medication use, there has been a desire to perform identification without changing equipment or software depending on the condition of the medications dispensed at the pharmacy. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 9-16681 [Patent Document 2] Japanese Patent Publication No. 5-245186 [Patent Document 3] WO2015 / 152225 [Patent Document 4] Japanese Patent Publication No. 2019-076787 [Patent Document 5] Japanese Patent Publication No. 2020-011122 [Patent Document 6] Japanese Patent Publication No. 2020-157077 [Patent Document 7] Japanese Patent Publication No. 2020-156746 [Overview of the project] [Problems that the invention aims to solve]

[0007] Conventional drug imaging devices have a problem when identifying continuously packaged drugs: if the device uses illumination to highlight the markings or printing on the drug packaging, the illumination light reflects off the packaging paper, causing glare and making the markings or printing difficult to see, thus making drug identification impossible.

[0008] Therefore, the present invention aims to provide a drug imaging device and drug identification software that can identify drugs in a continuous stream of individually packaged drugs. [Means for solving the problem]

[0009] --- First aspect of the present invention --- A first aspect of the present invention relates to a drug imaging device used for imaging drugs. This drug imaging device includes a mounting section on which drugs are placed, a camera for imaging the mounting section, and a light source for illuminating the mounting section. A mounting unit for placing packaged drugs is set in the mounting section. The mounting unit includes a transparent bottom and at least one polarizing member. The light source includes, at least a portion, a ring-shaped illumination. The light from the ring-shaped illumination is configured to reach the drug through the polarizing member.

[0010] The light source preferably includes a ring-shaped light source installed above the transparent bottom of the mounting unit, and the polarizing member more preferably includes a first polarizing member installed on the upper surface of the transparent bottom so as to be perpendicular to the transparent bottom.

[0011] The light source preferably includes a ring-shaped light source installed below the transparent bottom of the mounting unit. The polarizing member more preferably includes a second polarizing member installed on the underside of the transparent bottom so as to be inclined at a predetermined angle with the underside of the transparent bottom, and a third polarizing member installed on the underside of the transparent bottom so as to be parallel to the transparent bottom. Furthermore, one end of each of the first polarizing member, the second polarizing member, and the third polarizing member is configured to be adjacent to one another.

[0012] Furthermore, it is preferable that the polarization direction of the first polarizing member is set to be perpendicular to the transparent bottom, the polarization direction of the second polarizing member is set to be the same as the inclination angle of the second polarizing member with respect to the lower surface of the transparent bottom, and the polarization direction of the third polarizing member is set to be parallel to the transparent bottom.

[0013] The placement unit preferably further includes a guide section that allows for the placement and movement of multiple individually packaged drugs.

[0014] The mounting unit preferably further includes a handle for removing the mounting unit from the mounting section described above.

[0015] Preferably, the camera further comprises a first camera installed above the mounting section and a second camera installed below the mounting section.

[0016] ---- Second aspect of the present invention ---- A second aspect of the present invention relates to drug identification software that causes a computer to identify drugs. This drug identification software can cause the computer to perform the following steps: a step of accepting a selection of whether to switch between an individual mode for identifying drugs that are not packaged and a packaged mode for identifying drugs that are packaged; a step of acquiring an image of a drug placed on a placement item set in a placement unit when the individual mode is selected, and a step of acquiring an image of a drug packaged in a placement unit set in a placement unit when the packaged mode is selected; a step of extracting the area occupied by the drug from the image; a step of extracting markings and / or printing within the area; a step of identifying the drug based on the extracted markings and / or printing; and a step of displaying the identification result on a display device.

[0017] Preferably, after the step of acquiring an image of the drug, if a selection to switch preview display is accepted, the computer is further instructed to display a preview image of the acquired drug on the display device. In another embodiment, the computer is further instructed to display a preview image of the acquired drug on the display device.

[0018] When the drug to be identified is a capsule, it is preferable to obtain images of the drug taken from above and from below in the step of acquiring images of the drug. Furthermore, in the step of identifying the drug, it is preferable to perform template matching on the markings and / or printing extracted from each image, such as the image taken from above only, the image taken from below only, and a combination of the images taken from above and below, and compare the matching rates of each to determine which drug has the highest matching rate.

[0019] Furthermore, the system may also perform the steps of recording the matching rate in a learning database and performing automatic learning, and performing automatic learning multiple times, and outputting the learning result if the matching rate recorded in the learning database meets a predetermined threshold.

[0020] More preferably, the matching coincidence rate recorded in the learning database includes at least one high value among the highest matching coincidence rate obtained by multiple automatic learning and the latest matching coincidence rate.

[0021] ―――― Third Aspect of the Present Invention ―――― The third aspect of the present invention relates to drug discrimination software that causes a computer to perform drug discrimination. This drug discrimination software includes steps of causing a computer to acquire an image taken from above and an image taken from below of a packaged drug that moves to a placement unit set on a placement portion, extracting an area occupied by the drug in an image in which the drug can be detected from the image taken from above and the image taken from below, extracting the engraving and / or printing within the area, performing a search for the drug based on the extracted engraving and / or printing, and displaying the discrimination result on a display device.

[0022] In the step of extracting the area occupied by the drug in an image in which the drug can be detected, the step of cutting out the area including the engraving and / or printing of the drug by AI technology may be further performed.

[0023] It is preferable to further cause the computer to perform a step of displaying, on the display device as it is, an image obtained by cutting out an outer circumferential rectangular area along the contour including the engraving and / or printing of the drug from the taken image on the side where the drug cannot be detected in either the image taken from above or the image taken from below.

[0024] Furthermore, when there is no matching discrimination result as a result of searching for the first drug among the packaged drugs, it is preferable to cause the computer to perform steps of accepting an input of a candidate drug, storing the input candidate drug, and when there is no matching discrimination result as a result of searching for a second drug behind the first drug among the packaged drugs, displaying the stored candidate drug on the display device as the first candidate drug.

[0025] If neither the image taken from above nor the image taken from below can detect the drug, the mounting unit may be angled so that it is removed from the area where the drug cannot be detected.

[0026] Furthermore, if the image obtained from the packaged drug being moved to the angled placement unit described above is oblique, image processing may be used to correct it. [Effects of the Invention]

[0027] This invention enables the automation of drug identification, thereby reducing the burden of drug identification work. [Brief explanation of the drawing]

[0028] [Figure 1] Figure 1 is a schematic diagram illustrating the overview of the drug identification system.

[0029] Figures 2-8 primarily show the configuration of the drug imaging device. [Figure 2] Figure 2 is a perspective view of an embodiment of a drug imaging device. [Figure 3] Figure 3 is a perspective view showing the drug imaging device shown in Figure 2 with its front cover open. [Figure 4] Figure 4 is a perspective view showing the internal configuration of the drug imaging device shown in Figure 2. This figure mainly shows the frame and the components attached to the frame. [Figure 5] Figure 5 is a front view of the drug imaging device shown in Figure 4. [Figure 6] Figure 6(a) is a perspective view showing a first embodiment of a tray set in a drug imaging device. Figure 6(b) is a diagram showing an embodiment of the mounting unit. Figure 6(c) is a side view of the same tray. [Figure 7] Figure 7 shows the relationship between the light source and the deflection plate. [Figure 8] Figure 8 is a view of the mounting unit shown in Figure 7, from the left side.

[0030] Figures 9-21 show the screen layouts of computer images primarily generated by drug identification software. [Figure 9] Figure 9 shows an image of the initial stage after the drug identification software has been activated. [Figure 10] Figure 10 shows an image from the first stage of drug image differentiation. [Figure 11] Figure 11 shows an image from the second stage of drug image differentiation. [Figure 12] Figure 12 shows an image from the third stage of drug image differentiation. [Figure 13] Figure 13 shows an image from the fourth stage of drug image differentiation. [Figure 14] Figure 14 shows the fifth stage image for drug image differentiation. [Figure 15] Figure 15 shows an image from the sixth stage of drug image differentiation. [Figure 16] Figure 16 shows an image example used for grouping in drug image differentiation. [Figure 17] Figure 17 shows the first stage image of drug image identification, which is used when all drugs consist of the same type of drug. [Figure 18] Figure 18 shows the image of the second stage of drug image differentiation, which began in Figure 17. [Figure 19] Figure 19 shows the image of the third stage of drug image differentiation, which began in Figure 17. [Figure 20] Figure 20 shows the image of the fourth stage of drug image differentiation, which began in Figure 17. [Figure 21] Figure 21 shows an image obtained using the magnifying glass function of the drug identification software.

[0031] Figures 22-25 show the processing steps when drug identification software performs automatic drug identification. [Figure 22]Figure 22 is a flowchart showing the processing in the upstream part of the identification process. [Figure 23] Figure 23 is a flowchart showing the process for individual drug identification. [Figure 24] Figure 24 is a flowchart showing the process for identifying tablets. [Figure 25] Figure 25 is a flowchart showing the capsule identification process.

[0032] [Figure 26] Figure 26(a) is a perspective view of a calibration sheet used for camera calibration. Figure 26(b) is a plan view of the calibration sheet.

[0033] [Figure 27] Figure 27 is an exploded perspective view showing a second embodiment of a tray set in a drug imaging device. [Figure 28] Figure 28 is an image of another embodiment of the third stage of drug image identification. [Figure 29] Figure 29 is an image of an alternative embodiment of the sixth stage of drug image identification. [Figure 30] Figure 30 is a printed image of the identification results. [Figure 31] Figure 31 is a flowchart showing the automated learning procedure. [Figure 32] Figure 32 shows an example of the message displayed on the screen during automated learning. [Figure 33] Figure 33 shows the image of the fifth stage of drug image differentiation, which began in Figure 17. [Figure 34] Figure 34 shows the image of the sixth stage of drug image differentiation, which began in Figure 17. [Figure 35] Figure 35 is an image representing the seventh stage of drug image differentiation, which began in Figure 17. [Figure 36] Figure 36 shows an image of the message displayed before printing the identification results. [Figure 37]Figure 37(a) is a plan view of a second embodiment of a calibration sheet used for camera calibration. Figure 37(b) is a side view of the calibration sheet of the second embodiment. [Figure 38] Figure 38 is an image of the second embodiment of the seventh stage of drug image identification, which began in Figure 17. [Figure 39] Figure 39 is an image of the third embodiment of the seventh stage of drug image identification, which began in Figure 17. [Figure 40] Figure 40 is an image of the fourth embodiment of the seventh stage of drug image identification, which began in Figure 17. [Figure 41] Figure 41 is an image of a manual drug search. [Figure 42] Figure 42 is an image illustrating the process of identifying medications within individual packaging. [Figure 43] Figure 43 is an image of the packaging mode used to identify individually packaged medications. [Figure 44] Figure 44 is an image showing photographs of individually packaged medication taken from above and below. [Figure 45] Figure 45 is an image illustrating how to distinguish between individually packaged medications. [Figure 46] Figure 46 shows an image preview of the individually packaged medication. [Figure 47] Figure 47(a) is an image showing the identification of drugs during additional imaging. Figure 47(b) is an image showing the addition of drugs from other mounted units to the list of drugs to be identified during additional imaging. [Figure 48] Figure 48 is an image showing a screen for confirming the number of individually packaged medications. [Modes for carrying out the invention]

[0034] §1 Drug Identification System Figure 1 is a schematic diagram showing an overview of the drug identification system. As shown in the figure, the drug identification system 100 consists of a drug imaging device 200 and a computer 500. In this drug identification system 100, the drug imaging device 200 takes images of the drug, and the computer 500 identifies the drug based on the obtained drug image data. As explained in the background technology, drugs dispensed at pharmacies are sometimes dispensed as tablets or capsules wrapped in long, narrow paper packets. This drug identification device 200 is configured to enable identification in both cases: when tablets or capsules are opened and removed from the paper packets and identified directly, and when they are identified while still contained within the paper packets without opening them.

[0035] The computer 500 comprises a main unit 520 containing a CPU, storage device, recording device, etc., a display device 530 that functions as an output device, and a keyboard 541 and mouse 542 that function as input devices. The display device 530 also includes a screen 531 and a touchscreen 532, and the user can also input to the computer 500 via the touchscreen 532. Furthermore, drug identification software 600 is installed on the computer 500. From this perspective, the computer 500 with the drug identification software (also called a drug identification program) 600 installed can also be called a drug identification device. In this embodiment, the computer 500 is a so-called PC (personal computer), and the drug identification software 600 is stored in a recording device within the PC. However, the computer 500 may not be a PC, but for example, a microcomputer built into another device. In that case, the drug identification software 600 may be written to a storage device such as the ROM (read-only memory) of that device.

[0036] When identifying a drug, the user first places the drug to be identified into the drug imaging device 200. Next, triggered by the user's operation on the computer 500, the drug imaging device 200 takes an image of the drug. The captured image of the drug is transmitted as data to the computer 500. The computer 500 then uses this image data to search for the drug by referring to a database (not shown).

[0037] In one embodiment, the computer 500 has a built-in database. For example, if the comprehensive pharmaceutical database "MDbank" (trademark) manufactured by Yuyama Seisakusho Co., Ltd. is installed on the computer 500, the computer 500 can search for drugs by referring to MDbank without accessing an external database. In another embodiment, the computer 500 accesses the database via a network. For example, if the medication guidance support system "PharmaRoad" (trademark) manufactured by Yuyama Seisakusho Co., Ltd. is installed on another computer (server) in the same facility, the computer 500 can search for drugs by accessing PharmaRoad on the server via a LAN (local area network). Furthermore, in yet another embodiment, the computer 500 may access the database via the internet.

[0038] The drug imaging device 200 will be described in detail below. Following that, the drug identification software 600 will be described in detail.

[0039] §2 Drug imaging device Figure 2 is a perspective view showing the external appearance of the drug imaging device 200. In this figure, the interior is not visible due to the case 210, but a placement section 220 for placing drugs is provided approximately in the center of the interior of the drug imaging device 200. More specifically, the drugs are placed in a tray 300 as shown in Figure 6, and this tray 300 is set in the placement section 220. For the sake of explanation, the part of the drug imaging device 200 above the placement section 220 will be referred to as the upper part 201, and the part below it will be referred to as the lower part 202. The method of placing the drugs to be identified on the tray 300 of this drug identification device 200 differs depending on whether the drugs to be identified are removed from their packaging, such as individual packets, and identified directly, or whether the drugs are identified while still packaged in individual packets.

[0040] §2.1 External configuration of the drug imaging device As shown in Figure 2, the drug imaging device 200 has a lower case 211, a notch 212, a tray support member 221, a guide member 230, an upper case 213, and a cover 214 visible from the outside. This figure shows a long piece of packaging paper inserted, but when identifying unpackaged tablets or capsules, the drugs are placed in a petri dish (not shown) for identification.

[0041] The lower case 211 covers the lower part 202 of the drug imaging device 200. A horizontally extending notch 212 is formed on the upper surface of the lower case 211. This notch 212 is open to the left, right, and front of the drug imaging device 200. Through this notch 212, the user can insert a long piece of packaging paper into the drug imaging device 200. Specifically, the packaging paper is spread horizontally and inserted into the notch 212 from the front. Then, the packaging paper is moved to the vicinity of the center of the drug imaging device 200. This allows the built-in camera of the drug imaging device 200 to photograph the drug sealed inside the packaging paper without cutting the paper. Because the notch 212 allows the internal space of the drug imaging device 200 to communicate with the external space in the left-right direction, even if the packaging paper is longer than the width of the drug imaging device 200, the user can easily insert the packaging paper into the drug imaging device 200 while it is unwound.

[0042] A tray 300 specifically for dispensing paper is placed directly above the notch 212. The tray 300 for dispensing paper is used when the drug to be identified is packaged in a long form or in individual film packets. When the drug to be identified is removed from its packaging and identified directly, a roughly rectangular plate-shaped member is used to support the member that holds the drug. This roughly rectangular plate-shaped member has a circular hole in its center, allowing imaging from above and below. The invention in which tablets and capsules are brought in unpackaged is described in detail in the patent documents listed in the prior art section. In this embodiment, a tray 300 specifically for dispensing paper, with the dispensing paper sandwiched inside, is set in the drug imaging device 200 instead of a circular petri dish, and a major feature is that the user can perform the identification work without significantly changing the work method depending on the state of the drug.

[0043] Figure 6 shows a tray 300 specifically for dispensing paper. Figure 6(a) is a perspective view of the tray 300 specifically for dispensing paper, viewed from above in one direction. The tray 300 specifically for dispensing paper consists of a petri dish body 301 on which the dispensing paper is placed, and a handle portion 302 for placing the tray 300 directly above the notch portion 212. When the drug to be identified is in dispensing paper, the user can place this tray 300 on the drug imaging device 200 and insert the long dispensing paper through the notch 212 into the drug imaging device 200. The drug imaging device 200 can then photograph the drug inside the dispensing paper using cameras positioned vertically. The drug identification software 600 then identifies the drug inside the dispensing paper that has been photographed. Thus, the drug identification system 100 of this embodiment is very convenient for users who identify a large number of drugs in a day, as it can identify the drug inside the dispensing paper without tearing the paper. Furthermore, when identifying medications brought into the hospital in their original packaging, identifying them while they are still in their original packaging eliminates the need for repackaging before returning them to the patient, thus shortening the overall identification time. In addition, eliminating the need to open the packaging eliminates waste from repackaging after identification, preventing the generation of unnecessary waste and the waste of packaging paper for repackaging. Thus, this invention contributes to achieving the Sustainable Development Goals (SDGs). Moreover, the tray 300 is provided with multiple packaging paper guides to properly hold and move the packaging paper. Specifically, a front packaging paper guide 303 and a rear packaging paper guide 304 are provided near the handle portion 302. In this embodiment, the packaging paper is placed in a position where it is held in place while moving and identification is being performed, with the parallel grooves sandwiched between the front packaging paper guide 303 and the rear packaging paper guide 304. Furthermore, the tray 300 is equipped with a clip 305 at the base of the handle portion 302. This clip 305 is designed to hold the packaging paper, allowing identification work to be performed without having to individually cut off the long pieces of packaging paper. In particular, long pieces of packaging paper can easily slide or shift due to their weight, so it is necessary to maintain their position while the identification work is being performed.In this embodiment, there is no need to separate the individual packets according to their usage instructions, and daily usage confirmation can be performed by sliding the packets within the drug imaging device 200. Furthermore, when placing a single separated packet, it can be set in the drug imaging device 200, and when moving it from another workbench or other location, it can be secured with the clip 305, enabling stable transport and identification work.

[0044] Figure 6(b) is a perspective view of the tray 300, which is dedicated to dispensing paper, from another direction above. The petri dish body 301 is equipped with an imaging glass 306. Through this imaging glass 306, the dispensed drugs can be imaged not only from above but also from below. In this embodiment, an upper deflection plate 307 is provided longitudinally between the multiple rear dispensing paper guides 304, perpendicular to the direction in which the dispensing paper moves. This upper deflection plate 307 is installed to prevent glare on the drug surface caused by reflection of light from the upper ring illumination. In this embodiment, ring illumination is used in addition to bar illumination, but unlike bar illumination, it is difficult to install the deflection plate to cover the entire ring shape. Therefore, glare may occur due to reflection of light from the ring illumination. In particular, when trying to take an image that highlights the markings on the drug to be imaged, glare occurs due to reflection on the surface of the film-like dispensing paper. This glare caused the problem of the drug markings becoming difficult to see.

[0045] Figure 6(c) is a perspective view of the tray 300, which is dedicated to dispensing paper, as seen from the rear. In this embodiment, since ring illumination is also installed at the bottom, lower deflection plates 308 and 309 are installed. Similar to the top, glare may occur on the surface of the drug due to reflection of illumination from the lower ring illumination. In particular, when attempting to take an image that highlights the markings on the drug to be imaged, glare occurs due to reflection on the surface of the film-like dispensing paper. Since the dispensing paper is placed on the imaging glass 306, glare also occurs due to reflection from the imaging glass 306. For this reason, multiple lower deflection plates are provided, but the position and angle of these deflection plates are merely examples, and it is of course possible to change their position and angle depending on the arrangement of the ring illumination, but it is even more preferable that they have a length at least equal to the length of one dispensing paper. Furthermore, the tray 300 or imaging glass 306 is provided with a gentle slope, although this is not particularly limited, so that the drug in the packaging rolls towards the front packaging guide 303 or the rear packaging guide 304. This is because the instructions for use are often printed, pen-lined, or on a white background on the film of the packaging. By arranging the packaging so that the drug is concentrated in the transparent areas of the film that do not have printing, pen-lined, or a white background, the accuracy of identification can be improved. In this way, without significantly changing the structure of the drug imaging device described in the prior art literature section, by inserting a tray 300 specifically for packaging with a deflection plate into the drug imaging device 200, the user can easily perform identification work for both tablets and capsules that are brought in loose and those that are packaged.

[0046] As shown in Figure 2, the top surface and the upper front portion 201 of the drug imaging device 200 are covered by a cover 214. As shown in Figure 3, the cover 214 can be opened upward. In detail, the cover 214 is hinged to the upper rear end of the upper case 213 by a hinge 257 (see Figure 4), and the cover 214 is rotatable vertically around this hinge 257.

[0047] Now, let's return to Figure 3. Figure 3 shows the drug imaging device 200 with the cover 214 open. As shown in the figure, when the cover 214 is opened, the upper internal space 240, which is the upper part 201 of the internal space of the drug imaging device 200, is opened to the space in front of the drug imaging device 200. The user can then set the tray 300 into the fitting part 223 of the mounting part 220 through this upper internal space 240. As shown in Figure 3, the front outer surface of the upper case 213 has a roughly inverted U shape. This front outer surface is inclined, forming an inclined part 215. Specifically, the inclined part 215 is inclined such that the distance from the back of the drug imaging device 200 decreases as you go from bottom to top. This configuration makes it easier for the user to place the tray 300 into the upper internal space 240. Users often insert the tray 300 into the upper internal space 240 by moving it from an upward diagonal direction to an upward diagonal direction. When doing so, if the inclined portion 215 is tilted so that it moves inward as it goes upward, a larger space is secured in front of the upper part of the inclined portion 215, making it less likely for the tray 300 to hit the upper case 213. The front surface of the cover 214 is configured to cover this inclined portion 215. Therefore, when the cover 214 is closed, its front surface is located near the inclined portion 215 and is parallel to it. In this way, the top surface and front surface of the cover 214 also form an obtuse angle, corresponding to the obtuse angle between the top surface of the upper case 213 and the inclined portion 215.

[0048] The lower end of the inclined section 215 is located further back than the outer front surface of the guide member 230 and the lower case 211. In other words, the distance from the back of the drug imaging device 200 to the lower end of the inclined section 215 is smaller than the distance from the back of the drug imaging device 200 to the outer front surface of the guide member 230 and the outer front surface of the lower case 211. This configuration makes it easier for the user to insert the tray 300 into the upper internal space 240.

[0049] Furthermore, a separate inclined surface 216 is provided on the front outer surface of the upper case 213, distinct from the inclined section 215. More specifically, the inclined surface 216 is located inside the U-shape of the roughly inverted U-shaped inclined section 215. This inclined surface 216 is located in front of the first camera 410 (see Figure 4), which will be described later, and protects the first camera 410. This prevents the tray 300 from hitting the first camera 410 and causing the position or orientation of the first camera 410 to shift when the user inserts the tray 300 into the upper internal space 240. As shown in Figure 3, the inclined surface 216 has a roughly rectangular shape and extends in the left-right direction of the drug imaging device 200. The distance from the back of the drug imaging device 200 to the inclined surface 216 gradually decreases from top to bottom. That is, the horizontal position of the upper end of the inclined surface 216 is closer to the back of the drug imaging device 200 than the horizontal position of the lower end of the inclined surface 216. This makes it less likely for the tray 300 to hit the slope 216 during insertion, and makes it easier for the user to insert the tray 300 into the back of the upper internal space 240.

[0050] As shown in Figure 3, a pair of guide members 230 are provided between the upper case 213 and the mounting section 220. The outer surfaces of these guide members 230 extend vertically and are coplane with the outer surfaces of the upper case 213. In contrast, as shown in Figures 3-5, the inner surfaces of the guide members 230 are inclined with respect to the vertical, thereby forming a slope 231. This slope 231 has a roughly rectangular shape and extends in the front-to-back direction of the drug imaging device 200. The slope 231 also faces the upper internal space 240. One of the slopes 231 is located above and to the right of the mounting section 220, and the other is located above and to the left of the mounting section 220. The slope 231 is formed so that it approaches the central axis of the drug imaging device 200 as it goes from top to bottom. That is, in a plan view, the horizontal position of the lower end of the slope 231 is closer to the mounting section 220 than the horizontal position of the upper end. Furthermore, the distance between the lower ends of the two inclined surfaces 231 is approximately the same as or slightly larger than the width of the fitting portion 223 of the tray support member 221 (see Figure 6(a)). Specifically, it is preferable that the distance between the lower ends of the two inclined surfaces 231 be about 1 to 1.2 times the width of the fitting portion 223. When the drug imaging device 200 is viewed from above, a portion of the lower end of the inclined surface 231 is located in a position that is almost in contact with the periphery of the hole 222 of the tray support member 221. This configuration makes it easier for the user to place the tray 300 on the fitting portion 223 of the tray support member 221.

[0051] §2.2 Internal Configuration of Drug Imaging Device Figures 4 and 5 show the drug imaging device 200 with its casing removed, i.e., they show the main internal components of the drug imaging device 200. Figure 4 is a perspective view of the drug imaging device 200, and Figure 5 is a front view.

[0052] As shown in Figure 4, a frame 250 is provided on the rear side of the drug imaging device 200. Many of the main components of the drug imaging device 200 are directly or indirectly attached to this frame 250. The frame 250 consists of a first column 251, a second column 252, a third column 253, a first beam 254, a second beam 255, a third beam 256, and a hinge 257. When viewed from the front, the first column 251 is located on the left rear side of the drug imaging device 200, the second column 252 is in the center of the rear side, and the third column 253 is on the right rear side. These first column 251, second column 252, and third column 253 are connected by the first beam 254, second beam 255, and third beam 256. Furthermore, a hinge 257 that rotates the cover 214 (see Figure 3) is attached to the third beam 256, which is located at the top of the frame 250.

[0053] As shown in Figures 4 and 5, the drug imaging device 200 includes a first camera 410, a second camera 420, a first light source 430, a second light source 440, a third light source 450, a fourth light source 460, a tray support member 221, and a guide member 230. Of these, the tray support member 221 and the guide member 230 are directly attached to the frame 250, while the first camera 410, the second camera 420, the third light source 450, and the fourth light source 460 are attached to the frame 250 via support members. The first light source 430 is housed within the tray support member 221 and supported by the tray support member 221. Although not shown in the figures, the second light source 440 is attached to the lower case 211 (see Figure 2) via support members. In this way, the main internal components of the drug imaging device 200 are attached to the frame 250 rather than the case 210. This makes it easy to remove the case 210 and facilitates maintenance of the main internal components. While there are no specific restrictions, it is also possible to use the built-in camera of a mobile phone or tablet device as a camera for identification. In this case, the user can determine whether the drug or packaging has a single-sided or double-sided engraving by looking at the actual product, making the work easier compared to viewing an image. In other words, when displaying the identification results, candidates with single-sided engravings and candidates with double-sided engravings can be separated using tags or other means, and the engraving type displayed in the list can be standardized to one or the other. Since the user is constantly determining whether the engraving is single-sided or double-sided through images during the work, switching cameras in this way can lead to a reduction in work time.

[0054] The tray support member 221 is located near the center of the drug imaging device 200. A guide member 230 is located directly above the tray support member 221. A third light source 450 is located above the guide member 230. A first camera 410 is located at a higher position than the third light source 450, i.e., near the top surface of the drug imaging device 200. A second light source 440 is located below the tray support member 221 at a predetermined distance. A fourth light source 460 is located at a lower position than the second light source 440, near the side surface of the drug imaging device 200. A second camera 420 is located at a lower position than the fourth light source 460, i.e., near the bottom surface of the drug imaging device 200. Thus, in the drug imaging device 200, the upper light source is composed of the first light source 430 and the third light source 450. In addition, the lower light source is composed of the second light source 440 and the fourth light source 460.

[0055] Figure 7 is a cross-sectional view showing a sachet containing the drug to be identified, placed on the imaging glass 306. An upper deflection plate 307 is provided on the upper surface of the imaging glass 306. The sachet is placed to the right of the upper deflection plate 307. Lower deflection plates 308 and 309 are provided on the lower surface of the imaging glass 306. The first light source 430 and the second light source 440, which are ring illuminations, and the third light source bar illumination 450 and the fourth light source bar illumination 460 are positioned above and below the imaging glass 306, respectively. This figure is for illustrative purposes to explain the relationship between the illuminations and the deflection plates placed on the imaging glass 306; please refer to Figures 4 and 5 for the actual arrangement. The upper deflection plate 307 is installed to prevent glare caused by reflection of illumination from the upper ring illumination, and the lower deflection plates 308 and 309 are installed to prevent glare caused by reflection of illumination from the upper ring illumination. Normally, for bar lights 450 and 460, which have a linear shape and small surface area, deflection plates are installed near the bar lights. However, for ring lights 430 and 440, a deflection plate covering the entire ring light would be too large and impractical. Furthermore, ring lights have a complex light reflection pattern. For example, even if the light is only partially ring-shaped, not just perfectly circular, the light reflection pattern becomes complex. Therefore, deflection plates for ring lights are installed on the surface of the imaging glass 306. Here, the upper deflection plate 307 is manufactured to a height of 8 to 14 mm. The deflection direction of the upper deflection plate 307 is set to the vertical direction as shown in the figure. The lower deflection plate 308 is manufactured to a length of 8 to 13 mm. The deflection direction of the lower deflection plate 308 is set to the horizontal direction as shown in the figure. The lower deflection plate 309 is manufactured to a length of 1.5 to 4 mm and at a 45-degree angle to the surface of the imaging glass 306. The deflection direction of the lower deflection plate 309 is set at a 45-degree angle as shown in the figure. These dimensions, angles, and deflection directions are merely examples, and other angles may be used as long as they can suppress the glare reflected on the surface of the film-like packaging paper.

[0056] Figure 8 is a schematic diagram illustrating the effect of ring illumination and deflection plates. This figure shows the upper illumination in Figure 7 viewed from the left. A first light source 430, which is ring illumination, and a third light source 450, which is bar illumination, are positioned above the imaging glass 306 surface. A deflection plate for the left and right directions is positioned directly below the third light source 450, which is bar illumination. On the other hand, since the first light source 430, which is ring illumination, emits light from 360 degrees, glare due to reflection of the illumination may occur, as shown on the left side of (Equation 1) below. In particular, when attempting to take images that highlight the markings on the drug to be imaged, glare occurs due to reflection on the surface of the film-like packaging paper. This glare caused the drug markings to become difficult to see. For this reason, a vertical deflection plate 453 is installed on the imaging glass 306 surface to prevent glare from the ring illumination, as shown on the right side of (Equation 1) below.

number

number

[0057] As shown in Figures 4 and 5, a third light source 450 is provided above the guide member 230. This third light source 450 is composed of multiple bar lights, specifically a first bar light 451 and a second bar light 452. The first bar light 451 is attached to the third column 253 via a support member 453. The second bar light 452 is attached to the first column 251 via a support member 454. In other words, these first bar lights 451 and second bar lights 452 are provided to the side of the drug imaging device 200, specifically to the side of the mounting section 220. As shown in Figure 5, the first bar light 451 and second bar light 452 are installed in a direction parallel to the front-rear direction of the drug imaging device 200, and the light-emitting surfaces of the first bar light 451 and second bar light 452 are directed diagonally downward, more specifically toward the mounting section 220. Furthermore, the distance from the first bar light 451 and the second bar light 452 to the location on the mounting section 220 where the drug is placed is greater than the distance from the first ring light 431 to the location on the mounting section 220 where the drug is placed. Moreover, the first bar light 451 and the second bar light 452 are positioned closer to the side of the drug imaging device 200 than the lower end of the inclined surface 231 of the guide member 230 and the outer circumference of the hole 222 of the tray support member 221. This arrangement of the third light source 450 ensures a larger space in the upper internal space 240, making it easier for the user to place the tray 300 on the mounting section 220.

[0058] The first bar light 451 and the second bar light 452 are equipped with polarizing filters, and are configured so that diffused light reaches the drug through these polarizing filters. Specifically, direct light from the first bar light 451 and the second bar light 452 is cut off by the polarizing filters, and the light diffused by the polarizing filters is configured to reach the drug through the holes 222 in the tray support member 221, i.e., within the ring of the first ring light 431. In other words, the third light source 450 functions as a diffuse light source or an indirect light source. This allows the upper side of the drug to be suitably illuminated. The inventors have found that with the third light source 450 configured in this way, the printing on the surface of the drug can be clearly photographed. Moreover, the printing can be clearly photographed whether the drug is a tablet or a capsule.

[0059] As shown in Figures 4 and 5, a fourth light source 460 of the same type as the third light source 450 is provided below the second light source 440. This fourth light source 460 is composed of multiple bar lights, specifically a third bar light 461 and a fourth bar light 462. The third bar light 461 is attached to the third column 253 via a support member 463. The fourth bar light 462 is attached to the first column 251 via a support member 464. In other words, these third bar lights 461 and fourth bar lights 462 are provided on the side of the drug imaging device 200, similar to the first bar light 451 and second bar light 452. Also, the third bar light 461 and fourth bar light 462 are installed in a direction parallel to the front-to-back direction of the drug imaging device 200, similar to the first bar light 451 and second bar light 452. The light-emitting surfaces of the third bar light 461 and the fourth bar light 462 are directed diagonally upward, more specifically toward the mounting section 220. Similar to the relationship between the third light source 450 and the first light source 430, the distance from the third bar light 461 and the fourth bar light 462 to the mounting section 220 is greater than the distance from the second ring light 441 to the mounting section 220. The third bar light 461 and the fourth bar light 462, like the first bar light 451 and the second bar light 452, are equipped with polarizing filters, and the light diffused through these polarizing filters is configured to reach the chemical. Specifically, the diffused light from the third bar light 461 and the fourth bar light 462 is configured to pass through the ring of the second ring light 441 and reach the chemical. This illuminates the underside of the chemical so that the printing is clear. In this embodiment, the third light source 450 and the fourth light source 460 are each composed of two bar lights, but in other embodiments, four bar lights may be provided. In this case, it is preferable to arrange the bar lights so that the four bar lights form a square and surround the mounting section 220. Needless to say, in this case the arrangement of the deflection plates provided on the surface of the imaging glass 306 will be different.

[0060] A first camera 410 is installed above the first bar light 451 and the second bar light 452. This first camera 410 is attached to the second column 252 via a mounting member 411. The mounting member 411 fixes the first camera 410 so that it is positioned vertically above the mounting section 220 and faces the mounting section 220. When the drug imaging device 200 is viewed from above, the imaging area of ​​the first camera 410 includes the hole 222 of the tray support member 221 and the inside of the ring of the first ring light 431. As a result, the first camera 410 can suitably capture an image of the top surface of the drug placed on the mounting section 220, i.e., an image of the drug viewed from directly above.

[0061] A second camera 420 is installed below the third bar illumination 461 and the fourth bar illumination 462. This second camera 420 is attached to the second column 252 via a mounting member 421. The mounting member 421 fixes the second camera 420 so that it is positioned vertically below the mounting section 220 and faces the mounting section 220. When the drug imaging device 200 is viewed from below in a plan view, the imaging area of ​​the second camera 420 includes the inside of the ring of the second ring illumination 441. As a result, the second camera 420 can suitably capture an image of the underside of the drug placed on the mounting section 220, i.e., an image of the drug viewed from directly below. Both the first camera 410 and the second camera 420 described above are capable of capturing color images.

[0062] §2.3 Configuration of mounting items §2.3.1 First Embodiment of a Mounting Item The drug imaging device 200 uses a tray 300, as shown in Figure 6, as a mounting item for drugs. The tray 300 has a rectangular, flat body 320 with holes 321 formed in it, into which a transparent cylinder 322 is fitted. A petri dish 310, as shown in Figure 8, is placed inside this cylinder 322.

[0063] §3 Overview of Drug Identification Software As shown in Figure 1, the drug imaging device 200 described above is connected to a computer 500. This computer 500 has drug identification software 600, which controls the operation of the drug imaging device 200. Specifically, the computer 500 controls the lighting inside the drug imaging device 200 and causes the camera to take pictures of the drugs. The computer 500 then receives the images of the drugs taken from the drug imaging device 200. The computer 500 processes and modifies the drug image data to obtain information necessary for narrowing down and searching for drugs. Based on this information, the computer 500 narrows down and searches for drugs and displays the results as identification results on the screen 531 of the display device 530.

[0064] §3.1 Basic Operating Procedures Figure 9 shows screen 531 with the drug identification software 600 running. More specifically, the drug identification software is called by the medication guidance support system 610, and this software displays the tablet identification menu window 620 in the lower right corner of screen 531. From this window 620, identification according to the user's purpose is started. As shown in Figure 9, window 620 displays the "Tablet Image Identification" item 621, the "Returned Tablet Image Identification" item 622, and the "Magnifying Glass" item 623. Although window 620 is named "Tablet Identification," it should be noted that the drug identification software 600 can identify not only tablets but also capsules. In the example shown in Figure 9, the medication identification software is launched by another software, but the medication identification software can also be launched and operated independently, for example, by clicking its icon. When the user selects the "tablet image identification" item 621, the "tablet image identification" window 700, as shown in Figure 10, is displayed on screen 531. Message 701 is displayed within the "tablet image identification" window 700. This message 701 prompts the user to place the petri dish 310 containing the drug into the drug imaging device 200 and select the "imaging" item 751A. At this time, the user can switch between petri dish mode and drug packaging mode using the mode switch 751B. In this figure, drug packaging mode is already selected, and pressing the mode switch 751B switches to petri dish mode. The user places the tray 300 on the mounting section 220 of the drug imaging device 200. After placing the packaging paper to be identified in tray 300 and closing the cover 214 of the drug imaging device 200, the user touches (when inputting via touchscreen 532) or clicks (when inputting via mouse 542) the "Capture" item 751. This causes the drug imaging device 200 to capture an image of the placed drug. As a result, the "Tablet Image Identification" window 700 displays a display as shown in Figure 11. This mode switching allows the user to easily switch modes according to the state of the drug to be identified.As shown in Figure 11, the "Tablet Image Identification" window 700 is broadly divided into a first section 710, a second section 720, a third section 730, a fourth section 740, a first item display section 750, and a second item display section 760. The first section 710 and the second section 720 are located on the left side of the "Tablet Image Identification" window 700. By default, the widths of the first section 710 and the second section 720 are approximately the same. The third section 730 and the fourth section 740 are located on the right side of the "Tablet Image Identification" window 700. The widths of the third section 730 and the fourth section 740 are approximately the same. The first item display section 750 is located at the bottom of the "Tablet Image Identification" window 700. The second item display section 760 is located below the third section 730. These first item display section 750 and second item display section 760 display various items that can be selected by the user.

[0065] The first section 710 displays an image of the petri dish 310 taken by the drug imaging device 200. Preferably, this image is taken with the first light source 430 or the second light source 440 turned on. This is because when images are taken under such lighting, the contrast between the background and the drugs tends to be greater. This image contains areas that include multiple drugs that have been photographed. At the top of the first section 710 are the "View from Above" tab 711 and the "View from Below" tab 712. When the user selects the "View from Above" tab 711, an image of the drugs taken from above is displayed in the first section 710. When the user selects the "View from Below" tab 712, an image of the drugs taken from below is displayed in the first section 710. When the computer 500 acquires an image of the petri dish 310, the computer 500 automatically recognizes the areas occupied by each drug based on the image. The computer then automatically counts the recognized drugs and displays the resulting number of drugs in the gauge section 752 of the first item display section 750. Furthermore, the computer 500 automatically identifies each recognized drug sequentially in the background. The computer 500 then displays a bar on the gauge unit 752 indicating the number of drugs that have been automatically identified and the percentage of all drugs in the petri dish 310 that have been identified. In this embodiment, the progress of the identification is displayed graphically, so the user can grasp at a glance how far the identification has progressed. The user can also forcibly stop the automatic identification by the computer 500 by selecting the "Stop" item 753.

[0066] As shown in Figure 11, the computer 500 sequentially attaches a first marker 713 to each drug that has been identified, indicating that the identification is complete. This first marker 713 consists of, for example, a solid line surrounding the drug, and this line is drawn in a predetermined color (for example, red). When the user selects one of the drugs displayed in the first section 710, the top drug in the example shown in Figure 11, the "tablet image identification" window 700 changes as shown in Figure 12. First, the image of the selected drug is enlarged and displayed in the second section 720. At the top of the second section 720, an image of the selected drug taken from above is displayed, and at the bottom, an enlarged image of the selected drug taken from below is displayed. To save the user time, depending on the settings of the drug identification software 600, the computer 500 may automatically select the drug from among the drugs that have been identified.

[0067] Figure 43 shows the display screen when the user sets the tray 300 in the drug imaging device 200 and performs the drug packaging identification work in the drug packaging mode, which is labeled "Drug Packet" in the mode switch 751B described in Figure 10. As shown in Figure 2, the user sets the tray 300, which is specifically for drug packaging and contains the drug packaging, in the drug imaging device 200 instead of a circular petri dish, and performs the identification of the drug packaging. Then, the drug in its packaged state is displayed in the first section 710 of the "Tablet Image Identification" window 700.

[0068] Figure 44 shows a magnified view of the medication in its individual packaging, as displayed in the first section 710 mentioned above. In the packaging mode preview screen, a video is displayed, allowing the user to adjust the shooting position of the medication in the packaging within the indicated red frame area while viewing the screen. When photographing medication in individual packaging, the instructions for use are often printed, written with a pen liner, or on a white background on the film of the packaging. Therefore, if the medication is not photographed while avoiding such printed areas, pen liner, or white background areas, accurate information cannot be obtained (see Equation 3 below). In this implementation example, in packaging mode, semantic segmentation is used to detect the shooting area (see Equations 4 and 5 below). This prevents the detection of areas labeled as "background" (see Equations 6 and 7 below), making it possible to photograph only the area where the medication is displayed (see Equation 8 below). As a result, in (Equation 8), packaging mode makes the decision not to use the camera image on the side hidden by the white band. This means that if the instructions for use are printed, written with a pen liner, or placed on a white background on the film of the packaging paper, the system will not detect those areas, but will only detect and process the areas containing the medication. It should be noted that semantic segmentation is merely one example; any deep learning algorithm that associates labels or categories with image pixels will suffice, and is not limited to this. Any method that uses artificial intelligence to recognize clusters of pixels that form characteristic categories can realize this embodiment, and the most suitable method will be utilized as advancements in artificial intelligence progress.

number

number

number

number

number

number

[0069] As shown in Figure 28, in the second section 720A of the "Tablet Image Identification" window 700, an image of the drug is displayed, but because the outline of the drug was extracted, cases of distorted shapes occurred. Conventionally, the drug was extracted and displayed from the region acquired by image processing. Until now, since the outline of the drug was extracted using only image processing, distorted shapes were rare, but when identifying the packaging paper, as mentioned above, detection was decided to be done using AI (Semantic Segmentation). This allows the system to identify the region to be recognized as a tablet even when the drug overlaps with the printing, pen liner lines, or white background printed on the film of the packaging paper, by learning the background. However, on the other hand, as shown in the second section 720A of the tablet identification screen, it was found that tablets were sometimes displayed with distorted outer edges because they could not be recognized as circular. When detection is done using AI (Semantic Segmentation), the outline is extracted and displayed, which is why such distorted displays occur. Therefore, in the drug packaging mode, instead of using contours obtained through image processing, the image is cropped from the original image using a bounding rectangle that follows the contour. For example, even for solid white areas or printed drug information where objects cannot be detected through image processing, the area of ​​the drug is displayed by cropping a bounding rectangle that follows the contour from the original image, as shown in the second section 720B, resulting in a more easily viewable image. Also, as shown in Figure 48, in drug packaging mode, a drug count confirmation screen is displayed when you press the "New Shoot," "Additional Shoot," or "Reshoot" button for identification. This image uses AI (semantic segmentation) to extract the contours of all drugs on the packaging paper, and then overlays them on the overall image, drawing the contours with color-coded lines to make it easier to visually confirm whether there is a difference between the actual number of drugs and the number obtained by image processing. In addition, the drug count confirmation screen can display a magnified view of each individual tablet.

[0070] In Figure 45, the first section 710 and the second section 720 of the "Tablet Image Identification" window 700, which was photographed in drug packaging mode, are displayed, respectively. Some drug packaging papers have an image of the drug contained within printed on the back as a sample photograph, which is stored in the drug information database. In this case, as shown in Figure 42, an image of the drug actually contained within the packaging paper is displayed in an image taken from above. On the other hand, an image of the drug printed on the back of the packaging paper is displayed in an image taken from below. More specifically, in the first section 710, selecting the "View from above" tab 711 displays an image of the drug taken from above. Selecting the "View from below" tab 712 displays an image of the drug taken from below. In addition, the top of the second section 720 displays an enlarged image of the drug actually contained within the packaging paper. Then, at the bottom of the second section 720, an enlarged image of a sample photograph of the drug printed on the back of the packaging is displayed. In this case, the drug identification software 600 performs drug identification separately for each image in the background. As a result, two identification results are obtained within the drug identification software 600. From these, candidate drugs that receive high scores in both identifications are weighted in rank. That is, for candidate drugs that receive high scores in both identifications, an additional score is added to the sum of the scores from both identifications. As a result, candidate drugs that receive high scores in both identifications are displayed higher up in the list of candidate drugs shown in the third section 730. In this way, when the captured images include two types of images—an actual photograph of the drug and a photograph of a printed sample—the accuracy of identification is improved by performing drug identification independently for both and then combining the obtained identification results to create a list of candidate drugs. In particular, the accuracy of identification is further improved if the sample photograph is printed in color (especially full-color printing using an inkjet printer).

[0071] Returning to Figure 12, in the first section 710, a second marker 714 is attached to the selected drug so that it can be identified by the user or the computer 500. This second marker 714 looks different from the first marker 713. For example, the second marker 714 consists of a dashed line surrounding the drug. The second marker 714 may also be displayed as an animation to make it stand out which drug is selected. For example, the dashed line surrounding the drug may rotate around the drug. This makes it easy for the user to identify which of the multiple drugs displayed in the first section 710 is the drug displayed in the second section 720. The second marker 714 may be drawn in the same color as the first marker 713 (e.g., red) or in a different color (e.g., green). If the identification of the selected drug is complete, the computer 500 displays a list of candidate drugs obtained as a result of the identification in the third section 730. Each row in the list displays information for one candidate drug. By default, drugs that are statistically more likely to be correct are displayed higher up in the list. In other words, the list is sorted from top to bottom, from most likely to least likely. The score obtained from the search results is an indicator of how likely a drug is to be correct. Specifically, a high score for the probability of matching with the drugs found in the search results indicates a high probability that the drug is correct, while a low score indicates a low probability.

[0072] Each row in the list displays candidate drug information, from left to right: a score indicating the likelihood that this candidate drug is correct, an image of the drug stored in the database, a marking or printed code, and the drug name. By changing the settings, the ranking of this candidate drug's likelihood among all candidates can be displayed instead of the score. Furthermore, in addition to the drug name, a code identifying the drug (e.g., a YJ code) can also be displayed. Displaying information about a single drug on a single row in this way makes it easier for the user to understand. Additionally, the dimensions of the drug may be displayed on each row in the list.

[0073] Figure 46 shows the preview tabs for the drug packaging mode. In the first section 710 of the "Tablet Image Identification" window 700, where a drug is photographed in drug packaging mode, tabs labeled "Preview Top" and "Preview Bottom" are provided. In drug packaging mode, since the drug is expected to overlap with the printing, pen liner lines, and white background printed on the film of the packaging paper, the "Preview Top" tab automatically displays an image of the packaging paper taken from above, and the "Preview Bottom" tab displays an image of the packaging paper taken from below. By providing such tabs, the user can check the contents without actually selecting the "Photograph" item 751A, reducing the number of clicks required for the operation. This allows for quick identification of long packaging papers. The user can also select drugs from the drugs displayed in the first section 710 that have not yet been identified by the computer 500. For example, if a capsule displayed in the lower right of the first section 710 is selected, the selected drug is enlarged and displayed in the second section 720. However, since the computer 500 has not yet completed the identification process, the identification result is not displayed in the third section 730. Instead, the third section 730 displays a message indicating that the selected drug is currently being identified by the computer 500. Thus, in this embodiment, the user can also select drugs that have not yet been identified. This is convenient because it allows the user to immediately enlarge and view the image of the drug they are currently interested in. On the other hand, drugs that have been identified are marked with the first label 713, as shown in Figure 11. Therefore, during the process, it is easy for the user to see which drugs have been identified. If a drug that has been identified is selected, the user can immediately see the identification result from the computer 500 without having to wait. For this reason, the presence of the first label 713 greatly contributes to reducing the user's working time.

[0074] As shown in Figure 13, when the mouse cursor 543 is moved near an image of a drug in the list displayed in the third section, the image of the drug pointed to by the mouse cursor 543 is enlarged. In this way, by enlarging one of the images of multiple drugs displayed in the third section 730 in response to the user's operation, the image of the drug the user is currently interested in can be seen in detail immediately. This also makes it easy for the user to compare the drug images displayed in the second section 720 with the drug images displayed in the third section. When the mouse cursor 543 is moved away from the enlarged drug image, the enlargement is released and the drug image returns to its original size. The user then compares the information of each candidate drug listed in the third section 730 with the enlarged drug in the second section 720 and selects the drug that the user determines to be correct. Specifically, the user selects the row in the third section 730 that displays the information of the correct drug. Although not shown in the figure, when the user selects a row that displays drug information, at least a portion of the row is highlighted.

[0075] Subsequently, once the user is confident that the selected medication is correct, they select the "Select" item 761. As a result, as shown in Figure 14, the information of the selected medication is added to the fourth section 740. The added information, that is, the information displayed on the added line, is similar to the line selected in the third section 730. Specifically, the added line displays the image of the medication stored in the database, its marking or print code, the drug name, and the quantity of the medication. This design makes it easy for the user to later review which medications they have selected.

[0076] Furthermore, if the user believes that any of the candidate drugs listed in the third section 730 is clearly not a candidate drug, the user selects the row in the third section 730 that displays such information and selects the "Not Applicable" item 763. This removes the information for that candidate drug from the third section 730. Then, information for the next most likely correct candidate drug, which was not on the list, is added to the third section 730. Also, if the user wishes to find further information about a drug from the drug image displayed in the second section 720 or the drug information displayed in the third section 730, the user selects the "Manual Search" item 764. This allows the user to manually search for information about the drug.

[0077] For drugs selected in the third section 730 and displayed in the fourth section 740, the corresponding drug is marked with a third marker 715 in the first section 710. The third marker 715 indicates that the user has completed the selection and verification process. This third marker 715 has a different appearance from the first marker 713 mentioned above. For example, the third marker 715 consists of a solid line surrounding the drug, and this line is drawn in a different color (e.g., blue) than the first marker 713. Having different appearances for the first marker 713 and the third marker 715 allows the user to see at a glance which drugs have been verified and which have not, improving the user's work efficiency. In particular, if the first marker 713 and the third marker 715 are colored differently, the user can intuitively see at a glance how far along the drug verification process is, and can grasp how much more work is needed until all drugs have been identified.

[0078] If, with respect to a drug selected by the user and added to the fourth section 740, the user realizes they made a mistake in their judgment and wishes to cancel the selection, the user selects the row in the fourth section 740 displaying the information of the drug to be canceled, or the image of the drug to be canceled in the first section 710. Then, with the drug selected, the user selects the "Deselect" item 762. This removes the selected drug from the fourth section 740. In the first section 710, the third marker 715 attached to the selected drug is reverted to the first marker 713 or the second marker 714.

[0079] Once the verification process for one drug is complete, the user selects another drug that has not yet been verified in the first section 710 and performs the verification process for this selected drug in the same manner as described above. Depending on the settings of the drug identification software 600, once the verification process for one drug is complete, the computer 500 automatically selects the next drug from the drugs that have already been identified. In this case as well, the user performs the verification process for the selected drug in the same manner as described above. Once the user has completed the verification process for all drugs in this manner, the "Tablet Image Identification" window 700 will look like the one shown in Figure 15.

[0080] As shown in the figure, for all drugs displayed in the first section 710, once the user has completed all selections in the third section 730, or, depending on the settings, once the automatic identification by the computer 500 has been completed, the following items are added to the first item display section 750: "Register" item 754, "Additional Image" item 755, "Retake Image" item 756, and "Page Turn" item 757. When the user selects the "Register" item 754, the drug identification process is completed. That is, when the user selects the "Register" item 754, the identification result is sent to the medication guidance support system 610 (see Figure 9), and the "Tablet Image Identification" window 700 is closed. If there are any drugs that the user wants to identify further, the user places these drugs in the petri dish 310 and sets the petri dish 310 in the drug imaging device 200. Then, the user selects the "Additional Image" item 755. This allows for the imaging of additional drugs, and the user can perform drug identification on these drugs in the same manner as described above. If multiple image sets for Petri dish 310 are generated in this way, the user can move between the multiple image sets by selecting the "Page Turn" item 757. If the user wishes to redo the drug identification performed this time, they can restart from the drug image imaging stage by selecting the "Reshoot" item 756. Selecting the "End" item 758 will interrupt the drug identification process and close the "Tablet Image Identification" window 700.

[0081] In another embodiment, as shown in Figure 29, a "Print" item 759 is further displayed in the first item display section 750. When this "Print" item 759 is selected, the identification results are printed, as shown in Figure 30. Items to be printed include the title, date and time of identification, identification number of the drug identification device, patient name, drug serial number, drug name, markings (or printing), number of drugs, total number of drugs, etc. Furthermore, although not included in Figure 30, items to be printed may also include the name of the pharmacist in charge, a photographed image, or an image of the drug stored in the database. Regarding the identification results, there are times when a detailed identification report needs to be written, and times when it is sufficient to record the identification results in a simple document. Printing as shown in Figure 30, so-called journal printing, is well-suited for the latter use. Furthermore, there are also cases where the identification results want to be pasted into a handwritten medical record. Journal printing is also well-suited for such uses.

[0082] When the drug identification software 600 transmits the identification results to the medication guidance support system 610 (see Figure 9), the computer 500 may also transmit the number of drugs and their usage instructions to the medication guidance support system 610 in addition to the identification results. The number of drugs is obtained by "grouping," which will be described later. The computer 500 may also automatically group and count drugs that have been identified as the same drug through identification. The computer 500 may open a usage instruction input window on screen 531, where it may accept input of usage instructions for the identified drugs from the user. In the medication guidance support system 610, users often input usage instructions. Allowing the drug identification software 600 to input usage instructions at the same time as identifying drugs improves the convenience of user input.

[0083] Furthermore, if the bag in which the drug is packaged has information that identifies the drug, the drug identification software 600 does not need to perform the image identification described above. For example, if the bag in which the drug is packaged has a barcode, and the computer 500 can identify the drug inside the bag by scanning this barcode, the drug identification software 600 may omit the image identification described above and transmit the drug information obtained by scanning the barcode to the medication guidance support system 610. Adding such a function to the drug identification software 600 improves user convenience.

[0084] §3.2 Useful Features §3.2.1 Grouping During drug identification, a "grouping" item 765 is displayed in the second item display section 760, as shown in Figures 11-15. When this "grouping" item 765 is selected, a "grouping" window 630 opens on screen 531, as shown in Figure 16. Within this "grouping" window 630, the user can group multiple arbitrary drugs.

[0085] As shown in Figure 16, the "Grouping" window 630 consists of a first section 631 and an item display section 632. The first section 631 displays an enlarged version of the same image as the first section 710. Specifically, the first section 631 displays images of the petri dish 310 taken from above and images taken from below side by side.

[0086] The user selects multiple drugs that are judged to be of the same type from the images displayed in the first section 631 by touching or clicking. In another embodiment, the computer 500 determines whether the drugs displayed in the first section 631 are of the same type or not, and the computer 500 automatically selects multiple drugs that it has determined to be of the same type. The determination of whether they are of the same type or not can be made by performing template matching, described later, on each drug displayed in the first section 631. Drugs selected by the user or the computer 500 are marked with a grouping marker 633. In this state, if the user selects the "Group" item 634, the selected drugs are grouped together. That is, drugs that have been selected and marked with the grouping marker 633 are grouped together. If the user selects the "Deselect" item 635, the drug selection made by the user in the first section 631 is deselected, and the grouping marker 633 disappears from each drug. Also, if the user selects the "Ungroup All" item 636, any existing grouping of drugs is ungrouped. Finally, when the user selects the "Confirm" item 637, the grouping performed by the user becomes effective. Then, the "Grouping" window 630 is closed, and the original "Pill Image Identification" window 700 becomes active on screen 531. Within the now active "Pill Image Identification" window 700, the relevant drugs are grouped according to the grouping performed by the user. The user can also discard the grouping operation and return to the original "Pill Image Identification" window 700 by selecting the "Close" item 638.

[0087] When multiple drugs are grouped together, and the group includes drugs that have been identified by the computer 500 and those that have not yet been identified, the computer 500 omits the identification of the drugs that have not yet been identified. This reduces the time required for the computer 500 to identify all drugs. Also, when multiple drugs are grouped together, if the user selects one drug in the third section 730, the same selection is automatically made for the other drugs in the same group. Therefore, the effort required for user input is reduced, and the work time is shortened. As shown in Figure 15, once the user has completed the verification process for the grouped drugs and the drugs have been added to the fourth section 740, the number of drugs belonging to that group is displayed. Drugs that are not grouped are displayed as "1".

[0088] §3.2.2 Automated Learning The drug identification software 600 has a function to automatically learn the identification results. More specifically, the computer 500 has a learning database inside. The drug identification software 600 records the drug selected by the user (see Figure 14) in the learning database, and in subsequent automatic identifications, if the same drug is identified by the computer 500, the computer 500 will display the drug previously selected by the user in the third section 730 list with the highest rank.

[0089] Figure 31 shows the details of the learning function of the computer 500. The basic concept is that if the drug selected by the user in the list of the third section 730 has never been selected by the user before, the user's selection is recorded in the learning database. If the drug selected by the user in the list of the third section 730 has been selected by the user once before, the computer 500 displays a message to the effect of, "You selected this drug for this image last time as well. Please confirm once again if this drug is correct," as shown in Figure 32. If the drug selected by the user in the list of the third section 730 has been selected by the user two or more times before, the user has already confirmed the contents recorded in the learning database, so the computer 500 does nothing in particular to the learning database.

[0090] The following explanation is based on the flowchart shown in Figure 31. First, the computer 500 performs automatic identification based on the drug image (step 510). Then, as shown in Figure 12, the computer 500 displays the identification results in a list in the third section 730 (step 520). At this time, if the user's previous selection results are stored in the learning database, the previously selected drugs are displayed in a higher rank. Next, the user selects the correct drug from the list of candidate drugs displayed in the third section 730 (step 530). Next, the computer 500 determines whether this is the user's first, second, or third or later selection (step 540). If it is the first time, that is, if there is no record of the user's selection of the candidate drug selected by the user in the learning database, the computer 500 records the user's selection result in the candidate drug in the learning database (step 551), and also adds the selected candidate drug to the fourth section 740 as shown in Figure 14 (step 552). In the second instance, more specifically, if the learning results for the drug selected by the user are recorded in the learning database but the user-confirmed flag is not turned on, the computer 500 displays message 731 as shown in Figure 32 (step 561). This message 731 confirms with the user that this drug is one that was learned in the previous identification and that the learning results are correct (step 562). Message 731 also displays the name, markings (or printing), image of the drug photographed, and image of the drug stored in the database of the drug selected by the user in the previous identification. If the user selects the "yes" item 732, the computer 500 sets the user-confirmed flag for the learning results recorded in the learning database for that drug (step 563). The computer 500 also adds the selected candidate drug to the fourth section 740 as shown in Figure 14 (step 564). If the user selects the "No" item 733, the computer 500 returns to step 530, and as shown in Figure 12, gives the user the opportunity to select a different candidate drug from the list of candidate drugs displayed in the third section 730.Returning to step 540, in the case of the third time, more specifically, if the learning results for the drug selected by the user have been recorded in the learning database and the user-confirmed flag is turned on, the computer 500 adds the selected candidate drug to the fourth part 740 (step 571). Following this procedure reduces the likelihood that the user will continue to select the wrong drug even if they have previously made a mistake in selecting the wrong drug.

[0091] In another embodiment, whether or not to train the candidate drugs selected by the user in the learning database may be a setting made by the user. For example, the computer 500 may temporarily store the results of the user's selection of candidate drugs, and later (for example, after the end of the day's work), the user may review the selection results all at once, select the candidate drugs that the user wants to train in the learning database, and record only the selected ones in the learning database. Furthermore, in yet another embodiment, for the learning function that uses capsule image data as master data, up to three sets of data may be stored in drug packaging mode. In drug packaging mode, when photographing capsules, the markings may be photographed at an angle rather than directly facing the camera. In this case, when a tilted image becomes the master data, there is a problem that the matching score will not increase unless the image is tilted in the same way. Therefore, in addition to the main learning data, learning data sub1 and sub2 are set. By registering multiple tilted data, a higher matching score can be achieved. Furthermore, if the matching score is 0.6 or higher, and it is higher than the main matching score, the new information will be overwritten. By ensuring that higher data is successively overwritten from training data sub1 to sub2, it is possible to continuously update information with higher matching scores.

[0092] §3.2.3 Image Identification of Returned Tablets As shown in Figure 9, the tablet identification menu window 620 contains the "Returned Tablet Image Identification" item 622. When the user selects this "Returned Tablet Image Identification" item 622, the "Returned Tablet Image Identification" window 640 is displayed on screen 531, as shown in Figure 17. This returned tablet image identification is used when all of the multiple drugs to be identified are of the same type.

[0093] As shown in Figure 17, the "Returned Tablet Image Identification" window 640 consists of a first section 641, a second section 642, a third section 643, a counting and message display section 644, and an item display section 645. The functions and configurations of the first section 641, the second section 642, and the third section 643 are the same as those of the first section 710, the second section 720, and the third section 730 described above.

[0094] First, the user places the tray 310 containing the drug to be identified into the drug imaging device 200 and selects the "imaging" item 646. As a result, an image of the captured drug is displayed in the first section 641, as shown in Figure 18. The computer 500 then automatically counts the drugs and displays the counted number of drugs in the counting and message display section 644. Furthermore, the computer 500 displays message 647 in the counting and message display section 644. This message 647 prompts the user to select one drug from those displayed in the first section 641. Once the user selects one of the drugs displayed in the first section 641, the screen changes as shown in Figure 19. Depending on the settings, the computer 500 may automatically select one of the drugs displayed in the first section 641 as a representative drug on behalf of the user.

[0095] Once a selection is made, the computer 500 performs identification of the drug selected by the user or the computer 500. In the first section 641, a second label 648 is attached to the selected drug to make it easier to identify the drug selected by the user or the computer 500. Once the computer 500 has completed the automatic identification, the computer 500 displays a list of candidate drug information in the third section 643. Message 649 is displayed in the counting and message display section 644. This message 649 prompts the user to select the drug they believe to be correct from the drugs displayed in the third section 643 and select the "Start Identification" item 651. In another embodiment, as shown in Figure 33, the user can also manually search for the drugs displayed in the second section 642 by selecting the "Manual Search" item 656. The user can then add the drugs found through the manual search to the list displayed in the third section 643. This allows users to select medications from among those not initially listed in the third section 643.

[0096] When the user selects the "Manual Search" item 656, the "Manual Search" window 680 is displayed on screen 531, as shown in Figure 41. The "Manual Search" window 680 consists of a search expression input section 681, a search results display section 682, a database image display section 683, and an item display section 684. First, the user enters a keyword or search expression into the type section 685 of the search expression input section 681. In this embodiment, the user can search for drugs by either the markings or the drug name. Next, the user presses the search button 686 in the search expression input section 681. As a result, the search results from the computer 500 are displayed in a list in the search results display section 682. When the user selects one of the drugs displayed in the list, an image of that drug stored in the database is displayed in the database image display section 683. If the user selects the "Excluded" item 687 while a drug is selected in the search results display section 682, the selected drug is removed from the search results display section 682. Selecting the "Select" item 688 adds the selected drug to the third section 643 (see Figure 22). Selecting the "Add New" item 689 adds the selected drug to the learning database mentioned above (see §3.2.2). As a result, the drug selected in the search results display section 682 will always be displayed as a candidate drug in the third section 643 from the next differentiation. Selecting the "End" item 691 cancels the manual search and closes the "Manual Search" window 680.

[0097] Returning to Figure 19, when the user selects a drug in the third section 643, the row showing the selected drug is highlighted, as shown in Figure 20. Also, in the first section 641, the corresponding drug is marked with a third label 652, indicating that the user has completed the verification. If the user then selects the "Start Identification" item 651, the computer 500 performs identification on each drug that was not selected by the user. If the identification results for all remaining drugs are the same as those of drugs that the user has already verified, the "Return Tablet Image Identification" process completes successfully. If the user then selects the "Print" item 653, the counting results and the final identification results are printed. This printout includes a barcode (e.g., JAN code) that identifies the drug. Scanning this barcode with a barcode reader immediately shows which tablet cassette the identified drug should be returned to.

[0098] If the identification results show that the remaining medications include medications different from those already identified by the user, the computer 500 displays the message "Multiple different medications are present" on the counting and message display unit 644. In this case, the user removes the medications suspected to be different from the petri dish 310 and presses the "Retake Image" button 654. The user and the computer 500 then repeat the same operation. If all the medications in the captured image are determined to be the same, the medication identification software can successfully complete the "Returned Tablet Image Identification." Alternatively, the user can exit without completing the "Returned Tablet Image Identification" by selecting the "End" item 655.

[0099] Figure 34 shows the state after the user has selected the "Start Identification" item 651 shown in Figure 20. In other words, Figure 34 is the screen when the computer 500 is performing identification on each drug that was not selected by the user. As shown in the figure, the second section 642 displays the drug image, drug name, markings (or printing), and quantity of drug stored in the database. The third section 643 displays enlarged images of the drugs displayed in the first section 641. In the lower right corner of each drug image, the dimensions of the drug measured by the computer 500 from the drug image are displayed. The display of dimensions is the same as described in Figure 28.

[0100] Once the identification is complete, the "Returned Tablet Image Identification" window 640 changes as shown in Figure 35. As shown in the figure, if the identification results for all drugs differ from the identification results for the drug selected by the user, i.e., if a different type of drug is found, the computer 500 affixes fourth markers 657 and 658 to the drug with the different identification result. More specifically, the image of the drug with the different identification result displayed in the first section 641 is affixed with fourth marker 657. Then, the image of the corresponding drug displayed in the third section 643 is affixed with fourth marker 658. This makes it easy for the user to see if a different type of drug has been mixed in with the drugs being identified.

[0101] As shown in Figure 35, the computer 500 further displays message 659 in the third section 643. This message 659 prompts the user to visually inspect the images of all the drugs displayed in the third section 643. If the user finds a different drug as a result of the visual inspection, the user selects that drug. As a result, the fourth labels 657 and 658 are also attached to that drug. Alternatively, if the computer 500 determined that the drugs were different, but the user determined through visual inspection that they were the same drug, the user can remove the fourth labels 657 and 658 by selecting that drug. In other words, the user can also revoke the computer 500's identification result that the drugs were different.

[0102] When the "Print" item 653 displayed in item display section 645 is selected, a message like the one shown in Figure 36 is displayed on the screen. If the computer 500 performs an automatic identification of all drugs and finds no drugs with different identification results, a message like the one shown in Figure 36(a) is displayed on the screen. If the user selects "Yes", the identification results are printed. If the computer 500 performs an automatic identification of all drugs and determines that different drugs are included, a message like the one shown in Figure 36(b) is displayed on the screen. In other words, the computer 500 has determined that different drugs are included in the analysis, and asks the user if it is okay to print the results. If the user selects "Yes", the identification results, including information on the different drugs, are printed. Also, if the user has only viewed the image of the drug from above and has not yet viewed the image from below, a message like the one shown in Figure 36(c) is displayed on the screen, and printing does not occur.

[0103] Figure 38 shows the second embodiment in the "Returned Tablet Image Identification" window 640 shown in Figure 34, after identification has been completed. As shown in the figure, in the third section 643, a fifth marker 671 is attached to drugs that have the same identification result as the drug selected by the user (or computer 500), i.e., the correct drug. This fifth marker 671 is composed of, for example, a solid blue line. On the other hand, a fourth marker 658 is attached to drugs that have a different identification result than the drug selected by the user, i.e., drugs that have a different identification result than the first drug. This fourth marker 658 is composed of, for example, a solid red line. By attaching different colored markers to the correct drug and the incorrect drug in this way, it becomes possible to see at a glance which drug is the incorrect drug.

[0104] When the user selects one of the drugs displayed in the third section 643, the label attached to the selected drug changes to the second label 673. Also, in the first section 641, the drug corresponding to the drug selected in the third section 643, i.e., the drug with the second label 673, is marked with the second label 674. The second label 673 has a different appearance from the fourth label 658 and the fifth label 671. For example, the second label 673 consists of a dashed line surrounding the drug image, and this dashed line rotates around the image. More specifically, if the user selects the red fourth label 658, the second label 673 consists of a red dashed line rotating around the drug image. Correspondingly, in the first section 641, the second label 674 becomes a red dashed line rotating around the drug. Furthermore, if the user selects the blue fifth marker 671, the second marker 673 will consist of a blue dashed line that rotates around the image of the drug. Correspondingly, in the first section 641, the second marker 674 will be a blue dashed line that rotates around the drug. Normally, in the identification of returned drugs, the drugs placed in the petri dish 310 are almost always the same drug. For this reason, it is difficult to determine which drug the image of the drug displayed in the third section 643 is in the first section 641. Therefore, if the first section 641 is marked only for the drug selected by the user, it becomes easy for the user in the third section 643 to understand which drug in the petri dish 310 is what they are currently looking at. As a result, it becomes easier for the user to remove different drugs in the petri dish 310 using tweezers or similar tools while looking at the image in the first section 641.

[0105] Figure 39 shows the third embodiment in the "Returned Tablet Image Identification" window 640 shown in Figure 34, after identification has been completed. As shown in the figure, in the third section 643, the fifth mark 671 is attached to drugs whose identification result is the same as the identification result for the drug initially selected by the user (or computer 500), i.e., the correct drug. Correspondingly, in the first section 641, the fifth mark 672 is attached to the correct drug, i.e., the drug corresponding to the drug that was marked with the fifth mark 671 in the third section 643. On the other hand, returning to the third section 643, the fourth mark 658 is attached to drugs whose identification result is different from the identification result for the drug initially selected by the user, i.e., drugs whose identification result is different from the identification result for the first drug. Correspondingly, in the first section 641, the fourth mark 657 is attached to the different drug, i.e., the drug corresponding to the drug that was marked with the fourth mark 658 in the third section 643.

[0106] The user visually checks each drug image displayed in the third section 643 to determine if it is correct, i.e., whether it is the same drug initially selected by the user. If the user determines that it is the correct drug, they select that drug. As a result, the image of that drug is marked with the sixth mark 675. This sixth mark 675 indicates that the user has completed visual confirmation and determined that it is the correct drug. If the user selects the "All Remaining OK" item 676 displayed in the item display section 645, the sixth mark 675 is added to all drugs except those marked with the fourth mark 658. If the user selects the "Switch Front and Back All" item 677, the front and back displays of each drug image displayed in the third section 643 are switched. In this way, the user examines each drug image displayed in the third section 643, and once any incorrect drugs have been removed from the petri dish 310, and all remaining drugs have been examined and all drug images have been marked with the sixth label 675, the image identification of returned tablets is complete.

[0107] Figure 40 shows the fourth embodiment in the "Returned Tablet Image Identification" window 640 shown in Figure 34, after identification has been completed. As shown in the figure, the computer 500 automatically switches the display of the front and back of each drug so that the front and back are the same as the drug initially selected by the user, etc., for the drugs that have been automatically identified. Furthermore, the computer 500 automatically rotates the drugs that have been automatically identified in sequence, aligning the orientation of each drug so that it is the same as the drug initially selected by the user, etc. The orientation of the drugs can be aligned by rotating the image of the drugs to be identified so that it is the same orientation as the image of the drug initially selected by the user, etc., when performing pattern matching on the drugs to be identified (see step 380 described later). Alternatively, since the master image used to match the representative drug (the drug initially selected by the user, etc.) is an image that is correctly oriented up and down, the orientation of the representative drug can be corrected to match the orientation of the master image when matching with the master image, and the orientation of other drugs to be identified in the next phase can also be corrected when matching with the master image. If a drug could not be displayed up or down correctly at this time, or if it could not be displayed in the same orientation as the representative drug, it is highly likely that the orientation could not be adjusted because it was a different drug. Therefore, users can more easily spot such images with incorrect orientations, improving the accuracy and efficiency of visual inspection work. Automatic switching of the front and back display of drugs can be performed by the computer 500 swapping the front and back images of the drug to be identified if, as a result of pattern matching, the computer determines that the pattern on the back of the drug to be identified matches the pattern on the front of the drug initially selected by the user, or if the computer determines that the pattern on the front of the drug to be identified matches the pattern on the back of the drug initially selected by the user. This makes the visual inspection work by the user even easier, further improving the accuracy and safety of visual inspection.

[0108] §3.2.4 Magnifying glass function As shown in Figure 9, the tablet identification menu window 620 contains the "Magnifying Glass" item 623. When the user selects this "Magnifying Glass" item 623, the "Magnifying Glass" window 660 appears on screen 531, as shown in Figure 21. Using the functions of this "Magnifying Glass" window 660, the user can magnify and view the medication they wish to see.

[0109] As shown in Figure 21, the "Magnifying Glass" window 660 consists of a first section 661 and an item display section 662. The first section 661 displays an image of the mounting section 220 (see Figure 3) or the notch space 217 (see Figures 4 and 5) immediately adjacent to the mounting section 220. For example, if a drug is placed in the tray 300 and then placed on the mounting section 220, an enlarged image of the area including the petri dish 310 is displayed in the first section 651. The user can also insert a long package of paper through the notch 212 (see Figure 2) into the drug imaging device 200. In this case as well, an enlarged image of the area including the package of paper located directly below the mounting section 220 is displayed in the first section 661. As shown in Figure 21, the first section 661 displays an image selected by the user from among images of the subject taken from above and images of the subject taken from below. When the user selects the "Magnify" item 663, the image is zoomed in and the subject is displayed in an enlarged view. On the other hand, selecting the "Reduce" item 664 zooms out, displaying the subject in a reduced size. The current magnification of the displayed image is also shown in the item display area 662. Furthermore, by selecting the "Lighting Switch" item 665, the drug imaging device 200 switches the illumination of the drug between the first light source 430 and the second light source 440 and the third light source 450 and the fourth light source 460. Illuminating the drug with the first light source 430 and the second light source 440 makes the markings on the drug easier to see. Illuminating the drug with the third light source 450 and the fourth light source 460 makes the printing on the drug easier to see. Thus, in this embodiment, the user can magnify and observe the markings and printing on the drug inside the long-length packaging paper without dividing the packaging paper or removing the drug inside. Based on this, the user can also manually examine the drug inside the packaging paper without removing it. Note that selecting the "Exit" item 666 closes the "Magnifying Glass" window 660.

[0110] §3.2.5 Additional imaging function for other petri dishes and packaging paper This function allows you to add information from other petri dishes or dispensing sheets to the list of drugs to be identified in section 740. Normally, once the user is confident that the drug they selected is the correct drug, the information of the selected drug is added to section 740. However, it is sometimes necessary to identify multiple petri dishes or dispensing sheets as a whole, rather than just one petri dish or dispensing sheet. For example, in drug packaging mode, if there are multiple dispensing sheets for morning, noon, and evening use, you may want to identify them all together. Until now, in such cases, the only way to confirm was to switch to identifying other dispensing sheets during additional imaging. Therefore, by adding the items of other petri dishes or dispensing sheets to the list of drugs to be identified during additional imaging, it is now possible to see that they have been added to the list of drugs to be identified when imaging another petri dish or dispensing sheet. Figures 47(a) and (b) are image examples of adding drugs from other mounting units to the list of drugs to be identified during additional imaging. Here, in Figure 47(b), the item "Other Petri Dishes" added to the fourth section 740 shows "Present," indicating that the added drug is also present in the photograph of another Petri dish. This means that when adding a drug to the list of drugs to be differentiated, checking the items in other Petri dishes or packaging papers will show that if the same drug should be registered, and the item "Present" in the fourth section 740 indicates that it has been added to the list of drugs to be differentiated multiple times, thus preventing a selection error.

[0111] §3.2.6 Function to reduce manual searching of packaging labels This feature reduces the number of manual searches required in the medication packaging mode. If the user wants to find more information about the medication, they can manually search for information about the medication by selecting item 764, "Manual Search." If only one side of the packaging is photographed, automatic identification is not possible, and a manual search is required even if the user does not want to find more information about the medication. In fact, only about 4.8% of tablets can be identified with only one-sided markings. Therefore, in medication packaging mode, a manual search had to be performed for each package. To address this, the system now stores tablets identified through manual searches and uses this information to identify other packaging papers. Specifically, when identifying medications for each dosage (morning, noon, and evening), the system stores the tablets identified through manual searches when identifying the first morning packaging paper. When identifying the next noon packaging paper, if the same tablet appears, it will be displayed as the first candidate even if only one side of the markings is available. This significantly reduces the number of manual searches and greatly improves the efficiency of work in medication packaging mode.

[0112] §4 Drug Identification Algorithm The following details how the drug identification software 600 identifies each drug in the images captured by the drug imaging device 200. Figure 22 shows the upstream processing for drug identification. First, when the user inputs a message to start drug identification (step 110), the drug imaging device 200 captures multiple images of the drugs (step 120). Next, the computer 500 corrects the captured images (step 130). Then, the computer 500 extracts the regions where each drug is present from the corrected images (step 140). Finally, it performs individual identification on each extracted drug region (step 200).

[0113] <Step 110> User Input The user first places the drugs to be identified into the petri dish 310 of the tray 300 (see Figure 7). At this time, it is recommended that tablets be placed directly on the transparent part 315 of the bottom surface 311 of the petri dish 310, and capsules be placed on the capsule holder 314. Next, the user places this tray 300 on the mounting section 220 of the drug imaging device 200 and closes the cover 214. Then, the user selects the "imaging" item 751 shown in Figure 10.

[0114] <Step 120> Drug image acquisition When the computer 500 receives input from the user to photograph the drug, it drives the drug imaging device 200 to take an image of the petri dish 310. Specifically, the computer 500 acquires the four images shown below [Image 1] to [Image 4]. At this time, the computer 500 sets the camera shooting conditions based on the settings obtained in the "camera focus adjustment" and "brightness and color calibration" described later (for example, camera focus, exposure time, and RGB gain). This improves the reproducibility of image quality. [Image 1] An image of the petri dish 310 taken from above under illumination by the first light source 430. This image can be obtained by turning on the first ring illumination 431 and taking a picture with the first camera 410. [Image 2] An image of the petri dish 310 taken from below under illumination by the second light source 440. This image can be obtained by turning on the second ring illumination 441 and taking a picture with the second camera 420. [Image 3] An image of the petri dish 310 taken from above under illumination by the third light source 450. This image can be obtained by turning on the first bar illumination 451 and the second bar illumination 452 and taking a picture with the first camera 410. [Image 4] An image of the petri dish 310 taken from below under illumination by the fourth light source 460. This image can be obtained by turning on the third bar illumination 461 and the fourth bar illumination 462 and taking a picture with the second camera 420.

[0115] <Step 130> Image Correction As will be described later, the drug identification system 100 is pre-calibrated, and the computer 500 stores this calibration information as data. The computer 500 corrects the captured images based on this calibration information. Specifically, it corrects the coordinates of the captured images based on the calibration information. As a result, the coordinate axes of the image captured by the first camera 410 and the coordinate axes of the image captured by the second camera 420 match.

[0116] <Step 140> Drug Area Extraction After image correction, the areas occupied by each drug within the image are extracted. Images [1] and [2], which tend to have a large difference in brightness between the background and the drugs, are preferably used for this purpose. Region extraction is performed by binarizing the brightness of each pixel using a threshold (below the threshold = background, above the threshold = drug), and separating the areas above the threshold.

[0117] Figure 23 shows the flow of the individual drug identification process 200. The following processes are performed individually for each drug. First, the computer 500 matches the area occupied by a drug in the image taken from above with the area occupied by the same drug in the image taken from below (step 210). Next, the computer 500 determines whether the drug is a tablet or a capsule (step 220). If it is determined to be a tablet, the computer 500 performs tablet identification (step 300), and if it is determined to be a capsule, it performs capsule identification (step 400).

[0118] <Step 210> Correspondence between drugs in upper and lower images The drug placed directly on the transparent portion 315 of petri dish 310 is visible in both images taken from above ([Image 1] and [Image 3]) and images taken from below ([Image 2] and [Image 4]). This step associates the area occupied by a single drug in the image taken from above with the area occupied by the same drug in the image taken from below. Specifically, when one of the images corrected in step 130, [Image 1] or [Image 2], is flipped horizontally or vertically, the area occupied by the drug in one image will approximately coincide with the area occupied by the same drug in the other image. This matching area is the area occupied by a single drug in each of the two images. That is, one area in the unflipped image and the area that overlapped with it in the flipped image and then returned to their original state after the flip is reversed become the areas corresponding to a single drug. In this case, the overlapping area where the two images are flipped and overlapping can also be used as the corresponding area.

[0119] <Step 220> Tablet / Capsule Determination After performing the drug region mapping process, the computer 500 determines whether the drug is a tablet or a capsule. Since the tablet is placed in the transparent part 315 of the petri dish 310, its top surface is shown in the image taken from above, and its bottom surface is shown in the image taken from below. That is, when comparing the two images ([Image 1] and [Image 2]), if the drug is a tablet, there will be a region corresponding to the drug in both the image taken from above and the image taken from below. Therefore, if a single region can be mapped to the two images in step 210, that is, if the region occupied by the drug recognized in the image taken from above is mapped to the region occupied by the drug recognized in the image taken from below, the drug is determined to be a tablet.

[0120] On the other hand, the capsule is placed on an opaque capsule-holding member 314. Therefore, the top surface is visible in the image taken from above, but the capsule is not visible in the image taken from below. In other words, when comparing the two images ([Image 1] and [Image 2]), if the drug is a capsule, the image taken from above will show an area occupied by the drug, but the image taken from below will not show an area occupied by the drug. Therefore, if a correspondence between one area and the two images could not be made in step 210, that is, if the area corresponding to the area occupied by the drug recognized in the image taken from above was not recognized in the image taken from below, the drug is determined to be a capsule.

[0121] <Step 300> Tablet Identification If the computer 500 determines that one of the selected drugs is a tablet, the process proceeds to the tablet identification process 300. Figure 24 shows each step of the tablet identification process 300. First, the computer 500 determines whether the drug is round or not (step 310). Next, the computer 500 extracts any markings on the surface of the drug (step 320). Next, the computer 500 extracts any printing on the surface of the drug (step 330). Next, the computer 500 determines whether the information on the surface of the drug is a marking or printing (step 340). Next, the computer 500 extracts any score lines on the surface of the drug (step 350). Next, the computer 500 extracts the representative color of the drug (step 360). After that, the computer 500 narrows down the candidate drugs based on the refinement information regarding the drug (step 370). Finally, the computer 500 performs template matching and a final search based on the markings and / or printing information extracted from the drug image (step 380).

[0122] <Step 310> Yen Determination Computer 500 determines the circularity of the area occupied by the drug. If the circularity is greater than or equal to a predetermined value, Computer 500 determines that the drug is circular. If the circularity is less than the predetermined value, Computer 500 determines that the drug is not circular. If the drug is determined to be circular, Computer 500 calculates the center position and radius of the circle. If the drug is determined to be not circular, Computer 500 rotates the area occupied by the drug so that the major axis of the area is parallel to the X axis and the minor axis is parallel to the Y axis. Furthermore, Computer 500 calculates the lengths of the major axis and minor axis of the drug, and the ratio of the lengths of the major axis and minor axis.

[0123] <Step 320> Imprint Extraction Computer 500 performs mark extraction processing on [Image 1] and [Image 2], i.e., images captured under illumination by the first light source 430 and the second light source 440. The inventors have found that when a drug is photographed under illumination by the first light source 430 and the second light source 440, an image of the drug with the markings emphasized can be captured (see Equation 1 below).

[0124] <Step 330> Print Extraction Computer 500 performs print extraction processing on [Image 3] and [Image 4], i.e., images captured under illumination by the third light source 450 and the fourth light source 460. The inventors have found that when a drug is photographed under illumination by the third light source 450 and the fourth light source 460, an image of the drug with enhanced printing can be captured (see Equation 9 below).

number

[0125] <Step 340> Engraving / Printing Judgment As described above, when an imprint extraction process (step 320) is performed on an image taken under the illumination of the first ring illumination 431 and the second ring illumination 441, an image is obtained in which the imprint affixed to the surface of the drug is suitably extracted. Furthermore, when a print extraction process (step 330) is performed on an image taken under the illumination of the first bar illumination 451, the second bar illumination 452, the third bar illumination 461, and the fourth bar illumination 462, an image is obtained in which the print affixed to the surface of the drug is suitably extracted (see Equation 1 above). Based on these imprint extraction images and print extraction images, the computer 500 determines whether the information affixed to the surface of the drug is an imprint or a print, that is, whether or not there is an imprint and a print on the surface of the drug.

[0126] To determine whether a drug has an imprint, the computer 500 first binarizes the imprint extraction image obtained in step 320 based on a predetermined threshold. Next, the computer 500 uses this binarized image as a mask and takes the average value of the imprint extraction image. If the average value is greater than or equal to the threshold, the computer 500 determines that there is an imprint; if it is less than the threshold, it determines that there is no imprint. Similarly, to determine whether a drug has a print, the computer 500 first binarizes the print extraction image obtained in step 330 based on a predetermined threshold. Next, the computer 500 uses this binarized image as a mask and takes the average value of the print extraction image. If the average value is greater than or equal to the threshold, the computer 500 determines that there is a print; if it is less than the threshold, it determines that there is no print.

[0127] <Step 350> Extraction of secant region The computer 500 performs a secant region extraction process on the imprint extraction image obtained in step 320. Specifically, the computer 500 determines whether or not secants exist in the drug. If it determines that secants exist, it separates the drug into regions where secants exist and regions where they do not.

[0128] <Step 360> Representative Color Extraction Next, the computer 500 separates the representative color of the drug into the engraved or printed area and the other area, and extracts them separately. The extraction of the representative color can be preferably performed by clustering the colors that make up the pixels. In another embodiment, instead of extracting the representative color, the intensity of the color is determined as "white," "light," or "dark" for the drug image, or for the engraved or printed area and the other area. This can be done by determining that if the color in the area is less than the first threshold, it is "dark"; if it is between the first threshold and the second threshold, it is "light"; and if it is above the second threshold, it is "white."

[0129] <Step 370> Filtering Computer 500 accesses the database and searches for candidate medications based on the data obtained in the above steps. First, as the initial step, it narrows down the medications to be searched. Specifically, Computer 500 narrows down the medications to be searched based on at least one of the following: type of medication (tablet or other), shape of medication (round or non-round, ratio of major axis to minor axis length if non-round), size of medication (diameter if round, length of major axis and minor axis length if non-round), presence or absence of markings, presence or absence of printing, presence or absence of score lines, and representative color of medication (representative color of the marked area, representative color of the printed area, and representative color of the other area). If Computer 500 can access the patient's prescription history (e.g., medication record data), Computer 500 can also limit the medications to be searched to those included in the prescription history. For example, if Computer 500 can access the medication record data stored on the patient's smartphone, Computer 500 can access that medication record data and obtain information on the medications prescribed to the patient. Furthermore, if a printed medication record exists, it can be scanned, converted into text using OCR (optical character recognition), and used to narrow down the search for medications. This speeds up the medication search process.

[0130] <Step 380> Template Matching Subsequently, template matching (also called pattern matching) is performed between the imprinted image and / or printed image obtained in the previous step and the imprinted template image and / or printed template image of each drug stored in the database to pick out drugs that are most likely to be the drug to be differentiated. If the tablet is round, prior to template matching, the area of ​​the image occupied by the drug is moved and corrected so that the center of the circle is the center of the image. In another embodiment, the extracted image and template image may be corrected so that the center of gravity of the imprint and / or print is the center of the image. Also, if the tablet is round, the extracted image and / or template image are rotated so that the result of template matching is maximized. Then, template matching is performed on both images and a score is calculated. This score numerically indicates the degree to which the two images match. If the template matching score is above a predetermined value, or within a predetermined rank from the highest, the computer 500 picks out the drug to which the template image belongs as a candidate drug. If the tablet has a score line, both template matching with the score line present and template matching with the score line removed may be performed. Furthermore, if the printing or engraving consists of words, the computer 500 may perform template matching on each word as a unit, or it may divide the words into letters, numbers, and symbols and perform template matching on each letter, number, and symbol. In addition, the computer 500 may extract text information from the engraving or printed image using OCR and perform template matching based on this text information. Moreover, when using the built-in camera of a mobile phone or tablet device as a camera for identification, the user can determine whether the engraving is on one side or both sides by looking at the actual drug or packaging paper, making the work easier compared to viewing an image. Furthermore, when performing identification based on an image captured with the built-in camera of a mobile phone or tablet device, the matching efficiency can be improved when matching the acquired engraving with the extracted image.Specifically, if both drugs with single-sided and double-sided markings are displayed in the same list, it can be difficult for the correct drug to appear at the top of the list of candidates. When displaying the identification results, the user can separate candidates with single-sided markings from those with double-sided markings using tags or similar methods, and unify the list display to one type of marking. In this way, by allowing the user to select the type of marking on the drug, the frequency of similar drugs appearing as candidates can be reduced. By configuring the system in this way, the appearance rate of the correct drug candidate can be significantly improved.

[0131] <Step 400> Capsule Identification Returning to step 220 in Figure 23, if the computer 500 determines that one of the selected drugs is a capsule, the process proceeds to capsule identification 400. Figure 25 shows each step of capsule identification 400. First, the computer 500 divides the capsule into two regions (step 410). Next, the computer 500 extracts representative colors from each of the two regions (step 420). Next, the computer 500 extracts the printing on the surface of the drug from each of the two regions (step 430). Next, the computer 500 determines whether there is printing on the surface of the drug (step 440). After that, the computer 500 narrows down the candidate drugs based on the refinement information about the drug (step 450). Finally, the computer 500 performs template matching based on the printing information extracted from the drug image and performs a final search (step 460). Note that in drug packaging mode, the orientation in which the capsule is imaged is unknown. This was a factor that lowered the matching score. Therefore, in drug packet mode, if a capsule was identified, the matching method was changed. Specifically, the drug that achieved the highest match rate in the following three matching methods will now be displayed. 1. Matching information between images captured by the upper and lower cameras and the upper and lower master images. 2. Matching the information from the image captured by the upper camera with the master image on the top of the tablet. 3. Matching information between the image captured by the lower camera and the master image at the bottom of the tablet.

[0132] Specifically, we decided to compare scores using the following method and calculate the highest score. Note that the number of matches is merely an example and is not limited to this. First, the score of the first matching result between the image data and the master image was 0.8. Therefore, the main score criterion for the training data was set to 0.8. Next, the score of the second matching result between the image data and the master image was 0.9. Therefore, the main score criterion for the training data was changed to 0.9. The score of the third matching result between the image data and the master image was 0.7. Since this is lower than the main score, the main score was kept at 0.9. However, a region called Sub1 was created in the training data and 0.7 was stored there. The score of the fourth matching result between the image data and the master image was 0.6. As a result, the main score criterion for the training data was set to 0.6. This was also lower than the main score, so the main score remained at 0.9. However, a region called Sub2 was created in the training data, and 0.6 was stored there. The score of the matching result of the 5th imaging information with the master image was 0.8. This was also lower than the main score, so the main score remained at 0.9. However, since it was higher than Sub1 in the training data, Sub1 was swapped, and the data for Sub2 was set to 0.7. The score of the matching result of the 6th imaging information with the master image was 0.95. This was higher than the main score, so it was swapped with the main data. On the other hand, since it was higher than Sub1 in the training data, Sub1 was swapped, and the data for Sub2 was set to 0.8.

[0133] <Step 410> Region division First, the computer 500 divides the drug into two regions. Specifically, it clusters the color information of the drug's image, specifically the color of each pixel. Next, it groups the clustered color information, and the region to which each color group belongs is designated as a single region. In this way, a capsule composed of different colors in its right and left halves is appropriately divided into two regions.

[0134] <Step 420> Representative Color Extraction Next, the computer 500 extracts the representative color of each region of the drug. This can be preferably done by clustering the color information of the pixels that make up the region.

[0135] <Step 430> Print Extraction The computer 500 performs the same processing as in step 330 on each region divided in step 410 to extract prints from each region. The inventors have found that by dividing the capsule image into two regions and performing the print image extraction process separately on each region, prints can be extracted more favorably from the capsule.

[0136] <Step 440> Printing Result The computer 500 performs the same process as in step 340 on each region of the divided drug to determine whether or not printing is present in each region.

[0137] <Step 450> Filtering Computer 500 narrows down the candidate drugs in the same manner as in step 360. Specifically, Computer 500 narrows down the drugs to be searched based on at least one of the following: drug type (capsule or other drug), drug shape (ratio of the length of the drug's long axis to the length of its short axis), drug size (length of the drug's long axis and length of its short axis), representative color, and whether or not there is printing in the area to which each representative color belongs.

[0138] <Step 460> Template Matching The computer 500 extracts the printed portion from the print extraction image (input image) obtained in step 430. Then, in the same manner as in step 370, template matching is performed on the print extraction image for each region divided in step 410 to pick out candidate drugs.

[0139] §5 Additional processing of drug identification software §5.1 Additional processing in <Step 370> In step 370 described above, it is assumed that the database accessed by computer 500 stores template images corresponding to the imprinted image and / or printed image. However, in some cases, only a photograph of the drug may be stored. In that case, the region of the drug may be obtained from this image data, and printed and imprinted extraction may be performed in the same manner as described above, and the obtained printed and imprinted images may be used as template images. Alternatively, a template image to be stored in the database can be created using this method. The region of the drug can be obtained from the image data as follows: First, the background color is obtained, and the area other than the background color and text is determined to be the region occupied by the drug. If one or more regions are found, an error is recorded. Next, the position of the centroid of each region is determined, and the left and right sides of the drug are determined from the position of this centroid.

[0140] §5.2 Camera Calibration For the above embodiment to be suitably implemented, the first camera 410 and the second camera 420 must be in focus. Furthermore, the shooting areas of the first camera 410 and the second camera 420 must be aligned. In addition, the brightness and color of the captured image must be within predetermined ranges. Therefore, calibration as described below is performed before the drug identification system 100 leaves the factory or during setup at the delivery destination. Note that this calibration usually only needs to be performed once.

[0141] §5.2.1 Camera alignment In this calibration, the first step is to align the cameras. First, the tray 300, on which the petri dish 310 is set, is placed on the mounting section 220 of the drug imaging device 200. Next, the petri dish 310 is photographed with the first camera 410, and this image is displayed on the display device 530 of the computer 500. At this time, a virtual image with a cross shape at a predetermined position is superimposed on the image of the petri dish 310. Then, the position and angle of the first camera 410 are finely adjusted so that this cross shape fits inside the petri dish 310. The same operation is performed for the second camera 420.

[0142] §5.2.2 Camera Focus Adjustment Next, the camera's focus is adjusted. First, a template with the specified characters, symbols, or diagrams is placed on the mounting section 220 of the drug imaging device 200, and the template is photographed at a low magnification. Then, the camera's focus is roughly adjusted so that the markings on the template are as clear as possible in the captured image. Once this is complete, the magnification is increased and the same operation is performed to fine-tune the focus.

[0143] §5.2.3 Calibration of the imaging area This calibration allows the coordinates of the shooting areas of the upper and lower cameras to be aligned. It also allows the resolution of the cameras to be obtained. First, a calibration sheet 810, which is a calibration tool, is prepared as shown in Figure 26. This calibration sheet 810 is made of a plate or sheet of a predetermined thickness, on both sides, with m × n black circles of a predetermined size arranged in a matrix at predetermined intervals (3 × 4 circles in the example shown in Figure 23). The position and size of the black circles are the same on the front and back surfaces of the calibration sheet 810. During calibration, this calibration sheet 810 is set on the mounting section 220 of the drug imaging device 200, and the calibration sheet 810 is photographed with the first camera 410 and the second camera 420. Next, a transformation matrix is ​​obtained such that the black circles in the image are located at specified coordinates and have a specified size. This is done for both the image taken from above and the image taken from below the calibration sheet 810. In step 130 described above, the image of the petri dish 310 is corrected based on this transformation matrix. This ensures that the coordinate positions of the image areas in the images of Petri dish 310 taken from above and from below coincide. Furthermore, the image resolution (DPI), i.e., the number of pixels constituting a unit length, can also be obtained from the transformation matrix.

[0144] §5.2.4 Brightness and Color Calibration This calibration can improve the color reproduction accuracy of captured images. As described above, with the calibration sheet 810 placed on the mounting section 220, repeat the following steps (a) to (b). (a) Exposure adjustment: With the first light source 430 turned on, the first camera 410 takes a picture for a predetermined exposure time. Also, with the second light source 440 turned on, the second camera 420 takes a picture for a predetermined exposure time. Next, the brightness of each predetermined point in the captured image is calculated. If this brightness is higher than the target value, the camera's exposure time is reduced. If it is lower than the target value, the camera's exposure time is increased. (b) Color (white balance) adjustment: Calculate the RGB values ​​for each designated point in the captured image. If B (blue) is greater than R (red), increase the gain of R. If B is less than R, decrease the gain of R. Also, if B is greater than G (green), increase the gain of G. If B is less than G, decrease the gain of G. The above steps (a) to (b) are repeated until each value reaches the target value. This allows the camera settings at the time of shooting to be obtained. The same operation is also performed using the third light source 450 and the first camera 410, and using the fourth light source 460 and the second camera 420.

[0145] Figure 37 shows a second embodiment of the calibration sheet. This calibration sheet 810' has a configuration in which m × n black circles 811 of a predetermined size are arranged in a matrix at predetermined intervals on both sides of a circular plate or sheet of a predetermined thickness. In addition, near the outer edge of the calibration sheet 810', straight lines 812 of a predetermined length are arranged at equal intervals of 90° in four directions (up, down, left, and right), so as to be located on the centerlines of the circles. More specifically, one pair of straight lines 812 are arranged parallel to the row direction of the circles 811 and are positioned on the extension of the circles 811 arranged in the row direction. Another pair of straight lines 812 are arranged parallel to the column direction of the circles 811 and are positioned on the extension of the circles 811 arranged in the column direction. The rest of the configuration of the calibration sheet 810' is the same as that of the calibration sheet 810. The diameter of this calibration sheet 810' is approximately the same as the inner diameter of the ring 313 placed on a petri dish 310 (see Figure 8) or a flat plate 330 (see Figure 27). In other words, the planar shape of the calibration sheet 810' corresponds to the shape of the bottom surface of the mounting item for placing the drug. Using the calibration sheet 810' effectively prevents the position of the calibration sheet 810' from shifting when the mounting item is set on the mounting section 220 of the drug imaging device 200 after the calibration sheet 810' has been placed on it. Furthermore, by using the straight line 812 as a guide, it becomes easy to align the orientation of the circle 811 to a predetermined direction. [Explanation of Symbols]

[0146] 100... Drug Identification System

[0147] 200... Drug imaging device 210...cases 211...Lower case 212... Notch 213... Upper case 214...cover 215...Slope part 216...Slope 217...Space for cutout 220... Mounting section 221...Tray support member 222...hole 223...Matching part 230... Guide component 231...Slope 240····Upper internal space 250...frame 300, 300'... tray 301... Petri dish body 302...Handle section 303... Front packaging paper guide 304...Rear side packaging paper guide 305... clip 306...Imaging Glass 307...Top deflection plate 308, 309...lower deflection plate 310... Petri dish 311...Bottom 312... Side view 313... Ring 314... Capsule mounting component 315...Transparent part 316...recess 317...recess 320...Main unit 321...hole 322...tube 330...Flat plate 331...2nd cylinder 410...First Camera 420...Second camera 430...1st light source 431...First Ring Illumination 440...Second light source 441...Second Ring Lighting 442...Lower support plate 443...Upper support plate 450...Third light source 451...First bar lighting 452...Second bar lighting 460...4th light source 461...3rd bar lighting 462...4th bar lighting

[0148] 500...computer 520...Main unit 530...Display device 531... screen 532...Touchscreen 541... Keyboard 542...Mouse 543...Mouse cursor 600... Drug identification software 610...Window for the medication guidance support system 620...Tablet Identification Menu Window 621... "Tablet Image Identification" item 622... "Returned Tablet Image Identification" item 623... "Magnifying Glass" item 630... "Group" window 631...Part 1 632...Item display section 633... Grouping marker 634... "Grouping" items 635... "Deselect" item 636... "Ungroup All" item 637... "Confirmed" item 638 ··· "Close" item 640····"Returned Tablet Image Identification" window 641...Part 1 642... Part 2 643... Part 3 644...Counting and message display section 645... Item display section 646... "Photography" item 647...Message 648...Second sign 649...Message 651... "Inspection Started" Item 652...Third sign 653 ··· "Print" item 654... "Reshoot" item 655... "Finished" item 656 ··· "Manual Search" Item 657, 658...4th sign 659...Message 671, 672...5th sign 673, 674...Second sign 675...6th sign 676... "Remaining items can be purchased in one go" items 677... "Fully Reversible" Item 660····"Magnifying Glass" window 661...Part 1 662...Item display section 663... "Expand" item 664 ··· "Shrink" item 665... "Lighting Switch" item 666... ​​"Finished" item 680····"Manual Search" window 681... Search input section 682... Search results display section 683...Database image display section 684...Item display section 685...Type section 686... Search button 687... "Ineligible" items 688... "Selection" item 689... "Newly Added" Items 691... "Finished" item 700·····“Tablet Image Identification” window 701...Message 710...Part 1 713...1st sign 714...Second sign 715...Third sign 720...Part 2 721... Dimensions 730...Part 3 731...Message 740... Part 4 750...First item display section 751... "Photography" item 752... Gauge section 753... "Cancelled" item 754 ··· "Registered" item 755... "Additional shooting" item 756... "Reshoot" item 757... "Page Turning" Item 758... "Finished" item 759 ··· "Print" item 760...Second item display section 761... "Selection" item 762... "Deselect" item 763... "Ineligible" items 764 ··· "Manual Search" Item 765... "Grouping" items 810, 810'... Calibration Sheet 811... yen 812...straight line

Claims

1. A drug imaging device used to image a drug, A placement section on which the medicine is placed; A camera for photographing the placement portion; A light source for illuminating the placement portion, A placement unit for placing packaged medicines is set in the placement section, The mounting unit has a transparent bottom and at least one polarizing member; The light from the light source is configured to pass through the polarizing member and reach the drug. A pharmaceutical imaging device characterized by the above.

2. the light source includes a ring-shaped light disposed above the transparent bottom of the mounting unit; The pharmaceutical imaging device according to claim 1 , wherein the polarizing member includes a first polarizing member disposed on an upper surface of the transparent bottom portion so as to be perpendicular to the transparent bottom portion.

3. the light source includes a ring-shaped light disposed beneath a transparent bottom of the mounting unit; The polarizing member further includes a second polarizing member disposed on the lower surface of the transparent bottom so as to be inclined at a predetermined angle with respect to the lower surface of the transparent bottom, and a third polarizing member disposed on the lower surface of the transparent bottom so as to be parallel to the transparent bottom, The pharmaceutical imaging device of claim 2 , wherein one end of each of the first polarizing member, the second polarizing member, and the third polarizing member is configured to be adjacent to each other.

4. 4. The drug imaging device of claim 3, wherein the polarization direction of the first polarizing member is set perpendicular to the transparent bottom, the polarization direction of the second polarizing member is set equal to an inclination angle of the second polarizing member with respect to a lower surface of the transparent bottom, and the polarization direction of the third polarizing member is set parallel to the transparent bottom.

5. The medicine photographing device according to claim 1 , wherein the mounting unit further comprises a guide portion on which a plurality of individually packaged medicines can be mounted and which is adapted to be moved.

6. The medication photographing apparatus according to claim 1 , wherein the mounting unit further comprises a handle for removing the mounting unit from the mounting portion.

7. The drug photographing device according to claim 1 , wherein the camera further comprises a first camera disposed above the placement portion and a second camera disposed below the placement portion.

Images