Medicine identification device, medicine identification method and program, and method for generating learned model

Abstract

Provided are a medicine identification device and a medicine identification method and program that associate front and back of individual medicines from an image of a front surface and an image of a rear surface of a medicine one-dose package in which a plurality of medicines are packaged, and a method for generating a learned model. A front surface cut-out image and a rear surface cut-out image of each medicine one-dose package in which a plurality of medicines are packaged are generated, a pair of a front surface cut-out image and a rear surface cut-out image of an identification object medicine which is at least one medicine out of the plurality of medicines is created on the basis of a score indicating the probability that the front surface cut-out image and the rear surface cut-out image are images of both surfaces of the same medicine, and the medicine type of the identification object medicine is identified on the basis of the created pair of the front surface cut-out image and the rear surface cut-out image.

Description

Drug identification device, drug identification method and program, and method for generating trained model

[0001] The present invention relates to a drug identification device, a drug identification method and program, and a method for generating a trained model, and in particular to a technology for identifying individual drugs from images of both sides of a packaged drug package containing multiple drugs.

[0002] There is known a technology for identifying medicines based on captured images of the medicines to be identified. For example, Patent Literature 1 discloses a medicine identification system that captures images of both sides of the medicine using a camera above a mounting section on which the medicine to be identified is placed and a camera below the mounting section, and identifies the medicine based on the images captured by the upper and lower cameras.

[0003] JP 2023-105994 A

[0004] The medicine discrimination system described in Patent Document 1 has the advantage of being able to simultaneously capture images of the front and back of a single-packaged medicine because cameras are installed in two locations, above and below.However, it has the disadvantage of being expensive and requiring a complex device structure.

[0005] In contrast, a method for identifying drug types by photographing only the front or back of a single-packaged drug has the advantage of requiring only one camera, making the device simple and low-cost. However, with this method, there are cases where the drug cannot be identified with a single photograph due to the existence of drugs that have the same or similar markings on one side but different markings on the other side.

[0006] On the other hand, a method is conceivable in which a single camera is used to photograph the front and back of a medicine separately and use the images of both sides to identify the medicine. With this method, it is possible to distinguish between medicines that have the same or similar markings on one side but different markings on the other side.

[0007] This method requires the manual process of turning the sachet over to prevent the medication from flipping over when photographing the front and back of a single-packaged medication separately with a single camera. However, because the medication is freely movable inside the sachet, this manual process can result in the position of each medication changing between photographing the front and back sides. Therefore, when using this method, it is essential to match the same medications shown in the front and back package images. Note that even if the position of the same medication does not change at all between photographing the front and back sides, the medication positions in both images will be mirror-symmetrical. Therefore, this specification will discuss the difference in the position of the same medication when photographing the front and back sides, taking this fact into consideration.

[0008] The present invention has been made in consideration of these circumstances, and aims to provide a drug identification device, a drug identification method and program, and a method for generating a trained model that can identify individual drugs from images of the front and back of a packaged drug package containing multiple drugs.

[0009] To achieve the above object, the drug identification device of the present disclosure includes at least one processor and at least one memory that stores instructions to be executed by the processor, wherein the processor acquires a front image of one side of a single-packaged drug package containing multiple drugs, acquires a back image of the other side of the single-packaged drug package, generates a front cutout image of each of the multiple drugs from the front image, generates a back cutout image of each of the multiple drugs from the back image, creates a pair of front cutout image and back cutout image of at least one of the multiple drugs to be identified based on a score that indicates the likelihood that the front cutout image and the back cutout image are images of both sides of the same drug, and identifies the drug type of the drug to be identified based on the created pair of front cutout image and back cutout image.The drug identification device can associate the front and back of each drug from the front image and back image of the single-packaged drug package containing multiple drugs. Therefore, even if the position of the drug differs between the image of the front and back of a packaged drug, or if there are drugs with the same or similar markings on one side but different markings on the other side, the drug can be identified using the associated images of both sides.

[0010] To achieve the above object, the presently disclosed drug identification method includes at least one processor: acquiring a front image of one side of a single-drug package containing multiple drugs; acquiring a back image of the other side of the single-drug package; generating a front cutout image of each of the multiple drugs from the front image; generating a back cutout image of each of the multiple drugs from the back image; creating a pair of front and back cutout images of at least one of the multiple drugs as a target drug based on a score representing the likelihood that the front and back cutout images are images of both sides of the same drug; and identifying the drug type of the target drug based on the created pair of front and back cutout images. This drug identification method allows for matching of the front and back sides of individual drugs from the front and back images of a single-drug package containing multiple drugs. Therefore, even if the positions of the drugs differ between the front and back images of the single-drug package, or even if there are drugs with the same or similar markings on one side but different markings on the other side, the drugs can be identified using the matched back and front images.

[0011] To achieve the above object, the present disclosure provides a trained model generation method executed by a learning device including a processor, wherein the processor acquires a first training dataset including a set of actual pairs each consisting of an image of the front side of a drug and an image of the back side of the drug, and a first label corresponding to the actual pair; acquires a second training dataset including a set of non-existent pairs each consisting of an image of the front side of the drug and an image of the front or back side of a drug different from the drug, and a second label corresponding to the non-existent pair; and trains the first training dataset and the second training dataset through machine learning to generate a trained model that, when a first image of a first side of a drug and a second image of a second side of the drug are input, outputs a score representing the likelihood that the first and second sides are two sides of the same drug. The trained model generation method can generate a trained model that associates the front and back sides of individual drugs from images of the front and back sides of a single-dose drug.

[0012] According to the present invention, it is possible to identify individual medicines from images of the front and back of a single-package medicine package in which multiple medicines are packaged.

[0013] FIG. 1 is a block diagram showing the electrical configuration of a smartphone. FIG. 2 is a block diagram showing the functional configuration of a medicine identification device. FIG. 3 is a diagram showing an example of photographing a packaged medicine. FIG. 4 is a diagram for explaining the operation of the front / back pair determination AI. FIG. 5 is a flowchart showing the steps of a medicine identification method. FIG. 6 is a diagram showing an example of a GUI for a touch panel display. FIG. 7 is a diagram showing an example of a GUI for a touch panel display. FIG. 8 is a diagram showing an example of a GUI for a touch panel display. FIG. 9 is a diagram showing an example of a GUI for a touch panel display. FIG. 10 is a block diagram showing the electrical configuration of an image processing device including a learning device. FIG. 11 is a block diagram showing the functional configuration of the learning device. FIG. 12 is a flowchart showing the steps of a recognizer learning method. FIG. 13 is a diagram for explaining sampling of non-existent pair images.

[0014] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0015] <Overview of drug identification device> The drug identification device of the present disclosure is a device that acquires a front image of the front side of a packaged drug in which multiple drugs are packaged together, and a back image of the back side, generates a front cutout image of each of the multiple drugs from the front image, generates a back cutout image of each of the multiple drugs from the back image, creates a pair of front cutout image and back cutout image of the drug to be identified, which is at least one of the multiple drugs, based on a score that indicates the likelihood that the front cutout image and the back cutout image are images of both sides of the same drug, and identifies the drug type of the drug to be identified based on the created pair of front cutout image and back cutout image.

[0016] The front and / or back of the medicine may be engraved and / or printed. Note that "engraved and / or printed" means "either engraved or printed" or "both engraved and printed." Also, "the front and / or back of the medicine" means "either the front or back" or "both the front and back."

[0017] Here, "marked" means that identification information is formed by forming a groove, which is a recessed area, on the surface of the medicine. The groove is not limited to being formed by digging into the surface of the medicine, but may also be formed by pressing the surface of the medicine.

[0018] Furthermore, "printed" means that identification information is formed by applying edible ink or the like to the drug surface with or without contact. Here, "applied by printing" is synonymous with "applied by printing."

[0019] The surface of the medicine may have a score line without any distinguishing function, which is a groove formed on the surface of the medicine, for example, to divide the tablet.

[0020] The medication surface of the medication may be plain with no markings and / or printing.

[0021] As an example, the drug identification device is mounted on a portable terminal device. The portable terminal device includes at least one of a mobile phone, a PHS (Personal Handyphone System), a smartphone, a PDA (Personal Digital Assistant), a tablet computer terminal, a notebook personal computer terminal, and a portable game console. The drug identification device may also be mounted on a fixed desktop computer, a workstation, or configured in combination with a separate digital camera and display. Below, a drug identification device configured on a smartphone will be used as an example and described in detail with reference to the drawings.

[0022] <Electrical Configuration of Smartphone> Fig. 1 is a block diagram showing the electrical configuration of a smartphone 10. As shown in Fig. 1, the smartphone 10 includes a touch panel display 14, a speaker 16, a microphone 18, an in-camera 20, an out-camera 22, a light 24, and a switch 26, as well as a CPU (Central Processing Unit) 28, a wireless communication unit 30, a call unit 32, a memory 34, an external input / output unit 40, a GPS receiving unit 42, and a power supply unit 44.

[0023] The touch panel display 14 includes a display unit that displays images and the like, and a touch panel unit that is disposed in front of the display unit and accepts touch inputs. The display unit is, for example, a color LCD (Liquid Crystal Display) panel, which is an example of a display device.

[0024] The touch panel unit (an example of an "input device") is, for example, a capacitive touch panel provided in a planar form on a light-transmitting substrate body, and having light-transmitting position detection electrodes and an insulating layer provided on the position detection electrodes. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to a user's touch operation. Touch operations include a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

[0025] The speaker 16 is an audio output unit that outputs audio during a call and when playing back a video. The microphone 18 is an audio input unit that inputs audio during a call and when shooting a video. The in-camera 20 is an imaging device that captures videos and still images. The out-camera 22 is an imaging device that captures videos and still images. The light 24 is a light source that emits illumination light when shooting with the out-camera 22, and is configured, for example, by an LED (Light Emitting Diode).

[0026] The switch 26 is an input member that receives instructions from a user. The switch 26 is a push-button switch that turns on when pressed with a finger or the like and turns off when the finger is released due to the restoring force of a spring or the like.

[0027] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates in accordance with the control program and control data stored in the memory 34, and controls all the components of the smartphone 10. The CPU 28 has a mobile communication control function that controls all the components of the communication system to perform voice communication and data communication via the wireless communication unit 30, and an application processing function.

[0028] The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function visually conveys information such as still images, moving images, and text to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to a user's touch operation from the touch panel portion of the touch panel display 14. The CPU 28 also acquires an input signal from the switch 26.

[0029] The hardware structure of the CPU 28 is made up of various processors as follows: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0030] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple functional units may also be configured with a single processor. Examples of multiple functional units configured with a single processor include: a first configuration, as typified by a client or server computer, in which a single processor is configured with a combination of one or more CPUs and software, and this processor operates as multiple functional units; and a second configuration, as typified by a SoC (System on Chip), in which a processor is used to realize the functions of an entire system including multiple functional units on a single IC (Integrated Circuit) chip. In this way, the various functional units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0031] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0032] The in-camera 20 and the out-camera 22 capture moving images and still images in accordance with instructions from the CPU 28. The in-camera 20 and the out-camera 22 have the same internal configuration. The in-camera 20 and the out-camera 22 each have a photographing lens, an image sensor, and an image processing unit (not shown).

[0033] The in-camera 20 and the out-camera 22 each receive subject light passing through a photographing lens with an image sensor. The image sensor is a photoelectric conversion element such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge-Coupled Device), and has R (red), G (green), and B (blue) color filters (not shown) provided on its light-receiving surface. The subject light forms an image on the light-receiving surface of the image sensor, which then converts the subject light imaged on the light-receiving surface into an electrical signal based on the R, G, and B color signals. The image processing unit performs predetermined processing on the analog image signal output from the image sensor to convert it into a digital image signal.

[0034] The in-camera 20 and the out-camera 22 may convert the image data of the captured moving images and still images into compressed image data using MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group) formats.

[0035] The CPU 28 stores the video and still images captured by the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0036] Furthermore, the CPU 28 displays the video and still images captured by the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the video and still images captured by the in-camera 20 and the out-camera 22 in application software.

[0037] The CPU 28 may illuminate the subject with fill light by turning on the light 24 when capturing an image with the outer camera 22. The light 24 may be turned on and off by a touch operation on the touch panel display 14 or by operating the switch 26 by the user.

[0038] The wireless communication unit 30 performs wireless communication with a base station device included in the mobile communication network in accordance with instructions from the CPU 28. Using this wireless communication, the smartphone 10 transmits and receives various file data such as audio data and image data, e-mail data, and the like, and receives Web (abbreviation for World Wide Web) data, streaming data, and the like.

[0039] The speaker 16 and the microphone 18 are connected to the communication unit 32. The communication unit 32 decodes the audio data received by the wireless communication unit 30 and outputs it from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into audio data that can be processed by the CPU 28 and outputs it to the CPU 28.

[0040] The memory 34 stores instructions to be executed by the CPU 28. The memory 34 is composed of an internal storage unit 36 ​​built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are realized using known storage media. The external storage unit 38 may be a non-transitory computer-readable recording medium.

[0041] The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of emails sent and received, web data downloaded by web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

[0042] The external input / output unit 40 serves as an interface with external devices connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to other external devices by communication or the like via the external input / output unit 40. The external input / output unit 40 transmits data received from the external devices to each component within the smartphone 10 and transmits data within the smartphone 10 to the external devices.

[0043] Examples of communication means include a Universal Serial Bus (USB), Institute of Electrical and Electronics Engineers (IEEE) 1394, the Internet, a wireless local area network (LAN), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. Examples of external devices include a headset, an external charger, a data port, an audio device, a video device, a smartphone, a PDA, a personal computer, and earphones.

[0044] The GPS receiver 42 detects the position of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, . . . , STn.

[0045] The power supply unit 44 is a power supply source that supplies power to each component of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium-ion secondary battery. The power supply unit 44 may also include an A / D converter that generates a DC voltage from an external AC power supply.

[0046] The smartphone 10 configured in this manner is set to a shooting mode by a user's instruction input using the touch panel display 14 or the like, and can take videos and still images using the in-camera 20 and the out-camera 22.

[0047] When the smartphone 10 is set to the shooting mode, it enters a shooting standby state, and a video is shot by the in-camera 20 or the out-camera 22, and the shot video is displayed on the touch panel display 14 as a live view image.

[0048] The user can visually check the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject they want to photograph, and set the photographing conditions.

[0049] When the smartphone 10 is in a standby state for shooting and receives an instruction to take a picture by a user input using the touch panel display 14 or the like, the smartphone 10 performs AF (Autofocus) and (Auto Exposure) control, and takes and stores video and still images.

[0050] <Functional Configuration of Drug Identification Device> Figure 2 is a block diagram showing the functional configuration of the drug identification device 100 realized by the smartphone 10. As shown in Figure 2, the drug identification device 100 has a front image acquisition unit 102, a back image acquisition unit 104, a drug detection unit 106, a matching unit 108, a drug type identification unit 110, a drug database 112, a candidate output unit 114, and a determination unit 116. The functions of the front image acquisition unit 102, the back image acquisition unit 104, the drug detection unit 106, the matching unit 108, the drug type identification unit 110, the candidate output unit 114, and the determination unit 116 are realized by the CPU 28 executing a program stored in the memory 34. In addition, the function of the drug database 112 is realized by the memory 34.

[0051] The surface image acquisition unit 102 acquires a surface image of one side of a single-package drug package containing multiple drugs. A single-package drug package contains multiple drugs that are freely movable inside a sachet. The multiple drugs are not limited to drugs of the same type, but may be drugs of different types. The sachet may be transparent or translucent.

[0052] The back surface image acquisition unit 104 acquires a back surface image of the other side of the packaged medicine, that is, the back surface. Note that, although one side of the packaged medicine is referred to as the front surface and the other side as the back surface here, either side of the packaged medicine may be referred to as the front surface.

[0053] FIG. 3 is a diagram showing an example of photographing a single-package drug. F3A in FIG. 3 shows how the surface of a single-package drug is photographed. In F3A, a sachet S (an example of a "sachet drug") containing multiple drugs is placed on a placement surface T with its surface F facing vertically upward. The user photographs the sachet S from above the surface F of the sachet S in the vertical direction, with the optical axis of the outer camera 22 (not shown in FIG. 3) of the smartphone 10 facing vertically downward. The surface image acquisition unit 102 acquires the surface image photographed in F3A.

[0054] F3B in Fig. 3 shows the state of photographing the back side of a single-package medicine. In F3B, the sachet S is inverted from the state shown in F3A, and the sachet S is placed on the placement surface T with the back side B facing vertically upward. The user photographs the sachet S from vertically above the back side B of the sachet S, with the optical axis of the outer camera 22 of the smartphone 10 facing vertically downward. The back side image acquisition unit 104 acquires the back side image photographed in F3B.

[0055] The front and back images may be images acquired from another device via the wireless communication unit 30, the external storage unit 38, or the external input / output unit 40. The front and back images may also be images taken by a camera other than that of the smartphone 10, or may be images taken using a dedicated single-dose drug placement table and / or a lighting device.

[0056] The front and back images may be images of a single-dose drug package and a marker. The marker may be multiple, and may be an ArUco marker, a circular marker, or a square marker. The front and back images may be images of a single-dose drug package and a reference gray color.

[0057] The front and back images may be images taken at a standard shooting distance and viewpoint. The shooting distance can be expressed by the distance between the packaged drug and the shooting lens and the focal length of the shooting lens. The shooting viewpoint can be expressed by the angle between the printed surface of the marker and the optical axis of the shooting lens.

[0058] When the front image and the back image include markers, the front image acquisition unit 102 and the back image acquisition unit 104 may standardize the shooting distance and shooting viewpoint of the front image and the back image based on the markers to acquire standardized images. Furthermore, when the front image and the back image include a reference gray area, the front image acquisition unit 102 and the back image acquisition unit 104 may perform color correction on the front image and the back image based on the reference gray color.

[0059] Returning to the explanation of Figure 2, the drug detection unit 106 detects the drug region of each of the multiple drugs from the surface image of the packaged drug or the standardized image of the surface image acquired by the surface image acquisition unit 102. Furthermore, the drug detection unit 106 cuts out the detected drug region from the surface image or the standardized image of the surface image to generate a cut-out surface image of the drug.

[0060] The drug detection unit 106 detects the drug area of ​​each of the multiple drugs from the back image of the packaged drug or the standardized image of the back image acquired by the back image acquisition unit 104. The drug detection unit 106 also cuts out the detected drug area from the back image or the standardized image of the back image to generate a cut-out back image of the drug.

[0061] The matching unit 108 creates a pair of cut-out front and back images of the drug to be identified, which is at least one drug among the plurality of drugs, based on a score indicating the likelihood that the cut-out front and back images are images of both sides of the same drug that actually exists. The matching unit 108 may display the pair of cut-out front and back images of the drug to be identified on the touch panel display 14.

[0062] The matching unit 108 includes a front / back pair determination AI (Artificial Intelligence) 108A (an example of a "first trained model"). The front / back pair determination AI 108A is stored in the memory 34. When a first image, which is a photograph of one side of a drug, and a second image, which is a photograph of the second side of the drug, are input, the front / back pair determination AI 108A is a trained model that outputs a score representing the likelihood that the first and second images are images of both sides of the same actual drug, i.e., the likelihood that the first and second sides are images of both sides of the same actual drug. The score is, for example, a value greater than or equal to 0 and less than or equal to 1, with a larger value indicating a higher likelihood that the two input images are images of both sides of the same actual drug.

[0063] The front / back pair determination AI 108A is obtained by machine learning using a first training data set and a second training data set. The first training data set is a set of images of an actual pair consisting of an image of the front side of a first drug and an image of the back side of the first drug, and a first label corresponding to the actual pair. The second training data set is a set of images of an impractical pair consisting of an image of the front side of a second drug and an image of the front or back side of a third drug different from the second drug, and a second label corresponding to the impractical pair.

[0064] The front / back pair determination AI 108A can apply a convolutional neural network (CNN).

[0065] 4 is a diagram for explaining the operation of the front / back pair determination AI 108A. In F4A of FIG. 4, the front / back pair determination AI 108A receives images I1 and I2 and outputs a score of "0.9." That is, since images I1 and I2 are images of both sides of the same actual drug, a relatively high score is output.

[0066] 4, the front / back pair determination AI 108A receives images I3 and I4 and outputs a score of 0.1. That is, since images I3 and I4 are images of both sides of a non-existent drug, a relatively low score is output.

[0067] Here, the input images of the front and back pair determination AI 108A have been described as an example of a front surface cutout image and a back surface cutout image of the drug to be identified, but the input images of the front and back pair determination AI 108A may be either an imprint print extraction image in which the imprint and / or print of the drug to be identified is extracted from the captured image, or a shape extraction image in which the shape of the drug to be identified is extracted from the captured image, or a combination of these. The input image of the front and back pair determination AI 108A may be either one of a front imprint print extraction image, a back imprint print extraction image, and a shape extraction image, or a combination of two or more of these.

[0068] The matching unit 108 inputs the front and back cutout images into the front and back pair determination AI 108A to obtain a score. The matching unit 108 may input only the front and back cutout images that have been pre-screened into the front and back pair determination AI 108A, rather than inputting all the front and back cutout images generated by the drug detection unit 106 into the front and back pair determination AI 108A through a round-robin combination of all the front and back cutout images. For example, the matching unit 108 may compare the size and / or shape of the drugs shown in all the front and back cutout images generated by the drug detection unit 106 through a round-robin combination of the front and back cutout images as a pre-processing step, and screen out pairs of front and back cutout images that are relatively likely to depict the same drug. The size of the drug may be the long and short sides of a rotated rectangle corresponding to the drug area.

[0069] The drug type identification unit 110 acquires multiple drug type candidates for the drug to be identified based on the pair of cut-out front and back images of the drug to be identified created by the matching unit 108. The drug type candidates include drug identification information consisting of drug name, trade name, abbreviation, or a combination thereof.

[0070] The drug type identification unit 110 includes a drug type identification AI 110A (an example of a "second trained model"). The drug type identification AI 110A is stored in the memory 34. The drug type identification AI 110A is a trained model that, when a cut-out image of the front side and a cut-out image of the back side of a drug are given as input, outputs a score representing the likelihood that the drug type corresponds to each of the drug types of all drugs to be trained. The drug type identification AI 110A may be a trained model that, when at least one of a cut-out image of the front side and a cut-out image of the back side of a drug is given as input, outputs a score representing the likelihood that the drug type corresponds to each of the drug types of all drugs to be trained.

[0071] The drug type identification AI 110A is a training data set for a plurality of different drugs, and is obtained by machine learning using a training data set for learning, which is a set of cut-out images of the front and back sides of the drug, and the drug type of the drug. The drug type identification AI 110A may also be a training data set for a plurality of different drugs, and is obtained by machine learning using a training data set for learning, which is a set of cut-out images of the front and back sides of the drug, and the drug type of the drug. The drug type identification AI 110A can apply a CNN, as with the front / back pair determination AI 108A.

[0072] The drug type identification unit 110 inputs a pair of cut-out images of the front and back surfaces of the drug to be identified, created by the matching unit 108, to the drug type identification AI 110A, and acquires a plurality of drug type candidates for the drug to be identified.

[0073] The drug database 112 stores drug identification information and master images of both sides of all drugs to be identified, i.e., all drugs to be learned by the drug type identification AI 110A, in association with each other.

[0074] The candidate output unit 114 outputs a list of multiple drug type candidates for the drug to be identified acquired by the drug type identification unit 110. The candidate output unit 114, for example, causes the multiple drug type candidates for the drug to be identified to be selectable and displayed on the touch panel display 14. The candidate output unit 114 may acquire drug identification information of the multiple drug type candidates and master images of both sides from the drug database 112 and display them.

[0075] The confirmation unit 116 confirms a correct drug for the drug to be identified from a plurality of drug type candidates for the drug to be identified. The confirmation unit 116 confirms, for example, a drug type candidate selected by a user as the correct drug from a plurality of drug type candidates displayed on the touch panel display 14. When there is only one drug type candidate for the drug to be identified output by the candidate output unit 114, the confirmation unit 116 may confirm the drug type candidate as the correct drug.

[0076] The drug identification device 100 can associate the front and back of each drug from the images of the front and back of a packaged drug in which multiple drugs are packaged together. Therefore, even if the position of the drug differs between the images of the front and back of the packaged drug, or even if there are drugs with the same or similar markings on one side but different markings on the other side, the drug can be identified using the associated images of both sides.

[0077] <Drug Identification Method> Figure 5 is a flowchart showing the steps of a drug identification method using the drug identification device 100. The drug identification method is realized by the CPU 28 reading and executing a drug identification program from the memory 34. The drug identification program may be provided via the wireless communication unit 30 or the external input / output unit 40. The drug identification program may be stored in a non-transitory computer-readable recording medium such as a CD-ROM.

[0078] In step S1, the user uses the in-camera 20 or out-camera 22 of the smartphone 10 to capture images of both sides of a packaged drug containing multiple tablets (an example of a "drug") whose drug type is to be identified. For example, the user captures a surface image of the packaged drug as described in F3A of FIG. 3. The surface image acquisition unit 102 acquires the surface image.

[0079] Next, the user turns the single-dose drug package over so that the tablets do not turn over inside the package, and captures an image of the back side of the single-dose drug package as described in F3B of Fig. 3. The back side image acquisition unit 104 acquires the back side image.

[0080] The front image acquisition unit 102 and the back image acquisition unit 104 may display the acquired front image and back image on the touch panel display 14 .

[0081] In step S2, the drug detection unit 106 detects the area of ​​each of the multiple tablets from the surface image of the single-drug package acquired by the surface image acquisition unit 102. The drug detection unit 106 also cuts out the area of ​​the tablet detected from the surface image to generate a cut-out image of the tablet. Similarly, the drug detection unit 106 detects the area of ​​each of the multiple tablets from the back image of the single-drug package acquired by the back image acquisition unit 104. The drug detection unit 106 also cuts out the area of ​​the tablet detected from the back image to generate a cut-out image of the tablet.

[0082] The medicine detection unit 106 may display a cut-out image of the front surface and a cut-out image of the back surface of the tablet on the touch panel display 14.

[0083] In step S3, the matching unit 108 compares the size and shape of the tablet shown in the front cut-out image (an example of the "size and / or shape of the front surface of the drug") with the size and shape of the tablet shown in the back cut-out image (an example of the "size and / or shape of the back surface of the drug") for all front cut-out images and back cut-out images generated by the drug detection unit 106, and extracts candidate pairs of front cut-out images and back cut-out images that are relatively likely to show the same drug, i.e., pairs of tablets with similar sizes and shapes.

[0084] In addition, the matching unit 108 may extract pair candidates by comparing only the sizes of the tablets shown in the front cut-out image and the back cut-out image, or may extract pair candidates by comparing only the shapes of the tablets shown in the front cut-out image and the back cut-out image.

[0085] The processes in steps S4 and S5 are performed for each of all pair candidates extracted in step S3. The order in which the processes are performed for the pair candidates may be arbitrary.

[0086] In step S4, the matching unit 108 inputs the front and back cut-out images of the pair candidates into the front and back pair determination AI 108A and obtains a score representing the likelihood that the input front and back cut-out images are images of both sides of the same tablet.

[0087] In step S5, if the score obtained in step S4 is equal to or greater than a certain value, the matching unit 108 determines that the front cut-out image and the back cut-out image of the input pair candidate are images of both sides of the same tablet, and provisionally determines the pairing of the input pair candidate.

[0088] In addition, if the front and back cutout images of a pair candidate for which pairing has been tentatively determined are included in other pair candidates, the matching unit 108 may not perform the processes of steps S4 and S5 for that pair candidate, or may compare the scores of each pair candidate and tentatively determine the pairing of the pair candidate with the relatively higher score.

[0089] When the processing of steps S4 to S5 is completed for all pair candidates extracted in step S3, the drug identification device 100 proceeds to the processing of step S6. In step S6, the matching unit 108 displays the pair candidates that have been provisionally determined as pairings on the touch panel display 14 in an editable arrangement.

[0090] In step S7, if there is an error in the displayed pair candidate, the user corrects the pair candidate using the touch panel display 14. The user may delete the pair candidate or create a new pair candidate. Furthermore, in step S7, the matching unit 108 confirms the pairing of the displayed pair candidate and sets the front and back cutout images of the pair candidate as paired images. Confirming the pairing corresponds to creating a pair of the front and back cutout images of the drug to be identified.

[0091] The processes in steps S8 to S10 are performed for each of all paired images whose pairing has been confirmed in step S7. The order in which the paired images are processed may be arbitrary.

[0092] In step S8, the drug type identification unit 110 inputs the paired images to the drug type identification AI 110A.

[0093] In step S9, the drug type identification AI 110A calculates the scores of all learned drugs and sorts the drugs in descending order of score. The candidate output unit 114 selectably displays a plurality of drugs with the highest scores among the sorted drugs as candidate drugs for the tablets shown in the front and back cutout images on the touch panel display 14. The candidate output unit 114 may display drug identification information of a plurality of drug type candidates and master images of both sides.

[0094] In step S10, the user selects one drug candidate from the multiple drug candidate types displayed on the touch panel display 14. The confirmation unit 116 confirms the drug candidate selected by the user as the correct drug for the tablet shown in the front and back cutout images.

[0095] When the processes of steps S8 to S10 have been completed for all paired images for which pairing has been confirmed in step S7, the drug identification device 100 ends the process of this flowchart.

[0096] According to the drug identification method, it is possible to determine a pairing of the front and back images of the same drug from the front and back images of a packaged drug in which multiple drugs are packaged. Therefore, even if the position of the drug differs between the front and back images of the packaged drug, or even if there are drugs with the same or similar markings on one side but different markings on the other side, it is possible to identify the drug to be identified using the paired images where the pairing has been determined.

[0097] <GUI Example> FIGS. 6 to 9 are diagrams showing an example of a GUI (Graphical User Interface) of the touch panel display 14 in the medicine identifying method.

[0098] <<Display of Front and Back Cutout Images>> Fig. 6 shows an example of the display screen in step S6 of Fig. 5. The display screen has a tentative paired area A1, an unpaired area A2, and a confirmed paired area A3.

[0099] The tentative pair area A1 is an area that displays the front cutout image and the back cutout image of a pair candidate that have been provisionally determined to be paired by the matching unit 108, among the front cutout image and the back cutout image of a tablet included in a single-dose medicine package. In the tentative pair area A1, the front cutout image and the back cutout image of each pair candidate are displayed side by side in the horizontal direction of the screen.

[0100] The tentative pair area A1 includes pair candidate areas AP1 to AP4, in which each pair candidate is displayed. In pair candidate areas AP1 to AP4, a front cutout image of each pair candidate is displayed on the left side, and a back cutout image of each pair candidate is displayed on the right side. In the example shown in FIG. 6, of the front cutout image IF1 and the back cutout image IB1, which are pair candidates, the front cutout image IF1 is displayed to the left of the pair candidate area AP1, and the back cutout image IB1 is displayed to the right of the pair candidate area AP1. Furthermore, of the front cutout image IF2 and the back cutout image IB2, which are pair candidates, the front cutout image IF2 is displayed to the left of the pair candidate area AP2, and the back cutout image IB2 is displayed to the right of the pair candidate area AP2.

[0101] Similarly, of the pair candidates, front side cutout image IF3 and back side cutout image IB3, front side cutout image IF3 is displayed to the left of pair candidate area AP3, and back side cutout image IB3 is displayed to the right of pair candidate area AP3. Furthermore, of the pair candidates, front side cutout image IF4 and back side cutout image IB4, front side cutout image IF4 is displayed to the left of pair candidate area AP4, and back side cutout image IB4 is displayed to the right of pair candidate area AP4.

[0102] A scroll bar SB1 is displayed at the right end of the tentative pair area A1. The scroll bar SB1 includes a scroll area SA1 that extends linearly in the vertical direction of the screen and a scroll knob SN1 that can be moved in the vertical direction of the screen within the scroll area SA1. The tentative pair area A1 is scrolled by the user dragging the scroll knob SN1.

[0103] The unpaired region A2 is a region that displays the front surface cutout image and the back surface cutout image of the tablet contained in the single-dose medicine other than the pair candidate provisionally determined for pairing by the matching unit 108. The unpaired region A2 includes a front surface region A2A where the front surface cutout image is displayed and a back surface region A2B where the back surface cutout image is displayed.

[0104] 6, the surface area A2A displays a surface cutout image IF5, a surface cutout image IF6, and a surface cutout image IF7 aligned in the vertical direction of the screen. A scroll bar SB2 is also displayed at the right end of the surface area A2A. The scroll bar SB2 includes a scroll area SA2 that extends linearly in the vertical direction of the screen and a scroll knob SN2 that can be moved in the vertical direction of the screen within the scroll area SA2. The surface area A2A is scrolled by the user dragging the scroll knob SN2.

[0105] In the back surface area A2B, back surface crop image IB5, back surface crop image IB6, and back surface crop image IB7 are displayed side by side in the vertical direction of the screen. A scroll bar SB3 is displayed at the right end of the back surface area A2B. The scroll bar SB3 includes a scroll area SA3 that extends linearly in the vertical direction of the screen and a scroll knob SN3 that can be moved in the vertical direction of the screen within the scroll area SA3. The back surface area A2B is scrolled by the user dragging the scroll knob SN3.

[0106] The pair confirmation area A3 is an area where GUI components for confirming the pairing of the provisionally determined pair candidates are displayed. In the example shown in Fig. 6, a pair confirmation button BM is displayed in the pair confirmation area A3.

[0107] By tapping the pair confirmation button BM, the user can confirm the pairing of all pair candidates displayed in the tentative pair area A1 and set the front and back cutout images of the pair candidates as paired images. Note that pairing can be confirmed even if one or more front and back cutout images exist in the unpaired area A2. However, if only one of the front and back cutout images exists in the pair candidate area and the other is not selected, the pairing of only that one front or back cutout image is not confirmed.

[0108] <<Replacement and Correction within the Provisional Pair Area>> As shown in FIG. 6, the front surface cutout image IF2 (an example of a "first front surface cutout image") and the back surface cutout image IB2 (an example of a "first back surface cutout image") displayed in the pair candidate area AP2 are one pair candidate (an example of a "first pair"), and the front surface cutout image IF3 (an example of a "second front surface cutout image") and the back surface cutout image IB3 (an example of a "second back surface cutout image") displayed in the pair candidate area AP3 are a different pair candidate (an example of a "second pair"). However, in reality, the front surface cutout image IF2 and the back surface cutout image IB3 are images of both sides of the same tablet, and the front surface cutout image IF3 and the back surface cutout image IB2 are images of both sides of the same tablet. Here, the user corrects the pair candidate of the front surface cutout image IF2 and the back surface cutout image IB2 to the pair candidate of the front surface cutout image IF3 and the back surface cutout image IB3.

[0109] The user can drag the front and back cutout images displayed in the tentative pair area A1 and the unpaired area A2. Here, as shown in F7A in Fig. 7, the user drags the back cutout image IB2 displayed in the pair candidate area AP2 toward the pair candidate area AP3.

[0110] Thereafter, as shown in F7B, when the drag operation of back surface cutout image IB2 reaches pair candidate area AP3, back surface cutout image IB3 of pair candidate area AP3 automatically moves to and is displayed in the position where back surface cutout image IB2 of pair candidate area AP2 was displayed. When the user ends the drag operation in this state, back surface cutout image IB2 is displayed in the position where back surface cutout image IB3 of pair candidate area AP3 was displayed. As a result, as shown in F7C, the positions of back surface cutout image IB2 and back surface cutout image IB3 are swapped from the state shown in FIG. 6 .

[0111] As a result, the front surface cutout image IF2 and the back surface cutout image IB3 of the pair candidate area AP2 become pair candidates, and the front surface cutout image IF3 and the back surface cutout image IB2 of the pair candidate area AP3 become pair candidates.

[0112] Furthermore, the user can confirm the pairing of each pair candidate displayed in the pair candidate areas AP1 to AP4 of the tentative pair area A1 by tapping the pair confirmation button BM. In other words, the front cut-out image and the back cut-out image of each pair candidate can be made into pair images.

[0113] In addition, in the state shown in Figure 6, if the user drags the back side cutout image IB3 displayed in the pair candidate area AP3 to the position where the back side cutout image IB2 in the pair candidate area AP2 is displayed, the back side cutout image IB2 and the back side cutout image IB3 can be swapped in the same way.

[0114] 6, if the user drags the front-side cutout image IF2 displayed in the pair candidate area AP2 to the position where the front-side cutout image IF3 of the pair candidate area AP3 is displayed, the front-side cutout image IF2 and the front-side cutout image IF3 can be swapped. In this case, the front-side cutout image IF3 and the back-side cutout image IB2 of the pair candidate area AP2 become pair candidates, and the front-side cutout image IF2 and the back-side cutout image IB3 of the pair candidate area AP3 become pair candidates. In other words, the pair candidates can be corrected to the same as the example shown in F7C.

[0115] In this way, in the tentative pair area A1, by interchanging the front side cutout images or the back side cutout images of the pair candidate areas AP1 to AP4, the pair candidates can be corrected.

[0116] On the other hand, the front cutout image and the back cutout image cannot be swapped. For example, in the state shown in Figure 6, even if the back cutout image IB2 is dragged to the position where the front cutout image IF3 is displayed, the front cutout image IF3 will not move to the position where the back cutout image IB2 was displayed, and the back cutout image IB2 and the front cutout image IF3 cannot be swapped.

[0117] In the example shown in F7C, the front cutout image IF4 and the back cutout image IB5 are images of both sides of the same tablet. Here, the user corrects the pair candidate displayed in the pair candidate area AP4 to the pair candidate of the front cutout image IF4 and the back cutout image IB5.

[0118] First, the user drags the back side cutout image IB4 displayed in the pair candidate area AP4 toward the outside of the temporary pair area A1, as shown in F8A in Fig. 8. When the user stops the dragging operation when the back side cutout image IB4 reaches the outside of the pair candidate area AP4, as shown in F8B, the back side cutout image IB4 is deleted from the pair candidate area AP4 and displayed in the position where the back side cutout image IB5 in the back side area A2B was displayed in F8A.

[0119] Note that back side crop image IB5 is displayed at the position where back side crop image IB6 in back side area A2B was displayed in F8A, and back side crop image IB6 is displayed at the position where back side crop image IB7 in back side area A2B was displayed in F8A. The state shown in F8B is a state where the temporary pairing of front side crop image IF4 and back side crop image IB4 has been released, and the back side crop image corresponding to front side crop image IF4 has not been selected.

[0120] Next, as shown in F8C, the user drags the back side crop image IB5 displayed in the back side area A2B of the unpaired area A2 toward the pair candidate area AP4. When the user stops the dragging operation when the back side crop image IB5 reaches the pair candidate area AP4, the back side crop image IB5 is added to the pair candidate area AP4 of the temporary pair area A1 and displayed as shown in F8D. As a result, the front side crop image IF4 and the back side crop image IB5 become pair candidates.

[0121] As shown in F8D, the back surface cropped image IB6 is moved up and displayed at the position where the back surface cropped image IB5 of the back surface area A2B was displayed in F8B, and the back surface cropped image IB7 is moved up and displayed at the position where the back surface cropped image IB6 was displayed.

[0122] Here, the back side crop image IB4 is deleted from the temporary pair area A1, and then the back side crop image IB5 is added to the temporary pair area A1. However, the back side crop image IB5 may also be moved directly to the pair candidate area AP4 of the temporary pair area A1. In this case, when the drag operation of the back side crop image IB5 reaches the position of the back side crop image IB4, the back side crop image IB4 moves to and is displayed in the back side area A2B of the unpaired area A2. When the user ends the drag operation in this state, the back side crop image IB5 is displayed in the position where the back side crop image IB4 was displayed in the pair candidate area AP4. Even when the user performs this operation, the front side crop image IF4 and the back side crop image IB5 can be set as pair candidates.

[0123] <<Creating a New Pair Candidate>> F9A in FIG. 9 shows the state in which the scroll knob SN1 is dragged downward from the state shown in F8D to scroll the tentative pair area A1. In F9A, the pair candidate area AP1 is hidden in the tentative pair area A1, and a blank pair candidate area AP5 is displayed. In the example shown in F9A, the front cutout image IF6 and the back cutout image IB6 are images of both sides of the same tablet. Here, the user creates a new pair candidate for the front cutout image IF6 and the back cutout image IB6.

[0124] Here, as shown in F9B of Fig. 9, the user drags the surface cutout image IF6 displayed in the surface area A2A of the unpaired area A2 to the pair candidate area AP5. When the user ends the drag operation when the surface cutout image IF6 reaches the pair candidate area AP5, the surface cutout image IF6 can be displayed in the pair candidate area AP5, as shown in F9C of Fig. 9.

[0125] In the example shown in F9C, surface cutout image IF7 is moved up to the position where surface cutout image IF6 of surface area A2A was displayed, and surface cutout image IF8 is moved up to the position where surface cutout image IF7 was displayed.

[0126] Furthermore, similar to the examples described in F8C and F8D, the user can drag the back side cropped image IB6 displayed in the back side area A2B of the unpaired area A2 to move it to the pair candidate area AP5. This allows the front side cropped image IF6 and the back side cropped image IB6 to be new pair candidates.

[0127] When both the front and back cut-out images have been dragged into the blank area, a new blank pair candidate area for creating a new pair candidate is generated at the bottom of the temporary pair area A1.

[0128] 10 is a block diagram showing the electrical configuration of an image processing device 140 including a learning device that generates the front and back pair determination AI 108 A. As the image processing device 140, a personal computer or a workstation can be used.

[0129] As shown in FIG. 10, the image processing device 140 includes an image input unit 142 , a database 144 , an operation unit 146 , a display unit 148 , a CPU 150 , a RAM (Random Access Memory) 152 , and a ROM (Read Only Memory) 154 .

[0130] The image input unit 142 is an input interface for inputting the learning data set stored in the database 144, and includes wired and wireless communication interfaces.

[0131] The database 144 is a storage unit that stores the learning data set, and is configured by a large-capacity storage device.

[0132] The operation unit 146 is a user interface for the user to control the image processing device 140, and is composed of a keyboard and a pointing device.

[0133] The display unit 148 is an output interface that visually displays the status of the image processing device 140, and is configured by a display panel.

[0134] The CPU 150 is an example of a processor that executes instructions stored in the RAM 152 and the ROM 154, and has a similar hardware structure to the CPU 28. The RAM 152 is a memory device that temporarily stores data used by the CPU 150 for various calculations, and is made up of a semiconductor memory. The ROM 154 is a memory device that stores programs to be executed by the CPU 150, and is made up of a hard disk.

[0135] 11 is a block diagram showing the functional configuration of a learning device 170 implemented by the image processing device 140. As shown in FIG. 11, the learning device 170 includes a recognizer 172, an error calculation unit 174, and a parameter control unit 176.

[0136] A CNN is applied to the recognizer 172. The recognizer 172 can change from an untrained model to a trained model by updating parameters from initial values ​​to optimal values. The initial values ​​of the parameters of the recognizer 172 may be any values, or, for example, parameters of an existing trained model may be applied.

[0137] The recognizer 172 includes an input layer 172A, an intermediate layer 172B, and an output layer 172C, and each layer has a structure in which a plurality of nodes are connected by edges.

[0138] In the learning phase, paired images of a learning dataset are input to the input layer 172A. The paired images are a pair of a first image of a first side of a medicine and a second image of a second side, which is one of the sides of the medicine.

[0139] The intermediate layer 172B is a layer that extracts features from the first and second images input from the input layer 172A. The intermediate layer 172B includes multiple sets of convolutional layers and pooling layers, and a fully connected layer. The fully connected layer connects all of the nodes in the previous layer (here, the pooling layer).

[0140] The output layer 172C is a layer that outputs, as the recognition result of the recognizer 172, a score that represents the likelihood that the first side of the first image and the second side of the second image input from the input layer 172A are both sides of the same drug.

[0141] The error calculation unit 174 obtains the recognition result output from the output layer 172C of the recognizer 172 and the correct answer data indicating whether the first and second surfaces are the two surfaces of the same drug input from the input layer 172A, and calculates the error between them. Possible methods for calculating the error include, for example, softmax cross entropy or least squared error (MSE).

[0142] The parameter control unit 176 adjusts the parameters of the recognizer 172 (filter coefficients of the convolutional layer and deconvolutional layer, etc.) using the error backpropagation method based on the error calculated by the error calculation unit 174.

[0143] This parameter adjustment process is repeated, and learning is repeated until the difference between the output of the recognizer 172 and the correct data becomes small.

[0144] In the recognition phase, the recognizer 172 as the front / back pair determination AI 108A receives as input a first image of the first side, which is one side of the drug, and a second image of the second side, which is one side of the drug, and outputs a score representing the degree of certainty that the first side and the second side are two sides of the same actual drug.

[0145] <Method for generating a trained model> FIG. 12 is a flowchart showing steps in a method for generating the front / back pair determination AI 108A, i.e., a method for training the recognizer 172 (an example of a "method for generating a trained model"). The training method is implemented by the CPU 150 reading and executing a training program from the ROM 154. The training program may be provided via a network (not shown). The training program may be stored in a non-transitory computer-readable recording medium such as a CD-ROM.

[0146] In step S21, the user prepares a large number of paired images of the front and back of actual medicines. It is preferable that the user prepares paired images of the front and back of all actual medicines. The prepared paired images are stored in database 144 as first learning data.

[0147] In step S22, paired images of both sides of a real drug are sampled at a fixed ratio of paired images of both sides of a non-real drug. The fixed ratio may be, for example, a 1:1 ratio of real to non-real, or a 1:3 ratio of real to non-real. The sampled paired images are stored in database 144 as second training data.

[0148] 13 is a diagram for explaining sampling of non-existent paired images of both sides. In the example shown in FIG. 13, paired images of both sides of an actual medicine are prepared, including a front cutout image IF11 and a back cutout image IB11, a front cutout image IF12 and a back cutout image IB12, a front cutout image IF13 and a back cutout image IB13, ..., a front cutout image IFN and a back cutout image IBN.

[0149] The user samples a pair of non-existent double-sided images from this pool of paired images of both sides of an actual drug. In the example shown in Fig. 13, a front cutout image IF11 and a back cutout image IBN, a front cutout image IF12 and a back cutout image IB13, ... are sampled as paired images of non-existent double sides.

[0150] It is preferable to randomly switch the front and back sides of both the paired images of real medicines and the paired images of non-real medicines.

[0151] 12 , in step S23, the CPU 150 of the image processing device 140 assigns "1" as correct answer data when the paired images of both sides of an existing drug, which are the first learning data prepared in step S21, are input, and assigns "0" as correct answer data when the paired images of both sides of a non-existent drug, which are the second learning data sampled in step S22, are input, thereby training the recognizer 172 of the training device 170. That is, the recognizer 172 is trained to output "1" when the paired images of both sides of an existing drug are input, and to output "0" when the paired images of both sides of a non-existent drug are input.

[0152] The trained recognizer 172 is stored in the memory 34 of the smartphone 10 as the front / back pair determination AI 108A.

[0153] <Supplementary Notes> As can be understood from the detailed description of the embodiments above, this specification includes disclosure of various technical ideas including the inventions described below.

[0154] [Supplementary Note 1] A drug identification device comprising at least one processor and at least one memory that stores instructions to be executed by the processor, wherein the processor: acquires a front image of one side of a single-drug package containing multiple drugs; acquires a back image of the other side of the single-drug package; generates a front cut-out image of each of the multiple drugs from the front image; generates a back cut-out image of each of the multiple drugs from the back image; creates a pair of front cut-out image and back cut-out image of a drug to be identified, which is at least one of the multiple drugs, based on a score indicating the likelihood that the front cut-out image and the back cut-out image are images of both sides of the same drug; and identifies the drug type of the drug to be identified based on the created pair of front cut-out image and back cut-out image.

[0155] [Appendix 2] A drug identification device as described in Appendix 1, wherein the processor: detects the size and / or shape of the surface of each drug from the cut-out image of the surface of each drug; detects the size and / or shape of the back surface of each drug from the cut-out image of the back surface of each drug; extracts candidate pairs of the cut-out image of the surface and the cut-out image of the back surface based on the similarity between the size and / or shape of the surface and the size and / or shape of the back surface; and creates a pair of the cut-out image of the surface and the cut-out image of the drug to be identified from the candidate pairs.

[0156] [Supplementary Note 3] The drug identification device according to Supplementary Note 1 or Supplementary Note 2, wherein the processor displays the created pair on the display so that the pair can be modified by the input device.

[0157] [Appendix 4] The processor creates a first pair of a first front cutout image and a first back cutout image, and a second pair of a second front cutout image and a second back cutout image, and accepts an exchange of the first front cutout image and the second front cutout image, or an exchange of the first back cutout image and the second back cutout image. A drug identification device as described in Appendix 3.

[0158] [Supplementary Note 5] The drug identification device according to Supplementary Note 3 or Supplementary Note 4, wherein the processor accepts cancellation of the created pair.

[0159] [Supplementary Note 6] A drug identification device described in any one of Supplementary Note 3 to Supplementary Note 5, wherein the processor displays a front cutout image and a back cutout image different from the front cutout image and back cutout image of the created pair, and accepts the creation of a new pair.

[0160] [Appendix 7] A drug identification device as described in any of Appendices 1 to 6, comprising a first trained model that, when a first image of a first side of a drug and a second image of a second side of the drug are input, outputs a score representing the likelihood that the first side and the second side are two sides of the same drug, and a processor that inputs a front cut-out image and a back cut-out image into the first trained model to obtain the score.

[0161] [Appendix 8] A drug identification device as described in any of Appendices 1 to 6, comprising a second trained model that outputs candidate drug types when an image of a drug is input, and a processor inputs at least one of a front-side cutout image and a back-side cutout image of the drug to be identified into the second trained model to obtain candidate drug types of the drug to be identified.

[0162] [Supplementary Note 9] The drug identification device according to Supplementary Note 8, wherein the processor acquires a plurality of drug type candidates for the drug to be identified, displays a list of the plurality of drug type candidates on a display, and determines a drug type candidate selected from the displayed list by an input device as the correct drug for the drug to be identified.

[0163] [Appendix 10] A drug identification device described in any one of Appendices 1 to 9, wherein the processor acquires a back image in which the front and back of the packaged drug are inverted after the front and back images have been captured, thereby capturing the back side of the packaged drug.

[0164] [Supplementary Note 11] A drug identification method, comprising: at least one processor acquiring a front image of one side of a single-drug package containing multiple drugs; acquiring a back image of the other side of the single-drug package; generating a front cutout image of each of the multiple drugs from the front image; generating a back cutout image of each of the multiple drugs from the back image; creating a pair of front cutout image and back cutout image of at least one drug to be identified, which is at least one drug among the multiple drugs, based on a score representing a likelihood that the front cutout image and the back cutout image are images of both sides of the same drug; and identifying the drug type of the drug to be identified based on the created pair of front cutout image and back cutout image.

[0165] [Supplementary Note 12] A program that causes a computer to execute the drug identification method described in Supplementary Note 11.

[0166] [Supplementary Note 13] A method for generating a trained model executed by a learning device having a processor, wherein the processor: acquires a first training dataset consisting of a set of actual pairs each consisting of an image of the front side of a drug and an image of the back side of the drug, and a first label corresponding to the actual pair; acquires a second training dataset consisting of a set of non-existent pairs each consisting of an image of the front side of the drug and an image of the front or back side of a drug different from the drug, and a second label corresponding to the non-existent pair; and performs machine learning on the first training dataset and the second training dataset, thereby generating a trained model that, when a first image of the first side of the drug and a second image of the second side of the drug are input, outputs a score representing the likelihood that the first side and the second side are two sides of the same drug.

[0167] DESCRIPTION OF SYMBOLS 10: Smartphone 14: Touch panel display 16: Speaker 18: Microphone 20: In-camera 22: Out-camera 24: Light 26: Switch 28: CPU 30: Wireless communication unit 32: Call unit 34: Memory 36: Internal storage unit 38: External storage unit 40: External input / output unit 42: GPS receiving unit 44: Power supply unit 100: Drug identification device 102: Front image acquisition unit 104: Back image acquisition unit 106: Drug detection unit 108: Matching unit 108A: Front / back pair determination AI 110: Drug type identification unit 110A: Drug type identification AI 112: Drug database 114: Candidate output unit 116: Confirmation unit 140: Image processing device 142: Image input unit 144: Database 146: Operation unit 148: Display unit 150: CPU 152: RAM 154...ROM 170...Learning device 172...Recognizer 172A...Input layer 172B...Intermediate layer 172C...Output layer 174...Error calculation unit 176...Parameter control unit A1...Temporary pair area A2...Unpaired area A2A...Front area A2B...Back area A3...Confirmed pair area AI...Front / back pair determination AI108A...Front / back pair determination AI110A...Drug type identification AP1 to AP5...Pair candidate area B...Back side BM...Pair confirmation button F...Front side I1 to I4...Images IB1 to IB8, IB11 to IB13, IBN...Back side cut-out image IF1 to IF8, IF11 to IF13, IFN...Front side cut-out image S...Sachet S1 to S10...Steps of drug identification method S21 to S23...Steps of the recognizer learning method SA1 to SA3...Scroll area SB1 to SB3...Scroll bar SN1 to SN3...Scroll knob ST1, ST2...GPS satellite T...Placement surface

Claims

1. A drug identification device comprising at least one processor and at least one memory storing instructions to be executed by the processor, wherein the processor: acquires a front image of one side of a packaged drug in which multiple drugs are packaged together, acquires a back image of the other side of the packaged drug, generate a front cut-out image of each of the multiple drugs from the front image, generate a back cut-out image of each of the multiple drugs from the back image, create a pair of the front cut-out image and the back cut-out image of a drug to be identified, which is at least one of the multiple drugs, based on a score indicating a likelihood that the front cut-out image and the back cut-out image are images of both sides of the same drug, and identify the drug type of the drug to be identified based on the front cut-out image and the back cut-out image of the created pair.

2. The drug identification device of claim 1, wherein the processor: detects the size and / or shape of the surface of each of the drugs from the cut-out image of the surface of each of the drugs; detects the size and / or shape of the back side of each of the drugs from the cut-out image of the back side of each of the drugs; extracts candidates for pairs of the cut-out image of the surface and the cut-out image of the back side based on the similarity between the size and / or shape of the surface and the size and / or shape of the back side; and creates a pair of the cut-out image of the surface and the cut-out image of the back side of the drug to be identified from among the candidate pairs.

3. The drug identification device according to claim 1, wherein the processor causes the created pair to be displayed on a display so as to be modifiable by an input device.

4. The drug identification device of claim 3, wherein the processor creates a first pair of a first front cut image and a first back cut image, and a second pair of a second front cut image and a second back cut image, and accepts an exchange of the first front cut image and the second front cut image, or an exchange of the first back cut image and the second back cut image.

5. The drug identification device according to claim 3, wherein the processor accepts cancellation of the created pair.

6. The drug identification device of claim 3, wherein the processor displays the front cut-out image and the back cut-out image that are different from the front cut-out image and the back cut-out image of the created pair, and accepts the creation of a new pair.

7. A drug identification device as described in claim 1, comprising a first trained model that, when a first image of a first side of a drug and a second image of a second side of the drug are input, outputs a score representing the likelihood that the first side and the second side are two sides of the same drug, and the processor inputs the front cut-out image and the back cut-out image to the first trained model to obtain the score.

8. A drug identification device as described in claim 1, further comprising a second trained model that outputs candidate drug types for a drug when an image of the drug is input, and the processor inputs at least one of the cut-out front surface image and the cut-out back surface image of the drug to be identified into the second trained model to obtain candidate drug types for the drug to be identified.

9. The drug identification device according to claim 8, wherein the processor obtains a plurality of drug type candidates for the drug to be identified, displays a list of the plurality of drug type candidates on a display, and determines a drug type candidate selected from the displayed list by an input device as the correct drug for the drug to be identified.

10. The drug identification device of claim 1, wherein the processor is adapted to capture a back image in which the packaged drug is inverted after the front image is captured, and the back side of the packaged drug is captured.

11. A drug identification method, comprising: at least one processor: acquiring a front image of one side of a single packaged drug package in which multiple drugs are packaged together; acquiring a back image of the other side of the single packaged drug package, generating a front cut-out image of each of the multiple drugs from the front image; generating a back cut-out image of each of the multiple drugs from the back image; creating a pair of the front cut-out image and the back cut-out image of a drug to be identified, which is at least one of the multiple drugs, based on a score representing a likelihood that the front cut-out image and the back cut-out image are images of both sides of the same drug; and identifying the drug type of the drug to be identified based on the front cut-out image and the back cut-out image of the created pair.

12. A program for causing a computer to execute the drug identification method according to claim 11.

13. A non-transitory computer-readable recording medium having the program according to claim 12 recorded thereon.

14. A method for generating a trained model executed by a learning device having a processor, wherein the processor: acquires a first learning dataset comprising a set of an actual pair consisting of an image of the front side of a drug and an image of the back side of the drug, and a first label corresponding to the actual pair; acquires a second learning dataset comprising a set of an actual pair consisting of an image of the front side of the drug and an image of the front or back side of a drug different from the first drug, and a second label corresponding to the actual pair; and trains the first learning dataset and the second learning dataset by machine learning to generate a trained model that, when a first image of a first side of a drug and a second image of a second side of the drug are input, outputs a score representing the likelihood that the first side and the second side are two sides of the same drug.

Images