Drug identification device, drug identification method, and program

Abstract

To provide a medicine identification device, a medicine identification method, and a program capable of presenting which on a captured image of a second surface, which is a back side of a first surface, a medicine whose type has been determined from a captured image of the first surface of a one package pack corresponds to.SOLUTION: A storage device stores an identification stamp master including an image of an identification stamp on each surface of the front and the back of a medicine on which the identification stamp by marking or printing is put. A processor detects each medicine from a first surface image of one package pack in which a plurality of medicines is stored in subdividing bags, creates a second surface correct answer identification stamp image list including an identification stamp image of an identified medicine that appears on the second surface from the identification stamp master using information on the identified medicine whose type and, front and back have been identified, identifies the identified medicine on a second surface image by pattern matching of an identification stamp extraction image list including an identification stamp extraction image of each medicine extracted from the second surface image and the second surface correct answer identification stamp image list, and presents information for differentiating a non-identified medicine and the identified medicine.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates to a drug identification device, a drug identification method, and a program, and in particular to an image processing technique for identifying the type of drug from an image of the drug. [Background technology]

[0002] As one of the technologies for improving the efficiency of tasks such as distinguishing medications brought in by patients or conducting dispensing audits, development is underway of image processing technology for identifying medication types from photographed images of medications. Patent Document 1 describes a drug distinguishing method in which a mobile terminal photographs a medication placed on a mounting section of a medication photographing device, the mobile terminal transmits image data of the photographed medication to a server, and the server performs a drug distinguishing process for the medication in the image data transmitted from the mobile terminal based on correspondence data that associates the image data of the medication with medication information. According to Patent Document 1, the mobile terminal can be used to photograph the front and back surfaces of the medication placed on the mounting section from above and below the mounting section of the medication photographing device.

[0003] Furthermore, Patent Document 1 describes another method for acquiring image data of the front and back surfaces of a medicine, in which a portable terminal is fixed at the top or bottom, the front surface of the medicine is photographed, the mounting part is removed from the medicine photographing device, the medicine is turned over inside out in the petri dish, the mounting part is then reinserted into the medicine photographing device, and the back surface of the medicine is photographed with the portable terminal (paragraph

[0025] ). In this case, if the position of the medicine changes when the front and back surfaces of the medicine are turned over, the image of the front and back surfaces of the medicine will no longer correspond to each other, so it is necessary to turn the medicine over in the same position, and Patent Document 1 uses a textured sheet to prevent the medicine from shifting position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-182525 Summary of the Invention [Problem to be solved by the invention]

[0005] The markings or printing on a medicine are important information when identifying the medicine. Medicines with markings or printing have a front and back, and one or both of the front and back sides may not have enough information to identify the medicine. For example, if only numbers are marked or printed on one side of the medicine, it may be difficult to identify the medicine from that information alone. Also, if one side of the medicine is plain (no markings or printing information), it is difficult to identify the type of medicine from a photographed image of that side.

[0006] The size of a device for identifying single-dose medications is preferably as small as possible, considering portability and space-saving. It is easy to determine the correspondence between the front and back of a medication by simultaneously photographing both sides of the medication, but this requires a mechanism to simultaneously photograph both sides of the medication. This inevitably increases the size of the device, making it difficult to meet this requirement. On the other hand, if the device is configured with only a single smartphone, or a combination of a smartphone and a small, portable imaging stand, the requirements for portability and space-saving can be met. However, in this case, it becomes necessary to photograph the front and back of the single-dose pack separately.

[0007] When photographing a single-dose pack containing multiple medications in a single-dose dispenser, each side will be randomly facing upside down on the side of the pack facing the camera. Therefore, in order to completely identify all medications in a single sachet with a single-side photograph, it is necessary to manually turn each medication's side with the amount of information that can be identified through the bag (the medication-identifiable side) facing up before photographing.

[0008] However, when there are many medications in a sachet, it is both tedious and difficult to keep all the medications with their identifiable side facing up. Therefore, an alternative method is to take single-sided photographs of the front and back of the single-dose pack, and then use the two images taken of each side to identify the medications whose identifiable side is visible in each image.

[0009] However, with this method, because the medications can move around inside the sachet when the single-dose pack is turned inside out, the correspondence between the medications in the two images, the first-side image obtained by photographing the first side of the single-dose pack and the second-side image obtained by photographing the second side after turning it over, becomes unclear.As a result, it is often difficult to determine which medications in the second-side image correspond to the medications whose type has already been identified in the first-side image.

[0010] In particular, when there are a large number of medications, it is difficult for the user to remember the medications that have already been identified using the first-side image, and a system that requires the user to distinguish between the remaining medications to be identified in the second-side image and the medications that have already been identified places a heavy burden on the user.

[0011] The present disclosure has been made in consideration of these circumstances, and aims to provide a drug identification device, drug identification method, and program that can present which drug, whose type has been identified from the first-side image, corresponds to on the second-side image. [Means for solving the problem]

[0012] A drug identification device according to a first aspect of the present disclosure includes one or more processors and one or more storage devices, and the one or more storage devices store an identification mark master including images of the identification marks on the front and back of drugs that have been engraved or printed as identification marks. The one or more processors detect each drug from a first-side image of a single-dose pack in which multiple drugs are stored in a sachet, and extract the drug images extracted for each drug from the first-side image using identified drug information that identifies the drug type and the front and back of at least some of the multiple drugs. A second-side correct identification mark image list is created, which includes identification mark images showing the identification marks of identified drugs that appear on the second side, which is the reverse side of the first side of the packaged pack; each drug is detected from the second-side image in which the second side of the packaged pack is photographed; an identification mark extracted image list is created, which includes identification mark extracted images for each drug, in which the identification mark of each drug is extracted from the second-side image; pattern matching is performed using the second-side correct identification mark image list and the identification mark extracted image list to identify identified drugs on the second-side image; and information is presented that differentiates identified drugs from drugs whose drug type is unidentified in the second-side image.

[0013] According to the first aspect, one or more processors automatically identify which drugs in the second image correspond to drugs whose drug types have been identified in the first image, and for the multiple drugs shown in the second image, it is possible to clearly distinguish between identified drugs whose drug types have already been identified in the first image and unidentified drugs whose drug types have not yet been identified, and present them to the user. This allows the user to easily grasp which drugs to identify in the second image, and improves the efficiency of the work of identifying packaged drugs.

[0014] The identification mark may be, for example, an identification code that combines a company code and a product code, or may be either a company code or a product code.

[0015] Identifying the drug type of a drug may mean identifying an individual brand of the drug. The identified drug type may be, for example, an individual drug identification code, such as the YJ code. Identifying the drug type of a drug is synonymous with identifying the drug. The definition of the front and back of a drug may, for example, define the identifiable side of the drug, which allows the drug to be identified by an imprinted or printed identification mark, as the front, and the opposite side as the back. Also, on package inserts, which are usually official documents, the imprinted or printed designs on both sides are arranged side by side, either side, but the imprinted or printed mark on the left or top may be defined as the front, and the imprinted or printed mark on the right or bottom may be defined as the back. If both sides of a drug are identifiable, either side may be defined as the front.

[0016] The drug identification device of the second aspect may be configured in the drug identification device of the first aspect, wherein one or more processors identify the drug type and front and back of at least some of the multiple drugs using a drug identification model trained by machine learning to identify the drug type and front and back from a drug image.

[0017] The drug identification device of the third aspect may be configured such that, in the drug identification device of the first aspect, one or more processors use a drug identification model trained by machine learning to identify drug types from drug images to identify drug types for at least some of the multiple drugs, and use an identification mark master to identify the front and back of at least some of the multiple drugs.

[0018] The drug identification device of the fourth aspect may be configured such that, in the drug identification device of the third aspect, one or more processors identify the front and back by pattern matching the drug image or an identification mark image extracted from the drug image with an identification mark master.

[0019] The drug identification device of the fifth aspect may be configured such that, in the drug identification device of any one of the first to fourth aspects, one or more processors perform brute force pattern matching between identification mark images in the second-side correct identification mark image list and identification mark extraction images in the identification mark extraction image list, and identify identified drugs on the second-side captured image based on the matching score.

[0020] A drug identification device according to a sixth aspect is the drug identification device according to any one of the first to fifth aspects, wherein the pattern matching is template matching.

[0021] A drug identification device according to a seventh aspect may be configured such that, in the drug identification device according to any one of the first to sixth aspects, one or more processors receive input of an instruction to confirm the identified drug type for at least some of the multiple drugs, and identify the drug type of the target drug based on the input instruction.

[0022] A drug identification device according to an eighth aspect may be configured such that, in the drug identification device according to any one of the first to seventh aspects, the one or more processors create an identified list as information on identified drugs.

[0023] The drug identification device of the 9th aspect may be configured such that, in the drug identification device of any one of the 1st to 8th aspects, one or more processors mark the identified drug in the second-plane image as differentiating information to indicate that the drug type has been identified.

[0024] The drug identification device of the 10th aspect may be configured such that, in the drug identification device of any one of the 1st to 9th aspects, one or more processors mark unidentified drugs in the second-plane captured image as differentiating information to indicate that the drug type is unidentified.

[0025] The drug identification device according to the eleventh aspect may be configured to include a display that displays a second-side captured image including differentiating information in the drug identification device according to any one of the first to tenth aspects.

[0026] A medicine identification device according to a twelfth aspect may be configured such that the medicine identification device according to any one of the first to eleventh aspects further includes a camera that photographs the unit-dose pack.

[0027] A medicine identification method according to a thirteenth aspect of the present disclosure is a medicine identification method executed by one or more processors, which includes storing, in one or more storage devices, an identification mark master including images of the identification marks on the front and back of medicines that have been stamped or printed with an identification mark; acquiring a first-side image of a first side of a unit-dose pack in which a plurality of medicines are housed in a sachet; detecting each medicine from the first-side image and extracting a medicine image for each medicine; identifying the medicine type and the front and back of at least some of the plurality of medicines based on the medicine images extracted from the first-side image; and identifying the medicine using information on the identified medicines whose medicine types have been identified from the first-side image. The method includes creating a second-side correct identification mark image list from the mark master, which includes identification mark images showing the identification marks of identified drugs that appear on the second side, which is the back side of the first side of the unit-dose pack; acquiring a second-side photographed image of the second side of the unit-dose pack; detecting each drug from the second-side photographed image and extracting the identification mark of each drug, and creating an identification mark extracted image list including an identification mark extracted image for each drug; identifying identified drugs on the second-side photographed image by performing pattern matching using the second-side correct identification mark image list and the identification mark extracted image list; and presenting information that differentiates identified drugs from drugs whose drug type is unidentified in the second-side photographed image.

[0028] The medicine identifying method according to the thirteenth aspect may have a configuration including the same specific aspects as the medicine identifying device according to any one of the second to twelfth aspects.

[0029] A program according to a 14th aspect of the present disclosure is a program that causes a computer to execute the drug identification method according to aspect 13. A non-transitory computer-readable recording medium (computer-readable medium), which is a tangible object, on which the program according to the 14th aspect is recorded, is also included in the present disclosure.

[0030] The program according to the fourteenth aspect may be configured to include the same specific aspects as the drug identification device according to any one of the second to twelfth aspects. [Effects of the Invention]

[0031] According to the present disclosure, one or more processors execute a process for identifying which drugs in the second image correspond to drugs whose types have already been identified in the first image of a single-dose pack, and the identified drugs and unidentified drugs can be differentiated in the second image and presented to the user. This allows the user to easily understand which drug among the multiple drugs in the second image they need to identify. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a front perspective view of a smartphone. [Figure 2] FIG. 2 is a rear perspective view of the smartphone. [Figure 3] FIG. 3 is a block diagram showing the electrical configuration of the smartphone. [Figure 4] FIG. 4 is a block diagram showing the functional configuration of the drug identification device according to the embodiment. [Figure 5] FIG. 5 is an example of a first-side image in which the first side of a single-dose pack is photographed. [Figure 6] FIG. 6 is an explanatory diagram showing an example of an engraving image included in the engraving master. [Figure 7] FIG. 7 is an explanatory diagram showing an example of identifying the marking image on the second surface side of the identified medicine from the marking master. [Figure 8]FIG. 8 is an explanatory diagram showing an example of a list of correct stamp images for the second surface. [Figure 9] FIG. 9 is an example of a second-side photographed image of the second side of a single-dose pack. [Figure 10] FIG. 10 is an explanatory diagram showing an example of template matching processing using the second-side correct engraving image list and the second-side engraving extracted image list. [Figure 11] FIG. 11 is an explanatory diagram showing an example of presentation of information that differentiates between identified medicines and unidentified medicines on the second plane captured image. [Figure 12] FIG. 12 is a flowchart showing an example of a drug identification method carried out using the drug identification device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0033] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0034] [Outline of the drug identification device according to the embodiment] A drug identification device according to an embodiment of the present disclosure is an information processing device that performs processing to identify the drug type of each drug by photographing a single-dose pack in which multiple drugs are contained in a sachet bag using a first-side image and a second-side image obtained by photographing one side of the single-dose pack (a first side) and the other side (a second side) of the single-dose pack while the pack is still in the bag.

[0035] The drug identification device according to this embodiment first identifies the drug types of at least some of the drugs shown in the first-view image, and then identifies the remaining drugs using the second-view image. The drug identification device according to this embodiment includes one or more processors, which automatically identify which drugs in the second-view image correspond to those whose drug types have been identified in the first-view image. The one or more processors then execute a process to clearly differentiate between drugs whose drug types have already been identified in the first-view image and unidentified drugs whose drug types have not yet been identified (i.e., the remaining drugs to be identified in the second-view image) and present them to the user on the display screen of the second-view image. This allows the user to easily determine which drugs need to be identified by drug type in the second-view image, thereby streamlining the drug identification process for single-dose packs.

[0036] The term "identification" in relation to drugs includes the concepts of discrimination and auditing. The type of drug to be identified is a drug type that can be identified by identification information such as a YJ code (individual drug code) or drug name. Drug identification in this embodiment can be defined as the act of determining which YJ code the drug to be identified belongs to. This is one example of the definition of drug identification, and for example, the identification code may be defined using a type other than the YJ code. An identified drug whose drug type has been identified may also be called an "identified drug."

[0037] The drug identification device is mounted on a portable terminal device, for example. The portable terminal device includes at least one of a smartphone, a mobile phone, a PHS (Personal Handy-phone System), a PDA (Personal Digital Assistant), a tablet computer terminal, a notebook personal computer terminal, a wearable terminal, and a portable game console. Below, a drug identification device realized by the hardware and software of a smartphone is taken as an example and will be described in detail with reference to the drawings.

[0038] [Appearance of the smartphone] Fig. 1 is a front perspective view of a smartphone 10 that functions as a drug identification device according to an embodiment of the present disclosure. As shown in Fig. 1, the smartphone 10 has a flat housing 12. The smartphone 10 has a touch panel display 14, a speaker 16, a microphone 18, and an internal camera 20 on the front side of the housing 12.

[0039] The touch panel display 14 includes a display unit that displays images and the like, and a touch panel unit that is disposed in front of the display unit and accepts touch inputs. The display unit is, for example, a color LCD (Liquid Crystal Display) panel or a color organic EL (organic electro-luminescence) panel.

[0040] The touch panel unit is, for example, a capacitive touch panel provided in a planar form on a light-transmitting substrate body, and includes light-transmitting position detection electrodes and an insulating layer provided on the position detection electrodes. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to a user's touch operation. Touch operations include a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

[0041] The speaker 16 is an audio output unit that outputs audio during a call or when playing back a video. The microphone 18 is an audio input unit that inputs audio during a call or when shooting a video. The front camera 20 is an imaging device that captures videos and still images.

[0042] Fig. 2 is a rear perspective view of the smartphone 10. As shown in Fig. 2, the smartphone 10 is provided with an outer camera 22 and a light 24 on the rear surface of the housing 12. The outer camera 22 is an imaging device that captures moving images and still images. The light 24 is a light source that emits illumination light when capturing images with the outer camera 22, and is configured, for example, by an LED (Light Emitting Diode).

[0043] 1 and 2, the smartphone 10 is further provided with switches 26 on the front and side of the housing 12. The switches 26 are input members that accept instructions from the user. The switches 26 are push-button switches that turn on when pressed with a finger or the like and turn off when the finger is released due to the restoring force of a spring or the like.

[0044] The configuration of the housing 12 is not limited to this, and a configuration having a folding structure or a sliding mechanism may also be adopted.

[0045] [Smartphone electrical configuration] The smartphone 10 has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device and a mobile communication network.

[0046] Fig. 3 is a block diagram showing the electrical configuration of the smartphone 10. As shown in Fig. 3, the smartphone 10 includes the touch panel display 14, speaker 16, microphone 18, in-camera 20, out-camera 22, light 24, and switch 26 described above, as well as a CPU (Central Processing Unit) 28, a wireless communication unit 30, a call unit 32, a memory 34, an external input / output unit 40, a GPS (Global Positioning System) receiving unit 42, and a power supply unit 44.

[0047] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates in accordance with the control program and control data stored in the memory 34, and controls all the components of the smartphone 10. The CPU 28 has a mobile communication control function that controls all the components of the communication system to perform voice communication and data communication via the wireless communication unit 30, and an application processing function.

[0048] The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function visually conveys information such as still images, moving images, and text to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to a touch operation by the user from the touch panel portion of the touch panel display 14. The CPU 28 also acquires an input signal from the switch 26.

[0049] The in-camera 20 and the out-camera 22 each have a photographing lens, an aperture, an image sensor, an AFE (Analog Front End), an A / D (Analog to Digital) converter, a lens driver, etc. The in-camera 20 and the out-camera 22 capture moving images and still images according to instructions from the CPU 28.

[0050] The CPU 28 may convert the moving images and still images captured by the in-camera 20 and the out-camera 22 into compressed image data using MPEG (Moving Picture Experts Group) and JPEG (Joint Photographic Experts Group) formats, for example.

[0051] The CPU 28 stores the video and still images captured by the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0052] Furthermore, the CPU 28 displays the video and still images captured by the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the video and still images captured by the in-camera 20 and the out-camera 22 in application software.

[0053] The CPU 28 may illuminate the subject with fill light by turning on the light 24 when capturing an image with the outer camera 22. The light 24 may be turned on and off by a touch operation on the touch panel display 14 or by operating the switch 26 by the user.

[0054] The wireless communication unit 30 performs wireless communication with a base station device compatible with mobile communication networks of standards such as 4G (4th Generation) and 5G (5th Generation) in accordance with instructions from the CPU 28. The smartphone 10 uses this wireless communication to send and receive various file data such as audio data and image data, email data, and receive Web (abbreviation for World Wide Web) data and streaming data.

[0055] The speaker 16 and the microphone 18 are connected to the communication unit 32. The communication unit 32 decodes the audio data received by the wireless communication unit 30 and outputs it from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into audio data that can be processed by the CPU 28 and outputs it to the CPU 28.

[0056] The memory 34 stores instructions to be executed by the CPU 28. The memory 34 is composed of an internal storage unit 36 ​​built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are realized using known storage media.

[0057] The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of emails sent and received, web data downloaded by web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

[0058] The external input / output unit 40 serves as an interface with external devices connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to other external devices by communication or the like via the external input / output unit 40. The external input / output unit 40 transmits data received from external devices to each component within the smartphone 10, and also transmits data within the smartphone 10 to external devices.

[0059] Examples of communication means include Universal Serial Bus (USB), IEEE (Institute of Electrical and Electronics Engineers) 1394, the Internet, wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. Examples of external devices include headsets, external chargers, data ports, audio devices, video devices, smartphones, PDAs, personal computers, and earphones.

[0060] The GPS receiver 42 detects the position of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, . . . , STn.

[0061] The power supply unit 44 is a power supply source that supplies power to each unit of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium-ion secondary battery. The power supply unit 44 may also include an AC / DC converter that generates a DC voltage from an external AC power supply.

[0062] The smartphone 10 configured in this manner is set to a shooting mode in response to a user's instruction input using the touch panel display 14 or the like, and can capture moving images and still images using the in-camera 20 and the out-camera 22.

[0063] When the smartphone 10 is set to the shooting mode, it enters a shooting standby state, and a video is captured by the in-camera 20 or the out-camera 22, and the captured video is displayed on the touch panel display 14 as a live view image.

[0064] The user can visually check the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject they want to photograph, and set the photographing conditions.

[0065] When the smartphone 10 is in a standby state for shooting and receives an instruction to shoot by a user input using the touch panel display 14 or the like, the smartphone 10 performs AF (Autofocus) and AE (Auto Exposure) control, and shoots and stores videos and still images.

[0066] The memory 34 is an example of a "storage device" in the present disclosure. The touch panel display 14 is an example of a user interface and an example of a "display" in the present disclosure. Each of the in-camera 20 and the out-camera 22 is an example of a "camera" in the present disclosure.

[0067] [Functional configuration of the drug identification device] 4 is a block diagram showing the functional configuration of a drug identification device 100 realized by the smartphone 10. As shown in FIG. 4, the drug identification device 100 includes an image acquisition unit 102, a drug detection unit 104, a drug image extraction unit 106, a drug identification unit 108, a determination unit 110, an identified list creation unit 112, a second-page correct imprint image list creation unit 114, an imprint extraction unit 116, a second-page imprint extracted image list creation unit 118, a template matching unit 120, an identified drug identification unit 122, an identified drug information presentation unit 124, a touch panel display 14, and an imprint master 140. The functions of each unit of the drug identification device 100 can be realized by the hardware and software of the smartphone 10, and can be embodied by the CPU 28 executing instructions of a program stored in the memory 34.

[0068] The image acquisition unit 102 acquires a still image of a single-dose pack in which multiple medications are contained in a sachet by single-dose dispensing. The captured image may be an image captured by the external camera 22, for example. The captured image may also be an image acquired from another device via the wireless communication unit 30, the external memory unit 38, or the external input / output unit 40. The sachet may be entirely or partially transparent or semi-transparent. Each of the multiple medications contained in the sachet can be a medication to be identified.

[0069] The captured image acquired by the image acquisition unit 102 may be an image in which one or more markers are captured together with the unit-dose pack. The markers may be, for example, ArUco markers, circular markers, or rectangular markers. When performing image processing such as image area cropping and / or transformation processing on the captured image, it is preferable that the captured image contains multiple markers. The multiple markers are, for example, arranged at the four corners of a rectangular area of ​​the drug placement area on the surface on which the unit-dose pack is placed during imaging. It is preferable that the drug placement area be configured with, for example, a reference gray or black background.

[0070] The captured image may be an image captured at a standard shooting distance and shooting viewpoint. The shooting distance can be expressed by the distance between the drug to be identified and the shooting lens and the focal length of the shooting lens. The shooting viewpoint can be expressed by the angle between the drug placement surface (marker printed surface) and the optical axis of the shooting lens.

[0071] When photographing the unit-dose pack, a photography assistant may be used to set the smartphone 10 used for photography at or near the camera position that is the standard photography distance and photography viewpoint. This photography assistant includes a mounting base on which the unit-dose pack to be photographed is placed, and a light It is preferable that the configuration be combined with a lighting device that illuminates the image.

[0072] The image acquisition unit 102 may include an image correction unit (not shown). When a captured image includes markers, the image correction unit standardizes the shooting distance and shooting viewpoint of the captured image based on the markers to generate a standardized image. The standardized image may be an image obtained by cutting out an area inside a rectangle whose vertices are the four corner markers after the captured image has been standardized. For example, the image correction unit specifies the coordinates of the four vertices of a rectangle whose coordinates are identified by the markers after the shooting distance and shooting viewpoint are standardized. The image correction unit calculates a perspective transformation matrix that transforms these four vertices to the positions of the specified coordinates. Such a perspective transformation matrix is ​​uniquely determined if there are four points. For example, the getPerspectiveTransform function of OpenCV (Open Source Computer Vision Library) can be used to calculate the transformation matrix if there is a correspondence between the four points.

[0073] The image correction unit performs perspective transformation on the entire original captured image using the obtained perspective transformation matrix, and obtains the transformed image. Such perspective transformation can be performed using the warpPerspective function of OpenCV. The transformed image may be a standardized image in which the shooting distance and shooting viewpoint are standardized.

[0074] Furthermore, when the captured image includes an area of ​​a reference gray color, the image correction section may perform color correction on the captured image based on the reference gray color.

[0075] In the drug identification device 100 of this embodiment, first, a first-side captured image IM1 is obtained, which is a still image of the first side, which is one side of the unit-dose pack. Then, a process is performed to identify the drug type for drugs included in this first-side captured image IM1 that show a drug-identifiable side. A drug-identifiable side is a side of a drug that has been engraved or printed, and from which the drug can be identified (a side from which the drug type can be identified) from the information on the engraved or printed markings on that side. In contrast, a side from which it is difficult to identify the drug from the information on the engraved or printed markings on that side, or a side without any engraving or markings (a plain side), is called a drug-difficult-to-identify side. For drugs that show a drug-identifiable side in the first-side captured image IM1, the drug type can be identified from the first-side captured image IM1.

[0076] The drug identification device 100 then acquires a second-side image IM2, which is a still image of the second side opposite the first side of the same single-dose pack, and uses the second-side image IM2 to identify the drug types of the remaining (unidentified) drugs.

[0077] Each of the first plane image IM1 and the second plane image IM2 acquired via the image acquisition unit 102 may be a standardized image.

[0078] The medicine detection unit 104 detects the area of ​​each medicine from the captured image acquired via the image acquisition unit 102. The medicine detection unit 104 uses machine learning to perform a so-called object detection task. to Therefore, it may be configured using a trained and learned AI (Artificial Intelligence) model. The detection result of the drug detection unit 104 is displayed on the touch panel display 14. For example, the drug region detected from the first captured image IM1 may be displayed by a bounding box on the screen displaying the first captured image IM1.

[0079] FIG. 5 is an example of a first-side captured image IM1. Four medications D1 to D4 are shown in the first-side captured image IM1 shown in FIG. 5. The medications D2 and D4, enclosed by the dashed-line rectangle in FIG. 5, have a large amount of information about the markings, and the medication type can be identified in this first-side captured image IM1. In contrast, the medications D1 and D3 have a small amount of information about the markings or printing, making it difficult to identify the medications in this first-side captured image IM1. In other words, the first-side captured image IM1 shown in FIG. 5 is an image in which medications D2 and D4, whose medication-identifiable sides are photographed, and medications D1 and D3, whose medication-difficult-to-identify sides are photographed, are mixed together.

[0080] The user can specify the type of each drug by specifying drugs D2 and D4 whose drug identifiable surfaces are photographed from the first surface photographed image IM1 displayed on the touch panel display 14. Note that the drug image extraction unit 106 and the drug identification unit 108 may automatically perform processing on each drug detected by the drug detection unit 104 without waiting for the user to perform a drug designation operation.

[0081] The drug image extraction unit 106 extracts an image region of each drug from the captured image based on the detection result by the drug detection unit 104, and generates a drug image in which the drug image region is cut out for each drug.

[0082] The drug image extraction unit 106 may be incorporated into the drug detection unit 104. For example, the drug detection unit 104 includes a drug region extraction model. The drug region extraction model may be a trained model that, when a photographed image of a drug is given as input, outputs a drug image in which the drug region within the image is extracted. The drug region extraction model may be a segmentation model that has been machine-learned using a training data set of a plurality of different photographed images, the training data set being a set of drug images and drug regions included in the image. A convolutional neural network (CNN) can be applied as the drug region extraction model.

[0083] The drug identification unit 108 is a processing unit that identifies the drug type of a drug from the drug image extracted by the drug image extraction unit 106. The drug identification unit 108 may be, for example, an AI processing unit that uses a learned AI model (drug identification model) trained by machine learning to perform the object recognition task of identifying the drug type from an input drug image. The drug type identified by the drug identification unit 108 is, for example, a drug type that can be specified by identification information such as a YJ code (individual drug code) or drug name. Drug identification in this embodiment can be defined as the act of determining which YJ code the drug to be identified is. This is one example of the definition of drug identification, and for example, the identification code may be defined using a type other than the YJ code.

[0084] The drug identification model functions as a multi-class classifier that receives input drug images of drugs to be identified, identifies the type of drug, and classifies it into N previously trained drug types (classes). When a drug image is input, the drug identification model calculates a score value indicating the likelihood (certainty) that the drug to be identified is drug i for all N previously trained drug types i. The "i" in the notation "drug i" is an index that distinguishes the N previously trained drugs. The drug identification model calculates a score value for each drug i that serves as an index for determining whether the drug to be identified is drug i.

[0085] The drug identification model is configured using, for example, a neural network. CNN can be used as a machine learning model suitable for image recognition. The image input to the drug identification model may be a region image (drug image) of the drug to be identified, cut out from the captured image. In addition to the drug image, identification mark information such as an inscription or print extracted from the drug image may also be input to the drug identification model. The identification mark information may be an image or text information.

[0086] Based on the identification result by the medicine identifying unit 108, one or more medicine type candidates with the highest score values ​​are displayed on the touch panel display 14.

[0087] For example, when a user specifies a drug to be identified (e.g., drug D2) on the first-surface captured image IM1, a drug image of the specified drug to be identified is extracted from the first-surface captured image IM1 and input into the drug identification unit 108, and candidate drug types as the identification result by the drug identification unit 108 are displayed on the touch panel display 14.

[0088] The determination unit 110 receives an instruction to determine the drug type of the target drug from the touch panel display 14 or other user interface, and performs a process to determine the drug type of the target drug in accordance with the received instruction. This identifies the drug type of the target drug.

[0089] By identifying the drug type of the target drug, the front and back sides of the target drug can also be identified. For example, if the drug type of a drug is identified, the drug identifiable side is shown in the first side image IM1, so the drug identifiable side may be identified as the "front." Furthermore, for example, the front and back sides of the drug can be identified by referring to the marking master 140 based on the identified drug type.

[0090] The marking master 140 is a master database containing image information of the markings or printings affixed to the medicine, and stores images of the markings or printings of identification marks on the front and back of the medicine, linked to the medicine type (see FIG. 6). The marking master 140 is stored in memory 34, and the data is updated as necessary. The front and back of the medicine may be determined by template matching the medicine image extracted from the first-side captured image IM1, or the marking extraction image extracted from that medicine image, with the marking master 140.

[0091] Furthermore, the drug identification model may be a model trained to receive an input of an image of one side of a drug to be identified, photographed from one side (from one direction), and to be able to identify the drug type of the drug to be identified and even the front and back of the drug to be identified (whether it is the front or back side). For example, the drug identification model may be a classifier trained by machine learning to receive an input of a drug image and output a class that combines the YJ code and the front and back information. By using such a model, it is possible to identify the front and back at the time of identifying the drug type. In this case, instead of defining classes for all combinations of the YJ code and the front and back, the model may not define classes for combinations that include drug sides that are difficult to identify.

[0092] To increase the accuracy of identification by the medication identification model, template matching with the imprint master 140 may also be used to determine the front and back of the medication.

[0093] After detecting the medication in the first-surface captured image IM1, the user checks the identification result of the medication identification unit 108 for the first-surface captured image IM1 to identify the medication type and front / back. In the example of Figure 5, the user performs an operation to identify the medication type for each of medications D2 and D4. Alternatively, the user may identify the medication type using the search results by performing a text search in an imprint database (not shown) based on the information of the medication's imprint or print visible in the first-surface captured image IM1, without using the identification process of the medication identification unit 108. The imprint database is a collection of data that associates the imprint or print character string affixed to the medication with the medication type to which the imprint or print character string is affixed. The imprint database may be configured integrally with the imprint master 140.

[0094] The identified list creation unit 112 creates a list of identified drugs (hereinafter referred to as "identified list") whose drug types have been identified from the first captured image IM1 after processing by the determination unit 110. In the example of FIG. 5, an identified list is created that includes information indicating that drugs D2 and D4 have been identified. The identified list includes the YJ code and front / back information of the drugs identified in the first captured image IM1.

[0095] The second-side correct imprint image list creation unit 114 creates a second-side correct imprint image list, which is a list of correct imprint images on the second side of the identified medicine, from the identified list and the imprint master 140. Here, the "correct imprint image" means an imprint image that is expected to appear on the second side, and is an imprint image as an expected value for the second side. By reversing the front and back of the identified medicine identified from the first-side photographed image IM1, the second-side correct imprint image list can be created from the imprint master 140.

[0096] The term "imprint" in the terms "imprint master 140," "imprint database," and "imprint image" represents the concept of an identification mark formed by at least one of imprinting and printing, and includes the meaning of "imprinting and / or printing." In this specification, the term "imprinting" may be understood to include the concept of "printing" to the extent that this is not contradictory from the context.

[0097] Fig. 6 shows an example of the engraving master 140. Fig. 6 shows engraving images registered in the engraving master 140 for drugs D1 to D4. Although only four drugs D1 to D4 are shown here, the engraving master 140 may include data for all types of identifiable drugs.

[0098] In Figure 6, images marked with a star indicate images of the drug's identifiable side. Note that the drug stamped with "DD 444" shown at the bottom of Figure 6 has the same stamp on both the front and back, and both the front and back are drug identifiable sides. The stamp master 140 shown in Figure 6 is an example of an "identification mark master" in the present disclosure, and the stamp image shown in Figure 6 is an example of an "identification mark image" in the present disclosure.

[0099] The correct stamp image list creation unit 114 for the second surface refers to the stamp master 140 based on the identified list, extracts the stamp images on the second surface for the identified drugs from the stamp master 140, and creates a correct stamp image list for the second surface.

[0100] FIG. 7 is an explanatory diagram showing an example of identifying the imprint image on the second side of an identified drug from the imprint master 140. In the example of FIG. 5, if drugs D2 and D4 are identified drugs, the imprint images on the back side of these drugs will be surrounded by the rectangle shown by the thick line in FIG. 7. That is, the imprint expected to appear in the second side photographed image IM2 of the identified drug D2, whose obverse side is imprinted with "BB 222," is the imprint "50." Similarly, the imprint expected to appear in the second side photographed image IM2 of the identified drug D4, whose obverse side is imprinted with "DD 444," is the imprint "DD 444."

[0101] In this way, correct imprint images expected in the second-side photographed image IM2 of the identified medicine are extracted from the imprint master 140, and a list of correct imprint images for the second side is created by collecting these (see FIG. 8).

[0102] FIG. 8 is an example of the correct stamp image list LSC for the second side. The correct stamp image list LSC for the second side shown in FIG. 8 includes correct stamp images CE2 and CE4 (as expected values) that should appear on the second side of each of the identified drugs D2 and D4 described in FIG. 7. These correct stamp images CE2 and CE4 are extracted from the stamp master 140 using the identified list. Each of the stamp images CE2 and CE4 is an example of an "identification mark image" in the present disclosure, and the correct stamp image list LSC for the second side is an example of a "second side correct identification mark image list" in the present disclosure. Such a correct stamp image list LSC for the second side is used as a matching list to be applied to the template matching unit 120 when identifying the type of the remaining (unidentified) drug using the second side captured image IM2.

[0103] Next, the processing of the second plane captured image IM2 will be described.

[0104] After completing the specific operation of the medicine in the first side photographed image IM1, the user turns the same single-dose pack over and photographs the second side.

[0105] Fig. 9 is an example of a second-side photographed image IM2. The second-side photographed image IM2 shown in Fig. 9 is an example of a photographed image of the same single-dose pack as the first-side photographed image IM1 shown in Fig. 5, but taken upside down. In the second-side photographed image IM2 shown in Fig. 9, the drug identifiable sides of each of drugs D1 and D3 are photographed.

[0106] The image acquisition unit 102 acquires the second image IM2. As with the first image IM1, the drug detection unit 104 detects drug regions from the second image IM2. The drug image extraction unit 106 creates drug images that are cut-out images of each drug detected by the drug detection process. The mark extraction unit 116 extracts the mark from the drug image of each drug extracted from the second image IM2 to create a mark extraction image.

[0107] The second surface imprint extracted image list creating unit 118 creates a second surface imprint extracted image list by collecting the imprint extracted images of each medicine extracted from the second surface photographed image IM2.

[0108] An example of a second-side imprint extraction image list LS2 is shown on the left side of Figure 10. This second-side imprint extraction image list LS2 was created from the second-side photographed image IM2 shown in Figure 9. The second-side imprint extraction image list LS2 includes imprint extraction images EG1, EG2, EG3, and EG4 extracted from the respective drug images of drugs D1, D2, D3, and D4 included in the second-side photographed image IM2. Each of the imprint extraction images EG1 to EG4 is an example of an "identification mark extraction image" in the present disclosure, and the second-side imprint extraction image list LS2 is an example of an "identification mark extraction image list" in the present disclosure.

[0109] The template matching unit 120 performs template matching on the engraving extraction images EG1 to EG4 included in the second-side engraving extraction image list LS2 with the correct engraving images CE2 and CE4 included in the second-side correct engraving image list LSC in a round-robin manner, and evaluates the degree of match. The degree of match is evaluated based on the matching score calculated by template matching. Template matching is an example of "pattern matching" in this disclosure. Figure 10 shows the template matching process performed between the engraving images in the second-side correct engraving image list LSC and the second-side engraving extraction image list LS2.

[0110] Based on the processing result of the template matching unit 120, the identified medicine identifying unit 122 identifies, from the second-side imprint extracted image list LS2, those that have a high degree of match with the second-side correct imprint image list LSC as identified medicines.

[0111] In the example of Figure 10, the stamp extraction image EG2 and the stamp extraction image EG4 are evaluated as having a high degree of match with the correct stamp images CE2 and CE4 in the correct stamp image list LSC for the second surface, and in the second surface captured image IM2, the drugs D2 and D4 corresponding to these stamp extraction images EG2 and EG4 are identified as identified drugs.

[0112] The identified drug information presentation unit 124 performs a process of presenting to the user information that differentiates between identified drugs whose drug types have been identified and unidentified drugs whose drug types have not been identified, among the multiple drugs D1 to D4 in the second-surface captured image IM2, based on information about identified drugs identified by the identified drug identification unit 122.

[0113] FIG. 11 is a diagram showing an example of information presentation that differentiates between identified and unidentified drugs on the second-surface captured image IM2. In FIG. 11, a check mark CM is attached to each of identified drugs D2 and D4, indicating that they have been identified. The user can understand whether or not the drug type has been identified based on the presence or absence of the check mark CM. In other words, the user can easily recognize that drugs D2 and D4, which have check marks CM attached, have been identified, and understand that they only need to identify the remaining drugs D1 and D3, which do not have check marks CM attached.

[0114] With this configuration, drugs whose types have already been identified in the first-surface image IM1 can be identified on the second-surface image IM2, allowing the user to easily grasp the remaining drugs that need to be identified in the second-surface image IM2.

[0115] The check mark CM is an example of "differentiating information" in the present disclosure. FIG. 11 shows an example in which information indicating "identified" is attached to an identified drug, but instead of this, or in combination with this, information indicating "unidentified" may be attached to an unidentified drug. In addition, the manner of differentiation is not limited to the presence or absence of a check mark CM, but may also be the presence or absence of a bounding box or other frame surrounding the drug, a difference in the display color of the frame, or a different shape of the frame. difference The display may be a difference between flashing and constant display of marks, display of text information indicating whether a drug has been identified or not, or graying out identified drugs, or any suitable combination of these.

[0116] <Example of drug identification method> FIG. 12 is a flowchart showing an example of a drug identification method carried out using the drug identification device 100 according to this embodiment.

[0117] In step S1, the user photographs one side of the unit-dose pack, that is, the first side, using the camera (for example, outer camera 22) of the smartphone 10. The CPU 28 acquires a first-side photographed image IM1 in which the first side of the unit-dose pack is photographed.

[0118] In step S2, CPU 28 detects each medicine from first captured image IM1. The medicine detection process may be performed by, for example, medicine detection AI (Artificial Intelligence) using a trained model that extracts a medicine area from the input captured image.

[0119] Thereafter, the CPU 28 proceeds to a first loop LP1. The first loop LP1 includes steps S3 to S5. The CPU 28 executes steps S3 to S5 for each drug detected in step S2.

[0120] In step S3, the CPU 28 cuts out the area of ​​each detected medicine from the first photographed image IM1, and extracts medicine images, which are area images for each medicine.

[0121] In step S4, CPU 28 determines whether the extracted medicine image is an image of an identifiable surface that allows the medicine type to be identified. For example, if a user checks the imprint information on the display screen of first surface image IM1 and performs an operation to specify medicines D2 and D4 with a large amount of information, CPU 28 may determine that the specified medicines D2 and D4 have a "medicine identifiable surface."

[0122] If the determination result in step S4 is Yes, the process proceeds to step S5.

[0123] In step S5, the CPU 28 executes the drug identification process for the drug to be identified, and identifies the drug type and the front and back sides after the user's confirmation operation.

[0124] On the other hand, if the determination result in step S4 is No, step S5 is skipped.

[0125] The first loop LP1 is executed for each drug detected in the first image IM1, and the drug type and front / back are identified for each drug whose drug identifiable side is shown in the first image IM1. After the first loop LP1 processing is completed for each drug included in the first image IM1 and the first loop LP1 is exited, the process proceeds to step S6.

[0126] In step S6, the CPU 28 creates an identified list, which is a list of medicines whose types have been identified in the first captured image IM1.

[0127] In step S7, the CPU 28 creates a list of correct engraving images for the second surface from the specified list and the engraving master 140.

[0128] In step S8, the user turns the package pack over and photographs the second side of the package pack using the camera of the smartphone 10. The CPU 28 acquires a second-side photographed image IM2 in which the second side of the package pack is photographed.

[0129] In step S9, the CPU 28 performs a drug detection process on the acquired second plane captured image IM2. Step S9 may be the same process as step S2.

[0130] Thereafter, the CPU 28 proceeds to a second loop LP2, which includes steps S10 and S11. The CPU 28 executes steps S10 and S11 for each drug detected in step S9.

[0131] In step S10, the CPU 28 cuts out the area of ​​each detected drug from the second plane photographed image IM2, and extracts a drug image, which is an area image for each drug.

[0132] In step S11, the CPU 28 extracts the mark from the medicine image extracted in step S10.

[0133] After the processing of the second loop LP2 is completed for each drug included in the second plane photographed image IM2 and the second loop LP2 is exited, the process proceeds to step S12.

[0134] In step S12, the CPU 28 creates a second surface imprint extraction image list for matching, which is a list of the imprint extraction images extracted by the second loop LP2.

[0135] In step S13, the CPU 28 performs a round-robin template matching between the correct engraving image list for the second surface and the engraving extracted image list for the second surface.

[0136] In step S14, the CPU 28 identifies which drug on the second photographed image IM2 corresponds to the drug identified on the first plane based on the processing result of step S13.

[0137] In step S15, the CPU 28 presents to the user which drugs are identified on the second captured image IM2 in an easy-to-understand manner based on the identification result of step S14. For example, the CPU 28 presents identified drugs with check marks CM to differentiate them from unidentified drugs, as shown in FIG.

[0138] After step S15, the same process as in the first loop LP1 is performed on the unidentified medicine in the second plane photographed image IM2 to identify the medicine type of the unidentified medicine.

[0139] [Hardware configuration of each processing unit] The hardware structure of the processing unit that executes various processes, such as the image acquisition unit 102, drug detection unit 104, drug image extraction unit 106, drug identification unit 108, confirmation unit 110, identified list creation unit 112, second-side correct imprint image list creation unit 114, imprint extraction unit 116, second-side imprint extracted image list creation unit 118, template matching unit 120, identified drug identification unit 122, and identified drug information presentation unit 124 described in Figure 4, is various processors as shown below.

[0140] Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and function as various processing units, GPUs (Graphics Processing Units), which are processors specialized for image processing, Programmable Logic Devices (PLDs), which are processors whose circuit configuration can be changed after manufacture such as FPGAs (Field Programmable Gate Arrays), and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically to execute specific processes.

[0141] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0142] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0143] [Regarding the program that realizes the functions of the drug identification device 100] The processing functions of the drug identification device 100 are not limited to the smartphone 10, and can be realized using various types of information processing devices, such as a tablet computer, a personal computer, a workstation, or a server. The processing functions of the drug identification device 100 may be realized by a computer system including multiple computers. The processing functions of the drug identification device 100 may be realized using a cloud server.

[0144] A program that causes a computer to realize some or all of the processing functions of the drug identification device 100 described in the above embodiment can be recorded on a computer-readable medium that is a tangible, non-transitory information storage medium such as an optical disk, a magnetic disk, a semiconductor memory, or other such medium, and the program can be provided through this information storage medium.Instead of storing the program on such a tangible, non-transitory information storage medium and providing it, it is also possible to provide a program signal as a download service using a telecommunications line such as the Internet.

[0145] It is also possible to provide a part or all of the processing functions of the drug identification device 100 as an application server, and to provide a service that provides the processing functions via an electric communication line.

[0146] [Effects of the embodiment] According to the drug identification device 100 of the embodiment, the following effects can be obtained.

[0147] [1] The medicine identifying device 100 can automatically identify which medicine on the second image corresponds to a medicine whose type has been identified from the first image IM1 of the unit-dose pack.

[0148] [2] After identifying the drug types of some of the multiple drugs from the first-surface image IM1, when identifying the remaining unidentified drugs in the second-surface image IM2, it is possible to differentiate between identified drugs and unidentified drugs on the second-surface image IM2 and present them in an easy-to-understand manner for the user.

[0149] [3] The user can easily identify which of the multiple drugs in the second image IM2 should be identified. This reduces the burden of identifying the single-dose drugs and improves work efficiency.

[0150] [Variation 1] In the above embodiment, an example has been described in which the second image IM2 is acquired after identifying the drug types of some drugs in the first image IM1, but the timing of acquiring the first image IM1 and the second image IM2 is not limited to this example. For example, following the first image, the second image may be captured, and the first image IM1 and the second image IM2 may be acquired, and then the drugs in the first image IM1 may be identified.

[0151] Furthermore, there are no particular restrictions on the order in which the first and second sides are photographed; for example, the image photographed first may be the second side photographed image, and the image photographed later may be the first side photographed image.

[0152] [Variation 2] A possible configuration is to use the smartphone 10 to photograph the package pack, send the photographed image to a server, execute the processes of steps S2 to S7 and steps S9 to S15 in FIG. 12 on the server, and return the processing results to the smartphone 10.

[0153] [Variation 3] Although the above embodiment has been described as an example of a case where medicines are distinguished, the technology of the present disclosure can also be applied to a case where medicine inspection is performed.

[0154] 〔others〕 The technical scope of the present invention is not limited to the scope of the above-described embodiment and modifications. The configurations and the like in the embodiment and modifications can be changed without departing from the spirit of the present invention, and the embodiment and modifications can be combined as appropriate. [Explanation of symbols]

[0155] 10. Smartphones 12. Case 14 Touch panel display 16 speakers 18 microphones 20 In-camera 22 Rear camera 24 Light 26 Switch 28 CPU 30 Wireless Communication Department 32 Telephone section 34 memory 36 Internal storage 38 External memory unit 40 External input / output section 42 GPS receiver 44 Power supply section 100 Drug Identification Device 102 Image acquisition unit 104 Drug detection unit 106 Drug image extraction unit 108 Drug Identification Section 110 Determined part 112 Identified List Creation Department 114 Correct engraving image list creation unit for second surface 116 Stamp extraction part 118 Second surface stamp extraction image list creation unit 120 Template Matching Unit 122 Drug Specification Department 124 Drug Information Presentation Department 140 Engraving Master CE2, CE4 engraved image CM Checkmark D1, D2, D3, D4 drugs EG1,EG2,EG3,EG4 Engraving Extraction Image IM1 Front page image IM2 Second image LS2 Second surface stamp extraction image list LSC Correct Engraving Image List for the Second Page ST1,ST2,STn GPS satellite LP1: First loop in the drug identification method LP2: The second loop in the drug identification method S1~S15 Steps of the drug identification method

Claims

1. one or more processors; one or more storage devices; The one or more storage devices store an identification mark master including images of the identification mark on each of the front and back surfaces of a medicine on which an identification mark is engraved or printed; The one or more processors: Detecting each drug from a first-side image of a single-dose pack in which a plurality of drugs are contained in a sachet; and using information on identified drugs that identifies the drug type and the front and back sides of at least some of the drugs based on drug images extracted for each drug from the first-side photographed image, create a second-side correct identification mark image list from the identification mark master, which includes identification mark images showing the identification marks of the identified drugs that appear on the second side, which is the back side of the first side of the unit-dose pack. detecting each drug from the second surface image of the single-dose pack; creating an identification mark extraction image list including an identification mark extraction image for each of the medicines obtained by extracting the identification mark of each of the medicines from the second photographed image; performing pattern matching using the second-surface correct identification mark image list and the identification mark extracted image list to identify the identified drug on the second-surface photographed image; presenting information that differentiates between a drug whose type is unidentified and the identified drug in the second plane image; Drug identification device.

2. The one or more processors: identifying the drug type and the front and back of at least some of the plurality of drugs using a drug identification model trained by machine learning to identify the drug type and the front and back of the drug from the drug image; The drug identification device according to claim 1 .

3. The one or more processors: identifying drug types for at least some of the plurality of drugs using a drug identification model trained by machine learning to identify drug types from the drug images; using the identification mark master to identify the front and back of at least some of the medicines among the plurality of medicines; The drug identification device according to claim 1 .

4. The one or more processors: The front and back sides are identified by pattern matching the medicine image or an identification mark image extracted from the medicine image with the identification mark master. The drug identification device according to claim 3 .

5. The one or more processors: performing a round-robin pattern matching between the identification mark image in the second-side correct identification mark image list and the identification mark extracted image in the identification mark extracted image list; Identifying the identified drug on the second plane image based on the matching score. The drug identification device according to claim 1 .

6. The pattern matching is template matching. The drug identification device according to claim 1 .

7. The one or more processors: Accepting an input of an instruction to confirm the identified drug types for at least some of the plurality of drugs; Identifying the drug type of the target drug based on the input of the instruction; The drug identification device according to claim 1 .

8. The one or more processors: Creating an identified list as information on the identified drugs; The drug identification device according to claim 1 .

9. The one or more processors: a mark as the differentiating information is attached to the identified medicine in the second plane photographed image to indicate that the medicine type has been identified; The drug identification device according to claim 1 .

10. The one or more processors: a mark as the differentiating information is attached to the unidentified medicine in the second plane photographed image to indicate that the medicine type is unidentified; The drug identification device according to claim 1 .

11. a display for displaying the second-plane image including the differentiating information; The drug identification device according to claim 1 .

12. A camera is provided to photograph the single-dose pack. The drug identification device according to claim 1 .

13. 1. A method of medication identification executed by one or more processors, comprising: storing, in one or more storage devices, an identification mark master including images of the identification mark on each of the front and back surfaces of a medicine to which an identification mark is stamped or printed; acquiring a first-side image of a first side of a single-dose pack in which a plurality of medicines are contained in a sachet; Detecting each drug from the first plane captured image and extracting a drug image for each drug; Identifying the drug type and the front and back of at least some of the drugs based on the drug image extracted from the first-side captured image; Using information on the identified drug whose type has been identified from the first-side photographed image, a second-side correct identification mark image list is created from the identification mark master, which includes identification mark images showing the identification marks of the identified drug that appear on the second side, which is the reverse side of the first side of the unit-dose pack; acquiring a second-side image of the second side of the package; Detecting each medicine from the second photographed image, extracting the identification mark of each medicine, and creating an identification mark extracted image list including an identification mark extracted image for each medicine; Identifying the identified medicine on the second-surface photographed image by performing pattern matching using the second-surface correct identification mark image list and the identification mark extracted image list; presenting information that differentiates between a drug whose type is unidentified and the identified drug in the second plane photographed image; A drug identification method comprising:

14. A program that causes a computer to execute the drug identification method according to claim 13.

Images