Medicine identification device, medicine identification method, and program

Abstract

To provide a medicine identification device, a medicine identification method, and a program capable of presenting which on a captured image of a second surface, which is a back side of a first surface, a medicine whose type has been determined from a captured image of the first surface of a one package pack corresponds to.SOLUTION: A storage device stores an identification stamp master including an image of an identification stamp on each surface of the front and the back of a medicine on which the identification stamp by marking or printing is put. A processor detects each medicine from a first surface image of one package pack in which a plurality of medicines is stored in subdividing bags, creates a second surface correct answer identification stamp image list including an identification stamp image of an identified medicine that appears on the second surface from the identification stamp master using information on the identified medicine whose type and, front and back have been identified, identifies the identified medicine on a second surface image by pattern matching of an identification stamp extraction image list including an identification stamp extraction image of each medicine extracted from the second surface image and the second surface correct answer identification stamp image list, and presents information for differentiating a non-identified medicine and the identified medicine.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a drug identification device, a drug identification method, and a program, and in particular to an image processing technique for identifying the type of drug from an image of the drug. [Background technology]

[0002] As one of the techniques for improving the efficiency of tasks such as identification of medicines brought in by patients or prescription inspection, an image processing technique for identifying the type of medicine from an image of the medicine is being developed. Patent Document 1 describes a medicine identification method in which a portable terminal photographs a medicine placed on a placement section of a medicine photographing device, the portable terminal transmits image data of the photographed medicine to a server, and the server performs identification processing of the medicine in the image data transmitted from the portable terminal based on correspondence data that associates the image data of the medicine with the medicine information. According to Patent Document 1, the front and back surfaces of the medicine placed on the placement section can be photographed using a portable terminal from above and below the placement section of the medicine photographing device.

[0003] Furthermore, Patent Document 1 describes another method for acquiring image data of the front and back of a medicine, in which a portable terminal is fixed at the top or bottom, the front of the medicine is photographed, the mounting part is removed from the medicine imaging device, the medicine is turned over inside the petri dish, the mounting part is then inserted back into the medicine imaging device, and the back of the medicine is photographed with the portable terminal (paragraph

[0025] ). In this case, if the position of the medicine is changed when the front and back of the medicine are turned over, the image of the front and back of the medicine will no longer correspond to each other, so it is necessary to turn the medicine over in the same position, and Patent Document 1 uses a textured sheet to prevent the medicine from shifting position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-182525 A Summary of the Invention [Problem to be solved by the invention]

[0005] The markings or printing on a medicine are important information when identifying the medicine. Medicines with markings or printing have a front and back, and one or both of the front and back may not have enough information to identify the medicine. For example, if only numbers are marked or printed on one side of the medicine, it may be difficult to identify the medicine from that information alone. Also, if one side of the medicine is blank (no markings or printing information), it is difficult to identify the type of medicine from a photographed image of that side of the medicine.

[0006] The size of the device for identifying single-dose medications is preferably as small as possible, taking into consideration portability and space-saving. It is easy to grasp the correspondence between the front and back of a medication by simultaneously photographing both sides of the medication, but in this case a mechanism for simultaneously photographing both sides of the medication is required. This inevitably increases the size of the device, making it difficult to meet this requirement. On the other hand, if the device is configured with only one smartphone, or a combination of a smartphone and a small portable photography stand, the requirements for portability and space-saving can be met. However, in this case, it becomes necessary to photograph the front and back of the single-dose pack separately.

[0007] When photographing a single-dose pack containing multiple drugs in a single-dose package, each side will be randomly facing up or down on the side of the single-dose pack facing the camera. Therefore, in order to complete the identification of all drugs in a single package with a single-side photograph, it is necessary to manually turn each drug's side with the amount of information that can be identified through the package (drug identifiable side) up before photographing it.

[0008] However, when there are many medicines in a sachet, it is troublesome and difficult to keep all the medicines with their identifiable side facing up. Therefore, as an alternative method, it is possible to take single-sided photographs of the front and back of the single-dose pack, and use the two images taken of each side to identify the medicines whose identifiable side is shown in each image.

[0009] However, with this method, when the unitary package pack is turned over, the medicines may move inside the sachet, and the correspondence between the medicines in the two images, the first-side image obtained by photographing the first side of the unitary package pack and the second-side image obtained by photographing the second side after turning it over, becomes unclear. Therefore, it is often difficult to know which medicine in the turned-over second-side image corresponds to an identified (classified) medicine whose type has already been identified in the first-side image.

[0010] In particular, when there are a large number of medications, it is difficult for the user to remember the medications that have already been identified using the first-view captured image, and a system that requires the user to distinguish between the remaining medications to be identified in the second-view captured image and the medications that have already been identified places a heavy burden on the user.

[0011] The present disclosure has been made in consideration of the above circumstances, and aims to provide a drug identification device, drug identification method, and program that can present which drug type, whose type has already been identified from the first-side captured image, corresponds to on the second-side captured image. [Means for solving the problem]

[0012] A drug identification device according to a first aspect of the present disclosure includes one or more processors and one or more storage devices, and the one or more storage devices store an identification mark master including images of the identification marks on the front and back sides of a drug having an identification mark affixed thereto by stamping or printing. The one or more processors detect each drug from a first-side captured image in which a first side of a unit-dose pack in which a plurality of drugs are stored in a sachet is captured, and extracts a drug type and a front and back side of at least some of the plurality of drugs from the identification mark master using information on the identified drug based on drug images extracted for each drug from the first-side captured image. A second-side correct identification mark image list is created which includes identification mark images showing the identification marks of identified drugs which appear on the second side, which is the reverse side of the first side of the packaged pack, each drug is detected from the second-side image in which the second side of the packaged pack is photographed, and an identification mark extracted image list is created which includes identification mark extracted images for each drug obtained by extracting the identification mark of each drug from the second-side image. By performing pattern matching using the second-side correct identification mark image list and the identification mark extracted image list, identified drugs are identified on the second-side image, and information is presented which differentiates identified drugs from drugs whose drug type is unidentified in the second-side image.

[0013] According to the first aspect, one or more processors automatically identify which medicines in the second image correspond to medicines whose medicine types have been identified in the first image, and among the medicines in the second image, it is possible to clearly distinguish between identified medicines whose medicine types have already been identified in the first image and unidentified medicines whose medicine types have not yet been identified, and present them to the user. This allows the user to easily grasp which medicines to identify in the second image, and to streamline the work of identifying packaged medicines.

[0014] The identification mark may be, for example, an identification code that combines a company code and a product code, or may be either the company code or the product code.

[0015] Identifying the drug type of a drug may be identifying an individual brand of the drug. The identified drug type may be, for example, an individual identification code of the drug, such as the YJ code. Having identified the drug type of a drug is synonymous with having identified the drug. The definition of the front and back of a drug may be, for example, the drug identifiable side on which the drug can be identified from an identification mark by engraving or printing as the front, and the opposite side as the back. In addition, on a package insert, which is usually a public document, the designs of the engravings or prints on both sides are written side by side, left and right or top and bottom, but the engravings or prints written on the left or top may be defined as the front, and the engravings or prints written on the right or bottom may be defined as the back. When both sides of a drug are drug identifiable sides, either side may be defined as the front.

[0016] The drug identification device of the second aspect may be configured in the drug identification device of the first aspect, wherein one or more processors are configured to identify the drug type and front and back of at least some of the multiple drugs using a drug identification model trained by machine learning to identify the drug type and front and back from a drug image.

[0017] The drug identification device of the third aspect may be configured in such a way that, in the drug identification device of the first aspect, one or more processors use a drug identification model trained by machine learning to identify drug type from drug images to identify drug type for at least some of the multiple drugs, and use an identification mark master to identify the front and back of at least some of the multiple drugs.

[0018] The drug identification device of the fourth aspect may be configured in such a way that, in the drug identification device of the third aspect, one or more processors identify the front and back by pattern matching the drug image or an identification mark image extracted from the drug image with an identification mark master.

[0019] The drug identification device of the fifth aspect may be configured such that, in the drug identification device of any one of the first to fourth aspects, one or more processors perform brute force pattern matching between identification mark images in the second side correct identification mark image list and identification mark extraction images in the identification mark extraction image list, and identify identified drugs on the second side captured image based on the matching score.

[0020] A drug identification device according to a sixth aspect may be the drug identification device according to any one of the first to fifth aspects, in which the pattern matching is template matching.

[0021] A drug identification device according to a seventh aspect may be configured such that, in a drug identification device according to any one of the first to sixth aspects, one or more processors receive input of an instruction to confirm the identified drug type for at least some of the multiple drugs, and identify the drug type of the target drug based on the input instruction.

[0022] The drug identification device according to an eighth aspect may be configured such that, in the drug identification device according to any one of the first to seventh aspects, the one or more processors create an identified list as information on the identified drug.

[0023] A drug identification device according to a 9th aspect is a drug identification device according to any one of the 1st to 8th aspects, wherein one or more processors are configured to mark an identified drug in the second-plane image as differentiating information to indicate that the drug type has been identified.

[0024] The drug identification device of the 10th aspect is a drug identification device of any one of the 1st to 9th aspects, and the one or more processors may be configured to mark unidentified drugs in the second-plane image as differentiating information to indicate that the drug type is unidentified.

[0025] The drug identification device according to an eleventh aspect may be configured to include a display that displays a second-side captured image including differentiating information in the drug identification device according to any one of the first to tenth aspects.

[0026] A medicine identification device according to a twelfth aspect may be configured such that the medicine identification device according to any one of the first to eleventh aspects further includes a camera that photographs the unit dose pack.

[0027] A medicine identification method according to a thirteenth aspect of the present disclosure is a medicine identification method executed by one or more processors, which includes storing an identification mark master including images of the identification marks on the front and back sides of a medicine having an identification mark by engraving or printing in one or more storage devices, acquiring a first-side image of a first side of a unit-dose pack in which a plurality of medicines are stored in a sachet, detecting each medicine from the first-side image and extracting a medicine image for each medicine, identifying the medicine type and the front and back sides of at least some of the plurality of medicines based on the medicine images extracted from the first-side image, and identifying the medicine using information of the identified medicine whose medicine type has been identified from the first-side image. The method includes creating a second-side correct identification mark image list from the mark master, which includes identification mark images showing the identification marks of identified drugs that appear on the second side, which is the back side of the first side of the unit-dose pack; obtaining a second-side photographed image of the second side of the unit-dose pack; detecting each drug from the second-side photographed image, extracting the identification mark of each drug, and creating an identification mark extracted image list including an identification mark extracted image for each drug; identifying the identified drugs on the second-side photographed image by performing pattern matching using the second-side correct identification mark image list and the identification mark extracted image list; and presenting information that differentiates identified drugs from drugs whose drug type is unidentified in the second-side photographed image.

[0028] The medicine identifying method according to the thirteenth aspect may have a configuration including the same specific aspects as the medicine identifying device according to any one of the second to twelfth aspects.

[0029] A program according to a 14th aspect of the present disclosure is a program that causes a computer to execute the drug identification method according to the 13th aspect. A non-transitory computer-readable recording medium (computer-readable medium) that is a tangible object having the program according to the 14th aspect recorded thereon is also included in the present disclosure.

[0030] The program according to the fourteenth aspect may be configured to include the same specific aspect as the drug identification device according to any one of the second to twelfth aspects. Effect of the Invention

[0031] According to the present disclosure, one or more processors execute a process for identifying which drug in the second image corresponds to an identified drug whose type has already been identified from the first image of a single-dose pack, and it is possible to present the identified drug and the unidentified drug in the second image to the user in a differentiated manner. This allows the user to easily know which drug among the multiple drugs in the second image should be identified. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a front perspective view of a smartphone. [Diagram 2] FIG. 2 is a rear perspective view of the smartphone. [Diagram 3] FIG. 3 is a block diagram showing the electrical configuration of the smartphone. [Figure 4] FIG. 4 is a block diagram showing a functional configuration of the drug identification device according to the embodiment. [Diagram 5] FIG. 5 is an example of a first-side image in which the first side of a unit-dose pack is photographed. [Figure 6] FIG. 6 is an explanatory diagram showing an example of an engraving image included in the engraving master. [Figure 7] FIG. 7 is an explanatory diagram showing an example of identifying the marking image on the second surface side of an identified medicine from the marking master. [Figure 8]FIG. 8 is an explanatory diagram showing an example of a list of correct engraving images for the second surface. [Figure 9] FIG. 9 is an example of a second-side photographed image in which the second side of a unit-dose pack is photographed. [Figure 10] FIG. 10 is an explanatory diagram showing an example of template matching processing using the second surface correct engraving image list and the second surface engraving extracted image list. [Figure 11] FIG. 11 is an explanatory diagram showing an example of presentation of information differentiating identified medicines from unidentified medicines on the second plane captured image. [Figure 12] FIG. 12 is a flowchart showing an example of a drug identification method carried out using the drug identification device according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0034] [Overview of the drug identification device according to the embodiment] A drug identification device according to an embodiment of the present disclosure is an information processing device that performs processing to identify the drug type of each drug by photographing a first side image and a second side image obtained by photographing a single-dose pack in which multiple drugs are stored in a sachet bag using single-dose dispensing while the pack is still in the bag, the first side being one side, and the second side being the reverse side.

[0035] In the drug identification device according to this embodiment, the drug type of at least some of the drugs shown in the first-side photographed image is first identified, and then the remaining drugs are identified using the second-side photographed image. The drug identification device according to this embodiment includes one or more processors, which automatically identify which drug on the second-side photographed image corresponds to an identified drug whose drug type has been identified from the first-side photographed image, and executes a process of presenting the drugs whose drug type has already been identified in the first-side photographed image and unidentified drugs whose drug type has not yet been identified (i.e., the remaining drugs to be identified in the second-side photographed image) to the user in a differentiated manner so that they can be clearly distinguished from each other on the display screen of the second-side photographed image. This allows the user to easily grasp which drug to identify in the second-side photographed image, and the work of identifying drugs in a single-dose pack can be made more efficient.

[0036] The term "identification" with respect to drugs includes the concepts of discrimination and auditing. The type of drug to be identified is, for example, a drug type that can be specified by identification information such as a YJ code (individual drug code) or drug name. Drug identification in this embodiment can be defined as the act of determining which YJ code the drug to be identified is. This is one example of the definition of drug identification, and for example, the identification code may be defined using a type other than the YJ code. An identified drug whose drug type has been identified may be referred to as an "identified drug."

[0037] The drug identification device is mounted on a mobile terminal device as an example. The mobile terminal device includes at least one of a smartphone, a mobile phone, a PHS (Personal Handy-phone System), a PDA (Personal Digital Assistant), a tablet computer terminal, a notebook personal computer terminal, a wearable terminal, and a portable game machine. In the following, a drug identification device realized by the hardware and software of a smartphone is taken as an example and described in detail with reference to the drawings.

[0038] [Appearance of the smartphone] Fig. 1 is a front perspective view of a smartphone 10 that functions as a drug identification device according to an embodiment of the present disclosure. As shown in Fig. 1, the smartphone 10 has a flat housing 12. The smartphone 10 has a touch panel display 14, a speaker 16, a microphone 18, and an in-camera 20 on the front side of the housing 12.

[0039] The touch panel display 14 includes a display unit that displays images and the like, and a touch panel unit that is disposed in front of the display unit and receives touch inputs. The display unit is, for example, a color LCD (Liquid Crystal Display) panel or a color organic EL (organic electro-luminescence) panel.

[0040] The touch panel unit is, for example, a capacitive touch panel provided in a planar shape on a light-transmitting substrate body, and has a light-transmitting position detection electrode and an insulating layer provided on the position detection electrode. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to a touch operation by a user. The touch operation includes a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

[0041] The speaker 16 is an audio output unit that outputs audio during a call and when playing back a video. The microphone 18 is an audio input unit to which audio is input during a call and when shooting a video. The front camera 20 is an imaging device that shoots videos and still images.

[0042] Fig. 2 is a rear perspective view of the smartphone 10. As shown in Fig. 2, the smartphone 10 includes an outer camera 22 and a light 24 on the rear surface of the housing 12. The outer camera 22 is an imaging device that captures moving images and still images. The light 24 is a light source that emits illumination light when the outer camera 22 captures images, and is formed of, for example, an LED (Light Emitting Diode).

[0043] 1 and 2, the smartphone 10 is provided with switches 26 on the front and side of the housing 12. The switches 26 are input members that accept instructions from a user. The switches 26 are push-button switches that are turned on when pressed with a finger or the like and turned off by the restoring force of a spring or the like when the finger is released.

[0044] The configuration of the housing 12 is not limited to this, and a configuration having a folding structure or a sliding mechanism may be adopted.

[0045] [Smartphone electrical configuration] The smartphone 10 has, as its main function, a wireless communication function for performing mobile wireless communication between a base station device and a mobile communication network.

[0046] Fig. 3 is a block diagram showing the electrical configuration of the smartphone 10. As shown in Fig. 3, the smartphone 10 has a CPU (Central Processing Unit) 28, a wireless communication unit 30, a call unit 32, a memory 34, an external input / output unit 40, a GPS (Global Positioning System) receiving unit 42, and a power supply unit 44 in addition to the above-mentioned touch panel display 14, speaker 16, microphone 18, in-camera 20, out-camera 22, light 24, and switch 26.

[0047] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates according to the control program and control data stored in the memory 34, and controls each unit of the smartphone 10. The CPU 28 has a mobile communication control function that controls each unit of the communication system to perform voice communication and data communication through the wireless communication unit 30, and an application processing function.

[0048] The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function allows information such as still images, moving images, and text to be visually conveyed to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to a touch operation by the user from the touch panel section of the touch panel display 14. The CPU 28 also acquires an input signal from the switch 26.

[0049] The in-camera 20 and the out-camera 22 each have a photographing lens, an aperture, an image sensor, an AFE (Analog Front End), an A / D (Analog to Digital) converter, a lens driving unit, etc. The in-camera 20 and the out-camera 22 capture moving images and still images according to instructions from the CPU 28.

[0050] The CPU 28 may convert the moving images and still images captured by the in-camera 20 and the out-camera 22 into compressed image data using MPEG (Moving Picture Experts Group) and JPEG (Joint Photographic Experts Group) formats, for example.

[0051] The CPU 28 stores the video and still images captured by the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0052] Furthermore, the CPU 28 displays the moving images and still images captured by the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the moving images and still images captured by the in-camera 20 and the out-camera 22 in application software.

[0053] The CPU 28 may illuminate the subject with auxiliary light by turning on the light 24 when capturing an image with the outer camera 22. The light 24 may be turned on and off by a touch operation on the touch panel display 14 or an operation of the switch 26 by the user.

[0054] The wireless communication unit 30 performs wireless communication with a base station device compatible with a mobile communication network of a standard such as 4G (4th Generation) or 5G (5th Generation) in accordance with an instruction from the CPU 28. The smartphone 10 uses this wireless communication to transmit and receive various file data such as audio data and image data, e-mail data, and the like, and to receive Web (abbreviation for World Wide Web) data, streaming data, and the like.

[0055] The speaker 16 and the microphone 18 are connected to the communication unit 32. The communication unit 32 decodes the voice data received by the wireless communication unit 30 and outputs the decoded voice data from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into voice data that can be processed by the CPU 28 and outputs the voice data to the CPU 28.

[0056] The memory 34 stores instructions to be executed by the CPU 28. The memory 34 is composed of an internal storage unit 36 ​​built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are realized using known storage media.

[0057] The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of sent and received e-mails, Web data downloaded by Web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

[0058] The external input / output unit 40 serves as an interface with an external device connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to another external device by communication or the like via the external input / output unit 40. The external input / output unit 40 transmits data received from an external device to each component inside the smartphone 10, and transmits data inside the smartphone 10 to the external device.

[0059] Examples of the communication means include a Universal Serial Bus (USB), IEEE (Institute of Electrical and Electronics Engineers) 1394, the Internet, a wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. Examples of the external device include a headset, an external charger, a data port, an audio device, a video device, a smartphone, a PDA, a personal computer, and an earphone.

[0060] The GPS receiver 42 detects the position of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, . . . , STn.

[0061] The power supply unit 44 is a power supply source that supplies power to each unit of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium ion secondary battery. The power supply unit 44 may include an AC / DC conversion unit that generates a DC voltage from an external AC power source.

[0062] The smartphone 10 configured in this manner is set to a shooting mode in response to an instruction input from a user using the touch panel display 14 or the like, and can capture videos and still images using the in-camera 20 and the out-camera 22.

[0063] When the smartphone 10 is set to the shooting mode, it goes into a shooting standby state, a moving image is captured by the inner camera 20 or the outer camera 22, and the captured moving image is displayed on the touch panel display 14 as a live view image.

[0064] A user can visually confirm the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject that the user wants to photograph, and set the photographing conditions.

[0065] When the smartphone 10 is instructed to take a picture by a user inputting an instruction using the touch panel display 14 or the like in a shooting standby state, the smartphone 10 performs AF (Autofocus) and AE (Auto Exposure) control, and takes and stores a moving image and a still image.

[0066] The memory 34 is an example of a "storage device" in the present disclosure. The touch panel display 14 is an example of a user interface and is an example of a "display" in the present disclosure. Each of the in-camera 20 and the out-camera 22 is an example of a "camera" in the present disclosure.

[0067] [Functional configuration of the drug identification device] 4 is a block diagram showing the functional configuration of the drug identification device 100 realized by the smartphone 10. As shown in FIG. 4, the drug identification device 100 includes an image acquisition unit 102, a drug detection unit 104, a drug image extraction unit 106, a drug identification unit 108, a determination unit 110, an identified list creation unit 112, a second-side correct imprint image list creation unit 114, an imprint extraction unit 116, a second-side imprint extraction image list creation unit 118, a template matching unit 120, an identified drug identification unit 122, an identified drug information presentation unit 124, a touch panel display 14, and an imprint master 140. The functions of each unit of the drug identification device 100 can be realized by the hardware and software of the smartphone 10, and can be embodied by the CPU 28 executing the instructions of a program stored in the memory 34.

[0068] The image acquisition unit 102 acquires a still image of a single-dose pack in which multiple drugs are contained in a sachet by single-dose dispensing. The captured image may be an image captured by the outer camera 22, for example. The captured image may also be an image acquired from another device via the wireless communication unit 30, the external storage unit 38, or the external input / output unit 40. The sachet may be entirely or partially transparent or semi-transparent. Each of the multiple drugs contained in the sachet may be a drug to be identified.

[0069] The captured image acquired by the image acquisition unit 102 may be an image in which one or more markers are captured together with the unit-dose pack. The marker may be, for example, an ArUco marker, a circular marker, or a square marker. When performing image processing such as image area cutout processing and / or transformation processing on the captured image, it is preferable that the captured image contains multiple markers. The multiple markers are, for example, arranged at the four corners of a rectangular area of ​​a drug placement range on the surface on which the unit-dose pack is placed during imaging. It is preferable that the drug placement range is configured to have, for example, a reference gray color or a black background.

[0070] The captured image may be an image captured at a standard shooting distance and shooting viewpoint. The shooting distance can be expressed by the distance between the drug to be identified and the shooting lens and the focal length of the shooting lens. The shooting viewpoint can be expressed by the angle between the drug placement surface (marker printed surface) and the optical axis of the shooting lens.

[0071] When photographing the unit-dose pack, a photography assistant may be used to set the smartphone 10 used for photography at or near the camera position that is the standard photography distance and photography viewpoint. This photography assistant is preferably configured by combining a placement stand for placing the unit-dose pack to be photographed and a lighting device for illuminating the unit-dose pack.

[0072] The image acquisition unit 102 may include an image correction unit (not shown). When a marker is included in the captured image, the image correction unit standardizes the shooting distance and shooting viewpoint of the captured image based on the marker to generate a standardized image. The standardized image may be an image obtained by cutting out an area inside a rectangle having markers at the four corners as vertices after a standardization process is performed on the captured image. For example, the image correction unit specifies the coordinates of the four vertices of a rectangle whose coordinates are specified by the markers after standardization of the shooting distance and the shooting viewpoint. The image correction unit obtains a perspective transformation matrix that transforms these four vertices into the positions of the specified coordinates. Such a perspective transformation matrix is ​​uniquely determined if there are four points. For example, a transformation matrix can be obtained by the getPerspectiveTransform function of OpenCV (Open Source Computer Vision Library) if there is a correspondence between the four points.

[0073] The image correction unit performs perspective transformation on the entire original captured image using the obtained perspective transformation matrix, and obtains a transformed image. Such perspective transformation can be performed using the warpPerspective function of OpenCV. This transformed image may be a standardized image in which the shooting distance and shooting viewpoint are standardized.

[0074] Furthermore, when the captured image includes an area of ​​a reference gray color, the image correction section may perform color correction on the captured image based on the reference gray color.

[0075] In the drug identification device 100 of this embodiment, first, a first surface image IM1 is obtained, which is a still image of the first surface, which is one side of the single-dose pack. Then, a process is performed to identify the drug type of the drug that is included in the first surface image IM1 and has a drug identifiable surface. The drug identifiable surface is a surface of a drug that is engraved or printed, and the drug can be identified from the information of the engraved or printed markings on the surface (a surface where the drug type can be identified). In contrast, a surface where it is difficult to identify the drug from the information of the engraved or printed markings on the surface, or a surface without any engraved markings (a plain surface) is called a drug difficult to identify surface. For a drug that has a drug identifiable surface in the first surface image IM1, the drug type can be identified from the first surface image IM1.

[0076] The drug identification device 100 then acquires a second-side image IM2, which is a still image of the second side of the same unit-dose pack opposite the first side, and uses the second-side image IM2 to identify the drug types of the remaining (unidentified) drugs.

[0077] Each of the first plane image IM1 and the second plane image IM2 acquired via the image acquisition unit 102 may be a standardized image.

[0078] The medicine detection unit 104 detects the area of ​​each medicine from the captured image acquired via the image acquisition unit 102. The medicine detection unit 104 may be configured using a learned AI (Artificial Intelligence) model trained by machine learning to perform a so-called object detection task. The detection result of the medicine detection unit 104 is displayed on the touch panel display 14. For example, the medicine area detected from the first-side captured image IM1 may be displayed by a bounding box on a screen displaying the first-side captured image IM1.

[0079] FIG. 5 is an example of a first-side photographed image IM1. Four medicines D1 to D4 are shown in the first-side photographed image IM1 shown in FIG. 5. The medicines D2 and D4 surrounded by the rectangle shown by the dashed line in FIG. 5 have a large amount of information on the markings, and the medicine type can be identified in this first-side photographed image IM1. In contrast, the medicines D1 and D3 have a small amount of information on the markings or printings, and it is difficult to identify the medicines in this first-side photographed image IM1. In other words, the first-side photographed image IM1 shown in FIG. 5 is an image in which the medicines D2 and D4, which are photographed on the medicine-identifiable side, and the medicines D1 and D3, which are photographed on the medicine-difficult-to-identify side, are mixed together.

[0080] The user can specify the drugs D2 and D4 whose drug identifiable surfaces are photographed from the first surface photographed image IM1 displayed on the touch panel display 14, and perform an operation to identify the type of each drug. Note that, without waiting for a drug designation operation by the user, the drug image extraction unit 106 and the drug identification unit 108 may automatically perform processing for each drug detected by the drug detection unit 104.

[0081] The drug image extraction section 106 extracts an image area of ​​each drug from the captured image based on the detection result by the drug detection section 104, and generates a drug image in which the drug image area is cut out for each drug.

[0082] The drug image extraction unit 106 may be incorporated in the drug detection unit 104. For example, the drug detection unit 104 includes a drug region extraction model. The drug region extraction model may be a trained model that, when a photographed image of a drug is given as an input, outputs a drug image in which the region of the drug in the image is extracted. The drug region extraction model may be a segmentation model that is machine-learned using a learning data set for learning, which is a learning data set of a plurality of different photographed images and includes a drug image and a drug region included in the image. A convolutional neural network (CNN) can be applied as the drug region extraction model.

[0083] The drug identification unit 108 is a processing unit that identifies the drug type of the drug from the drug image extracted by the drug image extraction unit 106. The drug identification unit 108 may be, for example, an AI processing unit that uses a learned AI model (drug identification model) trained by machine learning to perform an object recognition task of identifying the drug type from an input drug image. The type of drug identified by the drug identification unit 108 is, for example, a drug type that can be specified by identification information such as a YJ code (individual drug code) or a drug name. Drug identification in this embodiment can be defined as an act of determining which YJ code the drug to be identified is. This is an example of the definition of drug identification, and for example, the identification code may be defined using a type other than the YJ code.

[0084] The drug identification model functions as a multi-class classifier that receives an input of a drug image of a drug to be identified, identifies the type of drug, and classifies it into the N types of drug types (classes) that have been trained. When a drug image is input, the drug identification model calculates a score value indicating the likelihood (certainty) that the drug to be identified is drug i for all N types of drug i that have been trained. In the notation "drug i," "i" is an index that distinguishes the N types of drugs that have been trained. The drug identification model calculates a score value for each drug i that serves as an index for determining whether the drug to be identified is drug i.

[0085] The drug identification model is configured using, for example, a neural network. CNN can be used as a machine learning model suitable for image recognition. The image input to the drug identification model may be a region image (drug image) of the drug to be identified, cut out from the captured image. In addition to the drug image, identification mark information of a mark or print extracted from the drug image may be input to the drug identification model. The identification mark information may be an image or text information.

[0086] Based on the identification result by the medicine identifying unit 108, one or more medicine type candidates with the highest score values ​​are displayed on the touch panel display 14.

[0087] For example, when a user selects a drug to be identified (e.g., drug D2) from the first-side captured image IM1, a drug image of the drug to be identified related to the selection is extracted from the first-side captured image IM1 and input to the drug identification unit 108, and candidate drug types as the recognition result by the drug identification unit 108 are displayed on the touch panel display 14.

[0088] The determination unit 110 receives an instruction to determine the drug type of the target drug from the touch panel display 14 or other user interface, and performs a process to determine the drug type of the target drug in accordance with the received instruction. This identifies the drug type of the target drug.

[0089] By identifying the drug type of the target drug, the front and back of the target drug can also be identified. For example, the drug type of the drug is identified, and the drug identifiable side is shown in the first side image IM1, so the drug identifiable side may be identified as the "front". Also, for example, the front and back of the drug can be identified by referring to the marking master 140 from the identified drug type.

[0090] The engraving master 140 is a master database that includes image information of the engraving or printing affixed to the medicine, and images of identification marks made by engraving or printing on the front and back of the medicine are stored in association with the medicine type (see FIG. 6). The engraving master 140 is stored in the memory 34, and data is updated as necessary. The front and back of the medicine may be determined by performing template matching between the engraving master 140 and a medicine image extracted from the first surface photographed image IM1 or an engraving extraction image extracted from that medicine image.

[0091] The drug identification model may be a model trained to receive an image of one side of a drug to be identified, taken from one side (from one direction), and to identify the drug type of the drug to be identified and even the front and back of the drug to be identified (whether it is the front or back). For example, the drug identification model may be a classifier trained by machine learning to receive an image of a drug and output a class that combines the YJ code and information on the front and back. By using such a model, the front and back can be identified at the time of identifying the drug type. In this case, the model may not define classes for combinations of all YJ codes and front and back combinations, but may not define classes for combinations that include difficult drug identification sides.

[0092] To improve the accuracy of the identification by the medication identification model, template matching with the imprint master 140 may be further used to determine the front and back of the medication.

[0093] After detecting the medicine in the first surface photographed image IM1, the user checks the identification result of the medicine identification unit 108 for the first surface photographed image IM1 to identify the medicine type and the front and back. In the example of FIG. 5, the user performs an operation to identify the medicine type for each of the medicines D2 and D4. The user may identify the type of medicine by using the search results, such as by performing a text search in an imprint database (not shown) based on the information of the medicine's imprint or print visible from the first surface photographed image IM1, without using the identification process of the medicine identification unit 108. The imprint database is a collection of data in which the character string of the imprint or print affixed to the medicine is associated with the medicine type to which the character string is imprinted or printed. The imprint database may be configured integrally with the imprint master 140.

[0094] The identified list creation unit 112 creates a list of identified drugs whose drug types have been identified from the first captured image IM1 after processing by the determination unit 110 (hereinafter referred to as the "identified list"). In the example of Fig. 5, an identified list is created that includes information indicating that drugs D2 and D4 have been identified. The identified list includes the YJ code and front / back information of the drugs identified in the first captured image IM1.

[0095] The second-side correct stamp image list creating unit 114 creates a second-side correct stamp image list, which is a list of correct stamp images on the second side of the identified medicine, from the identified list and the stamp master 140. The "correct stamp image" here means the stamp image expected to appear on the second side, and is the stamp image as the expected value of the second side. By inverting the front and back of the identified medicine identified from the first-side photographed image IM1, the second-side correct stamp image list can be created from the stamp master 140.

[0096] In addition, the term "imprint" in the terms imprint master 140, imprint database, and imprint image represents the concept of an identification mark formed by at least one of imprinting and printing, and includes the meaning of "imprinting and / or printing." In this specification, the term "imprint" may be understood to include the concept of "printing" to the extent that there is no contradiction from the context.

[0097] Fig. 6 shows an example of the engraving master 140. Fig. 6 shows engraving images registered in the engraving master 140 for drugs D1 to D4. Although only four drugs D1 to D4 are shown here, the engraving master 140 may include data for all types of identifiable drugs.

[0098] In Fig. 6, the image marked with an asterisk indicates that it is an image of the drug identifiable side of the drug. Note that the drug marked with "DD 444" shown at the bottom of Fig. 6 has the same marking on both the front and back, and both the front and back are drug identifiable sides. The marking master 140 shown in Fig. 6 is an example of an "identification mark master" in this disclosure, and the marking image shown in Fig. 6 is an example of an "identification mark image" in this disclosure.

[0099] The second surface correct stamp image list creating unit 114 refers to the stamp master 140 based on the identified list, extracts the stamp images on the second surface side for the identified medicines from the stamp master 140, and creates a second surface correct stamp image list.

[0100] FIG. 7 is an explanatory diagram showing an example of identifying the marking image on the second side of the identified medicine from the marking master 140. In the example of FIG. 5, when the medicine D2 and the medicine D4 are identified medicines, the marking image on the back side of these medicines is surrounded by a rectangle shown by a thick line in FIG. 7. That is, the marking expected to appear in the second side photographed image IM2 of the identified medicine D2, whose front side is marked "BB 222", is the marking "50". Also, the marking expected to appear in the second side photographed image IM2 of the identified medicine D4, whose front side is marked "DD 444", is the marking "DD 444".

[0101] In this way, the correct marking images expected in the second-side photographed image IM2 of the identified medicine are extracted from the marking master 140, and a list of correct marking images for the second side is created by collecting these (see FIG. 8).

[0102] FIG. 8 is an example of the correct stamp image list LSC for the second side. The correct stamp image list LSC for the second side shown in FIG. 8 includes the correct stamp images CE2 and CE4 (as expected values) that should appear on the second side of each of the identified drugs D2 and D4 described in FIG. 7. These correct stamp images CE2 and CE4 are extracted from the stamp master 140 using the identified list. Each of the stamp images CE2 and CE4 is an example of an "identification mark image" in this disclosure, and the correct stamp image list LSC for the second side is an example of a "second side correct identification mark image list" in this disclosure. Such a correct stamp image list LSC for the second side is used as a matching list to be applied to the template matching unit 120 when identifying the type of the remaining (unidentified) drug using the second side photographed image IM2.

[0103] Next, the processing of the second plane captured image IM2 will be described.

[0104] After completing a specific operation on the medicine in the first side image IM1, the user turns the same single-dose pack over and photographs the second side.

[0105] Fig. 9 is an example of a second-side photographed image IM2. The second-side photographed image IM2 shown in Fig. 9 is an example of a photographed image obtained by turning over the same unit-package pack as the first-side photographed image IM1 shown in Fig. 5. In the second-side photographed image IM2 shown in Fig. 9, the drug identifiable sides of the drug D1 and the drug D3 are photographed.

[0106] The image acquisition unit 102 acquires the second plane image IM2. As with the first plane image IM1, the medicine detection unit 104 detects a medicine area from the second plane image IM2. The medicine image extraction unit 106 creates a medicine image that is a cut-out image of each medicine detected by the medicine detection process. The mark extraction unit 116 extracts the mark from the medicine image of each medicine extracted from the second plane image IM2 to create an mark extraction image.

[0107] The second surface imprint extracted image list creating unit 118 creates a second surface imprint extracted image list by collecting the imprint extracted images of each medicine extracted from the second surface photographed image IM2.

[0108] An example of the second-side imprint extraction image list LS2 is shown on the left side of Fig. 10. This second-side imprint extraction image list LS2 is created from the second-side photographed image IM2 shown in Fig. 9. The second-side imprint extraction image list LS2 includes imprint extraction images EG1, EG2, EG3, and EG4 extracted from the respective drug images of the drugs D1, D2, D3, and D4 included in the second-side photographed image IM2. Each of the imprint extraction images EG1 to EG4 is an example of an "identification mark extraction image" in the present disclosure, and the second-side imprint extraction image list LS2 is an example of an "identification mark extraction image list" in the present disclosure.

[0109] The template matching unit 120 performs template matching on the engraving extraction images EG1 to EG4 included in the second surface engraving extraction image list LS2 and the correct engraving images CE2 and CE4 included in the correct engraving image list LSC for the second surface in a round-robin manner, and evaluates the degree of match. The degree of match is evaluated based on a matching score calculated by template matching. Template matching is an example of "pattern matching" in this disclosure. FIG. 10 shows the process of template matching between the engraving images in the correct engraving image list LSC for the second surface and the engraving extraction image list LS2 for the second surface.

[0110] Based on the processing result of the template matching unit 120, the identified medicine identifying unit 122 identifies, from the second surface imprint extracted image list LS2, those having a high degree of coincidence with the second surface correct imprint image list LSC as identified medicines.

[0111] In the example of Figure 10, the stamp extraction image EG2 and the stamp extraction image EG4 are evaluated as having a high degree of match with the correct stamp images CE2 and CE4 in the correct stamp image list LSC for the second surface, and in the second surface captured image IM2, the drugs D2 and D4 corresponding to these stamp extraction images EG2 and EG4 are identified as identified drugs.

[0112] The identified drug information presentation unit 124 performs a process of presenting to the user information differentiating between identified drugs whose drug types have been identified and unidentified drugs whose drug types have not been identified, among the multiple drugs D1 to D4 in the second-plane captured image IM2, based on information about identified drugs identified by the identified drug identification unit 122.

[0113] FIG. 11 is a diagram showing an example of information presentation that differentiates between identified and unidentified drugs on the second-plane captured image IM2. In FIG. 11, a check mark CM is attached to each of identified drugs D2 and D4. The user can understand whether the drug type has been identified or not based on the presence or absence of the check mark CM. That is, the user can easily recognize that the drugs D2 and D4 with the check mark CM have been identified, and can see that the remaining drugs D1 and D3 without the check mark CM should be identified.

[0114] According to this configuration, drugs whose types have already been identified in the first plane image IM1 can be identified on the image of the second plane image IM2, allowing the user to easily identify the remaining drugs that need to be identified in the second plane image IM2.

[0115] The check mark CM is an example of "differentiating information" in the present disclosure. FIG. 11 shows an example in which information indicating "identified" is added to an identified drug, but instead of or in combination with this, information indicating "unidentified" may be added to an unidentified drug. The mode of differentiation is not limited to the presence or absence of a check mark CM, but may also be the presence or absence of a bounding box or other frame surrounding the drug, a difference in the display color of the frame, a device in the shape of a frame, a difference between blinking and constant display of a mark, display of text information informing that the drug is identified or unidentified, or graying out an identified drug, or an appropriate combination of these.

[0116] <Examples of drug identification methods> FIG. 12 is a flowchart showing an example of a drug identification method carried out by using the drug identification device 100 according to this embodiment.

[0117] In step S1, the user photographs a first surface, which is one surface of the package pack, using a camera (for example, the outer camera 22) of the smartphone 10. The CPU 28 acquires a first surface photographed image IM1 in which the first surface of the package pack is photographed.

[0118] In step S2, the CPU 28 detects each medicine from the first captured image IM1. The medicine detection process may be performed by, for example, a medicine detection AI (Artificial Intelligence) using a trained model that extracts a medicine area from the input captured image.

[0119] After that, the CPU 28 proceeds to a first loop LP1. The first loop LP1 includes steps S3 to S5. The CPU 28 executes steps S3 to S5 for each drug detected in step S2.

[0120] In step S3, the CPU 28 cuts out the area of ​​each detected medicine from the first plane captured image IM1, and extracts a medicine image which is an area image of each medicine.

[0121] In step S4, the CPU 28 judges whether the extracted medicine image is an image of an identifiable surface that can identify the medicine type. For example, the user may check the marking information on the display screen of the first surface photographed image IM1 and perform an operation to specify the medicines D2 and D4 with a large amount of information, and the specified medicines D2 and D4 may be judged to be "medicine identifiable surfaces".

[0122] If the determination result in step S4 is Yes, the process proceeds to step S5.

[0123] In step S5, the CPU 28 executes a drug identification process for the drug to be identified, and identifies the drug type and the front and back sides after a confirmation operation by the user.

[0124] On the other hand, if the determination result in step S4 is No, step S5 is skipped.

[0125] The first loop LP1 is executed for each medicine detected in the first image IM1, and the medicine type and front / back are identified for the medicine whose medicine identifiable side is shown in the first image IM1. After the processing of the first loop LP1 is completed for each medicine included in the first image IM1 and the first loop LP1 is exited, the process proceeds to step S6.

[0126] In step S6, the CPU 28 creates an identified list, which is a list of medicines whose types have been identified in the first photographed image IM1.

[0127] In step S7, the CPU 28 creates a list of correct engraving images for the second surface from the specified list and the engraving master 140.

[0128] In step S8, the user turns the package pack over and captures an image of the second side of the package pack with the camera of the smartphone 10. The CPU 28 acquires a second-side captured image IM2 in which the second side of the package pack is captured.

[0129] In step S9, the CPU 28 performs a drug detection process on the acquired second plane image IM2. Step S9 may be the same process as step S2.

[0130] After that, the CPU 28 proceeds to a second loop LP2. The second loop LP2 includes steps S10 and S11. The CPU 28 executes steps S10 and S11 for each drug detected in step S9.

[0131] In step S10, the CPU 28 cuts out the area of ​​each detected medicine from the second plane photographed image IM2, and extracts a medicine image which is an area image of each medicine.

[0132] In step S11, the CPU 28 extracts the inscription from the medicine image extracted in step S10.

[0133] When the process of the second loop LP2 is completed for each medicine included in the second plane photographed image IM2 and the second loop LP2 is exited, the process proceeds to step S12.

[0134] In step S12, CPU 28 creates a second surface imprint extraction image list for matching, which is a list of the imprint extraction images extracted by the second loop LP2.

[0135] In step S13, the CPU 28 performs template matching on a round-robin basis between the second surface correct engraving image list and the second surface engraving extracted image list.

[0136] In step S14, the CPU 28 specifies which drug on the second plane photographed image IM2 corresponds to the drug specified on the first plane based on the processing result of step S13.

[0137] In step S15, the CPU 28 presents to the user which medicines are identified on the second photographed image IM2 in an easy-to-understand manner based on the identification result in step S14. For example, the CPU 28 presents the identified medicines with check marks CM to differentiate them from unidentified medicines, as shown in FIG.

[0138] After step S15, the same process as in the first loop LP1 is performed on the unidentified medicine in the second plane image IM2 to identify the medicine type of the unidentified medicine.

[0139] [Hardware configuration of each processing unit] The hardware structure of the processing unit that executes various processes such as the image acquisition unit 102, drug detection unit 104, drug image extraction unit 106, drug identification unit 108, determination unit 110, identified list creation unit 112, second side correct imprint image list creation unit 114, imprint extraction unit 116, second side imprint extraction image list creation unit 118, template matching unit 120, identified drug identification unit 122, and identified drug information presentation unit 124 described in Figure 4 is various processors as shown below.

[0140] Various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically to execute a specific process, such as an ASIC (Application Specific Integrated Circuit).

[0141] A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, a processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Also, multiple processing units may be composed of one processor. As an example of multiple processing units being composed of one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system including multiple processing units is used in one IC (Integrated Circuit) chip. In this way, the various processing units are composed of one or more of the above various processors as a hardware structure.

[0142] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0143] [Regarding the program for realizing the functions of the drug identification device 100] The processing function of the drug identification device 100 is not limited to the smartphone 10, and can be realized using various types of information processing devices, such as a tablet computer, a personal computer, a workstation, or a server. The processing function of the drug identification device 100 may be realized by a computer system including multiple computers. The processing function of the drug identification device 100 may be realized by a cloud server.

[0144] A program that causes a computer to realize some or all of the processing functions of the drug identification device 100 described in the above embodiment can be recorded on a computer-readable medium that is a non-transient information storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, and the program can be provided through this information storage medium. In addition, instead of providing the program by storing it on such a non-transient information storage medium that is a tangible object, it is also possible to provide a program signal as a download service using an electric communication line such as the Internet.

[0145] It is also possible to provide a part or all of the processing functions of the drug identification device 100 as an application server, and to provide a service that provides the processing functions via an electric communication line.

[0146] [Effects of the embodiment] According to the drug identification device 100 according to the embodiment, the following effects can be obtained.

[0147] [1] The medicine identification device 100 can automatically identify which medicine on the second image corresponds to a medicine whose type has been identified from the first image IM1 of the unit dose pack.

[0148] [2] After identifying the drug type of some of the multiple drugs from the first-view captured image IM1, when identifying the remaining unidentified drugs in the second-view captured image IM2, it is possible to differentiate between identified drugs and unidentified drugs on the second-view captured image IM2 and present them in an easy-to-understand manner for the user.

[0149] [3] The user can easily grasp which drug to identify among the multiple drugs in the second image IM2. This reduces the burden of identifying drug packages and improves the efficiency of the work.

[0150] [Variation 1] In the above embodiment, an example is described in which the second-side image IM2 is acquired after identifying the drug types of some drugs in the first-side image IM1, but the timing of acquiring the first-side image IM1 and the second-side image IM2 is not limited to this example. For example, following the first-side image capture, the second-side image may be captured, and the first-side image IM1 and the second-side image IM2 may be acquired, and then the drugs in the first-side image IM1 may be identified.

[0151] Furthermore, there are no particular restrictions on the order in which the first and second sides are photographed. For example, the image photographed first may be the second side image, and the image photographed later may be the first side image.

[0152] [Modification 2] A configuration is also possible in which the package pack is photographed using the smartphone 10, the photographed image is sent to a server, the processes of steps S2 to S7 and steps S9 to S15 in FIG.

[0153] [Modification 3] In the above embodiment, an example of identifying medicines has been described, but the technology of the present disclosure can also be applied to a case of inspecting medicines.

[0154] 〔others〕 The technical scope of the present invention is not limited to the scope of the above-mentioned embodiment and modified examples. The configurations and the like in the embodiment and modified examples can be changed without departing from the spirit of the present invention, and the embodiment and modified examples can be appropriately combined. [Explanation of symbols]

[0155] 10. Smartphones 12. Cabinet 14 Touch panel display 16 Speakers 18 Microphones 20 In-camera 22 Rear camera 24 Light 26 Switch 28 CPU 30 Wireless Communication Section 32 Telephone section 34 Memory 36 Internal storage 38 External memory unit 40 External input / output section 42 GPS receiver 44 Power supply section 100 Drug Identification Device 102 Image acquisition unit 104 Drug detection unit 106 Drug Image Extraction Unit 108 Drug Identification Section 110 Determined part 112 Identified List Creation Department 114 Second surface correct engraving image list creation unit 116 Stamp extraction part 118 Second surface engraving extraction image list creation unit 120 Template Matching Section 122 Drug Specifications Department 124 Drug Information Section 140 Engraving Master CE2, CE4 engraved image CM Checkmark D1, D2, D3, D4 drugs EG1,EG2,EG3,EG4 Engraving Extraction Image IM1 First page image IM2 Second plane image LS2 Second surface engraving extraction image list LSC Correct Engraving Image List for the Second Side ST1,ST2,STn GPS satellite LP1: The first loop in the drug identification method LP2: The second loop in the drug identification method S1-S15 Steps of the drug identification method

Claims

1. one or more processors; one or more storage devices; The one or more storage devices store an identification mark master including images of the identification mark on each of the front and back surfaces of a medicine on which an identification mark is imprinted or printed; The one or more processors: Detecting each medicine from a first surface photographed image in which a first surface of a unit dose pack in which a plurality of medicines are contained in a sachet is photographed; a second-side correct identification mark image list is created from the identification mark master using information on identified drugs that identifies the drug type and the front and back of at least some of the multiple drugs based on drug images extracted for each drug from the first-side photographed image, the second-side correct identification mark image list including identification mark images showing the identification marks of the identified drugs that appear on a second side, which is the back side of the first side of the unit-dose pack; Detecting each drug from a second surface image obtained by photographing the second surface of the unit-dose pack; creating an identification mark extraction image list including an identification mark extraction image for each of the medicines by extracting the identification mark of each of the medicines from the second photographed image; By performing pattern matching using the second-surface correct identification mark image list and the identification mark extraction image list, the identified medicine is identified on the second-surface photographed image; presenting information that differentiates between a drug whose type is unidentified and the identified drug in the second plane image; Drug identification device.

2. The one or more processors: Identifying the drug type and the front and back of at least a part of the plurality of drugs using a drug identification model trained by machine learning to identify the drug type and the front and back of the drug from the drug image. The drug identification device of claim 1 .

3. The one or more processors: identifying drug types for at least some of the plurality of drugs using a drug identification model trained by machine learning to identify drug types from the drug image; Identifying the front and back of at least some of the medicines of the plurality of medicines using the identification mark master. The drug identification device of claim 1 .

4. The one or more processors: The front and back are identified by pattern matching the medicine image or an identification mark image extracted from the medicine image with the identification mark master. The drug identification device according to claim 3.

5. The one or more processors: performing a brute force pattern matching between the identification mark image in the second surface correct identification mark image list and the identification mark extraction image in the identification mark extraction image list; Identifying the identified drug on the second plane image based on the matching score. The drug identification device of claim 1 .

6. The pattern matching is template matching. The drug identification device of claim 1 .

7. The one or more processors: receiving an input of an instruction to confirm the identified drug type for at least a portion of the plurality of drugs; Identifying the drug type of the target drug based on the input of the instruction; The drug identification device of claim 1 .

8. The one or more processors: Creating an identified list as information on the identified drugs; The drug identification device of claim 1 .

9. The one or more processors: A mark is added to the identified medicine in the second plane photographed image as the differentiating information to indicate that the medicine type has been identified. The drug identification device of claim 1 .

10. The one or more processors: a mark as the differentiating information is added to the unidentified medicine in the second plane photographed image to indicate that the medicine type is unidentified; The drug identification device of claim 1 .

11. a display for displaying the second-plane image including the differentiating information; The drug identification device of claim 1 .

12. A camera is provided for photographing the single-package pack. The drug identification device of claim 1 .

13. 1. A method for drug identification executed by one or more processors, comprising: storing, in one or more storage devices, an identification mark master including images of the identification mark on each of the front and back surfaces of a medicine having an identification mark by stamping or printing; acquiring a first-side image of a first side of a unit-dose pack in which a plurality of medicines are contained in a sachet; Detecting each medicine from the first plane captured image and extracting a medicine image for each medicine; Identifying the medicine type and the front and back of at least a part of the medicines based on the medicine image extracted from the first surface photographed image; creating a second-side correct identification mark image list from the identification mark master using information on the identified drug whose type has been identified from the first-side photographed image, the second-side correct identification mark image list including identification mark images showing the identification marks of the identified drugs that appear on the second side, which is the reverse side of the first side of the unit-dose pack; acquiring a second surface image of the second surface of the package; detecting each medicine from the second plane photographed image, extracting the identification mark of each medicine, and creating an identification mark extraction image list including an identification mark extraction image for each medicine; Identifying the identified medicine on the second-plane photographed image by performing pattern matching using the second-plane correct identification mark image list and the identification mark extracted image list; Presenting information that differentiates between a drug whose type is unspecified and the identified drug in the second plane photographed image; A drug identification method comprising:

14. A program for causing a computer to execute the medicine identification method according to claim 13.

Images