Drug identification device, drug identification method and program, learning device

Abstract

The present invention provides a drug identification device, a drug identification method, and a program, a trained model, and a learning device with which a partial drug is highly accurately identified. The present invention involves: acquiring a captured image capturing an identification target drug to which engravement and / or print is added and capturing a partial drug that is a portion obtained by dividing an undivided full drug into a plurally portions; detecting a region of the partial drug from the captured image; acquiring an engravement / print extraction image by processing at least the region of the partial drug in the captured image to extract the engravement and / or the print of the partial drug; inputting the engravement / print extraction image into a first trained model to infer the drug type of the partial drug; and acquiring a candidate of the drug type of the partial drug, wherein the first trained model is trained using a first image obtained by extracting engravement and / or print of the full drug to which engravement and / or print is added.

Description

Technical Field

[0001] The present invention relates to a drug identification device, a drug identification method and program, a learned model, and a learning device, and particularly relates to a technique for identifying divided drugs.

Background Art

[0002] In hospital facilities, pharmacies, etc., drug audits and identification of drugs brought in are carried out. Performing audits and identifications visually places a heavy workload on pharmacists and the like. For this reason, a technique for automatically identifying the drug type of a drug from a captured image of the drug is used.

[0003] Depending on conditions such as prescription details, drugs such as tablets may be divided. For this reason, a technique for identifying divided partial drugs from a captured image is also known.

[0004] For example, Patent Document 1 describes a learned model capable of identifying divided tablets such as 1 / 2 tablets or 1 / 4 tablets. Further, Patent Document 2 describes a technique for performing image matching processing using a master image of a half tablet obtained by dividing a drug into two parts.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Because partial drug components are not precisely divided down the middle, a variety of appearances can occur. Furthermore, the cutting line is not always straight and can have a stepped shape, resulting in a variety of appearances. Moreover, partial drug components, even if they are different drugs, may have the same or very similar appearance. For these reasons, the technologies described in Patent Documents 1 and 2 may not be able to distinguish between partial drug components, and there has been a need for more accurate identification of partial drug components.

[0007] This invention has been made in view of these circumstances, and aims to provide a drug identification device, a drug identification method and program, a trained model and a learning device that can identify partial drugs with high accuracy. [Means for solving the problem]

[0008] To achieve the above objective, the drug identification device according to the first aspect of this disclosure comprises at least one processor and at least one memory for storing instructions to be executed by the at least one processor, wherein the at least one processor acquires an image of a drug to be identified that has markings and / or printings attached to it, and the image of a partial drug that is a part of a whole drug that has been divided into multiple parts, detects the region of the partial drug from the image, processes at least the region of the partial drug in the image to obtain a marking and / or printing extraction image by extracting the markings and / or printings of the partial drug, inputs the marking and / or printing extraction image to a first trained model to infer the drug type of the partial drug, obtains multiple candidate drug types for the partial drug, presents multiple candidate drug types, and the first trained model is trained using a first image from which markings and / or printings have been extracted from a full drug that has markings and / or printings attached to it. According to the first aspect, partial drugs can be identified with high accuracy.

[0009] In the drug identification device according to the second aspect of this disclosure, it is preferable that the first trained model is trained using a second image which is a part obtained by dividing the first image at an arbitrary orientation and position, and which contains more information than a certain standard for the markings and / or printing. According to the second aspect, a second image which becomes noise during training because its information content is below a certain standard can be excluded from the training data.

[0010] A drug identification device according to a third aspect of this disclosure preferably has a second image as a part of the drug identification device according to a second aspect, wherein the second image is a part obtained by dividing the first image after it has been rotated and / or translated. According to the third aspect, the first trained model can be made robust to arbitrary rotational direction and arbitrary parallel position of the cutting axis.

[0011] A drug identification device according to a fourth aspect of this disclosure is preferably a drug identification device according to any of the first to third aspects, wherein the first trained model is trained using a first image and a second image which is a part of the first image divided at an arbitrary orientation and position, and which contains more information than a certain standard for markings and / or printing. According to the fourth aspect, both full drugs and partial drugs can be identified.

[0012] The drug identification device according to the fifth aspect of this disclosure is preferably a drug identification device according to any of the first to fourth aspects, wherein at least one processor acquires an image in which multiple drugs are captured, detects regions of multiple drugs from the image, and detects regions of partial drugs from the regions of multiple drugs. According to the fifth aspect, partial drugs contained in multiple drugs can be identified.

[0013] The drug identification device according to the sixth aspect of this disclosure is a drug identification device according to any of the first to fifth aspects, wherein at least one processor preferably acquires a first captured image of a first partial drug which is a divided part of a first drug, acquires a second captured image of a second partial drug which is a divided part of the first drug and is different from the first partial drug, acquires a composite image by aligning the region of the first partial drug in the first captured image and the region of the second partial drug in the second captured image, detects the region of the first drug from the composite image, processes at least the region of the first drug in the composite image to extract the markings and / or printing of the first drug to obtain a first marking and printing extraction image, inputs the first marking and printing extraction image to a first trained model to infer the drug type of the first drug, and obtains a candidate drug type of the first drug. According to the sixth aspect, drugs can be identified from captured images of multiple partial drugs.

[0014] The seventh embodiment of the present disclosure is a drug identification device according to any of the first to fifth embodiments, wherein at least one processor acquires a third image in which a first partial drug, which is a divided part of a first drug, and a second partial drug, which is a divided part of the first drug and is different from the first partial drug, are photographed side by side; detects the region of the first partial drug and the region of the second partial drug from the third image; processes at least the region of the first partial drug and the region of the second partial drug from the third image to obtain a second marking and printing extraction image by extracting the markings and / or printing of the first partial drug and the second partial drug; inputs the second marking and printing extraction image to a first trained model to infer the drug type of the first drug; and preferably obtains a candidate drug type of the first drug. According to the seventh embodiment, drugs can be identified from a photograph in which a plurality of partial drugs are arranged side by side.

[0015] The eighth aspect of this disclosure relates to a drug identification device according to any of the first to seventh aspects, wherein at least one processor preferably acquires a plurality of drug candidates with relatively high scores output by the first trained model. According to the eighth aspect, an interface can be provided that allows the user to select the correct drug from among the plurality of candidates.

[0016] To achieve the above objective, the trained model according to the ninth aspect of this disclosure is a trained model in which machine learning was performed using a second image which is a part of a first image from which the markings and / or printing of a drug has been extracted, divided at an arbitrary orientation and position, and which contains more information about the markings and / or printing than a certain standard. According to the ninth aspect, it is possible to identify a portion of the drug divided at an arbitrary rotational direction and parallel position.

[0017] To achieve the above objective, a learning device according to a tenth aspect of the present disclosure comprises at least one processor and at least one memory for storing instructions to be executed by the at least one processor, wherein the at least one processor trains a first trained model using a second image obtained by dividing a first image, which is obtained by extracting the markings and / or printing from a drug to which markings and / or printing have been added, into two parts at an arbitrary orientation and position, and the second image contains more information about the markings and / or printing than a certain standard. According to the tenth aspect, a first trained model that is robust to the arbitrary rotational direction and parallel position of the cutting axis can be trained.

[0018] To achieve the above object, a drug identification method according to the 11th aspect of the present disclosure is such that at least one processor acquires a captured image of a drug to be identified with engraving and / or printing added thereto, where the captured image is of a partial drug that is a part of a non-divided full drug divided into a plurality of parts, detects the region of the partial drug from the captured image, processes at least the region of the partial drug in the captured image to obtain an engraved / printed extraction image in which the engraving and / or printing of the partial drug is extracted, inputs the engraved / printed extraction image into a first pre-trained model to infer the drug type of the partial drug, obtains a plurality of drug type candidates for the partial drug, presents the plurality of drug type candidates, and the first pre-trained model is trained based on a first image in which the engraving and / or printing of a non-divided full drug with engraving and / or printing added thereto is extracted. According to the 11th aspect, the partial drug can be identified with high accuracy.

[0019] To achieve the above object, a program according to the 12th aspect of the present disclosure is a program that causes a computer to execute the drug identification method of the 11th aspect. According to this aspect, the partial drug can be identified with high accuracy. A non-temporary and computer-readable recording medium such as a CD-ROM (Compact Disk - Read Only Memory) storing the program according to the 12th aspect is also included in the present disclosure.

Advantages of the Invention

[0020] According to the present invention, the partial drug can be identified with high accuracy.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 1 is a diagram for explaining the identification of half tablets. [Figure 2] FIG. 2 is a front perspective view of a smartphone. [Figure 3] FIG. 3 is a rear perspective view of a smartphone. [Figure 4] FIG. 4 is a block diagram showing the electrical configuration of a smartphone. [Figure 5]FIG. 5 is a block diagram showing the functional configuration of the drug identification device. [Figure 6] FIG. 6 is a block diagram showing the electrical configuration of the learning device. [Figure 7] FIG. 7 is a diagram showing an example of a learning dataset for generating a learned model. [Figure 8] FIG. 8 is a block diagram showing the functional configuration of the learning device. [Figure 9] FIG. 9 is a flowchart showing the learning method of the learned model according to the first embodiment. [Figure 10] FIG. 10 is a flowchart showing the drug identification method according to the first embodiment. [Figure 11] FIG. 11 is a block diagram showing the functional configuration of the learning data acquisition unit according to the second embodiment. [Figure 12] FIG. 12 is a diagram for explaining the generation of an image of learning data according to the second embodiment. [Figure 13] FIG. 13 is a diagram for explaining the generation of an image of learning data according to the second embodiment. [Figure 14] FIG. 14 is a diagram for explaining another example of the generation of an image of learning data of an oval tablet. [Figure 15] FIG. 15 is a flowchart showing the learning method of the learned model according to the second embodiment. [Figure 16] FIG. 16 is a flowchart showing the drug identification method according to the second embodiment. [Figure 17] FIG. 17 is a flowchart showing the learning method of the learned model according to the third embodiment. [Figure 18] FIG. 18 is a flowchart showing the drug identification method according to the third embodiment. [Figure 19] FIG. 19 is a flowchart showing the learning method of the learned model according to the fourth embodiment. [Figure 20] FIG. 20 is a flowchart showing the drug identification method according to the fourth embodiment. [Figure 21]Figure 21 is a diagram illustrating an example of a photograph of half a tablet according to the fourth embodiment. [Figure 22] Figure 22 is a diagram illustrating another example of a photograph of a half-tablet according to the fourth embodiment. [Modes for carrying out the invention]

[0022] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0023] <Definition of Terms> In this specification, "full drug" refers to an undivided drug in its shipped state. An example of a full drug is an undivided tablet called a "full tablet." In this specification, "tablet" refers to a solid drug molded into a specific shape by compression molding. In this specification, "round tablet" refers to a full tablet that is circular in plan view. In this specification, "elliptical tablet" refers to a full tablet that is elliptical in plan view. It does not need to be a mathematically precise ellipse.

[0024] In this specification, "partial drug" refers to a portion of a drug that has been divided into multiple parts. An example of a partial drug is a "half tablet," which is one of two tablets obtained when a full tablet has been divided into two. A "half tablet" is not limited to a full tablet that has been divided into two equal parts; it can be any half of a tablet that has been divided into two. A "half tablet" may sometimes be written as "1 / 2 tablet."

[0025] Another example of a partial tablet is a "quarter tablet," which is one of four portions of a full tablet. A "quarter tablet" is not limited to a full tablet divided into four equal parts; any one of the four portions of a full tablet is acceptable. A "quarter tablet" may sometimes be written as "1 / 4 tablet."

[0026] The means of dividing the tablets are not limited. Tablets may be divided manually using scissors, a cutter, a multi-tablet / half-tablet cutter, etc. Tablets may also be divided automatically by a dividing function installed in a packaging machine.

[0027] In this specification, "marked" means that identification information is formed on the surface of the drug by creating grooves, which are recessed areas. The grooves are not limited to those formed by excavating the surface, but may also be formed by pressing the surface. Furthermore, the markings may include those that do not have an identification function, such as score lines.

[0028] In this specification, "printed" means that identification information has been formed by applying edible ink or the like to the surface of the drug, either in contact or non-contact. Here, "printed" is synonymous with "printed."

[0029] In this specification, "engraving and / or printing" means "either engraving or printing," or "both engraving and printing." In this specification, "engraving and / or printing" may be omitted and written as "engraving and printing."

[0030] <Identifying half a tablet> Figure 1 is a diagram illustrating the identification of half tablets. F1A in Figure 1 shows the front surface of a full tablet TF with the engraved markings oriented upright. A full tablet TF is, for example, a round tablet with a diameter of 7.0 mm and a thickness of 2.4 mm. The front surface of a full tablet TF is engraved with "FF111".

[0031] F1B in Figure 1 shows the front surface of a half-tablet TH1. A half-tablet TH1 is the upper part when a full tablet TF is divided into two parts vertically by a straight line in the state shown in Figure 1. The front surface of the half-tablet TH1 is marked with "FF".

[0032] The drug type identification according to this embodiment includes identifying the drug type of a full tablet TF solely from an image of a half tablet TH1.

[0033] F1C in Figure 1 shows the front surface of a half-tablet TH2. A half-tablet TH2 is the lower part when a full tablet TF is divided into two parts vertically in a straight line as shown in Figure 1. The front surface of the half-tablet TH2 is stamped with "111".

[0034] The drug type identification according to this embodiment includes identifying the drug type of a full tablet TF by combining an image of a half tablet TH1 and an image of a half tablet TH2. Furthermore, the drug type identification according to this embodiment also includes identifying the drug type of a full tablet TF from an image of half tablets TH1 and TH2 placed side by side.

[0035] A half tablet is not limited to one half of a tablet divided vertically. Nor is it limited to one half of a tablet divided linearly. F1D in Figure 1 shows the front surface of half tablet TH3. Half tablet TH3 is the right-hand part when a full tablet TF is divided horizontally in a non-linear manner in the state shown in Figure 1. F1E in Figure 1 shows the front surface of half tablet TH4. Half tablet TH4 is the left-hand part when a full tablet TF is divided horizontally in a non-linear manner in the state shown in Figure 1. F1E in Figure 1 shows the front surface of half tablet TH5. Half tablet TH5 is the upper left part when a full tablet TF is divided diagonally in a non-linear manner in the state shown in Figure 1. F1G in Figure 1 shows the front surface of half tablet TH6. Half tablet TH6 is the lower right part when a full tablet TF is divided diagonally in a non-linear manner in the state shown in Figure 1. Thus, a half tablet may be divided in any direction or at any position.

[0036] Tablets may have a score line. Half tablets are not limited to those divided in two along a score line.

[0037] <Difficulty in identifying the type of drug when half a tablet is used> For example, there are many white, round tablets of the same size. Therefore, identifying them requires identification information from markings / printings, and under the constraints of the following tasks 1 to 5, it is necessary to identify the tablets using only partial information from the markings / printings.

[0038] (Problem 1) When cutting, the division is not precisely down the middle, which can result in a variety of appearances.

[0039] (Challenge 2) The cutting line is not necessarily straight and can have a stepped shape, resulting in a variety of appearances.

[0040] (Problem 3) In the case of a circular lock, there is arbitrariness in the rotation direction of the cutting axis, which can result in a variety of appearances of the markings / printings.

[0041] (Problem 4) The design of the markings / printing on the drug tablets, combined with the way they are cut, can result in half tablets having significantly little or no identifying information such as markings / printing that would otherwise be used as clues for drug type identification. Furthermore, images of half tablets with significantly little or no identifying information become noise when used as training data. Therefore, if such data is included in training an AI (Artificial Intelligence) for drug type identification, the accuracy of the AI's identification will decrease.

[0042] (Problem 5) Half a tablet of the drug to be identified and half a tablet of a different drug may look identical or very similar.

[0043] Furthermore, in order to perform drug type identification using AI, it is necessary to prepare diverse visual training data for each of the approximately 10,000 drug types used as pharmaceuticals, which presents the challenge of requiring an enormous amount of time and effort for data collection and other related tasks.

[0044] <Configuration of the drug identification device and learning device> The drug identification device according to this embodiment identifies the type of drug to be identified from a captured image of the drug to be identified, which has markings and / or printing added to it, and identifies the correct drug. The drug to be identified includes partial drugs, which are parts of a full drug that has been divided into multiple parts.

[0045] A drug identification device is, for example, installed in a mobile terminal device. A mobile terminal device includes at least one of the following: a mobile phone, a PHS (Personal Handyphone System), a smartphone, a PDA (Personal Digital Assistant), a tablet computer, a notebook personal computer, and a portable game console. Below, a drug identification device consisting of a smartphone will be used as an example and described in detail with reference to the drawings.

[0046] [Smartphone appearance] Figure 2 is a front perspective view of a smartphone 10, which is a camera-equipped mobile terminal device according to this embodiment. As shown in Figure 2, the smartphone 10 has a flat housing 12. The smartphone 10 is equipped with a touch panel display 14, a speaker 16, a microphone 18, and an in-camera 20 on the front of the housing 12.

[0047] The touch panel display 14 comprises a display section for displaying images, etc., and a touch panel section located in front of the display section for receiving touch input. The display section is, for example, a color LCD (Liquid Crystal Display) panel.

[0048] The touch panel is a capacitive touch panel provided in a planar manner on a substrate body that is, for example, light-transmitting, and has light-transmitting position detection electrodes and an insulating layer provided on the position detection electrodes. The touch panel generates and outputs two-dimensional position coordinate information corresponding to the user's touch operation. Touch operations include tap operations, double-tap operations, flick operations, swipe operations, drag operations, pinch-in operations, and pinch-out operations.

[0049] Speaker 16 is an audio output unit that outputs sound during calls and video playback. Microphone 18 is an audio input unit that receives sound during calls and video recording. In-camera 20 is an imaging device that captures videos and still images.

[0050] Figure 3 is a rear perspective view of the smartphone 10. As shown in Figure 3, the smartphone 10 has an out-camera 22 and a light 24 on the back of the housing 12. The out-camera 22 is an imaging device that captures videos and still images. The light 24 is a light source that illuminates the out-camera 22 when it is taking pictures, and is composed of, for example, an LED (Light Emitting Diode).

[0051] Furthermore, as shown in Figures 2 and 3, the smartphone 10 is equipped with switches 26 on the front and side of the housing 12, respectively. The switches 26 are input components that receive instructions from the user. The switches 26 are push-button type switches that turn on when pressed with a finger, etc., and turn off when the finger is released due to a restoring force such as a spring.

[0052] The configuration of the housing 12 is not limited to this, and a configuration having a folding structure or a sliding mechanism may also be adopted.

[0053] [Electrical configuration of a smartphone] The primary function of smartphone 10 is to provide wireless communication capabilities that enable mobile wireless communication via base station equipment and a mobile communication network.

[0054] Figure 4 is a block diagram showing the electrical configuration of the smartphone 10. As shown in Figure 4, the smartphone 10 includes the aforementioned touch panel display 14, speaker 16, microphone 18, front camera 20, rear camera 22, light 24, and switch 26, as well as a CPU (Central Processing Unit) 28, wireless communication unit 30, call unit 32, memory 34, external input / output unit 40, GPS receiver unit 42, and power supply unit 44.

[0055] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates according to the control program and control data stored in the memory 34, and comprehensively controls each part of the smartphone 10. The CPU 28 is equipped with a mobile communication control function that controls each part of the communication system in order to perform voice communication and data communication through the wireless communication unit 30, and an application processing function.

[0056] Furthermore, the CPU 28 is equipped with an image processing function that displays videos, still images, and text on the touch panel display 14. This image processing function visually conveys information such as still images, videos, and text to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to the user's touch operation from the touch panel portion of the touch panel display 14. In addition, the CPU 28 acquires input signals from the switch 26.

[0057] The hardware structure of the CPU28 consists of various processors as follows: These processors include a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) and acts as various functional units; a GPU (Graphics Processing Unit), a processor specialized in image processing; a PLD (Programmable Logic Device), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing; and dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which have a circuit configuration specifically designed to perform particular processing.

[0058] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, multiple functional units may be composed of a single processor. Examples of composing multiple functional units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, as is typical of computers such as client or server computers, and this processor acts as multiple functional units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple functional units, on a single IC (Integrated Circuit) chip, as is typical of SoCs (System On Chip). Thus, various functional units are configured as hardware structures using one or more of the above-mentioned various processors.

[0059] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0060] The front camera 20 and rear camera 22 capture video and still images according to instructions from the CPU 28. The internal configuration of the front camera 20 and rear camera 22 is the same. The front camera 20 and rear camera 22 each have a shooting lens, image sensor, and image processing unit (not shown).

[0061] The front camera 20 and the rear camera 22 each receive subject light through their imaging lenses using their image sensors. The image sensor is a photoelectric conversion element such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge-Coupled Device), and is provided with R (red), G (green), and B (blue) color filters (not shown) on its light-receiving surface. The subject light is imaged onto the light-receiving surface of the image sensor, and the image sensor converts the imaged subject light onto the light-receiving surface into an electrical signal based on the R, G, and B color signals. The image processing unit performs predetermined processing on the analog image signal output from the image sensor to convert it into a digital image signal.

[0062] The front camera 20 and the rear camera 22 may each convert the captured video and still image data into compressed image data such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group).

[0063] The CPU 28 stores the videos and still images captured by the front camera 20 and the rear camera 22 in the memory 34. The CPU 28 may also output the videos and still images captured by the front camera 20 and the rear camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0064] Furthermore, the CPU 28 displays videos and still images captured by the front camera 20 and the rear camera 22 on the touch panel display 14. The CPU 28 may also use the videos and still images captured by the front camera 20 and the rear camera 22 within the application software.

[0065] The CPU 28 may also illuminate the subject with assistive lighting by turning on the light 24 when shooting with the rear camera 22. The light 24 may be turned on and off by the user's touch operation on the touch panel display 14 or by operating the switch 26.

[0066] The wireless communication unit 30 performs wireless communication with base station equipment connected to the mobile communication network in accordance with instructions from the CPU 28. The smartphone 10 uses this wireless communication to send and receive various file data such as voice data and image data, email data, etc., and to receive Web (World Wide Web) data and streaming data, etc.

[0067] The communication unit 32 is connected to a speaker 16 and a microphone 18. The communication unit 32 decodes the audio data received by the wireless communication unit 30 and outputs it from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into audio data that can be processed by the CPU 28 and outputs it to the CPU 28.

[0068] Memory 34 stores instructions for the CPU 28 to execute. Memory 34 consists of an internal storage unit 36 ​​built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are implemented using known storage media.

[0069] Memory 34 stores the CPU 28's control program, control data, application software, address data associated with the names and telephone numbers of communication partners, sent and received email data, web data downloaded through web browsing, and downloaded content data. Memory 34 may also temporarily store streaming data.

[0070] The external input / output unit 40 serves as an interface with external devices connected to the smartphone 10. The smartphone 10 is connected directly or indirectly to other external devices via communication or other means through the external input / output unit 40. The external input / output unit 40 transmits data received from external devices to the various internal components of the smartphone 10 and transmits data from inside the smartphone 10 to external devices.

[0071] Communication methods include, for example, Universal Serial Bus (USB), IEEE (Institute of Electrical and Electronics Engineers) 1394, the Internet, Wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. External devices include, for example, headsets, external chargers, data ports, audio equipment, video equipment, smartphones, PDAs, personal computers, and earphones.

[0072] The GPS receiver 42 detects the location of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, ..., STn.

[0073] The power supply unit 44 is a power source that supplies power to various parts of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium-ion secondary battery. The power supply unit 44 may also include an A / D converter that generates a DC voltage from an external AC power source.

[0074] The smartphone 10 configured in this way can be set to shooting mode by user input via a touch panel display 14 or the like, and can capture videos and still images using the front camera 20 and rear camera 22.

[0075] When the smartphone 10 is set to shooting mode, it enters a shooting standby state, and a video is recorded by the front camera 20 or rear camera 22. The recorded video is then displayed as a live view image on the touch panel display 14.

[0076] Users can view the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject they want to photograph, and set shooting conditions.

[0077] When the smartphone 10 is in shooting standby mode and receives a shooting command from the user via the touch panel display 14 or the like, it performs AF (Autofocus) and AE (Auto Exposure) control to shoot and store videos and still images.

[0078] [Functional configuration of the drug identification device] Figure 5 is a block diagram showing the functional configuration of the drug identification device 100 implemented by the smartphone 10. Each function of the drug identification device 100 is realized when the CPU 28 executes a drug identification program stored in the memory 34. As shown in Figure 5, the drug identification device 100 includes an image acquisition unit 102, a drug detection unit 104, an imprinting / print extraction unit 106, a drug type recognition unit 108, a candidate output unit 110, and a confirmation unit 112.

[0079] The image acquisition unit 102 acquires captured images of the identification target drug to which the engraved markings have been added. The captured images are, for example, images taken by the in-camera 20 or the out-camera 22. The captured images may also be images acquired from other devices via the wireless communication unit 30, the external storage unit 38, or the external input / output unit 40.

[0080] The captured image may be an image showing the target drug and a marker. There may be multiple markers, or they may be ArUco markers. The captured image may also be an image showing the target drug and a reference gray color.

[0081] The captured image may be one taken at a standard shooting distance and viewpoint. Shooting distance can be expressed as the distance between the target drug and the photographic lens, plus the focal length of the photographic lens. The viewpoint can be expressed as the angle formed by the marker printing surface and the optical axis of the photographic lens.

[0082] The captured image may contain multiple target drugs. These multiple target drugs are not limited to those of the same drug type, but may be from different drug types. The target drugs may include the whole drug or partial drugs.

[0083] The image acquisition unit 102 may, if the captured image contains a marker, standardize the shooting distance and shooting viewpoint of the captured image based on the marker and acquire a standardized image. Furthermore, if the captured image contains a reference gray area, the image acquisition unit 102 may perform color correction of the captured image based on the reference gray color.

[0084] The drug detection unit 104 detects the region of the drug to be identified from the captured image acquired by the image acquisition unit 102. If a standardized image is acquired by the image acquisition unit 102, the drug detection unit 104 detects the region of the drug to be identified from the standardized image. If the captured image contains multiple drugs to be identified, the drug detection unit 104 detects the region of each of the multiple drugs to be identified.

[0085] The marking extraction unit 106 processes at least the region of the target drug in the captured image to remove the outline edge information of the target drug that may become noise in drug identification, and obtains a marking extraction image in which the markings have been extracted. Here, the marking extraction image is an image in which the markings are emphasized by representing the brightness of the marking portion (hereinafter referred to as the marking portion) as being relatively higher than the brightness of the portion other than the marking portion. The marking extraction image may also be an image in which the marking portion is white and the portion other than the marking portion is black.

[0086] If the drug detection unit 104 detects the respective regions of multiple target drugs, the marking extraction unit 106 acquires multiple marking extraction images corresponding to each of the multiple target drugs.

[0087] The marking extraction unit 106 may include a trained model that, when given an image of a drug with markings attached as input, outputs an image from which the markings of the drug have been extracted.

[0088] The drug type recognition unit 108 receives an image of the engraved markings extracted from the input and infers the drug type of the drug to be identified, and obtains candidate drug types for the drug to be identified. The candidate drug types include drug identification information consisting of the drug name, trade name, abbreviation, or a combination thereof. If multiple images of engraved markings extracted from multiple drugs to be identified are input to the drug type recognition unit 108, it obtains candidate drug types corresponding to each of the multiple drugs to be identified.

[0089] The drug type recognition unit 108 includes a trained model 108A (an example of the "first trained model"). The trained model 108A is a trained model that, when given an image from which the drug markings have been extracted as input, outputs the drug type corresponding to the markings. The trained model 108A may output multiple drug types. Along with the drug type, the trained model 108A may also output a score or probability indicating the likelihood that it corresponds to that drug type.

[0090] The trained model 108A was trained using machine learning with images from which the markings were extracted from full-sized drugs with markings. For example, the trained model 108A is trained using a training dataset of multiple different drugs with markings, where the training dataset consists of a first image of the drug from which the markings were extracted and the drug type corresponding to the markings. The first image is, for example, an image from which the markings of the full-sized drug were extracted. The first image may also be an image obtained by processing the image from which the markings of the full-sized drug were extracted. The trained model 108A can be trained using a Convolutional Neural Network (CNN).

[0091] Thus, since the drug type recognition unit 108 performs recognition based on the information printed on the markings without using color information, it is robust to the influence of the shooting environment.

[0092] The candidate output unit 110 outputs candidate drug types for the drug to be identified, acquired by the drug type recognition unit 108. The candidate output unit 110 may output multiple candidate drug types for which the trained model 108A has a relatively high score. The candidate output unit 110 displays multiple candidate drug types for the drug to be identified on the touch panel display 14, for example, allowing selection.

[0093] The confirmation unit 112 determines the correct drug for identification from the candidate drug types for identification. The confirmation unit 112 determines the candidate drug type selected by the user from among multiple candidate drug types for identification displayed on the touch panel display 14 as the correct drug. If there is only one candidate drug type for identification output by the candidate output unit 110, the confirmation unit 112 may determine that candidate drug type as the correct drug.

[0094] [Electrical configuration of the learning device] Figure 6 is a block diagram showing the electrical configuration of the learning device 120 that generates the trained model 108A. A personal computer or workstation can be used as the learning device 120.

[0095] As shown in Figure 6, the learning device 120 includes a learning data acquisition unit 122, a database 124, an operation unit 126, a display unit 128, a CPU 130, a RAM (Random Access Memory) 132, and a ROM (Read Only Memory) 134.

[0096] The learning data acquisition unit 122 is an interface for acquiring learning datasets stored in the database 124, and includes wired and wireless communication interfaces.

[0097] Database 124 is a storage unit that stores the training dataset and is composed of a large-capacity storage device. The training dataset will be described later.

[0098] The operation unit 126 is a user interface for the user to control the learning device 120, and consists of a keyboard and a pointing device.

[0099] The display unit 128 is an output interface that visually displays the status of the learning device 120, and is composed of a display panel.

[0100] The CPU 130 is a processor that executes instructions stored in the RAM 132 and ROM 134, and its hardware structure is the same as that of the CPU 28. The RAM 132 is a memory device that temporarily stores data used by the CPU 130 for various calculations, and is composed of semiconductor memory. The ROM 134 is a memory device that stores programs for the CPU 130 to execute, and is composed of a hard disk.

[0101] Figure 7 shows an example of a training dataset input to the learning device 120 for generating a trained model 108A. The training dataset is stored in the database 124. Each training dataset consists of a set of engraved print extraction images of multiple different drugs, and the correct data, which includes the drug type and the front and back information of the engraved print extraction image. The engraved print extraction images included in the training dataset are of a fixed size corresponding to the trained model 108A. In the example shown in Figure 7, the engraved print extraction image I01, the drug type D01 of the engraved print extraction image I01, and the information of the front side constitute one training dataset. Similarly, the engraved print extraction image I02, the drug type D02 of the engraved print extraction image I02, and the information of the front side constitute one training dataset, and the engraved print extraction image I03, the drug type D03 of the engraved print extraction image I03, and the information of the back side constitute one training dataset.

[0102] Figure 8 is a block diagram showing the functional configuration of the learning device 120. As shown in Figure 8, the learning device 120 includes a recognition unit 140, an error calculation unit 142, and a parameter control unit 144.

[0103] The recognition unit 140 uses a Convolutional Neural Network (CNN). The recognition unit 140 changes from an untrained model to a trained model when its parameters are updated from their initial values ​​to optimal values. The initial values ​​of the parameters of the recognition unit 140 can be arbitrary values, or the parameters of an existing trained model can be applied.

[0104] The recognition unit 140 comprises an input layer 140A, an intermediate layer 140B, and an output layer 140C. The input layer 140A, the intermediate layer 140B, and the output layer 140C have a structure in which multiple nodes are connected by edges.

[0105] During the training phase, the input layer 140A receives a fixed-size image of a mark extracted from the training dataset.

[0106] The intermediate layer 140B is a layer that extracts features from the engraved markings extracted image input from the input layer 140A. The intermediate layer 140B has multiple sets of convolutional layers and pooling layers, and a fully connected layer. The fully connected layer connects all the nodes of the preceding layer (in this case, the pooling layer).

[0107] The output layer 140C is a layer that outputs the drug type, which is the recognition result of the recognition unit 140.

[0108] The error calculation unit 142 obtains the recognition result output from the output layer 140C of the recognition unit 140 and the drug type from the training dataset of the engraved print extraction image input to the input layer 140A, and calculates the error between the two. Possible methods for calculating the error include, for example, softmax cross-entropy or least squares error (MSE: Mean Squared Error).

[0109] The parameter control unit 144 adjusts the parameters of the recognition unit 140 using the backpropagation method based on the error calculated by the error calculation unit 142. The parameters include, for example, the coefficients of the convolutional layer filter.

[0110] This parameter adjustment process is repeated, and learning is performed repeatedly until the difference between the output of the recognition unit 140 and the correct data becomes small.

[0111] The learning device 120 performs the learning in this manner and generates a trained recognizer 140 with optimized parameters as a trained model 108A. A moderate amount of noise may be artificially added to the drug label printing extraction images in the training dataset. This makes it possible to generate a trained model 108A that is robust to variations in the shooting environment.

[0112] Here, the input image is an image of the markings extracted from one side of the drug, but a set of images of the markings extracted from both sides may also be used as the input image. In this case, the amount of information will be greater than when only one side is used, so improved accuracy can be expected, but usability will decrease because both sides must be photographed consecutively.

[0113] In the application phase, the trained model 108A, which is the recognition unit 140, outputs the type of drug corresponding to the markings when it receives an image of the drug's markings extracted from the drug.

[0114] <First Embodiment> [Learning Phase] Figure 9 is a flowchart showing the learning method for a learned model 108A according to the first embodiment in the learning device 120. The learning method is implemented by the CPU 130 reading and executing a learning program from the ROM 134. The learning program may be provided via an input interface (not shown) of the learning device 120.

[0115] In step S1, the learning device 120 prepares a training dataset. The training dataset includes images of drug markings extracted, and the correct data, which includes the drug type and information on the front and back of the marking images. Here, the learning device 120 prepares a large number of full tablet marking images linked to drug type information from the database 124.

[0116] In step S2, the learning device 120 takes an image of the imprinted markings of a full tablet as input and the drug type as output to train the recognition device 140. Specifically, the learning device 120 calculates the error between the output result of the output layer 140C when the imprinted markings of the learning dataset are input to the input layer 140A and the drug type in the learning dataset using the error calculation unit 142, and adjusts the parameters of the recognition device 140 using the parameter control unit 144 based on the calculated error.

[0117] By repeatedly performing learning until the difference between the output of the recognition unit 140 and the correct data becomes small, a trained recognition unit 140 with optimized parameters is generated. As a result, in step S3, a trained model 108A, which is a drug type identification AI that identifies the drug type of a full tablet, is completed.

[0118] [Utilization Phase] Figure 10 is a flowchart showing a drug identification method according to the first embodiment in the drug identification device 100. The drug identification method is implemented by the CPU 28 reading and executing a drug identification program from the memory 34. The drug identification program may be provided via the wireless communication unit 30 or the external input / output unit 40. Here, the identification process for half a tablet will be described.

[0119] In step S11, the drug identification device 100 photographs multiple tablets with the out camera 22. The multiple tablets may be packaged together in individual packets. The image acquisition unit 102 acquires the captured image of the multiple tablets.

[0120] In step S12, the drug detection unit 104 detects multiple tablet regions from the image captured in step S11.

[0121] The processes in steps S13 to S16 are repeated for each region of the tablet detected in step S12.

[0122] In step S13, the drug detection unit 104 extracts an image of a single tablet to be identified from the image captured in step S11. The tablet image is an image in which the region of each individual tablet is extracted from the captured image.

[0123] In step S14, the drug detection unit 104 determines from the tablet image extracted in step S13 whether the tablet shown is half a tablet or not. If it is determined in step S14 that it is a full tablet, i.e., if the determination in step S14 is No, the processing for this tablet is terminated and the process moves to step S13 for a different tablet. If it is determined in step S14 that it is half a tablet, i.e., if the determination in step S14 is Yes, the process moves to step S15.

[0124] In step S15, the marking extraction unit 106 acquires a marking extraction image from the tablet image extracted in step S13.

[0125] In step S16, the drug type recognition unit 108 inputs the marking extraction image acquired in step S15 into the trained model 108A generated in steps S1 to S3, and acquires multiple candidate drug types for the half-tablet shown in the image. The candidate output unit 110 displays on the touch panel display 14 multiple drug type candidates with relatively high scores from the multiple drug type candidates for the half-tablet acquired by the drug type recognition unit 108.

[0126] The confirmation unit 112 confirms the drug candidate selected by the user from among the displayed list of drug candidates as the correct drug.

[0127] The drug identification device 100 completes the processing in steps S13 to S16 for all tablets detected in step S12, and then terminates the processing shown in this flowchart.

[0128] [Mechanism of Action / Effect] The first embodiment has the following features:

[0129] (Feature A1) This is a machine learning-based identification method that uses an image with a mark extracted from it as the input image.

[0130] (Feature A2) During training of the pre-trained model 108A, full drug imprint extraction images are used, while during inference, only partial information is input. This utilizes the properties of deep learning that allow the correct drug type to be inferred even with partial information input.

[0131] (Feature A3) Presents multiple drug candidates with a score or probability above a certain level.

[0132] According to the first embodiment, the above-mentioned problems 1 and 2 can be solved by features A1 and A2, problems 3 and 4 can be solved by features A2 and the subsequent part of A2, and problem 5 can be solved by feature A3.

[0133] <Second Embodiment> [Generating training data] Functional configuration of the learning device Figure 11 is a block diagram showing the functional configuration of the learning data acquisition unit 122 according to the second embodiment. Here, an example of generating learning data in the learning data acquisition unit 122 of the learning device 120 is described, but the learning data may be generated in a device other than the learning device 120. The learning data acquisition unit 122 artificially generates a simulated image of the engraved markings extracted from a half-tablet as learning data.

[0134] As shown in Figure 11, the learning data acquisition unit 122 includes an engraved print extraction image acquisition unit 152, a random rotation unit 154, a random translation unit 156, a cutting unit 158, a determination unit 160, and an arrangement unit 162.

[0135] The marking extraction image acquisition unit 152 acquires a marking extraction image of a full tablet from the database 124. Alternatively, the marking extraction image acquisition unit 152 may acquire a photographic image of a full tablet, detect the area of ​​the full tablet from the acquired photographic image, and extract the markings from the detected area to obtain a marking extraction image of a full tablet.

[0136] The random rotation unit 154 performs a rotation process that rotates the engraved print extraction image acquired by the engraved print extraction image acquisition unit 152 at random angles, with the center of the engraved print extraction image as the axis.

[0137] The random translation unit 156 performs a translation process that translates the engraved print extraction image rotated by the random rotation unit 154 by a random amount of movement.

[0138] The cutting unit 158 ​​cuts the engraved print extraction image, which has been moved in parallel by the random translation unit 156, along a cutting axis whose orientation and position are fixed in advance, dividing it into two parts and generating two divided engraved print extraction images.

[0139] Furthermore, the generation of the segmented engraved print extraction image is not limited to the example where the engraved print extraction image is rotated and translated before being cut with a fixed cutting axis; it is sufficient if the relationship between the engraved print extraction image and the cutting axis is rotated and translated. For example, the cutting axis may be rotated at a random angle in the random rotation unit 154 and then translated in the random translation unit 156. In this case, the fixed engraved print extraction image is cut with the cutting axis after rotation and translation.

[0140] The determination unit 160 determines whether or not to adopt the segmented engraved print extraction image as training data. For example, the determination unit 160 determines whether to adopt the segmented engraved print extraction image that contains more information about the engraved print than a certain standard as training data. The certain standard is 20% of the information about the engraved print in the engraved print extraction image before segmentation, preferably 30%, and more preferably 40%. The sum of the pixel values ​​of the engraved print is used as the amount of information. For example, if the sum of the pixel values ​​of the engraved print in the segmented engraved print extraction image is greater than 30% of the sum of the pixel values ​​of the engraved print in the engraved print extraction image before segmentation, the determination unit 160 determines whether to adopt that segmented engraved print extraction image as training data. On the other hand, if the sum of the pixel values ​​of the engraved print in the segmented engraved print extraction image is 30% or less of the sum of the pixel values ​​of the engraved print in the engraved print extraction image before segmentation, the determination unit 160 determines whether to exclude that segmented engraved print extraction image from the training data.

[0141] The determination unit 160 determines whether or not to use each of the two divided marking extraction images generated by the cutting unit 158 ​​as training data.

[0142] The placement unit 162 generates a training image having the input size of the trained model 108A. The training image is an image in which the segmented marking extraction image that the determination unit 160 has decided to adopt as training data is centered. In the training image, the background area other than the area in which the segmented marking extraction image is placed has the same brightness as the part of the segmented marking extraction image other than the marking print portion. For example, if the part of the segmented marking extraction image other than the marking print portion is black, the background area of ​​the training image is black.

[0143] (In the case of a round lock) Figure 12 is a diagram illustrating the generation of training data images according to the second embodiment, and shows the case of a circular lock. In the case of a circular lock, training data images are generated by rotating and translating the captured image.

[0144] Figure 12, F12A shows the front surface of the circular lock TC with the engraved markings facing upright. The front surface of the circular lock TC has the engraving "FF111" inscribed on it. Figure 12, F12B shows the engraved marking extraction image I11 of the front surface of the circular lock TC. The engraved marking extraction image I11 is obtained by the engraved marking extraction process P1 in the engraved marking extraction image acquisition unit 152, which extracts the engraved markings from the captured image of the front surface of the circular lock TC. Here, the center of the engraved marking extraction image I11 coincides with the center of the circular lock TC.

[0145] F12C in Figure 12 shows the engraved print extraction image I12. The engraved print extraction image I12 is obtained by rotating the engraved print extraction image I11 by the random rotation process P2 in the random rotation unit 154. Here, the engraved print extraction image I12 is an image obtained by rotating the engraved print extraction image I11 clockwise by an angle θ. The random rotation unit 154 may determine the angle θ to any value between 0 degrees and 360 degrees. The background region BG1 of the engraved print extraction image I12 shown in F12C is an image processing area, and the rotation and translation of the target engraved print extraction image are performed within the background region BG1. In F12C, the center of the background region BG1 is at the same position as the center of the engraved print extraction image I11.

[0146] F12D in Figure 12 shows the engraved print extraction image I13. The engraved print extraction image I13 is obtained by the random translation process P3 in the random translation unit 156, which translates the engraved print extraction image I12. The background region BG1 is fixed and does not translate in the random translation process P3. Here, the engraved print extraction image I13 is an image obtained by translating the engraved print extraction image I12 by Δx in the horizontal direction and Δy in the vertical direction in Figure 12. The random translation unit 156 may determine Δx and Δy to any values ​​including 0.

[0147] F12E in Figure 12 shows how the engraved print extraction image I13 is cut along the center line LC, which is the cutting axis, by the cutting process P4 in the cutting section 158. The center line LC is a straight line parallel to the vertical direction in Figure 12 and passes through the center of the background region BG1.

[0148] F12F in Figure 12 shows the divided stamped image I14, which is one half of the stamped image I13 that was cut and separated by the cutting process P4. F12G in Figure 12 shows the other half of the divided stamped image I15 that was cut and separated by the cutting process P4. Divided stamped images I14 and I15 correspond to images that are parts of the stamped image I13 that were divided at arbitrary orientations and positions, respectively.

[0149] F12H in Figure 12 shows the result RT1 of the determination of the divided marking extraction image I14 by the determination process P5 in the determination unit 160. The determination process P5 is a process that determines whether to adopt a divided marking extraction image that contains more information than a certain standard of marking as training data. Result RT1 is that it is adopted as training data. In other words, the divided marking extraction image I14 is an image obtained by dividing the marking extraction image I13 at arbitrary orientations and positions, and corresponds to an image that contains more information than a certain standard of marking.

[0150] Furthermore, F12I in Figure 12 shows the result RT2, which is the result of the determination process P5 in which the segmented marking print extraction image I15 was determined. Result RT2 was not adopted (excluded) as training data.

[0151] F12J in Figure 12 shows a training image I16 generated from a segmented marking extraction image I14 (an example of a "second image"). The training image I16 is generated by the placement process P6 in the placement unit 162, with the marking portion of the segmented marking extraction image I14 positioned in the center. Alternatively, for the purpose of data augmentation, the marking portion of the segmented marking extraction image I14 may be randomly shifted from the center. The training image I16 has the input size of the trained model 108A.

[0152] In the case of an oval lock Figure 13 is a diagram illustrating the generation of training data images according to the second embodiment, and shows the case of an elliptic tablet. In the case of an elliptic tablet, the region of the elliptic tablet is divided in half along the short axis to generate the training data images.

[0153] F13A in Figure 13 shows the front surface of the elliptical lock TE1 with its major axis parallel to the horizontal direction in Figure 13. The front surface of the elliptical lock TE1 has the inscription "FF222" engraved on it. F13B in Figure 13 shows the engraved inscription extraction image I21 of the front surface of the elliptical lock TE1. The engraved inscription extraction image I21 is obtained by the engraved inscription extraction process P1 in the engraved inscription extraction image acquisition unit 152, which extracts the engraved inscription from the captured image of the front surface of the elliptical lock TE1. Here, the horizontal center of the engraved inscription extraction image I21 in Figure 13 coincides with the center of the elliptical lock TE1 in the major axis direction.

[0154] F13C in Figure 13 shows the segmented marking extraction images I22 and I23. The segmented marking extraction images I22 and I23 are obtained by the cutting process P4 in the cutting unit 158, which cuts the marking extraction image I21 along the center line LC parallel to the minor axis direction of the elliptical lock TE1. Furthermore, F13C in Figure 13 shows the results RT11 and RT12, respectively, of the determination process P5 in the determination unit 160, which determines the segmented marking extraction images I22 and I23. Results RT11 and RT12 are adopted as training data, respectively.

[0155] F13D in Figure 13 shows the segmented marking extraction image I22. F13E in Figure 13 shows the training image I24 generated from the segmented marking extraction image I22. The training image I24 is generated by the placement process P6 in the placement unit 162, with the marking portion of the segmented marking extraction image I22 positioned in the center. Alternatively, for the purpose of data augmentation, the marking portion of the segmented marking extraction image I22 may be randomly shifted from the center. The training image I24 has the input size of the trained model 108A. Although not explained here, the placement process P6 also generates training images from the segmented marking extraction image I23.

[0156] Figure 13, F13F, shows the front surface of elliptical tablet TE2, which has a similar marking to elliptical tablet TE1. The front surface of elliptical tablet TE2 has the marking "FF333". Elliptical tablets TE1 and TE2 share the "FF" in their markings. Therefore, the segmented marking extraction image I22 can also be generated from the marking extraction image of elliptical tablet TE2. In contrast, by training the system with the marking extraction image (or segmented marking extraction image) of elliptical tablet TE1 and the marking extraction image (or segmented marking extraction image) of elliptical tablet TE2, it is expected that both elliptical tablets TE1 and TE2 will be output as candidates. As a result, although it is not possible to automatically identify the drug type, it has the effect of narrowing down the candidates and presenting them to the user.

[0157] Figure 14 is a diagram illustrating another example of image generation from training data for elliptic tablets.

[0158] F14A in Figure 14 shows the front surface of the elliptical lock TE3. The front surface of the elliptical lock TE3 has the inscription "qq" engraved on it. F14B in Figure 14 shows the engraved marking extraction image I31. The engraved marking extraction image I31 is obtained by the engraved marking extraction process P1 in the engraved marking extraction image acquisition unit 152, which extracts the engraved markings from the captured image of the front surface of the elliptical lock TE3.

[0159] F14C in Figure 14 shows the segmented marking extraction images I32 and I33. The segmented marking extraction images I32 and I33 are obtained by the cutting process P4 in the cutting unit 158, which cuts the marking extraction image I31 along the center line LC. Also, F14C in Figure 14 shows the results RT21 and RT22, respectively, of the determination process P5 in the determination unit 160, which determines the segmented marking extraction images I32 and I33. Result RT21 is adopted as training data, and result RT22 is not adopted as training data.

[0160] Thus, the marking information in the segmented marking extraction image I33 is insufficient for drug type identification and is therefore excluded from the training data. The certain criteria used in the determination process P5 in the determination unit 160 may be the same as those for the round tablets.

[0161] [Learning Phase] Figure 15 is a flowchart showing the learning method for the trained model 108A according to the second embodiment in the learning device 120.

[0162] In step S21, the learning device 120 prepares a large number of full tablet imprinted and printed images linked to drug type information from the database 124.

[0163] In step S22, the learning device 120 takes a simulated half-tablet imprint extraction image as input and outputs the drug type to train the recognition device 140. The simulated half-tablet imprint extraction image is, for example, a divided imprint extraction image generated by randomly dividing a full tablet imprint extraction image into two using image processing to create various patterns of appearance. Furthermore, as explained using Figures 12 to 14, the divided imprint extraction image is used as training data in the determination process P5 of the determination unit 160.

[0164] By repeatedly performing learning until the difference between the output of the recognition unit 140 and the correct data becomes small, a trained recognition unit 140 with optimized parameters is generated. As a result, in step S23, the trained model 108A, which is a drug type identification AI, is completed. The trained model 108A thus trained becomes a drug type identification AI specifically for half tablets and is a different model from the drug type identification AI for full tablets.

[0165] [Utilization Phase] Figure 16 is a flowchart showing a drug identification method according to the second embodiment in the drug identification device 100.

[0166] In step S31, the drug identification device 100 photographs multiple tablets with the out camera 22. The multiple tablets may be packaged together in individual packets. The image acquisition unit 102 acquires the captured image of the multiple tablets.

[0167] In step S32, the drug detection unit 104 detects multiple tablet regions from the image captured in step S31.

[0168] The processing in steps S33 to S36 is repeated for each region of the tablet detected in step S32.

[0169] In step S33, the drug detection unit 104 extracts an image of one tablet to be identified from the image captured in step S31.

[0170] In step S34, the drug detection unit 104 determines from the tablet image extracted in step S33 whether the tablet shown is a half tablet or not. If it is determined in step S34 that it is a full tablet, i.e., if the determination in step S34 is No, the processing for this tablet is terminated and the process moves to step S33 for a different tablet. If it is determined in step S34 that it is a half tablet, i.e., if the determination in step S34 is Yes, the process moves to step S35.

[0171] In step S35, the marking extraction unit 106 acquires a marking extraction image from the tablet image extracted in step S33.

[0172] In step S36, the drug type recognition unit 108 inputs the imprinted image extracted in step S35 into the trained model 108A generated in steps S21 to S23, and obtains multiple drug type candidates for half tablets. The candidate output unit 110 displays on the touch panel display 14 multiple drug type candidates obtained by the drug type recognition unit 108 that have relatively high scores.

[0173] The confirmation unit 112 confirms the drug candidate selected by the user from among the displayed list of drug candidates as the correct drug.

[0174] The drug identification device 100 completes the processing in steps S33 to S36 for all tablets detected in step S32, and then terminates the processing shown in this flowchart.

[0175] [Mechanism of Action / Effect] Thus, in the second embodiment, the use of full tablet images for learning is the same as in the first embodiment, but the learning method has been improved to increase the accuracy of half-tablet identification. The second embodiment has the following features.

[0176] (Feature B1) This is a machine learning-based identification method that uses an image with a mark extracted from it as the input image.

[0177] (Feature B2) The pre-trained model 108A is trained using full-drug imprint extraction images. However, during training, it is trained using pseudo-half-drug imprint extraction images generated from full-drug imprint extraction images.

[0178] (Feature B3) For images generated by Feature B2, if the amount of information is low based on the pixel values, it will be automatically excluded from the training data.

[0179] (Feature B4) Presents multiple drug candidates with a score or probability above a certain level.

[0180] According to the second embodiment, problems 1 and 2 can be solved by feature B1, and problem 3 can be solved by feature B2. Furthermore, according to the second embodiment, problem 3 can be solved by features B2 and B3, and problem 5 can be solved by feature B4.

[0181] <Third Embodiment> [Learning Phase] Figure 17 is a flowchart showing the learning method for the trained model 108A according to the third embodiment in the learning device 120.

[0182] In step S41, the learning device 120 prepares a large number of full tablet imprinted and printed images linked to drug type information from the database 124.

[0183] In step S42, the learning device 120 receives an image of the imprinted markings of a full tablet, similar to that of the first embodiment, and an image of the imprinted markings of a pseudo-half tablet, similar to that of the second embodiment, as input, and outputs the drug type, thereby training the recognition device 140.

[0184] By repeatedly performing learning until the difference between the output of the recognition unit 140 and the correct data becomes small, a trained recognition unit 140 with optimized parameters is generated. As a result, in step S43, the trained model 108A, which is a drug type identification AI, is completed. The trained model 108A thus trained becomes a drug type identification AI that can distinguish both full tablets and partial drugs.

[0185] [Utilization Phase] Figure 18 is a flowchart showing a drug identification method according to the third embodiment in the drug identification device 100.

[0186] In step S51, the drug identification device 100 photographs multiple tablets with the out camera 22. The multiple tablets may be packaged together in individual packets. The image acquisition unit 102 acquires the captured image of the multiple tablets.

[0187] In step S52, the drug detection unit 104 detects multiple tablet regions from the image captured in step S51.

[0188] The processing in steps S53 to S55 is repeated for each area of ​​the tablet detected in step S52.

[0189] In step S53, the drug detection unit 104 extracts an image of one tablet to be identified from the image captured in step S51.

[0190] In step S54, the marking extraction unit 106 acquires a marking extraction image from the tablet image extracted in step S53.

[0191] In step S55, the drug type recognition unit 108 inputs the marking extraction image acquired in step S54 into the trained model 108A generated in steps S41 to S43, and acquires multiple candidate drug types for the tablet. The candidate output unit 110 displays on the touch panel display 14 multiple candidate drug types with relatively high scores from the multiple drug type candidates for the tablet acquired by the drug type recognition unit 108.

[0192] The confirmation unit 112 confirms the drug candidate selected by the user from among the displayed list of drug candidates as the correct drug.

[0193] The drug identification device 100 completes the processing in steps S53 to S55 for all tablets detected in step S52, and then terminates the processing shown in this flowchart.

[0194] [Mechanism of Action / Effect] Thus, in the third embodiment, the learning method is devised to improve the accuracy of identifying half tablets. According to the third embodiment, it is possible to have a drug type identification AI that can be used for both full tablets and half tablets, eliminating the need to classify each tablet cut image as either a full tablet or a partial tablet, or to separate the processing for full tablets from that for partial tablets. The third embodiment has the following features.

[0195] (Feature C1) This is a machine learning-based identification method that uses an image with a mark extracted from it as the input image.

[0196] (Feature C2) For training the pre-trained model 108A, in addition to the full drug marking extraction images, pseudo half-tablet marking extraction images generated from the full drug marking extraction images are mixed in and trained simultaneously.

[0197] (Feature C3) For images generated by Feature C2, if the amount of information is low based on the pixel values, it will be automatically excluded from the training data.

[0198] (Feature C4) Presents multiple drug candidates with a score or probability above a certain level.

[0199] According to the third embodiment, problems 1 and 2 can be solved by feature C1, and problem 3 can be solved by feature C2. Furthermore, according to the third embodiment, problem 3 can be solved by features C2 and C3, and problem 5 can be solved by feature C4.

[0200] <Fourth Embodiment> [Learning Phase] Figure 19 is a flowchart showing the learning method for the trained model 108A according to the fourth embodiment in the learning device 120.

[0201] In step S61, the learning device 120 prepares a large number of full tablet imprinted and printed images linked to drug type information from the database 124.

[0202] In step S62, the learning device 120, similar to the first embodiment, takes an image of the imprinted markings of a full tablet as input and outputs the drug type to train the recognition device 140.

[0203] By repeatedly performing learning until the difference between the output of the recognition unit 140 and the correct data becomes small, a trained recognition unit 140 with optimized parameters is generated. As a result, in step S63, a trained model 108A, which is a drug type identification AI that identifies the drug type of a full tablet, is completed.

[0204] [Utilization Phase] Figure 20 is a flowchart showing a drug identification method according to the fourth embodiment in the drug identification device 100.

[0205] In step S71, the drug identification device 100 captures a half-tablet image with the out-camera 22. The image acquisition unit 102 acquires the captured image of the half-tablet.

[0206] Figure 21 is a diagram illustrating an example of a half-tablet image acquired in step S71. F21A in Figure 21 shows half-tablets TH11 (an example of the "first partial drug") and TH12 (an example of the "second partial drug"), which were originally one full tablet TF (see Figure 1, an example of the "first drug"), placed side by side with their dividing lines physically close together to recreate the full tablet TF before splitting. F21B in Figure 21 shows an image I41 (an example of the "third image") taken with half-tablets TH11 and TH12 placed side by side as shown in F21A. In step S71, the image acquisition unit 102 acquires the image I41 taken in this manner.

[0207] Returning to the explanation of Figure 20, in step S72, the drug detection unit 104 detects the region of half tablet TH11 and the region of half tablet TH12 from the captured image I41 acquired in step S71.

[0208] The processing in steps S73 to S75 is repeated for each area of ​​the full tablet detected in step S72.

[0209] In step S73, the drug detection unit 104 extracts an image of one tablet to be identified from the captured image I41 acquired in step S71.

[0210] In step S74, the marking extraction unit 106 processes at least the area of ​​half tablet TH11 and the area of ​​half tablet TH12 of the tablet image extracted in step S73 to obtain a marking extraction image (an example of a "second marking extraction image").

[0211] In step S75, the drug type recognition unit 108 inputs the marking extraction image acquired in step S74 into the trained model 108A generated in steps S51 to S53, and acquires multiple candidate drug types for the original full tablet TF before it was split into half tablets TH11 and half tablets TH12. The candidate output unit 110 displays on the touch panel display 14 multiple candidate drug types from the full tablet TF acquired by the drug type recognition unit 108 that have relatively high scores.

[0212] The confirmation unit 112 confirms the drug candidate selected by the user from among the displayed list of drug candidates as the correct drug.

[0213] The processes in steps S73 to S75 are performed for all reproduced full tablets detected in step S72, and the drug identification device 100 terminates the process in this flowchart.

[0214] Figure 22 illustrates another example of a half-tablet image acquired in step S71. F22A in Figure 22 shows separately captured images I42 of half-tablet TH11 (an example of a "first image") and I43 of half-tablet TH12 (an example of a "second image"). F22B in Figure 22 shows a composite image I44 obtained by aligning the dividing lines of the half-tablet TH11 region of image I42 and the half-tablet TH12 region of image I43 to reproduce the original full tablet TF. In step S71, the image acquisition unit 102 acquires the composite image I44 thus combined. The image acquisition unit 102 may also acquire the composite image I44 by combining images I42 and I43 using a dedicated GUI (Graphical User Interface).

[0215] In step S72, the drug detection unit 104 detects the region of the full tablet TF from the composite image I44 acquired in step S71. In step S73, the drug detection unit 104 extracts the tablet image to be identified from the composite image I44 acquired in step S71.

[0216] In the subsequent step S74, the marking extraction unit 106 processes at least the area of ​​the full tablet TF of the tablet image extracted in step S73 to obtain a marking extraction image (an example of the "first marking extraction image"). Preferably, the marking extraction unit 106 processes the area of ​​the dividing line so as not to extract it as markings.

[0217] Finally, in step S75, the drug type recognition unit 108 inputs the imprinted image obtained in step S74 into the trained model 108A to obtain multiple drug type candidates for the full tablet TF. The candidate output unit 110 displays on the touch panel display 14 the multiple drug type candidates for the full tablet TF obtained by the drug type recognition unit 108 that have relatively high scores. The confirmation unit 112 confirms the drug type candidate selected by the user from the displayed multiple drug type candidates as the correct drug.

[0218] [Mechanism of Action / Effect] Thus, in the fourth embodiment, the AI ​​for identifying half a tablet can distinguish between full tablets and drug types. The fourth embodiment has the following features.

[0219] (Feature D1) This is a machine learning-based classification method that uses an image with a mark extracted from it as the input image.

[0220] (Feature D2) During training of the pre-trained model 108A, images of the markings on the full tablet are used, while during inference, only partial information is input. This utilizes the properties of deep learning that allow the correct drug type to be inferred even with this input.

[0221] (Feature D3) Presents multiple drug candidates with a score or probability above a certain level.

[0222] (Feature D4) The input image will be either (1) an image in which multiple divided tablets that were originally one full tablet are photographed separately and then arranged on a dedicated GUI to resemble one full tablet, or (2) an image in which multiple divided tablets that were originally one full tablet are physically arranged to resemble one full tablet and then photographed.

[0223] According to the fourth embodiment, problems 1 and 2 can be solved by features D1, D2, and D4, and problem 3 can be solved by features D2 and D4. Furthermore, according to the fourth embodiment, problem 4 can be solved by feature D4, and problem 5 can be solved by features D3 and D4.

[0224] <Other> Up to this point, we have explained the identification of half a tablet as an example of identifying partial drugs, but similarly, the identification of a quarter tablet can also be done by training a pre-trained model and using that trained model.

[0225] Up to this point, we have described an example in which a smartphone 10 alone constitutes a drug identification device 100 that identifies the correct drug among the target drugs to be identified by engraved markings. However, the drug identification device 100 may also consist of a smartphone 10 and a server that can communicate with the smartphone 10, or it may consist of a server alone.

[0226] The drug identification program and the learning program can also be provided stored on a non-temporary recording medium such as a CD-ROM (Compact Disk-Read Only Memory).

[0227] The technical scope of the present invention is not limited to the scope described in the embodiments above. The configurations and other elements in each embodiment can be appropriately combined with those in each embodiment without departing from the spirit of the present invention. [Explanation of symbols]

[0228] 10… Smartphone 12…Cabinet 14…Touch panel display 16...Speaker 18… Microphone 20… Front camera 22…Rear camera 24...Light 26…Switch 28…CPU 30… Wireless Communication Department 32...Telephone section 34…Memory 36...Internal storage 38…External storage unit 40...External input / output section 42…GPS receiver 44...Power supply section 100… Drug identification device 102...Image acquisition unit 104... Drug detection unit 106…Engraving print extraction part 108…Drug recognition section 108A... Pre-trained model 110... Candidate output section 112...Determined part 120...Learning device 122...Training data acquisition unit 124…Database 126...Operation unit 128...Display section 130...CPU 132...RAM 134...ROM 140...Recognizer 140A…Input layer 140B…Middle layer 140C…Output layer 142...Error calculation section 144...Parameter Control Unit 152...Engraving print extraction image acquisition unit 154... Random rotation section 156... Random translation section 158…cutting section 160…Judgment section 162...Arrangement section BG1…Background area I01…Image of engraved markings extracted I02…Image extracted from engraved markings I03…Image extracted from engraved markings I11…Image of engraved markings extracted I12…Image of extracted engraving print I13…Image of engraved markings extracted I14…Image of extracted divided engraving print I15…Image of extracted divided engraving print I16…Learning Images I21…Image of extracted engraving print I22... Extracted image of divided engraving print I23... Extracted image of divided engraving print I24... Learning image I31…Image of engraved markings extracted I32... Extracted image of divided engraving print I33... Extracted image of divided engraving print I41...Photographed image I42...Photographed image I43...Photographed image I44... Composite image LC…Center line P1…Engraving and printing extraction process P2... Random rotation process P3... Random translation process P4…Cutting process P5... Judgment process P6… Placement Process RT1…Result RT2…Result RT11…Result RT12…Result RT21…Result RT22…Result S1-S3... Learning process steps S11-S16…Process for drug identification method S21-S23... Learning Method Process S31-S36…Process for drug identification method S41-S43…Learning Method Process S51-S55…Process for drug identification method S61-S63…Learning Method Process S71-S75…Process for drug identification method ST1…GPS satellite ST2…GPS satellite TC... Round lock TE1…Oval Tablet TE2…Oval Tablet TE3…Oval Tablet TF...Full Tablets TH1…half a tablet TH2…half tablet TH3…half tablet TH4…half a tablet TH5…half a tablet TH6…half tablet TH11…half tablet TH12…half tablet

Claims

1. At least one processor, At least one memory for storing instructions to be executed by the aforementioned at least one processor, Equipped with, The aforementioned at least one processor is An image is taken of an identifiable drug to which an imprint and / or marking has been added, and an image is taken of a partial drug which is a part of a drug that has been divided into multiple parts, The region of the partial drug is detected from the captured image, By processing at least the region of the partial drug in the captured image, an image of the markings and / or prints of the partial drug is obtained. The first trained model is input to extract the engraved marking image to infer the drug type of the partial drug, and multiple candidate drug types of the partial drug are obtained. The aforementioned multiple drug candidates are presented, The first trained model is trained using a first image obtained by extracting the markings and / or printing from a full drug with markings and / or printing added. Drug identification device.

2. The first trained model is trained using a second image which is a part of the first image divided at an arbitrary orientation and position, and which contains more information than a certain standard for the markings and / or printing. The drug identification device according to claim 1.

3. The second image is a portion of the first image that has been rotated and / or translated and then divided. The drug identification device according to claim 2.

4. The first trained model is trained using the first image and a second image which is a part of the first image divided at an arbitrary orientation and position, and which contains more information than a certain standard for the markings and / or prints. The drug identification device according to claim 1.

5. The aforementioned at least one processor is We acquired images in which multiple drugs were captured, The regions of the multiple drugs are detected from the captured image, The region of the partial drug is detected from the regions of the plurality of drugs. The drug identification device according to claim 1.

6. The aforementioned at least one processor is A first image is obtained in which the first partial drug, which is a divided portion of the first drug, is photographed. A second image is taken of a second partial drug, which is a divided part of the first partial drug, and which is different from the first partial drug. A composite image is obtained by aligning the region of the first partial drug in the first captured image and the region of the second partial drug in the second captured image. The region of the first drug is detected from the composite image. A first marking and printing extraction image is obtained by processing at least the region of the first drug in the composite image to extract the marking and / or printing of the first drug. The first pre-trained model is input to infer the drug type of the first drug and obtain candidates for the drug type of the first drug. The drug identification device according to claim 1.

7. The aforementioned at least one processor is A third image is obtained in which a first partial drug, which is a divided portion of the first drug, and a second partial drug, which is a divided portion of the first drug and is different from the first partial drug, are photographed side by side. The region of the first partial drug and the region of the second partial drug are detected from the third captured image. A second marking and printing extraction image is obtained by processing at least the region of the first partial drug and the region of the second partial drug of the third captured image to extract the markings and / or printing of the first partial drug and the second partial drug. The second engraved print extraction image is input to the first trained model to infer the drug type of the first drug and obtain candidate drug types for the first drug. The drug identification device according to claim 1.

8. The aforementioned at least one processor is The first trained model obtains several drug candidates that have relatively high scores. A drug identification device according to any one of claims 1 to 7.

9. At least one processor, At least one memory for storing instructions to be executed by the aforementioned at least one processor, Equipped with, The aforementioned at least one processor is A first trained model is trained using a training dataset consisting of a first image from which the markings and / or printing have been extracted from a drug that has markings and / or printing added to it, divided into two images at an arbitrary orientation and position, wherein the second image contains more information from the markings and / or printing than a certain standard, and the drug type of the drug is the ground truth data. The first trained model outputs the drug type of the partial drug when an image of a partial drug is input. Learning device.

10. At least one processor, An image is taken of an identifiable drug to which an imprint and / or marking has been added, and an image is taken of a partial drug which is a part of a drug that has been divided into multiple parts, The region of the partial drug is detected from the captured image, By processing at least the region of the partial drug in the captured image, an image of the markings and / or prints of the partial drug is obtained. The first trained model is input to extract the engraved marking image to infer the drug type of the partial drug, and multiple candidate drug types of the partial drug are obtained. The aforementioned multiple drug candidates are presented, The first trained model is trained on a first image obtained by extracting the markings and / or printing from an undivided full drug with markings and / or printing added. Drug identification method.

11. A program that causes a computer to execute the drug identification method described in claim 10.

12. A non-temporary and computer-readable recording medium on which the program described in claim 11 is recorded.

Images