Drug identification device, drug identification method and program, drug placement stand, lighting device, learning device, and method for manufacturing output

Abstract

Provided are a drug identification device enabling highly accurate, easy, and highly usable identification of an engraved and / or printed drug, a drug identification method and program, a drug identification system, a drug loading platform, an illumination device, an imaging assistance device, a learned model, and learning device. A region of an engraved and / or printed drug to be identified is detected from a captured image including the drug to be identified, the region of the drug to be identified is processed to acquire an engraved / printed mark extraction image in which an engraved and / or printed mark of the drug to be identified is extracted, the engraved / printed mark extraction image is input to infer the drug type of the drug to be identified, and a drug type candidate for the drug to be identified is acquired.

Description

[Technical field]

[0001] The present invention relates to a drug identification device, a drug identification method and program, a drug identification system, a drug placement stand, a lighting device, an imaging assistance device, a trained model, and a learning device, and in particular to a technology for identifying drugs from photographed images of the drugs. [Background technology]

[0002] There is known a drug identification device that identifies a drug from a captured image of the drug (see Patent Documents 1 to 3). In addition, Non-Patent Document 1 describes a drug image recognition system that uses a smartphone. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2015-535108 [Patent Document 2] Special Publication No. 2016-523405 [Patent Document 3] Special Publication No. 2015-523561 [Non-patent literature]

[0004] [Non-Patent Document 1] MobileDeepPill: A Small-Footprint Mobile Deep Learning System for Recognizing Unconstrained Pill Images,Xiao Zeng, K. Cao, Mi Zhang,Published 2017,Computer Science,Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to realize a drug identification device that uses a mobile device with a camera, has simple or no need for photography / lighting equipment, can be used in a variety of lighting environments, and is easy to operate and use, making it highly user-friendly.

[0006] As of January 2021, there are just under 10,000 types of tablet and capsule-type medicines in Japan. With a few exceptions (plain tablets without markings or printing, etc.), each medicine is characterized by a combination of external information (shape (circle, ellipse, etc.), size (diameter, major axis, minor axis, etc.), identification information from markings or printing added to the medicine, and color information of the medicine. In other words, it is theoretically possible to identify the type of medicine by extracting this information from an image of the medicine.

[0007] Although color information of a drug is an information source that characterizes the drug, when assuming use under various light environments, color reproducibility on individual photographed images of the same drug cannot be expected. In addition, regarding external shape information, the majority of drugs are circular or elliptical, and almost all drugs are between about 4 millimeters and about 3 centimeters in size, and there are many drugs with similar shapes. Furthermore, users do not necessarily photograph drugs from directly above, and drugs have thickness and various shapes in the cross-sectional direction, so drug shape information on the photographed image may be information with low reproducibility and cannot be said to be reliable information.

[0008] On the other hand, the identification information added to the surface of the medicine contains a wealth of information (combinations of multiple letters and symbols, layout, etc.) that characterizes each individual medicine. There are mainly two types of identification information added to the surface of medicine: engraved type and printed type. It is relatively easy to extract the printed information from the photographed image of the printed type medicine, because the contrast between the printed and non-printed parts on the medicine is high. On the other hand, with the engraved type medicine, it can be difficult to ensure the contrast between the engraved and non-engraved parts on the medicine from the photographed image. This tendency is particularly noticeable when a dedicated lighting system is not used or when the markings are shallow / thin / small, etc.

[0009] Considering these circumstances, in order to achieve highly accurate drug identification under various lighting environments, the challenge is to use the imprinted or printed information as the primary information source, with color information and size / shape information as secondary information sources, and then robustly extract identification information from imprinted drugs.

[0010] In the invention described in Non-Patent Document 1, in order to extract the marking information, the marking or printing information of the medicine is obtained from the gradient information (edge ​​information) by image processing. In this case, the gradient information contains not only the marking or printing information but also the outline information of the medicine. Although the outline information is also useful information for identifying medicine, as mentioned above, in the case of marked type medicines (especially when the marking is shallow / small / thin), the marking information useful for identification decreases relative to the outline information, leading to a decrease in accuracy.

[0011] The present invention has been made in consideration of the above circumstances, and aims to provide a drug identification device, a drug identification method and program, a drug identification system, a drug placement stand, a lighting device, a photography assistance device, a trained model, and a learning device that enable highly accurate, simple, and easy-to-use identification of drugs that have been engraved and / or printed. [Means for solving the problem]

[0012] One aspect of the drug identification device for achieving the above object is a drug identification device having an image acquisition unit that acquires a captured image of a drug to be identified with a mark and / or print, a drug detection unit that detects the area of ​​the drug to be identified from the captured image, an imprint extraction unit that processes at least the area of ​​the drug to be identified in the captured image to acquire an imprint extraction image in which the mark and / or print of the drug to be identified is extracted, and a first drug type recognition unit that inputs the imprint extraction image to infer the drug type of the drug to be identified and acquires candidates for the drug type of the drug to be identified. According to this aspect, it is possible to realize highly accurate, simple, and highly usable identification of drugs to which a mark and / or print has been added.

[0013] It is preferable that the imprint extraction unit includes a first trained model that inputs a first image of the drug with the imprint and / or print and outputs a second image in which the imprint and / or print of the drug is extracted, and the first drug type recognition unit includes a second trained model that inputs the second image and outputs the drug type of the drug corresponding to the imprint and / or print of the drug. By using the trained model, it is possible to obtain a highly accurate imprint extraction image and to obtain candidates for the drug type of the drug to be identified in a short time.

[0014] It is preferable that the apparatus further includes a second drug type recognition unit that inputs at least the area of ​​the drug to be identified in the captured image and infers the drug type of the drug to be identified, the first drug type recognition unit integrates the inference result of the second drug type recognition unit with the inference result of the first drug type recognition unit to obtain candidates for the drug type of the drug to be identified, and the second drug type recognition unit includes a third trained model that inputs the first image and outputs the drug type of the drug. By integrating the inference result of the first drug type recognition unit with the drug type of the drug to be identified that is inferred from the area of ​​the drug to be identified based on shape information, color information, etc., it is possible to obtain candidates for the drug type of the drug to be identified with higher accuracy.

[0015] It is preferable that the device has a drug-associated information acquisition unit that acquires drug-associated information including at least one of the shapes, sizes, and colors of a plurality of drugs, and the first drug type recognition unit acquires candidates for the drug type of the drug to be identified by integrating the drug-associated information with the inference result of the first drug type recognition unit. By integrating the drug-associated information with the inference result of the first drug type recognition unit, it is possible to acquire candidates for the drug type of the drug to be identified with higher accuracy.

[0016] The image acquisition unit preferably has an image correction unit that acquires a captured image of the drug to be identified and the marker, and standardizes the shooting distance and shooting viewpoint of the captured image based on the marker to acquire a standardized image, and the drug detection unit preferably detects the area of ​​the drug to be identified from the standardized image. By using the standardized image in which the shooting distance and shooting viewpoint have been standardized, it is possible to detect a certain area of ​​the drug to be identified regardless of the shooting environment. In addition, template matching with a master image acquired at the standardized shooting distance and shooting viewpoint is possible.

[0017] The image acquisition unit preferably acquires a captured image in which a plurality of ArUco markers, a plurality of circular markers, or a plurality of rectangular markers are captured. By using the ArUco marker, a standardized image can be acquired more appropriately and robustly. By using the circular marker or the rectangular marker, detection based on deep learning becomes easier.

[0018] It is preferable that the circular marker includes concentric circles and the quadrangular marker includes concentric quadrangles, which makes it easier to determine the center point coordinates of the marker and allows a standardized image to be obtained that is robust against various noises.

[0019] It is preferable that the image acquisition unit acquires a captured image of the drug to be identified and a reference gray color, and the image correction unit performs color correction of the captured image based on the gray color. By using the reference gray color, the color correction of the captured image can be performed appropriately. Furthermore, it is expected that the contrast between the engraved part and the non-engraved part can be ensured, especially in the case of engraved type drugs.

[0020] One aspect of the drug identification system for achieving the above object is a drug identification system having a mobile terminal and a server capable of communicating with each other, the mobile terminal has an image acquisition unit that acquires a captured image of a drug to be identified with a mark and / or print added thereto, the mobile terminal or the server has a drug detection unit that detects the area of ​​the drug to be identified from the captured image, the server has an imprint-print extraction unit that processes at least the area of ​​the drug to be identified in the captured image to acquire an imprint-print extraction image in which the imprint and / or print of the drug to be identified is extracted, and a first drug type recognition unit that inputs the imprint-print extraction image to infer the drug type of the drug to be identified and acquires candidates for the drug type of the drug to be identified, and the mobile terminal has a display control unit that displays the candidates for the drug type of the drug to be identified on a display. According to this aspect, it is possible to realize highly accurate, simple, and highly usable identification of drugs to which a mark and / or print is added.

[0021] It is preferable that the imprint extraction unit includes a first trained model that inputs a first image of the drug with the imprint and / or print and outputs a second image in which the imprint and / or print of the drug is extracted, and the first drug type recognition unit includes a second trained model that inputs the second image and outputs the drug type of the drug corresponding to the imprint and / or print of the drug. By using the trained model, it is possible to obtain a highly accurate imprint extraction image and to obtain candidates for the drug type of the drug to be identified in a short time.

[0022] The server preferably has a second drug type recognition unit that inputs at least the area of ​​the drug to be identified in the captured image to infer the drug type of the drug to be identified, and the first drug type recognition unit integrates the inference result of the first drug type recognition unit with the inference result of the second drug type recognition unit to obtain candidates for the drug type of the drug to be identified, and the second drug type recognition unit preferably includes a third trained model that inputs the first image and outputs the drug type of the drug. By integrating the inference result of the first drug type recognition unit with the drug type of the drug to be identified inferred from the area of ​​the drug to be identified, it is possible to obtain candidates for the drug type of the drug to be identified with higher accuracy.

[0023] The server preferably has a drug-associated information acquisition unit that acquires drug-associated information including at least one of the shapes, sizes, and colors of a plurality of drugs, and the first drug type recognition unit preferably acquires candidates for the drug type of the drug to be identified by integrating the drug-associated information with the inference result of the first drug type recognition unit. By integrating the drug-associated information with the inference result of the first drug type recognition unit, it is possible to acquire candidates for the drug type of the drug to be identified with higher accuracy.

[0024] It is preferable that the mobile terminal has a camera and a display, the image acquisition unit acquires an image of the drug to be identified and the marker photographed by the camera, the mobile terminal or the server has an image correction unit that standardizes the photographing distance and the photographing viewpoint of the photographed image based on the marker to acquire a standardized image, and the drug detection unit detects the area of ​​the drug to be identified from the standardized image. This allows the server to recognize candidates for the drug type of the drug to be identified photographed by the camera of the mobile terminal and display them on the display of the mobile terminal.

[0025] It is preferable that the medicine to be identified is placed on a gray-colored placement surface, and the mobile terminal has an exposure correction unit that corrects the exposure of the camera based on the gray color. By performing exposure correction using a reference gray color, the color of the medicine can be appropriately acquired. Furthermore, it is expected that the contrast between the engraved part and the non-engraved part can be ensured, especially in the case of engraved type medicine.

[0026] The image acquisition unit preferably acquires images captured at a standard shooting distance and shooting viewpoint. By using images captured at a standard shooting distance and shooting viewpoint, shape information of the medicine can be extracted with high reproducibility, and improvement in identification accuracy can be expected.

[0027] It is preferable that the image acquisition unit acquires a captured image including a plurality of identification target drugs, the drug detection unit detects the regions of each of the plurality of identification target drugs, the imprint extraction unit acquires a plurality of imprint extraction images corresponding to each of the plurality of identification target drugs, and the first drug type recognition unit acquires drug type candidates corresponding to each of the plurality of identification target drugs. In this way, even if a captured image is taken in which a plurality of identification target drugs are simultaneously captured, it is possible to acquire drug type candidates corresponding to each of the plurality of identification target drugs.

[0028] It is preferable that the first drug type recognition unit obtains a plurality of candidate drug types for the drug to be identified, obtains a master image for each of the candidate drug types, and performs template matching between the extracted image of the imprinted mark and the master image. By using template matching, it is possible to narrow down the candidate drug types for the drug to be identified with higher accuracy. It is preferable that template matching is performed in the rotation direction and the parallel movement direction.

[0029] The display control unit preferably has a display that displays at least one of an image of at least the region of the drug to be identified in the captured image, an image of the extracted imprint, and an image of the drug to be identified with the imprint and / or print highlighted, and the display control unit further preferably causes a master image of a candidate drug type for the drug to be identified to be selectable on the display. By selectably displaying the master image of the candidate drug type for the drug to be identified on the display, the user can easily select the correct drug for the drug to be identified with high visibility.

[0030] The display control unit preferably displays at least one of the images of the region of the drug to be identified in the captured image, the image of the extracted marking and / or the image of the drug to be identified with the marking and / or the image of the drug to be identified with the highlighted marking and / or the image of the candidate drug type of the drug to be identified, with the marking and / or the image aligned in direction. By displaying the marking and / or the image aligned in direction, the user can easily select the correct drug from the drug to be identified.

[0031] It is preferable that the display control unit displays a search window on the display into which a character string can be input, and the first drug type recognition unit identifies the drug type of the drug to be identified based on the character string input in the search window. By using the search window, even if the drug candidates obtained by drug identification based on the captured image do not include the correct drug, the correct drug of the drug to be identified can be identified with high accuracy.

[0032] One aspect of the medicine mounting table for achieving the above object is a medicine mounting table used for taking an image of the above medicine identification device or medicine identification system, the medicine mounting table having a mounting surface on which the medicine to be identified is placed, the mounting surface being gray in color, and having a plurality of markers arranged on the mounting surface. According to this aspect, a standardized image can be appropriately obtained, color correction of the captured image can be appropriately performed, and the contrast between the engraved portion and the non-engraved portion can be ensured for engraved type medicines.

[0033] Each of the multiple markers is preferably a circular marker or a rectangular marker, which facilitates deep learning-based detection.

[0034] It is preferable that the circular marker includes concentric circles and the quadrangular marker includes concentric quadrangles, which makes it easier to determine the center point coordinates of the marker and allows a standardized image to be obtained that is robust against various noises.

[0035] The placement surface is preferably provided with a recessed structure on which the drug to be identified is placed. The recessed structure includes a recess, a groove, a dent, and a hole. This allows the drug to be placed thereon. The drug placement table may be made of paper, synthetic resin, fiber, rubber, or glass. The drug placement table can be used for identifying (distinguishing) drugs brought by hospitalized patients, drugs taken by patients when they are transported by ambulance, and remaining drugs at home, or for sorting drugs that have been packaged but are unused due to a prescription change or the like, or packaged drugs that have been returned.

[0036] One aspect of the illumination device for achieving the above object is an illumination device used for capturing an image of the above drug identification device or drug identification system, the illumination device having a plurality of light sources that irradiate the drug to be identified with illumination light from a plurality of different directions. According to this aspect, it is possible to capture a plurality of images each irradiated with illumination light from a plurality of directions, and it is possible to capture a desired image.

[0037] One aspect of the photographing auxiliary device for achieving the above object is a photographing auxiliary device used for photographing an image of the above-mentioned drug identification device or drug identification system, the photographing auxiliary device having a drug placement table having a placement surface on which the drug to be identified is placed, and a lighting device for irradiating the drug to be identified placed on the placement surface with illumination light, the placement surface being gray in color, a plurality of markers being arranged on the placement surface, and the lighting device having a plurality of light sources for irradiating the drug to be identified from a plurality of different directions. According to this aspect, it is possible to photograph a desired photographing image, to appropriately obtain a standardized image, to appropriately perform color correction of the photographing image, and to ensure contrast between the engraved portion and the non-engraved portion of an engraved type drug.

[0038] One aspect of the drug identification method for achieving the above object is a drug identification method including an image acquisition step of acquiring a photographed image of a drug to be identified having a mark and / or print, a drug detection step of detecting an area of ​​the drug to be identified from the photographed image, an imprint extraction step of acquiring an imprint extraction image in which the mark and / or print of the drug to be identified is extracted by processing at least the area of ​​the drug to be identified in the photographed image, and a first drug type recognition step of inputting the imprint extraction image to infer the drug type of the drug to be identified and acquire candidates for the drug type of the drug to be identified. According to this aspect, it is possible to realize highly accurate, simple, and highly usable identification of drugs to which a mark and / or print has been added.

[0039] The imprint extraction step preferably acquires an imprint extraction image using a first trained model that inputs a first image of a drug with an imprint and / or print and outputs a second image in which the imprint and / or print of the drug is extracted, and the first drug type recognition step preferably infers the drug type of the drug to be identified using a second trained model that inputs the second image and outputs the drug type of the drug corresponding to the imprint and / or print of the drug. By using the trained model, it is possible to acquire an imprint extraction image with high accuracy and to acquire candidates for the drug type of the drug to be identified in a short time.

[0040] One aspect of the program for achieving the above object is a program for causing a computer to execute the above-described drug identification method. This aspect may also include a computer-readable non-transitory storage medium in which this program is recorded. According to this aspect, it is possible to realize highly accurate, simple, and highly usable identification of drugs to which markings and / or printing have been added.

[0041] One aspect of the trained model for achieving the above object is a trained model in which machine learning is performed using a training dataset that includes a second image in which the imprint and / or print of a medicine to which the imprint and / or print has been added is extracted, and a drug type of the medicine corresponding to the imprint and / or print. By outputting the drug type corresponding to the imprint and / or print from an image in which the imprint and / or print of a medicine to which the imprint and / or print has been added is extracted, the drug type of the medicine can be recognized without being affected by the shooting environment.

[0042] It is preferable that noise is added to the second image, which allows a trained model to be trained with robustness against changes in the shooting environment.

[0043] One aspect of the learning device for achieving the above object includes a learning data collection unit that collects a re-learning dataset that is a set of an inscription and / or print extraction image in which the inscription and / or print of the drug to be identified is extracted and information on the correct drug type of the drug to be identified, and a re-learning unit that re-learns a second trained model using the collected re-learning dataset, where the second trained model is a learning device that inputs a second image in which the inscription and / or print of the drug is extracted and outputs the drug type corresponding to the inscription and / or print of the drug. By re-learning the second trained model, the drug type can be output with even greater accuracy.

[0044] Another aspect of the learning device for achieving the above object is a learning device having a learning data collection unit that collects a re-learning dataset that is a set of images of the identification target drugs with markings and / or prints and information on the drug type of the identification target drugs acquired by the above drug identification device or drug identification system, and a re-learning unit that re-learns a third trained model using the collected re-learning dataset. By re-learning the third trained model, it is possible to output the drug type with even greater accuracy.

[0045] Another aspect of the learning device for achieving the above object includes a learning data collection unit that collects a learning data set including a captured image of a drug to be identified with a mark and / or print and information on the drug type of the drug to be identified acquired by the drug identification device or drug identification system, and a learning unit that trains a fourth trained model using the collected learning data set, and the fourth trained model is a learning device that inputs a first image of the drug to which a mark and / or print is added and outputs the drug type of the drug. According to this aspect, by using the fourth trained model newly trained in place of the third trained model, it is possible to output candidates for the drug type of the drug to be identified.

[0046] Another aspect of the drug identification device for achieving the above object is a drug identification device having an image acquisition unit that acquires a captured image of a drug to be identified with a mark and / or print added thereto, a drug detection unit that detects the area of ​​the drug to be identified from the captured image, an imprint extraction unit that processes at least the area of ​​the drug to be identified in the captured image to acquire an imprint extraction image in which the mark and / or print of the drug to be identified is extracted, and a display control unit that displays at least one image of the image of at least the area of ​​the drug to be identified in the captured image, the imprint extraction image, and the image of the drug to be identified with the mark and / or print emphasized, with the mark and / or print oriented upright. According to this aspect, the image related to the drug to be identified can be displayed in an easy-to-view manner for the user by uprighting the mark and / or print.

[0047] The imprint extraction unit preferably includes a first trained model that inputs a first image of the medicine with the imprint and / or print and outputs a second image in which the imprint and / or print of the medicine is extracted. By using the trained model, a highly accurate imprint extraction image can be obtained.

[0048] The display control unit preferably acquires a master image of the drug to be identified, in which the direction of the marking and / or printing is upright, and performs template matching to compare the rotation direction between the master image and at least one of the images of at least the region of the drug to be identified in the captured image, the image of the marking and printing extracted, and the image of the drug to be identified in which the marking and / or printing is emphasized. By using template matching to compare the rotation direction, the direction of the marking and / or printing can be appropriately uprighted.

[0049] The display control unit preferably includes a fifth trained model that inputs a first image of the medicine with the marking and / or printing and outputs a third image in which the marking and / or printing of the medicine is upright. By using the trained model, the marking and / or printing can be properly uprighted.

[0050] The display control unit has a search unit that displays a search window into which character information can be input on the display and searches for candidates for the drug type of the drug to be identified based on the character information input in the search window, and a first drug type recognition unit that inputs an extracted image of the marking print to infer the drug type of the drug to be identified and acquires candidates for the drug type of the drug to be identified, and it is preferable that the display control unit automatically inputs character information indicating the most likely candidate for the drug type of the drug to be identified acquired by the first drug type recognition unit into the search window. By automatically inputting character information indicating the most likely candidate for the drug type of the drug to be identified into the search window, the user can quickly search for the drug to be identified.

[0051] The display control unit preferably displays the candidate drug types of the drugs to be identified searched for by the search unit, thereby allowing the user to visually confirm the search results.

[0052] Another aspect of the drug identification system for achieving the above object is a drug identification system having a mobile terminal and a server capable of communicating with each other, the mobile terminal has an image acquisition unit that acquires a captured image of a drug to be identified with a mark and / or print added thereto, the mobile terminal or the server has a drug detection unit that detects the area of ​​the drug to be identified from the captured image, the server has an inscription and print extraction unit that processes at least the area of ​​the drug to be identified in the captured image to acquire an inscription and print extraction image in which the inscription and / or print of the drug to be identified is extracted, and the mobile terminal has a display control unit that displays at least one image of the image of at least the area of ​​the drug to be identified in the captured image, the inscription and print extraction image, and the image of the drug to be identified with the inscription and / or print emphasized, with the inscription and / or print oriented upright. According to this aspect, the image related to the drug to be identified can be displayed in an easy-to-see manner for the user by inverting the inscription and / or print.

[0053] Another aspect of the drug identification method for achieving the above object is a drug identification method having an image acquisition step of acquiring a photographed image of the drug to be identified to which a mark and / or print has been added, a drug detection step of detecting the area of ​​the drug to be identified from the photographed image, an imprint extraction step of acquiring an imprint extraction image in which the mark and / or print of the drug to be identified is extracted by processing at least the area of ​​the drug to be identified in the photographed image, and a display control step of displaying at least one image of at least the area of ​​the drug to be identified in the photographed image, the imprint extraction image, and the image of the drug to be identified in which the mark and / or print is emphasized, on a display with the mark and / or print oriented upright. According to this aspect, by oriented upright, the image related to the drug to be identified can be displayed in an easy-to-view manner for the user.

[0054] One aspect of the output product for achieving the above object is an output product used for taking a photographed image of the above medicine identification device or the above medicine identification system, the output product having a personal information display area in which information for identifying an individual is displayed, and a placement area for placing the medicine to be identified by the individual. According to this aspect, the individual can be identified and the medicine of the individual can be identified.

[0055] The information for identifying an individual preferably includes a bar code, which can be read by a reader to automate the identification of the individual.

[0056] It is preferable that the color of the placement area is gray, and that a plurality of markers are arranged in the placement area. By taking an image of the medicine placed in the placement area, a standardized image can be appropriately obtained, color correction of the taken image can be appropriately performed, and the contrast between the engraved part and the non-engraved part can be ensured for the engraved type medicine.

[0057] One aspect of the method for producing an output product to achieve the above object is a method for producing an output product having a step of acquiring information for identifying an individual, and a step of arranging and printing a personal information display area and a placement area on a printing medium. According to this aspect, an output product that can identify an individual and identify the individual's medication can be printed on the printing medium. Effect of the Invention

[0058] According to the present invention, it is possible to realize highly accurate, simple and highly usable identification of medicines having markings and / or printing. [Brief description of the drawings]

[0059] [Figure 1] FIG. 1 is a front perspective view of a smartphone. [Diagram 2] FIG. 2 is a rear perspective view of the smartphone. [Diagram 3] FIG. 3 is a block diagram showing the electrical configuration of the smartphone. [Figure 4] FIG. 4 is a block diagram showing the internal configuration of the in-camera. [Diagram 5] FIG. 5 is a block diagram showing a functional configuration of the drug identification device. [Figure 6] FIG. 6 is a flow chart showing steps of a medicine identifying method using the medicine identifying device. [Figure 7] FIG. 7 is a diagram showing the process of the drug identification method. [Figure 8] FIG. 8 is a block diagram showing a functional configuration of the drug identification device. [Figure 9] FIG. 9 is a flow chart showing steps of a medicine identifying method using the medicine identifying device. [Figure 10] FIG. 10 is a diagram showing the process of the drug identification method. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of the medicine identification device. [Figure 12] FIG. 12 is a diagram showing an example of a master image of a certain medicine stored in the master image storage unit. [Figure 13] FIG. 13 is a flow chart showing steps of a medicine identifying method using the medicine identifying device. [Figure 14] FIG. 14 is a diagram showing a screen display of the touch panel display. [Figure 15] FIG. 15 is a diagram showing a screen display of the touch panel display. [Figure 16] FIG. 16 is a diagram showing a screen display of the touch panel display. [Figure 17] FIG. 17 is a diagram showing a screen display of the touch panel display. [Figure 18] FIG. 18 is a diagram showing a screen display of the touch panel display. [Figure 19] FIG. 19 is a diagram showing a screen display on a display of a tablet computer terminal. [Figure 20] FIG. 20 is a diagram showing a screen display on a display of a tablet computer terminal. [Figure 21] FIG. 21 is a diagram showing a screen display on a display of a tablet computer terminal. [Figure 22] FIG. 22 is a diagram showing a screen display on the display of a tablet computer terminal. [Figure 23] FIG. 23 is a top view of the photography auxiliary device. [Figure 24] FIG. 24 is a cross-sectional view taken along line 24-24 of FIG. [Diagram 25] FIG. 25 is a diagram showing an example of a data set of an ArUco marker having four dots vertically and horizontally. [Figure 26] FIG. 26 is a top view of the shooting assist device with the auxiliary light source removed. [Figure 27] FIG. 27 is a top view of the photography auxiliary device. [Figure 28] FIG. 28 is a cross-sectional view taken along line 28-28 of FIG. [Figure 29] FIG. 29 is a top view of the shooting assist device with the auxiliary light source removed. [Diagram 30]FIG. 30 is a diagram showing the configuration of a medicine identification system. [Diagram 31] FIG. 31 is a block diagram showing the functional configuration of the medicine identification system. [Diagram 32] FIG. 32 is a flow chart showing steps of a medicine identification method using the medicine identification system. [Diagram 33] FIG. 33 is a block diagram showing the electrical configuration of the image processing device. [Diagram 34] FIG. 34 is a diagram illustrating an example of a training data set for generating a first trained model. [Diagram 35] FIG. 35 is a block diagram showing the functional configuration of the image learning device. [Diagram 36] FIG. 36 is a diagram illustrating an example of a learning dataset for generating a second trained model. [Figure 37] FIG. 37 is a block diagram showing the functional configuration of the image learning device. [Figure 38] FIG. 38 is a diagram illustrating an example of a learning dataset for generating a third trained model. [Figure 39] FIG. 39 is a block diagram showing a functional configuration of the drug identification device. [Diagram 40] FIG. 40 is a flow chart showing steps of the medication identification method. [Diagram 41] FIG. 41 is a diagram showing a screen display on a touch panel display of a smartphone. [Diagram 42] FIG. 42 is a diagram showing a screen display on a touch panel display of a smartphone. [Diagram 43] FIG. 43 is a diagram showing a screen display on a touch panel display of a smartphone. [Diagram 44] FIG. 44 is a diagram showing a screen display on a touch panel display of a smartphone. [Diagram 45] FIG. 45 is a diagram showing a screen display on a touch panel display of a smartphone. [Figure 46] FIG. 46 is a block diagram showing the configuration of a medicine identification system. [Figure 47] FIG. 47 is a flow chart showing steps of a medication identification method. [Figure 48] FIG. 48 is a diagram showing an example of a brought-medicine identification request form. [Figure 49] FIG. 49 shows how a single dose of a patient's own medication is placed on the placement area of ​​the brought-medicine discrimination request form and photographed. [Figure 50] FIG. 50 is a diagram showing a drug in a captured image and a bounding box of the correct answer data thereof. [Figure 51] FIG. 51 is a diagram for explaining detection of a reference marker when the reference marker is a rectangle. [Figure 52] FIG. 52 is a diagram for explaining marker detection when the reference marker is circular. [Diagram 53] FIG. 53 is a diagram showing a specific example of a circular fiducial marker. [Figure 54] FIG. 54 is a top view of the medicine placing platform using a circular marker as the reference marker. [Figure 55] FIG. 55 is a top view of a medicine placing table using a circular marker according to a modified example. [Figure 56] FIG. 56 is a diagram showing a specific example of a rectangular reference marker. [Figure 57] FIG. 57 is a block diagram showing a functional configuration of a drug identification device realized by a smartphone. [Figure 58] FIG. 58 is a diagram showing an example of an image included in the first learning data set. [Figure 59] FIG. 59 is a diagram showing an example of an image included in the second learning data set. [Figure 60] Figure 60 is a flowchart showing an example of a learning method for the sixth trained model. [Figure 61] Figure 61 is a flowchart showing an example of an inference method using the sixth trained model. [Figure 62] FIG. 62 is a diagram showing a medicine placing stand provided with a recess structure. [Figure 63]FIG. 63 shows a medicine placement platform having a recess structure for capsules. [Figure 64] FIG. 64 shows a medicine placement platform provided with a recessed structure for oval shaped tablets. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0060] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0061] <First embodiment> The drug identification device according to this embodiment is a device that identifies the drug type of a drug to be identified that has been marked and / or printed, and identifies the correct drug. Note that "marking and / or printing" means "either marking or printing, or both marking and printing."

[0062] Here, the marking means that identification information is formed by forming a groove, which is a recessed area, on the surface of the medicine. Carving The marking is not limited to one formed by pressing the surface, but may be one formed by pressing the surface. Also, the marking may include one that does not have an identification function such as a dividing line.

[0063] In addition, "printed" means that identification information is formed by applying edible ink or the like to the surface of the medicine with or without contact. Here, "applied by printing" is synonymous with "applied by printing."

[0064] The drug identification device is mounted on a mobile terminal device as an example. The mobile terminal device includes at least one of a mobile phone, a PHS (Personal Handyphone System), a smartphone, a PDA (Personal Digital Assistant), a tablet computer terminal, a notebook personal computer terminal, and a portable game machine. In the following, a drug identification device configured by a smartphone is taken as an example and described in detail with reference to the drawings.

[0065] [Appearance of the smartphone] Fig. 1 is a front perspective view of a smartphone 10, which is a camera-equipped mobile terminal device according to this embodiment. As shown in Fig. 1, the smartphone 10 has a flat housing 12. The smartphone 10 has a touch panel display 14, a speaker 16, a microphone 18, and an inner camera 20 on the front side of the housing 12.

[0066] The touch panel display 14 includes a display unit that displays images and the like, and a touch panel unit that is disposed in front of the display unit and receives touch inputs. The display unit is, for example, a color LCD (Liquid Crystal Display) panel.

[0067] The touch panel unit is, for example, a capacitive touch panel provided in a planar shape on a light-transmitting substrate body, and has a light-transmitting position detection electrode and an insulating layer provided on the position detection electrode. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to a touch operation by a user. The touch operation includes a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

[0068] The speaker 16 is an audio output unit that outputs audio during a call and when playing back a video. The microphone 18 is an audio input unit to which audio is input during a call and when shooting a video. The front camera 20 is an imaging device that shoots videos and still images.

[0069] Fig. 2 is a rear perspective view of the smartphone 10. As shown in Fig. 2, the smartphone 10 includes an outer camera 22 and a light 24 on the rear surface of the housing 12. The outer camera 22 is an imaging device that captures moving images and still images. The light 24 is a light source that emits illumination light when the outer camera 22 captures images, and is formed of, for example, an LED (Light Emitting Diode).

[0070] 1 and 2, the smartphone 10 is provided with switches 26 on the front and side of the housing 12. The switches 26 are input members that accept instructions from a user. The switches 26 are push-button switches that are turned on when pressed with a finger or the like and turned off by the restoring force of a spring or the like when the finger is released.

[0071] The configuration of the housing 12 is not limited to this, and a configuration having a folding structure or a sliding mechanism may be adopted.

[0072] [Smartphone electrical configuration] The smartphone 10 has, as its main function, a wireless communication function for performing mobile wireless communication between a base station device and a mobile communication network.

[0073] Fig. 3 is a block diagram showing the electrical configuration of the smartphone 10. As shown in Fig. 3, the smartphone 10 has a CPU (Central Processing Unit) 28, a wireless communication unit 30, a call unit 32, a memory 34, an external input / output unit 40, a GPS receiver 42, and a power supply unit 44 in addition to the above-mentioned touch panel display 14, speaker 16, microphone 18, front camera 20, rear camera 22, light 24, and switch 26.

[0074] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates according to the control program and control data stored in the memory 34, and controls each unit of the smartphone 10. The CPU 28 has a mobile communication control function that controls each unit of the communication system to perform voice communication and data communication through the wireless communication unit 30, and an application processing function.

[0075] The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function allows information such as still images, moving images, and text to be visually conveyed to the user. The CPU 28 also acquires two-dimensional position coordinate information corresponding to a touch operation by the user from the touch panel section of the touch panel display 14. The CPU 28 also acquires an input signal from the switch 26.

[0076] The hardware structure of the CPU 28 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and acts as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0077] A processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functional units may be configured with one processor. As an example of configuring multiple functional units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor acts as multiple functional units. Second, as represented by a SoC (System On Chip), there is a form in which a processor is used that realizes the functions of the entire system including multiple functional units with one IC (Integrated Circuit) chip. In this way, the various functional units are configured using one or more of the above various processors as a hardware structure.

[0078] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0079] The in-camera 20 and the out-camera 22 capture moving images and still images according to instructions from the CPU 28. Fig. 4 is a block diagram showing the internal configuration of the in-camera 20. The internal configuration of the out-camera 22 is the same as that of the in-camera 20. As shown in Fig. 4, the in-camera 20 has a photographing lens 50, an aperture 52, an image sensor 54, an AFE (Analog Front End) 56, an A / D (Analog to Digital) converter 58, and a lens driver 60.

[0080] The photographing lens 50 is composed of a zoom lens 50Z and a focus lens 50F. The lens driving unit 60 drives the zoom lens 50Z and the focus lens 50F forward and backward in response to a command from the CPU 28 to perform optical zoom adjustment and focus adjustment. The lens driving unit 60 also controls the aperture 52 in response to a command from the CPU 28 to adjust exposure. The lens driving unit 60 corresponds to an exposure correction unit that performs exposure correction of the camera based on the color gray, which will be described later. Information such as the positions of the zoom lens 50Z and the focus lens 50F and the opening degree of the aperture 52 is input to the CPU 28.

[0081] The image sensor 54 has a light receiving surface on which a large number of light receiving elements are arranged in a matrix. The subject light transmitted through the zoom lens 50Z, the focus lens 50F, and the aperture 52 is imaged on the light receiving surface of the image sensor 54. R (red), G (green), and B (blue) color filters are provided on the light receiving surface of the image sensor 54. Each light receiving element of the image sensor 54 converts the subject light imaged on the light receiving surface into an electrical signal based on the R, G, and B color signals. In this way, the image sensor 54 obtains a color image of the subject. The image sensor 54 can be a photoelectric conversion element such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge-Coupled Device).

[0082] The AFE 56 performs noise removal and amplification of the analog image signal output from the imaging element 54. The A / D converter 58 converts the analog image signal input from the AFE 56 into a digital image signal with a gradation range. An electronic shutter is used as a shutter that controls the exposure time of the incident light on the imaging element 54. In the case of an electronic shutter, the exposure time (shutter speed) can be adjusted by controlling the charge accumulation period of the imaging element 54 by the CPU 28.

[0083] The in-camera 20 may convert the captured image data of moving images and still images into compressed image data in a format such as Moving Picture Experts Group (MPEG) or Joint Photographic Experts Group (JPEG).

[0084] Returning to the explanation of FIG. 3, the CPU 28 stores the video and still images captured by the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0085] Furthermore, the CPU 28 displays the moving images and still images captured by the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the moving images and still images captured by the in-camera 20 and the out-camera 22 in application software.

[0086] The CPU 28 may illuminate the subject with auxiliary light by turning on the light 24 when capturing an image with the outer camera 22. The light 24 may be turned on and off by a touch operation on the touch panel display 14 or an operation of the switch 26 by the user.

[0087] The wireless communication unit 30 performs wireless communication with a base station device accommodated in a mobile communication network in accordance with an instruction from the CPU 28. Using this wireless communication, the smartphone 10 transmits and receives various file data such as audio data and image data, e-mail data, and the like, and receives Web (abbreviation for World Wide Web) data, streaming data, and the like.

[0088] The speaker 16 and the microphone 18 are connected to the communication unit 32. The communication unit 32 decodes the voice data received by the wireless communication unit 30 and outputs the decoded voice data from the speaker 16. The communication unit 32 converts the user's voice input through the microphone 18 into voice data that can be processed by the CPU 28 and outputs the voice data to the CPU 28.

[0089] The memory 34 stores instructions to be executed by the CPU 28. The memory 34 is composed of an internal storage unit 36 ​​built into the smartphone 10 and an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are realized using known storage media.

[0090] The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of sent and received e-mails, Web data downloaded by Web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

[0091] The external input / output unit 40 serves as an interface with an external device connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to another external device by communication or the like via the external input / output unit 40. The external input / output unit 40 transmits data received from an external device to each component inside the smartphone 10, and transmits data inside the smartphone 10 to the external device.

[0092] Examples of the communication means include a Universal Serial Bus (USB), IEEE (Institute of Electrical and Electronics Engineers) 1394, the Internet, a wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), and infrared communication. Examples of the external device include a headset, an external charger, a data port, an audio device, a video device, a smartphone, a PDA, a personal computer, and an earphone.

[0093] The GPS receiver 42 detects the position of the smartphone 10 based on positioning information from GPS satellites ST1, ST2, . . . , STn.

[0094] The power supply unit 44 is a power supply source that supplies power to each unit of the smartphone 10 via a power supply circuit (not shown). The power supply unit 44 includes a lithium ion secondary battery. The power supply unit 44 may include an A / D conversion unit that generates a DC voltage from an external AC power supply.

[0095] The smartphone 10 configured in this manner is set to a shooting mode in response to an instruction input from a user using the touch panel display 14 or the like, and can capture videos and still images using the in-camera 20 and the out-camera 22.

[0096] When the smartphone 10 is set to the shooting mode, it enters a shooting standby state, a moving image is captured by the inner camera 20 or the outer camera 22, and the captured moving image is displayed on the touch panel display 14 as a live view image.

[0097] A user can visually confirm the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject that the user wants to photograph, and set the photographing conditions.

[0098] When the smartphone 10 is instructed to take a picture by a user inputting an instruction using the touch panel display 14 or the like in a shooting standby state, the smartphone 10 performs AF (Autofocus) and AE (Auto Exposure) control, and takes and stores a moving image and a still image.

[0099] [Functional configuration of the drug identification device] Fig. 5 is a block diagram showing a functional configuration of the drug identification device 100 realized by the smartphone 10. Each function of the drug identification device 100 is realized by the CPU 28 executing a program stored in the memory 34. As shown in Fig. 5, the drug identification device 100 has an image acquisition unit 102, a drug detection unit 106, an imprinted character extraction unit 108, a first drug type recognition unit 110, a candidate output unit 112, and a determination unit 114.

[0100] The image acquisition unit 102 acquires a captured image of the drug to be identified to which the mark and / or print is added. The captured image is, for example, an image captured by the in-camera 20 or the out-camera 22. The captured image may be an image acquired from another device via the wireless communication unit 30, the external storage unit 38, or the external input / output unit 40.

[0101] The captured image may be an image of the drug to be identified and a marker. The marker may be a plurality of markers, and may be an ArUco marker, a circular marker, or a rectangular marker. The captured image may be an image of the drug to be identified and a reference gray color.

[0102] The photographed image may be an image photographed at a standard photographing distance and photographing viewpoint. The photographing distance can be expressed by the distance between the identification target drug and the photographing lens 50 and the focal length of the photographing lens 50. The photographing viewpoint can be expressed by the angle between the marker printed surface and the optical axis of the photographing lens 50.

[0103] The captured image may include a plurality of identification target drugs. The plurality of identification target drugs are not limited to identification target drugs of the same drug type, and may be identification target drugs of different drug types.

[0104] The image acquisition unit 102 includes an image correction unit 104. When a captured image includes a marker, the image correction unit 104 standardizes the shooting distance and shooting viewpoint of the captured image based on the marker to acquire a standardized image. When a captured image includes a reference gray color area, the image correction unit 104 performs color correction of the captured image based on the reference gray color.

[0105] The medicine detection unit 106 detects the area of ​​the medicine to be identified from the captured image acquired by the image acquisition unit 102. When a standardized image is acquired by the image correction unit 104, the medicine detection unit 106 detects the area of ​​the medicine to be identified from the standardized image. 。 When a captured image contains a plurality of drugs to be identified, the drug detection unit 106 detects the regions of each of the drugs to be identified.

[0106] The imprint extraction unit 108 processes at least the area of ​​the drug to be identified in the captured image, removes the outer edge information of the drug to be identified, and obtains an imprint extraction image in which the imprint and / or print are extracted. Here, the imprint extraction image is an image in which the imprint and / or print are emphasized by expressing the brightness of the imprint part or print part relatively higher than the brightness of the part other than the imprint part or print part.

[0107] When the medicine detection unit 106 detects the regions of the plurality of target medicines, the mark extraction unit 108 acquires a plurality of mark extraction images corresponding to the plurality of target medicines, respectively.

[0108] The imprint extraction unit 108 includes a first trained model 108A. The first trained model 108A is a trained model that outputs a second image in which the imprint and / or print of the medicine is extracted when a first image of the medicine with imprint and / or print is given as an input. The first trained model 108A is a training data set of a plurality of different medicines with imprint and / or print, and is machine learning performed using a training data set for learning that is a set of an image of the medicine with imprint and / or print and an image in which the imprint and / or print of the medicine is extracted. A convolutional neural network (CNN) can be applied to the first trained model 108A.

[0109] Here, since the image correction unit 104 standardizes the shooting distance and the shooting viewpoint of the captured image and corrects the color tone, the first trained model 108A can be expected to operate stably.

[0110] The first drug type recognition unit 110 inputs the stamp print extraction image, infers the drug type of the drug to be identified, and acquires candidates for the drug type of the drug to be identified. The candidates for drug type include drug identification information consisting of drug name, product name, abbreviation, or a combination of these. When multiple stamp print extraction images corresponding to multiple drugs to be identified are input, the first drug type recognition unit 110 acquires candidates for the drug type corresponding to each of the multiple drugs to be identified.

[0111] The first drug type recognition unit 110 includes a second trained model 110A. The second trained model 110A is a trained model that outputs a drug type corresponding to the drug's imprint and / or print when a second image from which the drug's imprint and / or print is extracted is given as an input. The second trained model 110A is a training data set of multiple different drugs to which imprints and / or prints are added, and machine learning is performed using a training data set for learning that is a set of an image of a drug from which the imprint and / or print is extracted and a drug type corresponding to the imprint and / or print of the drug. The second trained model 110A can apply CNN, similar to the first trained model 108A.

[0112] In this way, the first drug type recognition unit 110 performs recognition based on the inscription and / or printed information without using color information, and is therefore robust against the influence of the imaging environment.

[0113] The candidate output unit 112 outputs candidates of drug types of the drug to be identified acquired by the first drug type recognition unit 110. The candidate output unit 112 (an example of a display control unit) displays, for example, a plurality of candidates of drug types of the drug to be identified on the touch panel display 14 in a selectable manner.

[0114] The confirmation unit 114 confirms the correct drug of the drug to be identified from the candidates of the drug type of the drug to be identified. The confirmation unit 114 confirms, as the correct drug, a drug type candidate selected by the user from a plurality of candidates of the drug type of the drug to be identified displayed on the touch panel display 14, for example. When there is only one candidate of the drug type of the drug to be identified output by the candidate output unit 112, the confirmation unit 114 may confirm the candidate of the drug type as the correct drug.

[0115] [Drug Identification Method] Fig. 6 is a flow chart showing steps of a drug identification method using the drug identification device 100. Fig. 7 is a diagram showing a process of the steps of the drug identification method. The drug identification method is realized by the CPU 28 reading and executing a drug identification program from the memory 34. The drug identification program may be provided via the wireless communication unit 30 or the external input / output unit 40.

[0116] In step S1, which is an image acquisition step, the image acquisition unit 102 acquires a captured image of a drug to be identified to which a mark and / or a print is added. Here, the image acquisition unit 102 acquires a captured image I captured by the outer camera 22. C Obtain 1. Photographed image I C 1 may be an image in which illumination light is irradiated onto the drug to be identified by a light 24, or it may be a fully illuminated image or a partially illuminated image captured using a photography auxiliary device 70 (see Figures 23 and 24) described below.

[0117] Figure 7 shows the captured image I C Figure 1 shows an example of tablets T1, T2, and T3, which are drugs to be identified. Tablets are solid drugs formed into a certain shape by compression molding. Tablets T1, T2, and T3 each have identification information added by engraving and / or printing.

[0118] Also, the captured image I C In the image I, the background BG is captured, which is a reference gray color area. C Four markers M1, M2, M3, and M4 are shown in Fig. 1. Markers M1 to M4 are ArUco markers. Three tablets T1, T2, and T3 are shown in the image surrounded by markers M1 to M4.

[0119] In this way, the captured image I C The image I1 is an image in which the drug to be identified, the markers M1 to M4, and a reference gray color area are simultaneously captured. The image correction unit 104 corrects the captured image I1 based on the markers M1 to M4. C The shooting distance and the shooting viewpoint of the image I1 are standardized, and the captured image I is calculated based on the gray color of the background BG. C After color correction of 1, standardized image I S Get 1 (process P1).

[0120] Here, the standardization process of the shooting distance and the shooting viewpoint is performed as follows.

[0121] First, the image correction unit 104 corrects the captured image I C The image correction unit 104 acquires the coordinates of the four vertices V1, V2, V3, and V4 of the four markers M1, M2, M3, and M4 in the image of No. 1. Next, the image correction unit 104 specifies the coordinates of the four vertices used for alignment. Here, the image correction unit 104 specifies the coordinates of the upper left vertex V1 of the marker M1 arranged at the upper left, the upper right vertex V2 of the marker M2 arranged at the upper right, the lower right vertex V3 of the marker M3 arranged at the lower right, and the lower left vertex V4 of the marker M4 arranged at the lower left.

[0122] Furthermore, the image correction unit 104 specifies the coordinates of the four vertices V1 to V4 whose coordinates have been specified, after standardization of the shooting distance and the shooting viewpoint. The image correction unit 104 obtains a perspective transformation matrix that transforms the four vertices V1 to V4 whose coordinates have been specified, into the positions of the specified coordinates. Such a perspective transformation matrix is ​​uniquely determined if there are four points. For example, the transformation matrix can be obtained by the getPerspectiveTransform function of OpenCV if there is a correspondence between the four points.

[0123] The image correction unit 104 uses the obtained perspective transformation matrix to correct the original captured image I C 1 is perspective-transformed to obtain the transformed image. Such perspective transformation can be performed using the warpPerspective function of OpenCV. The transformed image is called the standardized image I, where the shooting distance and shooting viewpoint are standardized. S The answer is 1.

[0124] In step S2, which is a drug detection step, the drug detection unit 106 detects the standardized image I S 7, the medicine detection unit 106 detects the region of the tablets T1 to T3 from the region image I1. The medicine detection unit 106 may detect the region of the tablets T1 to T3 using a trained model for medicine detection. R 1 is obtained (process P2).

[0125] In addition, the captured image I C Since the marker and the gray area are not shown in the image I1, the image correction unit 104 corrects the standardized image I S If the medicine detection unit 106 does not acquire the photographed image I C 1, the areas of tablets T1 to T3 can be detected.

[0126] In step S3, which is a stamp extraction step, the stamp extraction unit 108 processes the regions of the tablets T1 to T3 detected in step S2 to obtain a stamp extraction image in which the stamp and / or print of each of the tablets T1 to T3 is extracted. The stamp extraction unit 108 may obtain the stamp extraction image by the first trained model 108A. FIG. 7 shows the stamp extraction unit 108 extracting the region image I R Image I, which is an image of tablet T1 extracted by processing image 1 E This shows an example of acquiring the stamp print extraction image I1 (process P3). E 1 is an image in which the brightness of the engraved or printed portion is relatively higher than the brightness of the portion other than the engraved or printed portion.

[0127] In step S4, which is the first drug type recognition step, the first drug type recognition unit 110 infers the drug type of each of the tablets T1 to T3 from the stamped and printed extracted images of the tablets T1 to T3 acquired in step S3, and acquires candidates for the drug type of the tablets T1 to T3. The first drug type recognition unit 110 uses the second trained model 110A to infer the drug type of the tablet T1. D 7 shows that the first medicine type recognition unit 110 acquires an image I of the stamped print extracted from the tablet T1. E 1 to Tablet T1 drug candidate R D In the example shown in FIG. 7, the candidate drug type R1 for tablet T1 is obtained (process P4). D 1 is for two drugs, "AB tablets 1 mg" and "AC tablets 1 mg", and the probability of these being score values ​​is "0.5" and "0.4", respectively.

[0128] In step S5, which is a candidate output step, the candidate output unit 112 outputs the candidate R D Here, the candidate output unit 112 outputs the candidate drug type R for the tablet T1 acquired in step S4. D 1 is selectably displayed on the touch panel display 14. This allows the user to recognize that the candidate drug types for tablet T1 are "AB tablets 1 mg" and "AC tablets 1 mg."

[0129] The candidate output unit 112 may display on the touch panel display 14 all the candidate drug types for each of the tablets T1 to T3.

[0130] In step S6, which is the confirmation step, the confirmation unit 114 selects the drug candidate R displayed in step S5. D 1, the correct medicine for tablet T1 is determined. Here, the determination unit 114 determines the medicine selected by the user via the touch panel display 14 from "AB tablets 1 mg" and "AC tablets 1 mg" as the correct medicine for tablet T1.

[0131] According to the drug identification device 100 of the first embodiment, drug-specific identification information without external shape information can be extracted, so that highly accurate drug recognition is possible with only the identification information added by marking and / or printing. Therefore, the drug type can be recognized without being affected by the shooting environment. In addition, since the marking and printing extraction image can be obtained by one shooting, the effort of shooting is reduced and usability is high.

[0132] <Second embodiment> [Functional configuration of the drug identification device] Fig. 8 is a block diagram showing a functional configuration of the drug identification device 120 realized by the smartphone 10. The same reference numerals are used for the parts common to Fig. 5, and detailed description thereof will be omitted. As shown in Fig. 8, the drug identification device 120 has a second drug type recognition unit 122, a drug-associated information storage unit 123, and a drug-associated information acquisition unit 124.

[0133] The second drug type recognition unit 122 inputs at least the area of ​​the drug to be identified in the captured image and infers the drug type of the drug to be identified. When multiple areas corresponding to multiple drugs to be identified are input, the second drug type recognition unit 122 infers the drug types corresponding to the multiple drugs to be identified. The image input to the second drug type recognition unit 122 is not limited to the image of the area of ​​the drug to be identified, but may be an image (composite extraction image) obtained by inverting the brightness of the imprint extraction image generated by the imprint extraction unit 108 and combining it with the image of the area of ​​the drug to be identified.

[0134] The second drug type recognition unit 122 includes a third trained model 122A. The third trained model 122A is a trained model that outputs the drug type of a drug when a first image of a drug with a mark and / or print is given as an input. The third trained model 122A is a training data set of a plurality of different drugs with a mark and / or print, and machine learning is performed using a training data set for learning that is a set of an image of a drug with a mark and / or print and a drug type of the drug corresponding to the mark and / or print of the drug. The third trained model 122A can apply CNN, like the first trained model 108A and the second trained model 110A.

[0135] The second drug type recognition unit 122 inputs at least the area of ​​the drug to be identified in the captured image, and recognizes not only the identification information based on the marking and / or printing of the drug to be identified, but also at least one of the color, size, and shape.

[0136] The medicine-associated information storage unit 123 stores medicine-associated information for a plurality of medicines. The medicine-associated information includes at least one of the following information: the name and identification code of the medicine, the shape, the size, and the color of the medicine. The medicine-associated information may include information on medicines with similar markings and / or printing.

[0137] The drug-associated information acquiring section 124 acquires drug-associated information of a predetermined drug from the drug-associated information storage section 123 .

[0138] [Drug Identification Method] Fig. 9 is a flow chart showing steps of a medicine identifying method using the medicine identifying device 120. Fig. 10 is a diagram showing the process of steps of the medicine identifying method. Note that parts common to Fig. 6 and Fig. 7 are given the same reference numerals, and detailed explanations thereof will be omitted.

[0139] The processing in steps S1 to S4 and processes P1 to P4 is similar to that in the first embodiment.

[0140] In step S11, which is the second drug type recognition step, the second drug type recognition unit 122 processes each region of the tablets T1 to T3 detected in step S2 to infer the drug type of each of the tablets T1 to T3. The second drug type recognition unit 122 may infer the drug type of each of the tablets T1 to T3 using a third trained model 122A. FIG. 10 shows the second drug type recognition unit 122 detecting the region image I of the tablet T1. R 1 to Tablet T1 drug candidate R D 2 is obtained (process P5).

[0141] In the example shown in Figure 10, the drug candidate R for tablet T1 D 2 is two drugs, “AB tablets 1 mg” and “AC tablets 1 mg”, and the probability of being score values ​​is “0.9” and “0.3”, respectively.

[0142] The order of the first drug type recognition step and the second drug type recognition step is not particularly limited, and either step may be performed first.

[0143] In step S12, which is a drug-associated information acquisition step, the drug-associated information acquisition unit 124 acquires drug-associated information of drugs of related drug types from the drug-associated information storage unit 123 based on the inference result of the first drug type recognition unit 110 acquired in step S4 and the inference result of the second drug type recognition unit 122 acquired in step S5. D 1 and candidate R D 1 shows an example of obtaining drug-related information INF for “AB tablets 1 mg” and “AC tablets 1 mg” (process P6).

[0144] In the example shown in FIG. 10, the acquired medicine-associated information INF is size information, and the size information of "AB tablet 1 mg" and "AC tablet 1 mg" is "diameter 12 mm" and "diameter 10 mm", respectively.

[0145] In step S13, which is a candidate acquisition process, the first drug type recognition unit 110 integrates the inference result of the second drug type recognition unit 122 acquired in step S11 and the drug-associated information acquired in step S12 with the inference result of the first drug type recognition unit 110 acquired in step S4 to acquire candidates for the drug type of each of the tablets T1 to T3.

[0146] The method of integrating the inference result of the first drug type recognition unit 110 and the inference result of the second drug type recognition unit 122 may be a method of weighting both inference results to obtain a discrimination result. The weighting of both may be based on experience or may be performed for each drug based on machine learning.

[0147] The method of integrating the inference result of the first drug type recognition unit 110 and the inference result of the second drug type recognition unit 122 is to sum the score value of the first drug type recognition unit 110 and the score value of the second drug type recognition unit 122, Candidates for drugs Alternatively, the inference results of the first drug type recognition unit 110 and the second drug type recognition unit 122 may be integrated in a manner that multiplies the score value of the first drug type recognition unit 110 and the score value of the second drug type recognition unit 122 by a coefficient common to all drugs, takes the sum, and arranges them in descending order. The inference results of the first drug type recognition unit 110 and the inference results of the second drug type recognition unit 122 may be integrated in a manner that multiplies the score value of the first drug type recognition unit 110 and the score value of the second drug type recognition unit 122 by a coefficient specific to the drug type, takes the sum, and arranges them in descending order, or may be integrated in a manner that multiplies the score value of the first drug type recognition unit 110 and the score value of the second drug type recognition unit 122 by a coefficient specific to the drug type that has been learned to maximize the drug type identification performance, takes the sum, and arranges them in descending order.

[0148] FIG. 10 shows the first drug type recognition unit 110 detecting candidate R D 2, and the drug-related information INF is a candidate R D 1 and merge to candidate R D In the example shown in FIG. 10, the candidate drug type R for tablet T1 is obtained (process P7). D 3 is "AB tablet 1 mg" and "AC tablet 1 mg", and the probability of being the score value is "0.99" and "0.2", respectively.

[0149] The subsequent processes of steps S14 and S15 are similar to the processes of steps S5 and S6 in the first embodiment.

[0150] Since the first drug type recognition unit 110 performs recognition based on the inscription and / or printing information without using color information, there is a possibility that the accuracy of identifying drugs with the same or similar inscription and printing information but different colors may decrease. On the other hand, since the second drug type recognition unit 122 uses color information, there is a possibility that it may be affected by the shooting environment. According to the drug identification device 100 of the second embodiment, by integrating the inference result of the first drug type recognition unit 110, the inference result of the second drug type recognition unit 122, and the drug identification information, it is possible to complement each other's shortcomings and obtain a more accurate drug type candidate. Note that the drug identification device 100 may at least integrate the inference result of the second drug type recognition unit 122 with the inference result of the first drug type recognition unit 110.

[0151] <Third embodiment> [Functional configuration of the drug identification device] Fig. 11 is a block diagram showing a functional configuration of the drug identification device 130 realized by the smartphone 10. The same reference numerals are used for the parts common to Fig. 8, and detailed description thereof will be omitted. As shown in Fig. 11, the drug identification device 130 has a master image storage unit 126. In addition, the first drug type recognition unit 110 of the drug identification device 130 has a master image acquisition unit 127 and a template matching unit 128.

[0152] The master image storage unit 126 stores master images of a plurality of medicines. Fig. 12 is a diagram showing an example of a master image of a certain medicine stored in the master image storage unit 126. The master image storage unit 126 stores a surface area image I R A, Extracted image of the surface imprint I E A, Composite extracted image of surface I W A, Image of the back area I R B, Extracted image of the stamp on the back I E B, and a composite extracted image I on the back W B, six master images are stored.

[0153] Surface Area Image I R A and back area image I R B is an image of the area of ​​the drug detected from the image of the front side of the drug and the back side of the drug, respectively. The original images were taken in a controlled environment with a known shooting distance and shooting viewpoint.

[0154] Extracted image of the surface imprint I E A, and the backside stamped and printed extracted image I E B are surface area images I R A and back area image I R This is an image obtained by performing the inscription extraction process on B. The inscription extraction process is a process in which the brightness of the inscription or print part is expressed relatively higher than the brightness of the part other than the inscription or print part, and may be performed by image processing or manually by the user. Inscription extraction image I E A, and the backside stamped and printed extracted image I E In this embodiment, B is an image that serves as a reference for each medicine when performing template matching for the medicine, and may be called an imprint master image.

[0155] Surface synthesis extraction image I W A is the surface imprinted print extraction image I E Invert the brightness of A to get the surface area image I R A is the image superimposed on A. Similarly, the back side composite extracted image I W B is the backside stamped print extraction image I E Invert the brightness of B to create the backside area image I R This is an image superimposed on B.

[0156] For convenience, the master image storage unit 126 stores six images of images with rotational symmetry such as capsules. However, the stamped and printed extracted images are stored as blank files. Capsules are medicines in which powder or granules are filled into a capsule base.

[0157] Returning to the explanation of FIG. 11, the master image acquisition unit 127 acquires an imprint master image of a predetermined medicine type from the master image storage unit 126.

[0158] The template matching unit 128 performs template matching between the stamp print extraction images of the drugs of multiple candidate drug types and the stamp master image acquired by the master image acquisition unit 127. Template matching is an image processing that searches for an engraving master image that matches the stamp print extraction image by comparing the stamp print extraction image of the drug with the engraving master image, and identifies which drug type the drug in the stamp print extraction image is. In this embodiment, the image correction unit 104 generates a standardized image and then acquires a region image, so that template matching can be performed with the engraving master image generated from a captured image with a known shooting distance and shooting viewpoint.

[0159] Fig. 13 is a flow chart showing steps of a medicine identifying method using the medicine identifying device 130. Note that parts common to Fig. 9 are given the same reference numerals, and detailed description thereof will be omitted.

[0160] The processes in steps S1 to S4 and steps S11 to S13 are similar to those in the second embodiment.

[0161] In step S21, which is a master image acquisition step, the master image acquisition unit 127 acquires from the master image storage unit 126 the engraved master images of the candidate drug types for each of the tablets T1 to T3 acquired in step S13.

[0162] For example, in the example described in the second embodiment, the candidate drug type R for tablet T1 D 3 are "AB tablets 1 mg" and "AC tablets 1 mg". Therefore, the master image acquisition unit 127 acquires the imprint master image of "AB tablets 1 mg" and the imprint master image of "AC tablets 1 mg" for tablet T1. When there are many candidate drug types, the imprint master images of the drug candidates with the top N scores (for example, N=10) may be acquired.

[0163] In step S22, which is a template matching step, the template matching unit 128 performs template matching between the stamp master image acquired in step S21 and the stamp print extracted image of each of the tablets T1 to T3.

[0164] For example, the template matching unit 128 may match the stamped print extraction image I E Template matching between 1 and the stamp master image of "AB Tablets 1mg", and stamp print extraction image I E 1 and the engraved master image of "AC tablet 1 mg", and identify whether the drug type of tablet T1 is "AB tablet 1 mg" or "AC tablet 1 mg".

[0165] Here, the template matching unit 128 extracts the marking and printing image I E The engraved master image is rotated and translated relative to 1 to obtain the engraved print extraction image I E 1 is compared with each of the imprint master images, and the drug with the highest score indicating the degree of match is identified. E The direction of the marking and / or printing of 1 is arbitrary. Therefore, in order to accommodate the arbitrary rotation of the medicine, the template matching unit 128 performs template matching by rotating the marking master image in increments of 1°, for example.

[0166] Furthermore, the master image acquisition unit 127 may add drugs with similar inscriptions and / or printing to the candidate drugs of the drug type acquired in step S13 as well as the candidate drugs, and acquire the inscription master image of the drugs with similar inscriptions and / or printing. For example, by performing template matching on the inscription master image in advance in a round-robin manner, information on drugs with similar inscriptions and / or printing is stored in the inscription master image. With this information, the master image acquisition unit 127 can acquire the inscription master image of the drugs with similar inscriptions and / or printing when acquiring the inscription master image of the drug. In the template matching unit 128, in addition to the candidates of the drugs with the highest scores, template matching is performed on the candidates of the drugs similar to those drugs, so that all drugs can be extracted without omission, and the accuracy of drug recognition can be improved.

[0167] In step S23, which is a secondary candidate acquisition step, the first drug type recognition unit 110 acquires further narrowed down drug type candidates of the tablets T1 to T3 from the result of the template matching in step S22.

[0168] The subsequent processes of steps S24 and S25 are similar to the processes of steps S5 and S6 in the first embodiment. Note that the medicine identified by template matching in step S22 may be determined as the correct medicine for tablets T1 to T3.

[0169] If the inscription information can be extracted, highly accurate matching is possible by template matching with the inscription master image. However, template matching has the drawback that it takes a very long time (several tens to several hundreds of seconds) when the search targets all drug types and the rotation angle is arbitrary. On the other hand, when matching is performed using a trained model, the identification result can be obtained in a very short time (several tens of milliseconds), but the accuracy rate is not as high as that of template matching with the inscription master image. According to the drug identification device 130 of the third embodiment, template matching is performed for the top-ranked drugs among the candidates acquired by the first drug type recognition unit 110, so it is possible to acquire candidates for the drug type of the drug to be identified while achieving both accuracy and speed.

[0170] <Capsules> Capsules come in the same color, shape, and size, and it is necessary to recognize the identification code printed on the capsule to identify the drug type. However, capsule images contain a huge number of patterns due to the arbitrariness of rotation and alignment. For this reason, both machine learning-based and template matching-based methods are Identification symbol recognition is difficult.

[0171] In consideration of the above, the drug identification device 100 (120, 130) performs a first stage of identification of capsules based on their color, size, and shape, and narrows down the drug type candidates to a small number. The drug type candidates are the capsules with the top N scores (e.g., N=5) or capsules with a certain score or higher.

[0172] The drug identification device 100 (120, 130) presents the master image, identification symbol, and drug name of these few candidates to the user in an arranged manner. In addition, the drug identification device 100 (120, 130) displays the captured image in an enlarged manner and simultaneously displays an image with the marking and / or print extracted, to make it easier for the user to select the correct drug.

[0173] <Mobile terminal device GUI (Graphical User Interface)> 14 to 18 are diagrams showing screen displays D1 to D5 of the touch panel display 14 of the smartphone 10 in each step of the medicine identifying method.

[0174] FIG. 14 shows a screen display D1 in the image acquisition process. The screen display D1 shows a live view image I LV , and shooting button B S 1 is displayed.

[0175] Live View Image I LV is an image in which the area captured by the outer camera 22 is displayed in real time. LVincludes tablets T1 to T3, markers M1 to M4, and a reference gray background BG.

[0176] The user presses the capture button B S When 1 is tapped, actual shooting is performed by the outer camera 22, and the screen display D1 is switched to the screen display D2. The captured image, which is a still image captured actually, is stored in the memory .

[0177] FIG. 15 shows a screen display D2 in the drug detection process. The screen display D2 shows a standardized image I S , and Retake button B S 2 is displayed. Standardized Image I S The standardized image I is an image obtained by cutting out an area surrounded by straight lines connecting the markers M1 to M4 after standardization processing is performed on the captured image. S Frames F1, F2, and F3 are displayed which surround the areas of tablets T1, T2, and T3 detected from the image.

[0178] In the screen display D2, the standardized image I S is enlarged in response to a pinch operation on the touch panel display 14 by the user.

[0179] When any of the areas of the tablets T1 to T3 (any of the areas surrounded by the frames F1 to F3) is tapped on the screen display D2, the screen display D2 is switched to the screen display D3. S When 2 is tapped, the screen display changes from D2 to D1.

[0180] 16 shows a screen display D3 in the candidate output step. Here, the case where a candidate drug type for tablet T1 is acquired by tapping the area for tablet T1 (area surrounded by frame F1) in screen display D2 is shown.

[0181] The screen display D3 shows an area image I of a tablet T1, which is a drug to be identified and which has been tapped on the screen display D2. R , and region image I RExtracted image of the stamp I E are displayed side by side. Furthermore, a composite extracted image of the tablet T1 may be displayed side by side.

[0182] Also, on screen display D3, there is a text box B B , Search button B S 3. Back button B S 4 and reshoot button B S 2 is displayed.

[0183] Back button B S When the button B4 is tapped, the screen display changes from D3 to D2. S When 2 is tapped, the screen changes from D3 to D1.

[0184] Furthermore, the screen display D3 includes a candidate drug display area A C 1. Candidate drug display area A C In the candidate drug display area A1, a plurality of candidate drugs recognized as candidates for the drug type of the tablet T1 are displayed in order of the highest score from top to bottom so that the candidate drug can be selected. C In the candidate drug display area A2, candidate drugs with lower scores are displayed in response to a user's swipe operation from the bottom of the screen to the top of the screen. C 1, candidate drugs with higher scores are displayed depending on the user's subsequent swiping action from the top of the screen to the bottom.

[0185] Candidate drug display area A C 1 contains candidate drug C for tablet T1 T 1~C T The composite extracted images of the front and back of each of the master images of Tablet T1 and Tablet T2 are displayed, as well as a superimposed image in which the inscription-printed extracted image of the tablet T1 and the inscription-printed extracted image of the target surface of the master image are superimposed. The superimposed image is displayed based on the orientation of the master image for the rotation angle and translation position of the master image that has the highest score by template matching, and the composite extracted image is displayed in an orientation in which the inscription and / or print are aligned with the superimposed image. Here, the candidate drug C displayed at the top is TIn 1, the extracted image of the imprinted mark on tablet T1 matches the extracted image of the imprinted mark on the master image, so the superimposed image is clearly displayed.

[0186] Thus, the screen display D3 shows the area image I of the tablet T1. R , and the stamped and printed extracted image I E Together with the candidate drug C for tablet T1 T 1~C T By displaying a composite extracted image of the front and back of the master image 5, as well as an overlaid image at the rotation angle and translation position that produced the highest matching score in template matching, the user can easily determine whether the recognition result is correct or not.

[0187] Candidate drug display area A on screen display D3 C When any one of the candidate drugs in 1 is tapped, the screen display D3 transitions to the screen display D4.

[0188] Text Box B B is a search window in which a character string can be input, and is an interface for identifying the drug type of tablet T1 by the character string input by the user. C If there is no correct drug in the area I of the tablet T1 in the screen display D3, the user R , and the stamped and printed extracted image I E Refer to the text box B. B Enter the identification code of tablet T1 in the text box and click the Search button B. S By tapping on the text box B3, candidates for the drug type of the input character string can be obtained. The input text may be an identification symbol such as "AB12" or a drug name. The smartphone 10 also has a text box B B When is selected, the keyboard will be displayed at the bottom of the screen. B The text may be input by a keyboard or by so-called voice input.

[0189] Text box B on screen display D3 BText is entered into and search button B is pressed S When 3 is tapped, the screen changes from D3 to D5.

[0190] FIG. 17 shows a screen display D4 which is a transition from the screen display D3. Here, the candidate drug C T 1 indicates the case where a tap operation is performed.

[0191] The screen display D4 shows an area image I of a tablet T1, which is a drug to be identified. R , and the stamped and printed extracted image I E In addition, the upper transition button B S 5. Downward transition button B S 6. Confirm button B S 7, and back button B S 8 is displayed.

[0192] Back button B S When 8 is tapped, the screen changes from D4 to D3.

[0193] In addition, the screen display D4 is a selected drug display area A S Selected drug display area A S In the selected drug display area A, information on the drug tapped on the screen display D3 is displayed. S Candidate drug C T 1 name, synthetic extraction image, identification code, pharmacological classification, dosage form, YJ code (individual drug code), main ingredient, drug price, and candidate drug C such as predecessor and generic T Information related to 1 is displayed. Side effects information, interaction information, etc., as well as candidate drug C T Information related to 1 may be displayed.

[0194] Upward transition button B S When 5 is tapped, the selected drug display area A S The button B displays candidate drugs with a score one level higher than the candidate drug currently being displayed. S When 6 is tapped, the selected drug display area A SThe candidate drug with a score one level lower than the candidate drug being displayed is displayed. Also, the Confirm button B S When the 7 is tapped, the correct drug of the tablet T1, which is the drug to be identified, is displayed in the selected drug display area A S The drug will be confirmed as shown in the table.

[0195] FIG. 18 shows screen display D5, which is a transition from screen display D3. In screen display D3, a search button B S When the text box B3 is tapped, the first drug type recognition unit 110 searches the drug-associated information storage unit 123 for the input text to obtain a candidate drug type corresponding to the input text. B After "AB12" is entered, press the search button B S 3 shows the case where a tap operation is performed.

[0196] The screen display D5 shows an area image I of a tablet T1, which is a drug to be identified. R , and the stamped and printed extracted image I E In addition, text box B B , Search button B S 3. Back button B S 4 and reshoot button B S 2 is displayed.

[0197] Back button B S When the button B4 is tapped, the screen display changes from D5 to D2. S When 2 is tapped, the screen changes from D5 to D1.

[0198] In addition, the screen display D5 shows the candidate drug display area A C 2. Candidate drug display area A C 2. In the screen display D3, text box B B Master images of multiple candidate medicines corresponding to the input text are displayed from top to bottom in order of the degree of match with the text, allowing selection.

[0199] Here, candidate drug display area AC 2, candidate drug C T 11~C T A composite extract of the front and back of the 15 master images is displayed.

[0200] Candidate drug display area A C In the candidate drug display area A2, candidate drugs with a relatively low degree of match in the text search results are displayed in response to a user's swipe operation from the bottom of the screen to the top of the screen. C In 2, candidate drugs with a relatively high degree of match in the text search results are displayed in response to the user's subsequent swiping action from the top of the screen to the bottom.

[0201] Candidate drug display area A in screen display D5 C When any candidate drug in 2 is tapped, the screen changes from D5 to D4.

[0202] In this way, even if the drug type candidate cannot be properly obtained, the text box B B By using this, the user can identify the region image I R , and the stamped and printed extracted image I E Candidate drugs can be obtained by performing a text search while referring to the above.

[0203] In addition, candidate drug display area A C If there is no correct answer in text box B, the user can enter a different text in text box B. B Enter and press the Search button B S By tapping 3, you can search again for the correct medication.

[0204] As described above, the drug identification device may be configured as a tablet computer terminal. Although a detailed description of the tablet computer terminal is omitted, the tablet computer terminal includes a display (not shown) that is relatively larger in area than the smartphone 10, and a camera (not shown).

[0205] 19 to 22 are diagrams showing screen displays D11 to D14 of the display of a tablet computer terminal in each step of the medicine identifying method, respectively. Note that the same reference numerals are used to designate parts common to those in Figs. 14 to 18, and detailed description thereof will be omitted.

[0206] 19 shows a screen display D11 in the image acquisition process. The screen display D11 shows a live view image I LV When the user operates a shooting button (not shown), actual shooting is performed and the screen display D11 is changed to screen display D12.

[0207] FIG. 20 shows a screen display D12 in the drug detection step. The screen display D12 shows a standardized image I S When an arbitrary drug is selected on the screen display D12, the screen display D12 is switched to the screen display D13.

[0208] 21 shows a screen display D13 in the candidate output step. Here, a case is shown in which the tablet T1 is selected in the screen display D12, and a candidate drug type for the tablet T1 is acquired.

[0209] Standardized image I of screen display D13 S In the screen display D12, the frame F1 of the tablet T1, which is the drug to be identified and selected, is displayed with a thicker line than the frames F2 and F3. The frame of the drug to be identified may be displayed in a color different from the frame lines of the other drugs.

[0210] In addition, the screen display D13 is a candidate drug display area A C 3. Candidate drug display area A C In the candidate drug display area A3, a number of candidate drugs recognized as candidates for the drug type of the tablet T1 are displayed from left to right in descending order of score. C 3, candidate drug C for tablet T1 T 21~C T A master image in which the printing or engraving on the surface of 30 is emphasized is displayed. CIn 3, candidate drugs with lower scores may be displayed in response to a user swiping operation from the right side of the screen to the left side of the screen.

[0211] In the screen display D13, candidate drug C T 21~C T When any one of 30 is selected, the tablet T1, which is the identification target drug, is determined to be the correct drug.

[0212] FIG. 22 shows a screen display D14 in the secondary candidate acquisition step. The screen display D14 shows a candidate drug display area A C 4. Candidate drug display area A C In FIG. 4, the photographed image of tablet T1 and the extracted image of the printed mark are arranged side by side, and the candidate drugs C, which are the top 5 out of the 10 candidate drugs for tablet T1, are shown. T 21~C T The 25 master images are displayed from top to bottom in order of highest template matching score.

[0213] The master images of each candidate drug displayed alongside the photographed image of tablet T1 and the engraved print extracted image are, from left to right, a composite extracted image of the front side, a composite extracted image of the back side, and an engraved print extracted image of the target surface (here, the front side). In addition, a superimposed image is displayed alongside the master image of each candidate drug, in which the engraved print extracted image of tablet T1 is superimposed on the engraved print extracted image of the target surface of the master image. These images are displayed in the candidate drug display area A of the screen display D3. C As in 1, the rotation angle and translation position of the master image that has the highest score by template matching are displayed based on the orientation of the master image. T In the case of 21, the extracted image of the imprinted mark on the tablet T1 matches the extracted image of the imprinted mark on the master image, so that the superimposed images are clearly displayed.

[0214] In the screen display D14, candidate drug C T 21~C T When any one of 25 is selected, the tablet T1, which is the identification target drug, is determined to be the correct drug.

[0215] <Shooting support equipment> Fig. 23 is a top view of the photographing assistant device 70 for photographing an image to be input to the drug identification devices 100, 120, and 130. Fig. 24 is a cross-sectional view taken along line 24-24 in Fig. 23. Fig. 24 also shows the smartphone 10 that photographs an image of a drug using the photographing assistant device 70.

[0216] As shown in FIGS. 23 and 24, the photographing auxiliary device 70 has a housing 72, a medicine placing stand 74, a main light source 75, and an auxiliary light source 78.

[0217] The housing 72 is composed of a square bottom plate 72A supported horizontally, and four rectangular side plates 72B, 72C, 72D, and 72E fixed vertically to the ends of each side of the bottom plate 72A.

[0218] The medicine placement stand 74 is fixed to the upper surface of the bottom plate 72A of the housing 72. The medicine placement stand 74 is a member having a surface on which medicines are placed, and here is a plastic or paper thin plate-like member having a square shape when viewed from above, and the placement surface on which the medicine to be identified is placed has a reference gray color. When expressed in 256 gradation values ​​from 0 (black) to 255 (white), the reference gray color is, for example, a gradation value in the range of 130 to 220, and more preferably a gradation value in the range of 150 to 190.

[0219] Generally, when a medicine is photographed with the smartphone 10 against a white or black background, the automatic exposure adjustment function causes colors to be washed out, and sufficient imprint information may not be obtained. With the medicine placement table 74, the placement surface is gray, so the colors are not washed out and the details of the imprint can be captured. In addition, by obtaining the gray pixel values ​​captured in the photographed image and correcting them against the true gray gradation values, color correction or exposure correction of the photographed image can be achieved.

[0220] At the four corners of the mounting surface of the medicine mounting table 74, black and white reference markers 74A, 74B, 74C, and 74D are attached or printed. Any reference markers may be used for the reference markers 74A, 74B, 74C, and 74D, but here, the ArUco marker is used, which has high detection robustness and is simple. The ArUco marker is a square two-dimensional marker with a code portion consisting of 4 to 8 dots in the vertical and horizontal directions, and is preferable because the data set of 4 dots in the vertical and horizontal directions is simple and has high detection robustness. Each dot is preferably composed of multiple pixels.

[0221] Fig. 25 is a diagram showing an example of a data set of ArUco markers of 4 dots vertically and horizontally. Fig. 25 shows 50 types of ArUco markers from id0 to id49. In this embodiment, reference markers 74A, 74B, 74C, and 74D are ArUco markers of id3, id13, id48, and id30, respectively. By arranging these four ArUco markers at these positions, higher detection robustness can be achieved.

[0222] Here, the fiducial markers 74A, 74B, 74C, and 74D preferably have a size of 3 to 30 mm in the vertical and horizontal directions, and more preferably have a size of 5 to 15 mm.

[0223] Moreover, the distance between reference marker 74A and reference marker 74B, and the distance between reference marker 74A and reference marker 74D are each preferably 20 to 100 mm, and more preferably 20 to 60 mm.

[0224] The bottom plate 72A of the housing 72 may also serve as the medicine placing stand 74.

[0225] The main light source 75 and the auxiliary light source 78 constitute a lighting device used to capture an image of the drug to be identified. The main light source 75 is used to extract the markings on the drug to be identified. The auxiliary light source 78 is used to accurately capture the color and shape of the drug to be identified. The photography auxiliary device 70 does not necessarily have to include the auxiliary light source 78.

[0226] FIG. 26 is a top view of the photographing assist device 70 with the assist light source 78 removed.

[0227] The main light source 75 is composed of multiple LEDs 76. Each LED 76 is a white light source with a light-emitting portion having a diameter of 10 mm or less. Here, six LEDs 76 are arranged horizontally at a certain height on each of the four rectangular side panels 72B, 72C, 72D, and 72E. This allows the main light source 75 to irradiate the drug to be identified with illumination light from at least four directions. It is sufficient for the main light source 75 to irradiate the drug to be identified with illumination light from at least two directions.

[0228] The angle θ between the light emitted by the LED 76 and the top surface (horizontal plane) of the drug to be identified is preferably within the range of 0° to 45° in order to extract the marking. Note that the main light source 75 may be composed of rod-shaped light sources with a width of 10 mm or less that are horizontally arranged on each of the four rectangular side panels 72B, 72C, 72D, and 72E.

[0229] The main light source 75 may be always on. This allows the photography assistant device 70 to irradiate the identification target drug with illumination light from all directions. An image taken with all LEDs 76 turned on is called a fully illuminated image. The fully illuminated image makes it easy to extract the printing of the identification target drug that has printing added thereto.

[0230] The main light source 75 may be an LED 76 that is turned on and off depending on the timing, or an LED 76 that is turned on and off may be switched by a switch (not shown). This allows the photography assistant device 70 to irradiate the identification target drug with illumination light from a plurality of different directions using a plurality of main light sources 75.

[0231] For example, an image taken with only the six LEDs 76 on the side panel 72B turned on is called a partial illumination image. Similarly, a partial illumination image taken with only the six LEDs 76 on the side panel 72C turned on, a partial illumination image taken with only the six LEDs 76 on the side panel 72D turned on, and a partial illumination image taken with only the six LEDs 76 on the side panel 72E turned on can be taken to obtain four partial illumination images each illuminated with illumination light from a different direction. A plurality of partial illumination images illuminated with illumination light from a plurality of different directions facilitates the extraction of the marking of the identification target drug to which the marking is added.

[0232] The auxiliary light source 78 is a flat, planar white light source having a square outer shape and a square opening in the center. The auxiliary light source 78 may be an achromatic reflector that diffuses and reflects the light emitted by the main light source 75. The auxiliary light source 78 is disposed between the smartphone 10 and the medicine placement table 74 so that the light is uniformly emitted from the shooting direction (the optical axis direction of the camera) onto the medicine to be identified. The illuminance of the light emitted by the auxiliary light source 78 onto the medicine to be identified is relatively lower than the illuminance of the light emitted by the main light source 75 onto the medicine to be identified.

[0233] FIG. 27 is a top view of a photography auxiliary device 80 according to another embodiment. FIG. 28 is a top view of the photography auxiliary device 80 according to the second embodiment of FIG. 8 -2 8 28 is a cross-sectional view. Fig. 28 also shows a smartphone 10 that takes an image of a medicine using a photographing auxiliary device 80. Note that the same reference numerals are used for parts common to Figs. 23 and 24, and detailed description thereof will be omitted. As shown in Figs. 27 and 28, the photographing auxiliary device 80 has a housing 82, a main light source 84, and an auxiliary light source 86. The photographing auxiliary device 80 does not need to have the auxiliary light source 78.

[0234] The housing 82 has a cylindrical shape and is composed of a circular bottom plate 82A supported horizontally and a side plate 82B fixed vertically to the bottom plate 82 A. The medicine placement stage 74 is fixed to the upper surface of the bottom plate 82A.

[0235] The main light source 84 and the auxiliary light source 86 constitute an illumination device used to take an image of the drug to be identified. Figure 29 is a top view of the photography auxiliary device 80 with the auxiliary light source 86 removed.

[0236] The main light source 84 is configured by 24 LEDs 85 arranged in a ring shape at a certain height and at certain intervals in the horizontal direction on the side panel 82B. The main light source 84 may be always lit, or the LEDs 85 may be switched on and off.

[0237] The auxiliary light source 86 is a flat, planar white light source having a circular outer shape and a circular opening in the center. The auxiliary light source 86 may be an achromatic reflector that diffusely reflects the light emitted by the main light source 84. The illuminance of the light emitted by the auxiliary light source 86 on the drug to be identified is relatively lower than the illuminance of the light emitted by the main light source 84 on the drug to be identified.

[0238] The photographing assistant device 70 and the photographing assistant device 80 may include a fixing mechanism (not shown) that fixes the smartphone 10 that photographs the identification target drug at a standard shooting distance and shooting viewpoint position. The fixing mechanism may be configured to be able to change the distance between the identification target drug and the camera according to the focal length of the photographing lens 50 of the smartphone 10.

[0239] <Drug Identification System> [Configuration of drug identification system] So far, an example of configuring the drug identification device 100 that identifies a correct drug among drugs to be identified that have been marked and / or printed using only a portable terminal device has been described, but the device may be configured with a portable terminal device and a server that can communicate with the portable terminal device, or may be configured with the server alone. Here, a drug identification system realized by a portable terminal device and a server that can communicate with each other will be described.

[0240] Fig. 30 is a diagram showing a configuration of a drug identification system 200. As shown in Fig. 30, the drug identification system 200 has a smartphone 10 and a server 210. The smartphone 10 and the server 210 are connected to each other so as to be able to communicate data with each other via a network 1 such as the Internet or a LAN (Local Area Network). Although only one smartphone 10 is shown in Fig. 30, the drug identification system 200 may include a plurality of smartphones 10.

[0241] The server 210 includes a communication unit 212 , a CPU 214 , and a memory 216 .

[0242] The communication unit 212 communicates with the smartphone 10 via the network 1. The CPU 214 is a processor that executes instructions stored in the memory 216, and has a hardware structure similar to that of the CPU 28. The memory 216 stores instructions to be executed by the CPU 214. The memory 216 also stores data necessary for identifying medicines.

[0243] [Functional configuration of the drug identification system] Fig. 31 is a block diagram showing the functional configuration of the drug identification system 200. The same reference numerals are used for the parts common to Fig. 11, and detailed description thereof will be omitted. Each function of the drug identification system 200 is realized by executing a program stored in the memory 34 of the smartphone 10 by the CPU 28, and by executing a program stored in the memory 216 of the server 210 by the CPU 214.

[0244] 31, the drug identification system 200 has an image acquisition unit 102, a candidate output unit 112, and a determination unit 114 in the smartphone 10. The drug identification system 200 also has a drug detection unit 106, an imprint extraction unit 108, a first drug type recognition unit 110, a second drug type recognition unit 122, a drug associated information storage unit 123, a drug associated information acquisition unit 124, and a master image storage unit 126 in the server 210.

[0245] [Drug Identification Method] Fig. 32 is a flow chart showing steps of a medicine identifying method using the medicine identifying system 200. Note that parts common to Fig. 13 are given the same reference numerals, and detailed explanations thereof will be omitted.

[0246] In step S1, the image acquisition unit 102 of the smartphone 10 acquires a captured image of a drug to be identified that has been marked and / or printed. The image correction unit 104 acquires a standardized image from the captured image. The smartphone 10 transmits the standardized image to the server 210. The image correction unit 104 may be located in the server 210, and the standardization process may be performed by the server 210.

[0247] In step S2, the drug detection unit 106 of the server 210 detects the area of ​​the drug from the standardized image received from the smartphone 10. Thereafter, the server 210 performs the processes of steps S3 to S23 in the same manner as in the third embodiment. The server 210 transmits the candidate drug types acquired in step S23 to the smartphone 10.

[0248] In step S24, the candidate output unit 112 of the smartphone 10 displays the candidate drug types received from the server 210 on the touch panel display 14. In step S25, the determination unit 114 of the smartphone 10 determines the drug selected by the user via the touch panel display 14 as the correct drug for the tablet T1.

[0249] As described above, according to the medication identification system 200, by communicating with the server 210 using the smartphone 10, it is possible to realize highly accurate, simple, and highly usable identification of medications bearing markings and / or printing.

[0250] Here, the drug detection unit 106 is provided in the server 210, but may be provided in the smartphone 10. In this case, the smartphone 10 detects the drug from the captured image, and transmits an image of the detected area of ​​the drug to the server 210. In this way, by transmitting only the image of the drug area, the amount of communication can be reduced compared to the case of transmitting the entire captured image.

[0251] Although not described, the medicine identification system 200 can realize all the functions of the medicine identification device 100.

[0252] <Learning device> 33 is a block diagram showing the electrical configuration of an image processing device 140 including a learning device. As the image processing device 140, a personal computer or a workstation can be used.

[0253] As shown in FIG. 33, the image processing device 140 includes an image input unit 142, a database 144, an operation unit 146, a display unit 148, a CPU 150, a RAM (Random Access Memory) 152, and a ROM (Read Only Memory) 154.

[0254] The image input unit 142 is an input interface for inputting a learning data set to be stored in the database 144, and includes wired and wireless communication interfaces.

[0255] The database 144 is a storage unit that stores the learning data set, and is configured from a large-capacity storage device.

[0256] The operation unit 146 is a user interface for the user to control the image processing device 140, and is composed of a keyboard and a pointing device.

[0257] The display unit 148 is an output interface that visually displays the state of the image processing device 140, and is configured with a display panel.

[0258] CPU 150 is a processor that executes instructions stored in RAM 152 and ROM 154, and has the same hardware structure as CPU 28. RAM 152 is a memory device that temporarily stores data used by CPU 150 for various calculations, and is configured from a semiconductor memory. ROM 154 is a memory device that stores programs to be executed by CPU 150, and is configured from a hard disk.

[0259] FIG. 34 is a diagram showing an example of a learning data set for generating a first trained model. The learning data set is a set of area images of a plurality of different types of medicines and an imprint extraction image, which is the correct answer data. In the example shown in FIG. 34, the area image I I 01, and the stamp print extraction image I, which extracts the stamp from the area image II01 I 11 constitutes one training data set. Similarly, region image I I 02, and area image I I Image I of the stamp 02 I 12 constitute one training dataset, and the region image I I 03, and area image I I Extracted image I of the engraving 03 I 13 constitute one training data set.

[0260] Fig. 35 is a block diagram showing a functional configuration of a learning device 160 realized by the image processing device 140. The learning device 160 is a device for generating a first trained model. As shown in Fig. 35, the learning device 160 includes a recognizer 162, a loss value calculation unit 164, and a parameter control unit 166.

[0261] A CNN model is applied to the recognizer 162. The recognizer 162 can change from an unlearned model to a trained model by updating parameters from initial values ​​to optimal values. The initial values ​​of the parameters of the recognizer 162 may be any values, or, for example, parameters of an existing trained model may be applied.

[0262] The recognizer 162 includes an input layer 162A, an intermediate layer 162B, and an output layer 162C, and each layer has a structure in which a plurality of nodes are connected by edges.

[0263] In the learning phase, region images of the learning data set are input to the input layer 162A.

[0264] The first half of the intermediate layer 162B is a layer that extracts features from the region image input from the input layer 162A. The first half of the intermediate layer 162B has multiple sets, each of which is a set of a convolution layer and a pooling layer. The convolution layer performs a convolution operation using a filter on nearby nodes in the previous layer to obtain a feature map. The pooling layer reduces the feature map output from the convolution layer to create a new feature map. The convolution layer plays a role in extracting features such as edge extraction from an image, and the pooling layer plays a role in providing robustness so that the extracted features are not affected by translation or the like. Note that the first half of the intermediate layer 162B is not limited to the case where the convolution layer and the pooling layer are set as one set, and may include a continuous convolution layer or a normalization layer.

[0265] The latter half of the intermediate layer 162B is a layer that increases the resolution of the extracted feature map to restore the region image. The latter half of the intermediate layer 162B has multiple sets of an unpooling layer and a deconvolution layer. The unpooling layer expands the feature map to create a new feature map. The deconvolution layer performs a deconvolution operation to restore the feature amount contained in the feature map. The final deconvolution layer outputs an imprint extraction image of the same size as the region image. Note that the latter half of the intermediate layer 162B is not limited to the case where the unpooling layer and the deconvolution layer are one set, and may include a continuous deconvolution layer or a normalization layer.

[0266] The output layer 162C is a layer that outputs the stamp print extraction image, which is the recognition result of the recognizer 162.

[0267] The loss value calculation unit 164 obtains the stamp-printed extracted image output from the output layer 162C of the recognizer 162 and the stamp-printed extracted image, which is a learning data set of the area image input to the input layer 162A, and calculates the loss value between the two. For example, the Jaccard coefficient or the Dice coefficient may be used as a method for calculating the loss value.

[0268] Based on the loss value calculated by the loss value calculation unit 164, the parameter control unit 166 adjusts the parameters of the recognizer 162 (such as the filter coefficients of the convolution layer and deconvolution layer) in order to minimize the distance in the feature space between the marking / printing extraction image output from the output layer 162C of the recognizer 162 and the marking / printing extraction image of the correct answer data, or to maximize the similarity, using the error backpropagation method.

[0269] This parameter adjustment process is repeated, and learning is repeated until the loss value calculated by the loss value calculation unit 164 converges.

[0270] Using the learning data set stored in the database 144 in this way, a trained recognizer 162 with optimized parameters is generated as a first trained model. Note that appropriate noise may be artificially added to the medicine area image and the marking print extraction image for training. This makes it possible to generate a recognizer 162 that has acquired robustness against fluctuations in the imaging environment.

[0271] In the recognition phase, the recognizer 162, which is the first trained model, outputs an image of the medicine's inscription mark extracted from an input area image of any medicine.

[0272] FIG. 36 is a diagram showing an example of a learning dataset for generating a second trained model. The learning dataset is a set of stamped and printed extracted images of multiple different types of medicines, and information on the drug type and the front and back of the stamped and printed extracted images, which are correct answer data. In the example shown in FIG. 36, the stamped and printed extracted image I I 11, and the stamped and printed extracted image I I Drug type D of 11 drugs C1 and the surface information constitute one learning data set. Similarly, the stamp print extraction image I I 12, and the stamped and printed extracted image I I 12 Drug Types D C 2 and the surface information constitute one learning data set, and the stamp print extraction image I I 13, and the stamped and printed extracted image I I 13 Drug Types D C 3 and the information on the reverse side make up one training data set.

[0273] Fig. 37 is a block diagram showing a functional configuration of a learning device 170 realized by the image processing device 140. The learning device 170 is a device for generating a second trained model. As shown in Fig. 37, the learning device 170 includes a recognizer 172, an error calculation unit 174, and a parameter control unit 176.

[0274] A CNN model is applied to the recognizer 172. The recognizer 172 can change from an unlearned model to a trained model by updating parameters from initial values ​​to optimal values. The initial values ​​of the parameters of the recognizer 172 may be any values, or, for example, parameters of an existing trained model may be applied.

[0275] The recognizer 172 includes an input layer 172A, an intermediate layer 172B, and an output layer 172C, and each layer has a structure in which a plurality of nodes are connected by edges.

[0276] In the learning phase, the input layer 172A receives the engraving print extraction images of the learning data set.

[0277] The intermediate layer 172B is a layer that extracts features from the marking extraction image input from the input layer 172A. The intermediate layer 172B has multiple sets, each of which is a convolution layer and a pooling layer, and a fully connected layer. The fully connected layer connects all the nodes of the previous layer (here, the pooling layer).

[0278] The output layer 172C is a layer that outputs the drug type of the medicine, which is the recognition result of the recognizer 172.

[0279] The error calculation unit 174 obtains the recognition result output from the output layer 172C of the recognizer 172 and the drug type of the medicine, which is the learning data set of the imprinted and printed extraction image input to the input layer 172A, and calculates the error between them. The error calculation method may be, for example, softmax cross entropy or least squared error (MSE: Mean Squared Error), etc.

[0280] Based on the error calculated by the error calculation unit 174, the parameter control unit 176 adjusts the parameters of the recognizer 172 (such as filter coefficients of the convolution layer and deconvolution layer) by the error backpropagation method.

[0281] This parameter adjustment process is repeated, and learning is repeated until the difference between the output of the recognizer 172 and the correct data becomes small.

[0282] Using the learning data set stored in the database 144 in this way, a trained recognizer 172 with optimized parameters is generated as a second trained model. Note that an appropriate amount of noise may be artificially added to the medicine imprint extraction image for training. This allows the generation of a second trained model that has acquired robustness against fluctuations in the imaging environment.

[0283] Here, the input image is an image of the imprinted text on one side of the medicine, but a set of images of the imprinted text on both sides may be used as the input image. In this case, the amount of information is greater than when only one side is used, so improved accuracy can be expected, but usability is reduced because both sides must be photographed consecutively.

[0284] In the recognition phase, the recognizer 172, which is the second trained model, outputs an image of the medicine's inscription mark extracted from an input region image of any medicine.

[0285] The image input unit 142 (an example of a learning data collection unit) may collect a re-learning data set in the database 144 using the stamp print extraction image acquired in the first to third embodiments as an input image and the correct drug determined in the first to third embodiments as correct data. That is, if the recognition by the first drug type recognition unit 110 is correct, the correct drug becomes the correct data, and if the recognition by the first drug type recognition unit 110 is incorrect, the correct drug determined thereafter becomes the correct data. The learning device 170 (an example of a re-learning unit) can re-learn the second trained model 110A using this collected learning data set.

[0286] In addition, the image input unit 142 may collect manually created imprinted and printed extracted images and a re-learning data set of candidate drug types in the database 144. The learning device 170 can re-learn the second trained model 110A even by using a manually created training data set.

[0287] FIG. 38 is a diagram showing an example of a learning dataset for generating a third trained model. The learning dataset is a set of area images of multiple different types of medicines and information on the drug type and the front and back of the area image, which is correct answer data. In the example shown in FIG. 38, I 01, and area image I I Drug type D of 01 C The information that is 1 and the surface constitutes one training data set. Similarly, the region image I I 02, and area image I I Drug type D of 02 C 2 and the surface information constitute one training data set, and the region image I I 03, and area image I I Drug type D of 03 C 3 and the information on the reverse side make up one training data set.

[0288] The third trained model 122A can be generated by training this training data set by the training device 170. Note that an appropriate amount of noise may be artificially added to the area image of the medicine for training. This allows the third trained model 122A to be generated that is robust to changes in the imaging environment.

[0289] The learning data sets for training the first trained model 108A, the second trained model 110A, and the third trained model 122A can be obtained, for example, from an existing packaged audit support system.

[0290] In the single-dose inspection support system, in order to perform highly accurate drug recognition, images are taken from a fixed distance and viewpoint. In addition, partial illumination images with different lighting methods and full illumination images are processed and synthesized for the marking extraction process. For this reason, a large number of appropriate marking extraction images are accumulated as learning data.

[0291] On the other hand, in this embodiment, a standardized image is generated by the image correction unit 104, and then the area image and the stamped and printed extraction image are generated. Therefore, by performing a zoom conversion so that the actual dimensions on the standardized image match the actual dimensions on the image obtained from the packaged inspection support system, the image from the packaged inspection support system can be used as is.

[0292] Therefore, appropriate learning can be performed by using a set of all illumination images, stamped and printed extraction images, composite extraction images, and corresponding drug identification information collected through the single-dose inspection support system.

[0293] The image input unit 142 (an example of a learning data collection unit) may use the area images acquired in the first to third embodiments as input images, and collect a re-learning data set in the database 144, with the correct drug determined in the first to third embodiments as correct data. The learning device 170 (an example of a re-learning unit) may use this collected learning data set to re-learn the third trained model 122A.

[0294] Furthermore, the image input unit 142 may collect manually created area images and a re-learning data set of candidate drug types in the database 144. The learning device 170 can also re-learn the third trained model 122A using a manually created training data set.

[0295] Furthermore, the image input unit 142 may collect a re-learning data set in the database 144 using the area image acquired in the first to third embodiments as an input image and the correct drug determined in the first to third embodiments as correct data, and the learning device 170 (an example of a learning unit) may train a new fourth trained model using this collected training data set. The fourth trained model trained in this way may be used in the second drug type recognition unit 122 in place of the third trained model 122A.

[0296] <Fourth embodiment> When performing a drug search based on the actual drug to be identified, a user generally visually reads the identification characters and symbols printed and / or engraved on the drug, inputs the text information to a search system, and queries the identification character and symbol database. On the other hand, the actual drug is small and difficult to see, and it is particularly difficult to distinguish the identification characters of engraved drugs. In addition, when the search system is a personal computer, input is performed using a keyboard while looking at the personal computer screen, which causes eye movement and hand movement between the actual drug, the personal computer screen, and the keyboard, etc., which causes problems in terms of usability. Therefore, in the fourth embodiment, a highly usable drug search is realized using a mobile terminal device.

[0297] Typically, mobile terminal devices such as smartphones are equipped with cameras, and when taking a picture, it is easy to zoom in and display the enlarged image on the screen. However, since inscribed medicines can still be difficult to read even when enlarged, an AI-based inscription emphasis process is used to present the medicine to the user. The AI ​​used is inscribed emphasis AI, which estimates the inscribed emphasis image from the full-illumination captured image, and inscribed emphasis AI, which calculates the medicine identification information from the inscribed emphasis image alone. acquisition There is an engraving identification AI that performs this function. These AIs may be AIs that artificially add a moderate amount of noise during learning to make them more robust to changes in the shooting environment. Since the engraving emphasis AI learns from teacher images obtained under certain conditions, it is desirable to perform inference under conditions as close as possible to those conditions. This can be achieved by standardizing the viewpoint and distance using ArUco markers, etc., to bring the conditions closer to those under which the teacher images were obtained. In addition, by placing four ArUco markers on a mount and placing multiple medicines inside them, it is possible to stabilize the identification and extraction process of the medicine placement range as an image processing method.

[0298] The medicine identification device according to the fourth embodiment will be described in detail below.

[0299] [Functional configuration of the drug identification device] Fig. 39 is a block diagram showing a functional configuration of a drug identification device 300 realized by the smartphone 10 (see Fig. 1). The same reference numerals are used for parts common to Fig. 5, and detailed description thereof will be omitted. As shown in Fig. 39, the drug identification device 300 has a search character information acquisition unit 302 and a search unit 304.

[0300] The search character information acquisition unit 302 acquires character information of the most likely candidate drug type among the candidate drug types of the drug to be identified output by the candidate output unit 112. The most likely candidate drug type among the candidate drug types of the drug to be identified is, for example, the drug type with the highest score value among multiple candidate drug types of the drug to be identified. The search character information acquisition unit 302 identifies the most likely candidate drug by using a drug recognition AI such as the third trained model 122A, or a combination of drug recognition AI and template matching, and acquires the engraving information of the identified drug type as character information from an engraving database (not shown). The candidate output unit 112 writes the character information of the most likely candidate drug type acquired by the search character information acquisition unit 302 into a text box B B (See Figure 43) and display it.

[0301] The search unit 304 is a text box B B Based on the character information input to the search unit 304, a drug database (not shown) is searched for candidates for the drug type of the drug to be identified. This drug database (not shown) must be a database having the same drug identification key as the above-mentioned engraving database (not shown). The engraving database and the drug database may be a single database. The candidates for the drug type of the drug to be identified searched for by the search unit 304 are displayed in a selectable manner on the touch panel display 14 (see FIG. 1) in the candidate output unit 112.

[0302] In addition, the candidate output unit 112 makes the direction of the marking and / or printing of the image of the medicine displayed on the touch panel display 14 upright (aligns the upright direction). The candidate output unit 112 includes a fifth trained model 112A. The fifth trained model 112A is a trained model that outputs a third image in which the direction of the marking and / or printing of the medicine is upright when a first image of the medicine with the marking and / or printing is given as an input. The fifth trained model 112A is a training data set of multiple different medicines with the marking and / or printing added, and is machine learning performed using a training data set for learning that is a set of an image of the medicine with the marking and / or printing added in any direction and an image of the medicine with the marking and / or printing in the upright direction. The fifth trained model 112A can be applied with CNN, as with the first trained model 108A.

[0303] In addition, the fifth trained model 112A further changes the current rotation angle in order to upright the orientation of the imprint and / or print of the medicine when a first image of the medicine with the imprint and / or print is given as an input. Drugs The fifth trained model 112A may be a trained model that outputs a real value of the rotation angle to be rotated. In this case, the fifth trained model 112A is a training data set of a plurality of different medicines to which markings and / or printings are added, and machine learning is performed using a training data set for learning that is a set of an image of a medicine to which markings and / or printings are added in any orientation and a rotation angle required to further upright the orientation of the markings and / or printings of the medicine from that orientation. The fifth trained model 112A can apply CNN, as with the first trained model 108A.

[0304] The candidate output unit 112 may orient the inscription and / or printing of the image of the medicine to be displayed on the touch panel display 14 in an upright position by checking the rotation direction with a master image through template matching.

[0305] [Drug Identification Method] Fig. 40 is a flowchart showing steps of a medicine identifying method using the medicine identifying device 300. Note that parts common to Fig. 6 are given the same reference numerals and detailed description thereof will be omitted. Figs. 41 to 45 are diagrams showing screen displays D21 to D25 of the touch panel display 14 of the smartphone 10 in each step of the medicine identifying method, respectively.

[0306] In step S1, the image acquisition unit 102 acquires a captured image of a drug to be identified that has been marked and / or printed. FIG. 41 is a diagram showing a screen display D21 when the drug to be identified is captured in step S1. The screen display D21 shows a live view image I of the subject to be captured. LV 2. Tap position marker M T , Brightness correction slider bar B SL , and shooting button B S 1 is displayed.

[0307] Live View Image I LV The live view image I2 is an image of the area captured by the rear camera 22 in real time. LV 2 includes tablets T11, T12, T13, markers M1 to M4, and a reference gray background BG.

[0308] Live View Image I LV When you tap on the pill in 2, the camera adjusts its focal length to focus on that area. T indicates the position on the touch panel display 14 when a tap operation is performed to focus on the pill portion. The tap operation is also used as a trigger to turn on the brightness correction slider bar B SL is displayed. Brightness correction slider bar B SL Adjusts the brightness of the captured image. Brightness correction slider bar B SLincludes a slider SL that can be moved by a user's dragging operation on the touch panel display 14. When the slider SL is moved leftward in Fig. 41, the captured image is corrected to be darker, and when it is moved rightward, the captured image is corrected to be brighter. By changing the position of the slider SL, the user can adjust the brightness of the captured image to a brightness that makes it easy to identify the markings and / or printing of the drug to be identified.

[0309] When the user taps the capture button BS1 on the screen display D21, actual capture is performed by the outer camera 22, and the process proceeds to step S2. The captured image, which is an actual captured still image, is stored in the memory 34 (see FIG. 3).

[0310] In step S2, the medicine detection unit 106 detects the areas of the tablets T11 to T13 from the captured image. Furthermore, in step S3, the imprint extraction unit 108 processes the areas of the tablets T11 to T13 detected in step S2 to obtain imprint extraction images in which the imprints and / or imprints of the tablets T11 to T13 are extracted. Here, the imprint extraction unit 108 further generates a composite extraction image by inverting the brightness of each imprint extraction image and combining it with the image of the areas of the tablets T11 to T13.

[0311] 42 shows a screen display D22 in step S2. The screen display D22 shows a standardized image I of the photographed image actually photographed in step S1. S 2 is displayed. Standardized Image I S The standardized image I2 is an image obtained by cutting out an area surrounded by lines connecting the markers M11 to M14 after standardization processing is performed on the captured image. S The frames F11, F12, and F13 surrounding the areas of the tablets T11, T12, and T13 detected from the standardized image I2, and the numbers N11, N12, and N13 corresponding to the frames F11, F12, and F13 are included in the standardized image I2. S In this example, number N11 is "1," number N12 is "2," and number N13 is "3."

[0312] In addition, standardized image I of screen display D22 S Below 2 are composite extracted images I corresponding to tablets T11, T12, and T13, respectively. W 11, I W 12, and I W 13 is displayed. Synthetic Extraction Image I W 11 corresponds to an image of the drug to be identified in which the marking and / or printing is emphasized, and is an image in which the brightness of the marking extraction image of tablet T11 is inverted and superimposed on the area image of tablet T11. The marking extraction image is an image in which a marking extraction process is performed on the area image. The marking extraction process is a process in which the brightness of the marking or printing part is expressed relatively higher than the brightness of the part other than the marking or printing part.

[0313] Similarly, the synthetic extraction image I W 12 is an image obtained by inverting the brightness of the stamp print extraction image of tablet T12 and superimposing it on the area image of tablet T12, and the composite extraction image I W The image I13 is an image obtained by inverting the brightness of the stamped and printed extracted image of the tablet T13 and superimposing it on the area image of the tablet T13. W 11, I W 12, and I W 13 are displayed from left to right in the order of the values ​​indicated by the numbers N11, N12, and N13.

[0314] On the screen display D22, the standardized image I S Any of the areas of tablets T11 to T13 of No. 2 (any of the areas surrounded by frames F11 to F13), or the composite extracted image I W 11, I W 12, and I W When any one of the buttons 13 is selected by the user tapping the touch panel display 14, the process proceeds to step S4.

[0315] In step S4, the first drug type recognition unit 110 acquires candidates for the drug type from the stamp print extraction image of the drug selected in step S3.

[0316] Next, in step S31 which is a search character information acquisition step, the search character information acquisition unit 302 acquires character information of the most likely candidate drug type among the candidates of the drug type of the identification target drug acquired in step S4.

[0317] FIG. 43 shows a screen display D23 in step S31. Here, the standardized image I S This shows the case where the area of ​​tablet T13 (area surrounded by frame F13) is selected.

[0318] At the top of the screen display D23, there is a region image I of the tablet T13, which is the drug to be identified. R 13. Area Image I R 13 Engraved Print Extracted Images I E 13, and a composite extracted image of tablet T13 I W 13 are displayed side by side. R 13. Engraved print extraction image I E 13, and synthetic extraction image I W At least one of 13 must be displayed.

[0319] Here, the candidate output unit 112 outputs the region image I using the fifth trained model 112A. R 13. Engraved print extraction image I E 13, and synthetic extraction image I W 13, the direction of the engraved and / or printed characters is made upright (an example of a display control step). As a result, in the screen display D23, the area image I R 13. Engraved print extraction image I E 13, and synthetic extraction image I W 13, the engraved and / or printed characters are displayed upright.

[0320] The candidate output unit 112 may upright each image by comparing the rotation direction of each image to be uprighted with that of the master image. R 13. Engraved print extraction image I E 13, and synthetic extraction image I WThe orientation of the engraved and / or printed characters 13 may be changeable by the user rotating the touch panel display 14 on the screen display D23.

[0321] Furthermore, on screen display D23, text box B B , Search button B S 3. Clear button B S 9, and the software keyboard KB is displayed.

[0322] Text Box B B is a search window in which the user can input a character string, and is an interface for identifying the drug type of tablet T13 by the input character string. Note that the character string is not limited to a string consisting of multiple characters, and includes a string consisting of a single character.

[0323] The user inputs a character into the text box B by using the software keyboard KB or the microphone 18 (see FIG. 1). B The user can input and modify a character string in the region image I of the tablet T13 on the screen display D23. R 13. Engraved print extraction image I E 13, and synthetic extraction image I W Refer to 13 and edit text box B. B Enter the text of the engraved and / or printed identification code of tablet T13 and press the Search button B S By tapping 3, the user can obtain candidates for the drug type of the input character string. In addition, the user can tap the clear button B S Tap 9 to open text box B. B You can erase the string entered in .

[0324] Here, Text Box B B In the example shown in FIG. 43, the text information of the most likely candidate drug type acquired in step S31 is automatically input in the text box B BThe character string "k12:#" is automatically entered in the text box B. The part before the ":" in "k12:#" indicates the inscription and / or printing on the front side, which here indicates the characters "k12". The part after the ":" in "k12:#" indicates the inscription and / or printing on the back side, where the "#" indicates a symbol. The automatically entered character string can be corrected using the software keyboard KB or microphone 18. Since the character information of the most likely candidate drug type is entered in advance, even if the AI ​​mistakenly presents a drug with similar inscription characters to the user as the most likely drug, the user can slightly correct the character string and input the character information of the correct drug, which minimizes the user's effort and achieves high usability. In addition, B The character string entered in the Clear button B S It is also possible to erase it with 9.

[0325] Furthermore, the screen display D23 includes a candidate drug display area A C 5. Candidate drug display area A C In 5, multiple candidate drugs are displayed from top to bottom in order of highest score as candidates for the drug type of tablet T13, with the score calculated based on the degree of match between the input character string and the engraved character string in the engraved character database.

[0326] On the screen display D23, the user presses the search button B S When 3 is tapped, the process proceeds to step S5.

[0327] In step S5, the candidate output unit 112 selects the search button B S When 3 is tapped, text box B B Engraving and / or printing based on the character string entered in ofThe search method for obtaining drug type candidates from an input character string may be any of an exact match search, a forward match search, a backward match search, a partial match search, a fuzzy search, a search method that scores the degree of match of a character string according to a specific algorithm, etc. Also, the user may specify a search method to be applied from a selection of multiple search methods.

[0328] FIG. 44 shows a screen display D24 in step S5. Here, the text box B B With the string "k12:#" entered, press Search button B S 3 shows the case where a tap operation is performed.

[0329] At the top of the screen display D24, the area image I of the tablet T13, which is the drug to be identified, is displayed. R 13. Area Image I R 13 Engraved Print Extracted Images I E 13, and a composite extracted image of tablet T13 I W 13 are displayed side by side.

[0330] In addition, in the screen display D24, the software keyboard KB is hidden and the candidate drug display area A C In the candidate drug display area A5, a plurality of candidate drugs having a marking and / or a printed character string similar to the character string "k12:#" are displayed in order of the score of the degree of matching of the marking character string from top to bottom so that the candidate drugs can be selected. C 5, candidate drug C for tablet T13 T 31~C T The master images of the front and back of each of the candidate drug display areas A and B are displayed. The candidate drug display area A and B are displayed. The candidate drug display area B and B are displayed. C In 5, candidate drugs with lower scores are displayed in response to the user swiping from the bottom of the screen to the top.

[0331] In this way, in the screen display D24, the area image I of the tablet T13 R 13. Engraved print extraction image IE 13, and synthetic extraction image I W 13, along with candidate drug C for tablet T13 T 31~C T By displaying the master images of the front and back of tablet 35, it becomes easier for the user to compare these images and select the correct drug for tablet T13.

[0332] In the screen display D24, the user selects a candidate drug display area A C When one drug is selected from the multiple candidate drugs displayed in 5, the process proceeds to step S6.

[0333] In step S6, the determination unit 114 determines the drug selected by the user as the correct drug.

[0334] FIG. 45 shows a screen display D25 in step S6. Here, the candidate drug C is T This shows the case where 31 is selected.

[0335] In screen display D25, candidate drug C T Detailed information on No. 31: Candidate drug C T The information displayed includes the name of the 31 drugs, the original / generic version, master images of the front and back, a two-dimensional barcode showing the identification code, the drug classification, generic name, attributes, manufacturer, and distributor. The user can check this information for the selected drug.

[0336] According to the fourth embodiment, a highly usable drug search can be realized as follows.

[0337] When a user visually reads the identification symbol engraved and / or printed on a drug and searches by text input or voice input, character information extracted from the captured image is automatically entered into a text box, allowing the correct drug to be quickly determined for the drug to be identified.

[0338] The drug search task can be completed on one mobile device, eliminating the need to move one's eyes or change hands. In addition, it is possible to provide an enlarged drug image, an image enhanced by the AI ​​for enhancing the engraving, and an image that combines the two, improving the visibility of the identification characters and symbols for the user.

[0339] Furthermore, multiple drugs to be searched for can be photographed at once, reducing the effort required for the user to take photographs.

[0340] <Fifth embodiment> When identifying medications brought to the hospital by patients (medications brought in by patients) or remaining medications at home, it is important to identify the target patient and link the patient to the medication to be identified correctly. For this reason, it is important to identify the individual, identify the medications owned by the individual, correctly link the patient to the medication to be identified, and then ensure that the data is sent to a management system, etc., during the identification process.

[0341] For example, when a patient with medications brought in by the patient is admitted to a hospital, the patient information is entered into the hospital's electronic medical record system or other management system, a request form for identification of medications brought in by the patient is issued, which lists the patient's name and patient ID (IDentification), and the request form and the medications are given as a set to a pharmacist or other identifier. The identifier identifies the medications brought in by the patient and enters the results of the identification into a management system such as an electronic medical record. The identifier then writes (e.g., stamps) a note of confirmation on the request form for identification of medications brought in by the patient, and the request form for identification of medications brought in by the patient and the identified medications are returned to the requester.

[0342] In accordance with this flow, the fifth embodiment provides a patient-brought medication identification request form and a medication identification system for identifying patients and reliably identifying medications brought in by patients.

[0343] [Configuration of drug identification system] 46 is a block diagram showing the configuration of a drug identification system 310 according to the fifth embodiment. As shown in FIG. 46, the drug identification system 310 includes a drug identification device 300, a discrimination request form generating device 320, and an electronic medical record system 340.

[0344] As described in the fourth embodiment, the drug identification device 300 is realized by, for example, the smartphone 10. The drug identification device 300 according to the fifth embodiment includes a barcode identification unit 306. The barcode identification unit 306 identifies a barcode included in an image. Here, the barcode included in an image captured by the in-camera 20 (see FIG. 1) or the out-camera 22 (see FIG. 2) of the smartphone 10 is identified.

[0345] The identification request generating device 320 is a device for generating an identification request for a brought-in drug to request an identifier to identify a brought-in drug of a patient. A personal computer or a workstation can be used as the identification request generating device 320.

[0346] As shown in FIG. 46, the discrimination request form generating device 320 includes a personal identification information acquiring unit 322 , an output unit 324 , an operation unit 326 , a display unit 328 , a CPU 330 , a RAM 332 , and a ROM 334 .

[0347] The personal identification information acquisition unit 322 is an input interface that acquires information that identifies the individual who is the owner of the brought-in medicine to be identified, and includes wired and wireless communication interfaces.

[0348] The output unit 324 is a device that outputs the brought-medicine discrimination request form, for example, a printing device that prints the brought-medicine discrimination request form on a print medium. The output unit 324 may be a printing device provided separately from the discrimination request form generating device 320, or may be a display device such as electronic paper or an organic EL (Electro-Luminescence) film display that displays the generated brought-medicine discrimination request form.

[0349] The operation unit 326 is a user interface for the user to control the discrimination request form generating device 320, and is composed of a keyboard and a pointing device.

[0350] The display unit 328 is an output interface that visually displays the status of the identification request form generating device 320, and is configured with a display panel.

[0351] The hardware structures of the CPU 330 , the RAM 332 , and the ROM 334 are similar to those of the CPU 150 , the RAM 152 , and the ROM 154 .

[0352] The electronic medical record system 340 is a system for managing all of the medical information of patients as electronic information. As the electronic medical record system 340, a personal computer or a workstation can be used.

[0353] As shown in FIG. 46, the electronic medical record system 340 has an input unit 342, a display unit 344, a database 346, a CPU 348, a RAM 350, and a ROM 352.

[0354] The input unit 342 is a user interface for the user to control the electronic medical record system 340, and is composed of a keyboard and a pointing device. The input unit 342 includes an identification code reader 342A that reads an identification code such as a GS1 code or a two-dimensional code.

[0355] The display unit 344 is an output interface that visually displays the status of the electronic medical record system 340, and is configured with a display panel.

[0356] The database 346 is a storage device in which electronic medical records of multiple patients are stored, and may be, for example, a hard disk drive.

[0357] The hardware structures of the CPU 348 , the RAM 350 , and the ROM 352 are similar to those of the CPU 150 , the RAM 152 , and the ROM 154 .

[0358] The drug identification device 300, the identification request form generating device 320, and the electronic medical record system 340 are connected to each other so as to be able to perform data communication via a network 1. The identification request form generating device 320 may be included in the electronic medical record system 340.

[0359] [Drug Identification Method] 47 is a flowchart showing the steps of the medicine identification method using the medicine identification system 310. Here, an example will be described in which, when a patient who brings his / her own medicine is admitted to a hospital, the patient information is linked to the information of the brought-in medicine and registered in the electronic medical record.

[0360] In step S41, the personal identification information acquisition unit 322 of the identification request generation device 320 acquires information for identifying a patient. Here, the identification requester, who is a hospital staff member, inputs information for identifying a patient to the personal identification information acquisition unit 322 using the operation unit 326. The information for identifying a patient is, for example, a patient ID. The patient ID is, for example, an identifier assigned to each patient to identify the patient. In addition to the identification information for identifying the patient, the patient ID may be added with additional information related to the identification, such as information on the time of taking the drug or a serial number of the same individual, issued from the electronic medical record system 340. The personal identification information acquisition unit 322 acquires patient information from the electronic medical record system 340 based on the acquired patient ID. The patient information includes information on the patient's name, sex, date of birth, ward, room, medical department, and attending physician.

[0361] In step S42, the output unit 324 uses a printing device to print a brought-medicine discrimination request form (an example of an output) by arranging a personal information display area and a placement area on a printing medium. The printing medium is, for example, A4 size (297 mm length × 210 mm width) high-quality paper. FIG. 48 is a diagram showing an example of a brought-medicine discrimination request form D30 printed by the output unit 324. As shown in FIG. 48, the brought-medicine discrimination request form D30 includes information such as the request date, the requester, the personal information display area A, and the like. I , Placement area A P , and a space for the verifier to stamp it.

[0362] Personal information display area A Iis an area for displaying the patient information acquired in step S41. I The patient ID includes information on the patient's name, sex, date of birth, ward, room, medical department, and doctor in charge. The patient ID also includes a barcode BC indicating the patient ID.

[0363] Placement area A P is an area for placing the patient's own medication, which is the medication to be identified, when photographing the medication. P The placement area A includes a background BG, which is a reference gray color area, and four markers M11, M12, M13, and M14 arranged at the four corners of the background BG. P The request form for identification of brought-in medication may not be printed by a printing device, but may instead be placed on a separately prepared drug placement platform made of paper, fiber, rubber, glass, or plastic.

[0364] The brought-medicine discrimination request form D30 printed in step S42 is handed over from the discrimination requester to the discriminator.

[0365] In step S43, the discriminator uses the in-camera 20 or out-camera 22 of the medicine discrimination device 300 (smartphone 10) to display the personal information display area A of the brought-medicine discrimination request form D30 output in step S42. I The barcode BC indicating the patient ID is captured in the captured image. The barcode identification unit 306 identifies the barcode BC included in the captured image, and specifies the patient ID.

[0366] In step S44, the discriminator further selects the place area A of the brought-medicine discrimination request form D30. P The patient places the patient's own medicine on the medicine identification request form D30 and photographs the patient's own medicine using a camera (not shown) of the medicine identification device 300. P This is a diagram showing how a patient's medicine is placed in a single dose and photographed in the placement area A. PFour tablets T21, T22, T23, and T24 are placed on the patient's brought-in medication identification request form D30. The unit of photography for one session may be a single dose per time point, a daily dose, or another unit of photography, as long as it matches the unit expected as input by the electronic medical record system 340. In addition, when the amount of medicine per session is large, the medicines at the same time point may be photographed separately on multiple brought-in medication identification request forms D30, or multiple placement areas A on the brought-in medication identification request form D30 may be used. P The layout may be such that multiple images are taken of a single brought-in medication identification request form D30.

[0367] In step S45, the drug identification device 300 determines the drug types of the four tablets T21, T22, T23, and T24 based on the images captured in step S43, similarly to the fourth embodiment.

[0368] Medications other than single-dose drugs (such as PTP sheets) that have a GTIN (Global Trade Identification Number, a GS1 standard barcode) attached and whose barcode can be read may be identified by comparing the photographed image with a master image, or those whose drug name is known may be identified by a text search.

[0369] In step S46, the drug identification device 300 inputs the patient ID identified in step S43 as the identification result of the brought-in drug and the drug types of the four tablets T21, T22, T23, and T24 identified in step S45 in association with each other to the electronic medical record system 340. The electronic medical record system 340 registers the patient ID and the drug types of the tablets T21, T22, T23, and T24 in association with each other.

[0370] The drug identification device 300 may compile the discrimination results into an electronic file format and transmit it to the electronic medical record system 340 via the network 1. The drug identification device 300 may display text information of the drug name in the form of a barcode near the read drug name, and input the barcode to the electronic medical record system 340 as text information by having an identification code reader 342A of the electronic medical record system 340 read the barcode.

[0371] Thereafter, the discriminator affixes a seal to the brought-medicine discrimination request form D30, returns the brought-medicine discrimination request form D30 and the discriminated tablets T21 to T24 to the discriminator, and ends the process of this flowchart.

[0372] As described above, according to the fifth embodiment, it is possible to identify the patient and link the patient to the drug to be identified, thereby reliably identifying the drug brought by the patient. In the drug identification method of the fifth embodiment, steps S41 and S42 constitute a method for producing an output product.

[0373] <Other aspects of the fiducial marker> Deep learning (an example of "machine learning") could be used to further improve the accuracy of detecting fiducial markers. However, the ArUco markers, which have a logical structure, have the disadvantage that they are not very compatible with deep learning.

[0374] When detecting the area of ​​the drug to be identified from a captured image using deep learning, the format of the teacher data used in detection is generally (center point coordinates, width, height), (center point coordinates, width, height, rotation angle), or a mask image that fills in the object shape, etc. Note that "height" here means the length in the direction perpendicular to "width".

[0375] Of these, the use of (center point coordinates, width, height, rotation angle) is considered to be the top priority because it is relatively light in information volume and is suitable for detecting oval-shaped medicines. If the rotation angle is not used, when multiple oval-shaped medicines are lined up diagonally, medicines other than the medicine to be detected may be included in the bounding box of the medicine to be detected.

[0376] Fig. 50 is a diagram showing the medicine in the photographed image and the bounding box of the correct answer data. F50A shown in Fig. 50 shows the correct answer data by (center point coordinates, width, height, rotation angle). The bounding box B1 shown in F50A is a rectangle that rotates in the photographed image at a rotation angle according to the orientation of the oval tablet T31 to be detected and circumscribes the area of ​​the tablet T31.

[0377] Also, F50B shown in FIG. 50 indicates the correct answer data based on (center point coordinates, width, height). The bounding box B2 shown in F50B is a rectangle that circumscribes the area of ​​tablet T31 without rotation. In the example shown in F50B, the bounding box B2 includes not only the area of ​​tablet T31 but also the area of ​​tablet T32.

[0378] For circular drugs, due to their infinite symmetry, an infinite number of correct answers (center point coordinates, width, height, rotation angle) can be defined; however, by always setting the rotation angle to 0 degrees when creating training data, it is possible to configure it so that a unique inference can be made.

[0379] The reference markers arranged on the placement surface of the medicine placement table 74 may be detected using deep learning in the same way as the medicine. In this case, it is necessary to create teacher data for a captured image (pre-standardized image) that is not standardized in terms of shooting distance and shooting viewpoint. Therefore, it is necessary to consider how the reference markers captured together with the medicine are captured in the pre-standardized image and how they are detected.

[0380] Fig. 51 is a diagram for explaining detection of a reference marker when the reference marker is a rectangle. F51A shown in Fig. 51 shows a captured image IC2 that is a pre-standard image including a rectangular reference marker. F51B shown in Fig. 51 shows a marker region IM1 that is the region of the reference marker acquired from the captured image IC2, and bounding boxes B11, B12, B13, and B14 that correspond to the marker region IM1.

[0381] As shown in F51A, when the reference marker is a rectangle, in the captured image IC2, in the marker area corresponding to one reference marker, there may be cases where none of the four sides of the rectangle are parallel to each other. When defining the correct answer data (center point coordinates, width, height, rotation angle) in such a quadrilateral marker area IM1, as shown in F51B, many types of bounding boxes may be defined. The existence of this arbitrariness is undesirable because it may cause conflicts in learning, reducing the overall detection accuracy, and has disadvantages such as making it difficult to create training data.

[0382] On the other hand, when a circular reference marker is used, the circle is distorted into an ellipse with a sense of perspective in the pre-standardized image, but the rotation angle can always be set to zero degrees (center point coordinates, width, height, rotation angle) and given as training data.

[0383] Fig. 52 is a diagram for explaining marker detection when the reference marker is circular. F52A in Fig. 52 shows a marker region IM2 acquired from a pre-standardized image and a bounding box B21 corresponding to the marker region IM2. The bounding box B21 is a rectangle circumscribing the marker region IM2, and has a rotation angle of 0 degrees. F52B in Fig. 52 shows the marker region IM2 and a bounding box B22 corresponding to the marker region IM2. The bounding box B22 is a rectangle circumscribing the marker region IM2, and is rotated in the captured image by a rotation angle according to the orientation of the marker region IM2.

[0384] Fig. 53 is a diagram showing a specific example of a reference marker with a circular outer shape. F53A shown in Fig. 53 shows a circular marker MC1. The circular marker MC1 has an outer perfect circle C1 with a relatively large diameter, and an inner perfect circle C2 that is arranged concentrically with the perfect circle C1 and has a relatively smaller diameter than the perfect circle C1. That is, the perfect circles C1 and C2 are circles with different radii arranged at the same center. In addition, the inside of the perfect circle C2 of the circular marker MC1 is white, and the area inside the perfect circle C1 and outside the perfect circle C2 is filled in black.

[0385] The diameter of the perfect circle C1 is preferably 3 to 20 millimeters. The diameter of the perfect circle C2 is preferably 0.5 to 5 millimeters. Furthermore, the ratio of the diameters of the perfect circle C1 and the perfect circle C2 (diameter of the perfect circle C1 / diameter of the perfect circle C2) is preferably 2 to 10. In order to further improve the estimation accuracy of the center coordinates, a black circle (not shown) (for example, a perfect circle) with a relatively smaller diameter than the perfect circle C2 may be placed concentrically inside the perfect circle C2.

[0386] F53B shown in FIG. 53 indicates a marker region IM3 which is the region of the circular marker MC1 acquired from a pre-standardization image including the circular marker MC1.

[0387] There is a possibility that the coordinates of the true center point of an object in a pre-standardized image may deviate from the coordinates of the center point estimated by machine learning. By providing the coordinates of the center point of the inner perfect circle C2 as training data, as in the case of the circular marker MC1, machine learning can accurately and easily estimate the center coordinates of the true object. In addition, the presence of the relatively large outer perfect circle C1 can significantly reduce the possibility of false detection due to dust or the like attached to the circular marker MC1.

[0388] FIG. 54 is a top view of the medicine placement table 74 using a circular marker MC1. As shown in FIG. 54, the medicine placement table 74 has circular markers MC1 arranged at four corners as reference markers 74A, 74B, 74C, and 74D. F54A shown in FIG. 54 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a square, and F54B shown in FIG. 54 shows an example in which the centers of the reference markers 74A, 74B, 74C, and 74D form the four vertices of a rectangle. The reason for arranging four reference markers is that the coordinates of four points are required to determine a perspective transformation matrix for standardization.

[0389] Here, the four reference markers 74A, 74B, 74C, and 74D are the same size and color, but the sizes and colors may be different. If the sizes are different, it is preferable to arrange the adjacent reference markers 74A, 74B, 74C, and 74D so that their centers form the four vertices of a square or rectangle. By making the sizes or colors of the reference markers different, it becomes easier to identify the shooting direction.

[0390] Furthermore, at least four circular markers MC1 may be arranged, and five or more may be arranged. When five or more circular markers MC1 are arranged, it is preferable that the centers of the four circular markers MC1 form the four vertices of a square or a rectangle, and the centers of the additional circular markers MC1 are arranged on the sides of the square or the rectangle. By arranging five or more reference markers, it becomes easier to specify the shooting direction. Furthermore, by arranging five or more reference markers, even if detection of any of the reference markers fails, it is possible to increase the probability of simultaneously detecting the minimum four points required to obtain a perspective transformation matrix for standardization, and there is an advantage in that the effort of re-shooting is reduced.

[0391] FIG. 55 is a top view of the medicine placement table 74 using a circular marker according to a modified example. F55A shown in FIG. 55 is a diagram showing an example in which a circular marker MC2 is used as the reference markers 74A, 74B, 74C, and 74D. The circular marker MC2 is arranged (concentrically) inside a perfect circle filled with black, with a cross-shaped figure consisting of two white straight lines that intersect at right angles, so that the intersection of the two straight lines coincides with the center of the perfect circle. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC2 are arranged so that their respective centers form the four vertices of a square, and the straight lines of the cross-shaped figure of the circular marker MC2 are arranged parallel to the sides of this square. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC2 may be arranged so that their respective centers form the four vertices of a rectangle. The thickness of the lines of the cross-shaped figure of the circular marker MC2 can be determined appropriately.

[0392] On the other hand, F55B shown in FIG. 55 is a diagram showing an example in which a circular marker MC3 is used as the reference markers 74A, 74B, 74C, and 74D. The circular marker MC3 has two perfect circles with different radii arranged at the same center, the inside of the inner perfect circle is white, and the inside of the outer perfect circle and the outside of the inner perfect circle are filled in black. Furthermore, the circular marker MC3 has a cross-shaped figure consisting of two black straight lines that are perpendicular to each other arranged inside the inner perfect circle so that the intersection of the two straight lines coincides with the center of the perfect circle. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC3 are arranged so that their respective centers form the four vertices of a square, and the straight lines of the cross-shaped figure of the circular marker MC3 are arranged parallel to the sides of this square. The reference markers 74A, 74B, 74C, and 74D of the circular marker MC3 may be arranged so that their respective centers form the four vertices of a rectangle. The thickness of the lines of the cross-shaped figure of the circular marker MC3 can be determined appropriately.

[0393] The circular markers MC2 and MC3 can improve the accuracy of estimating the center coordinates. In addition, the circular markers MC2 and MC3 have a different appearance from the medicine, making the markers easier to recognize.

[0394] Fig. 56 is a diagram showing a specific example of a reference marker with a rectangular outline. F56A shown in Fig. 56 shows a rectangular marker MS. The rectangular marker MS has an outer square SQ1 with a relatively long side length, and an inner square SQ2 that is arranged concentrically with the square SQ1 and has a relatively shorter side length than the square SQ1. That is, the squares SQ1 and SQ2 are quadrangles with different side lengths arranged at the same center (center of gravity). In addition, the inside of the square SQ2 of the rectangular marker MS is white, and the area inside the square SQ1 and outside the square SQ2 is filled in black.

[0395] The length of one side of the square SQ1 is preferably 3 to 20 millimeters. The length of one side of the square SQ2 is preferably 0.5 to 5 millimeters. Furthermore, the ratio of the lengths of one side of the squares SQ1 and SQ2 (length of one side of the square SQ1 / length of one side of the square SQ2) is preferably 2 to 10. To further improve the accuracy of estimating the center coordinates, a black rectangle (not shown, for example a square) with a side length relatively shorter than that of the square SQ2 may be placed concentrically inside the square SQ2.

[0396] F56B shown in Fig. 56 shows a top view of the medicine placing table 74 using a quadrilateral marker MS. As shown in F56B, the medicine placing table 74 has quadrilateral markers MS arranged at four corners as reference markers 74A, 74B, 74C, and 74D. Here, an example is shown in which the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D form a square, but the lines connecting the centers of adjacent reference markers 74A, 74B, 74C, and 74D may form a rectangle.

[0397] Circular markers MC and rectangular markers MS may be mixed on the medicine placing table 74. By mixing the circular markers MC and rectangular markers MS, it becomes easier to specify the imaging direction.

[0398] As described above, when detecting reference markers based on deep learning, it is preferable to adopt circular or rectangular markers with simpler structures.

[0399] Furthermore, it is preferable to employ circular markers rather than square markers, because when detecting the reference markers using a mobile terminal device such as a smartphone, the following requirements (a) to (c) arise. (a) Since mobile terminal devices have capacity limitations, it is desirable to perform marker detection and drug detection using the same trained model. (b) In drug detection, we would like to perform angle inference because of the presence of oval tablets. Due to request (a), we need to prepare training data for reasonable rotation angles for marker detection as well. (c) In the case of a rectangular marker, it is difficult to create reasonable training data for the rotation angle in the pre-standardized image, which is the input image when detecting the marker, but this is possible for a circular marker.

[0400] In addition, by making the circular marker concentric, the estimation accuracy of the marker's center coordinate can be improved. In the pre-standardized image, a simple circular marker is distorted in shape, which makes it easy for errors to occur in estimating the center coordinate, but the inner circle of the concentric circle has a narrower range, so the trained model can easily identify the center coordinate even in a distorted pre-standardized image. Furthermore, the outer circle of the concentric circle has the advantage that the structure is large, making it easy for the trained model to find, and it is robust against noise and debris. In addition, by making the square marker concentric, it is possible to improve the estimation accuracy of the marker's center point coordinate.

[0401] Sixth embodiment In the sixth embodiment, in response to the above requirement (a), two detection tasks, namely, detection of reference markers in pre-standardized images and drug detection in standardized images, are performed using one trained model.

[0402] [Functional configuration of the drug identification device] FIG. 57 is a block diagram showing the functional configuration of a drug identification device 400 realized by the smartphone 10 (see FIG. 1). Note that parts common to FIG. 5 are given the same reference numerals, and detailed description thereof will be omitted. As shown in FIG. 57, the image correction unit 104 and the drug detection unit 106 of the drug identification device 400 include a sixth trained model 104A. Note that it is sufficient for the image correction unit 104 and the drug detection unit 106 to commonly use a single sixth trained model 104A stored in the memory 34 (see FIG. 3), and it is not necessary for the image correction unit 104 and the drug detection unit 106 to each have the same model.

[0403] The sixth trained model 104A is a trained model that, when an image (pre-standardized image, standardized image) is given as an input, outputs a rotated rectangular bounding box (center coordinates, width, height, rotation angle) corresponding to the area of ​​the detected object, the object class, and the probability of object-likeness. The sixth trained model 104A can apply CNN, as with the first trained model 108A. The sixth trained model 104A is not particularly limited as long as it can estimate the rotated rectangular bounding box (center coordinates, width, height, rotation angle) of an object, the object class, and the probability of object-likeness.

[0404] The object classes include at least "markers" and "medicines". The "markers" and "medicines" may be divided into smaller groups. For example, in order to distinguish the four positions of the markers, markers of different shapes and sizes may be classified into different classes. In addition, in anticipation of the subsequent medicine identification process, the medicines may be classified into "round tablets", "oval tablets", and "capsules".

[0405] [Creating a learning dataset] The creation of a learning data set for training the sixth trained model 104A will be described. To train the sixth trained model 104A, a first learning data set that is a learning data set of a reference marker and a second learning data set that is a learning data set of a drug are required.

[0406] The first training dataset consists of multiple images that satisfy the following conditions:

[0407] Pre-standardized images taken at various shooting distances and viewpoints.

[0408] Only multiple circular markers MC or multiple square markers MS are visible (the drug is not visible).

[0409] The center coordinates, width, height, and rotation angle of the rotated rectangular bounding box indicating the area of ​​the circular marker MC and the square marker MS are given as the ground truth data.

[0410] For the circular marker MC and the quadrilateral marker MS, a rotated rectangular bounding box with a rotation angle of 0 degrees is always assigned as the ground truth data. The rotated rectangular bounding box is assigned so that a rectangle circumscribes the circular marker MC or the quadrilateral marker MS.

[0411] The class "marker" is assigned to the circular marker MC and the square marker MS.

[0412] The reason why a pre-standardized image that does not show the medication is used is that it is difficult to create training data for the rotated rectangular bounding box of the medication in a distorted pre-standardized image.

[0413] FIG. 58 is a diagram showing an example of an image included in the first learning data set. F58A shown in FIG. 58 shows a captured image which is a pre-standardized image, and F58B shown in FIG. 58 shows a captured image to which correct answer data has been added. As shown in F58A, the captured image IC3 which is a pre-standardized image includes four marker regions IM4, IM5, IM6, and IM7. Also, as shown in F58B, in the captured image IC4 in which correct answer data has been added to the captured image IC3, rotated rectangular bounding boxes B41, B42, B43, and B44 which are correct answer data are added to the four marker regions IM4, IM5, IM6, and IM7, respectively.

[0414] On the other hand, the second training data set is composed of multiple images that satisfy the following conditions:

[0415] -Standardized shooting distance and viewpoint Kaga image.

[0416] The marker is not visible, only the drug.

[0417] The center point coordinates, width, height, and rotation angle of the rotated rectangular bounding box of the drug are given as ground truth data.

[0418] Drugs are given the class "Drug".

[0419] Fig. 59 is a diagram showing an example of an image included in the second learning data set. As shown in Fig. 59, the standardized image IS3 includes the regions of three tablets T41, T42, and T43. Rotated rectangular bounding boxes B51, B52, and B53 are added to the regions of the three tablets T41 to T43, respectively, and the center point coordinates, width, height, and rotation angle of the rotated rectangular bounding boxes B51 to B53 are given as correct answer data.

[0420] [Learning method] 60 is a flowchart showing an example of a learning method (learning phase) of the sixth trained model 104A. The learning method is realized by the CPU 28 reading and executing a learning program from the memory 34. The learning program may be provided via the wireless communication unit 30 or the external input / output unit 40 (see FIG. 3 for each). The learning method may be realized by a computer different from the drug identification device.

[0421] In step S51, the CPU 28 creates a plurality of first learning data sets. In addition, in step S52, the CPU 28 creates a plurality of second learning data sets. The processes in steps S51 and S52 may be performed sequentially instead of in parallel.

[0422] In step S53, the multiple first learning data sets created in step S51 and the multiple second learning data sets created in step S52 are randomly shuffled. In addition, in step S54, a CNN (an example of a "learning model") is trained using the first learning data sets and the second learning data sets in the order shuffled in step S53.

[0423] In step S55, the CNN trained in step S54 is output as the sixth trained model 104A.

[0424] [Inference method] 61 is a flowchart showing an example of an inference method (inference phase) by the sixth trained model 104 A. Each step of the inference method is included in the image acquisition step and the drug detection step of the drug identification method.

[0425] In step S61, a pre-standardized image in which a marker and a medicine are captured and the capturing distance and the capturing viewpoint are indefinite is input to the image correction unit 104 (see FIG. 57). The pre-standardized image is, for example, an image captured by the outer camera 22 of the smartphone 10.

[0426] In step S62, the image correction unit 104 performs a first inference for marker detection using the sixth trained model 104A on the pre-standardized image input in step S61. The image correction unit 104 extracts only objects whose class is determined to be "marker" in the inference result, and extracts four center point coordinates of the marker. Note that objects whose class is inferred to be "medicine" are discarded here.

[0427] In step S63, the image correction unit 104 performs perspective transformation based on the four center point coordinates extracted in step S62, and obtains a post-perspective transformation image (standardized image) in which the shooting distance and shooting viewpoint are standardized. Note that when an image is cut out based on the center points of the four markers, one-fourth of the marker is cut out at each of the four corners of the standardized image, and this portion may be masked as necessary.

[0428] In step S64, the drug detection unit 106 performs a second inference on the standardized image acquired in step S63. In step S65, the drug detection unit 106 acquires a rotated rectangular bounding box of the drug appearing in the standardized image as the inference result of step S64.

[0429] The drug detection unit 106 cuts out the area of ​​each drug from the standardized image using the bounding box acquired in step S64. This allows the drug identification device 400 to perform the above-mentioned imprint extraction step, the first drug type recognition step, and the like.

[0430] The sixth embodiment has the following advantages.

[0431] The sixth pre-trained model detects markers on pre-standardized images and Kaga This allows the system to perform two tasks: detecting drugs in images and detecting viruses. This is advantageous when running trained models on mobile devices, which are subject to the capacity limitations of trained models.

[0432] Because the pre-standardized image is distorted, it is difficult to define training data (especially the rotation angle) for the rotated rectangular bounding box of oval tablets and capsules in the pre-standardized image. However, in this method, the training data that shows the medicine is standardized. Kaga This problem does not occur because only the image needs to be created and only the markers appear in the pre-standardized image.

[0433] In this method, training data on the rotation angle of the rotated rectangular bounding box must be added to the markers in the pre-standardized image, but since it is sufficient to simply provide the markers with correct answer data whose rotation angle is always 0 degrees, it is possible and easy to create training data. Note that it is also possible to provide correct answer data whose rotation angle is 0 degrees when using a square marker, but since the square marker has straight lines, the sixth trained model may be confused during inference. In this regard, it is preferable to use a circular marker, which has a smooth structure.

[0434] Since the sixth trained model has learned to detect markers and drugs, it is possible to use a pre-standardized image that contains both drugs and markers even during the first inference.

[0435] From the above description, it is possible to understand the drug identification device and trained model described below.

[0436] [Appendix 1] an image acquisition unit that acquires an image of the drug to be identified and the marker; a marker detection unit that detects a region of the marker from the captured image using a sixth trained model that receives an image of an object and outputs at least a class of the object and a region of the object; an image correction unit that standardizes a shooting distance and a shooting viewpoint of the captured image based on the marker to obtain a standardized image; A drug detection unit that detects a region of the drug to be identified from the standardized image using the sixth trained model; A drug identification device having:

[0437] [Appendix 2] The image acquisition unit acquires the captured image in which at least four circular markers or at least four rectangular markers are captured. 2. A drug identification device as described in appendix 1.

[0438] [Appendix 3] The at least four circular markers or the at least four quadrangular markers are arranged at positions where their centers form four vertices of a square or a rectangle. 3. A drug identification device as described in appendix 2.

[0439] [Appendix 4] The circular marker includes concentric circles, and the square marker includes concentric squares. 4. A drug identification device as described in appendix 2 or appendix 3.

[0440] [Appendix 5] The drug to be identified has a mark and / or a print added thereto; a drug detection unit that detects a region of the drug to be identified from the standardized image; an imprint extraction unit for processing at least the region of the drug to be identified in the standardized image to obtain an imprint extraction image in which the imprint and / or print of the drug to be identified is extracted; a first drug type recognition unit for inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates for the drug type of the drug to be identified; 5. A drug identification device according to any one of claims 1 to 4, comprising:

[0441] [Appendix 6] At least one processor; at least one memory storing instructions for execution by said at least one processor; Equipped with The at least one processor: Acquire a photographed image of the drug to be identified and the marker; Detecting the area of ​​the marker from the captured image using a sixth trained model that inputs an image of an object and outputs at least an object class and an object area; standardizing the shooting distance and the shooting viewpoint of the captured image based on the marker to obtain a standardized image; Detecting a region of the drug to be identified from the standardized image using the sixth trained model; Drug identification device.

[0442] [Appendix 7] an image acquisition step of acquiring an image of the drug to be identified and the marker; a marker detection step of detecting a region of the marker from the captured image using a sixth trained model that inputs an image of an object and outputs at least a class of the object and a region of the object; an image correction step of standardizing a shooting distance and a shooting viewpoint of the captured image based on the marker to obtain a standardized image; A drug detection step of detecting a region of the drug to be identified from the standardized image by the sixth trained model; A drug identification method comprising the steps of:

[0443] [Appendix 8] A trained model that has been machine-learned using a first training dataset that is a set of captured images of multiple markers placed on a mounting surface and the center point coordinates, width, height, rotation angle, and correct answer data for the class indicating the markers, and a second training dataset that is a set of standardized images in which the shooting distance and shooting viewpoint of a captured image of a medication are standardized and the center point coordinates, width, height, rotation angle, and correct answer data for the class indicating the medication.

[0444] [Appendix 9] A step of training a learning model using a first learning data set including a set of captured images of a plurality of markers arranged on a placement surface, center point coordinates, width, height, and rotation angle of the markers, and correct answer data of a class indicating the markers; A step of training the learning model using a second learning data set including a standardized image in which the shooting distance and shooting viewpoint of the image of the medicine are standardized, and a center point coordinate, width, height, rotation angle, and correct answer data of the class indicating the medicine; How to create a trained model, including

[0445] <Other aspects of the medicine placement table> The placement surface of the medicine placement table of the photographing assistant device may be provided with a recess structure for placing the medicine. The recess structure includes a recess, a groove, a depression, and a hole.

[0446] Fig. 62 is a diagram showing a medicine mounting stand 410 having a recess structure, which is used instead of or in addition to the medicine mounting stand 74 (see Fig. 23). The medicine mounting stand 410 is made of paper, synthetic resin, fiber, rubber, or glass. F62A shown in Fig. 62 is a top view of the medicine mounting stand 410. As shown in F62A, the mounting surface of the medicine mounting stand 410 is gray, and reference markers 74A, 74B, 74C, and 74D are arranged at the four corners.

[0447] Furthermore, a total of nine recesses 410A, 410B, 410C, 410D, 410E, 410F, 410G, 410H, and 410I are provided in a 3 row x 3 column arrangement on the mounting surface of the medicine mounting stand 410. The recesses 410A to 410I are each circular and of the same size when viewed from above.

[0448] F62B shown in Fig. 62 is a 62-62 cross-sectional view of the medicine placing stand 410. As shown in F62B, the recesses 410A, 410B, and 410C have a hemispherical bottom surface and are the same depth. The same is true for the recesses 410D to 410I.

[0449] Also, F62B shows tablets T51, T52, and T53 placed in recesses 410A, 410B, and 410C, respectively. The tablets T51 and T52 are medicines that are circular in top view and rectangular in side view. The tablet T53 is a medicine that is circular in top view and elliptical in side view. In top view, the tablets T51 and T53 have the same size, and the tablet T52 is relatively smaller than the tablets T51 and T53. As shown in F62B, the tablets T51 to T53 are placed still by fitting into the recesses 410A to 410C. The tablets T51 to T53 may be circular in top view, and may be linear in left and right and arc-shaped in top and bottom in side view.

[0450] In this way, the medicine mounting table 410 has a hemispherical recess structure on the mounting surface, which prevents the circular medicine from moving when viewed from above and allows it to be placed still. In addition, the position of the medicine when photographing it can be determined to be the position of the recess structure, making it easy to detect the area of ​​the medicine.

[0451] The medicine placement table may have a recess structure for rolling medicine such as capsules. Figure 63 is a diagram showing a medicine placement table 412 provided with a recess structure for capsules. Note that the same reference numerals are used for the parts common to Figure 62, and detailed explanations thereof will be omitted.

[0452] F63A shown in Fig. 63 is a top view of the medicine mounting stand 412. As shown in F63A, a total of six recesses 412A, 412B, 412C, 412D, 412E, and 412F in a 3-row x 2-column arrangement are provided on the mounting surface of the medicine mounting stand 412. The recesses 412A to 412F are each rectangular and of the same size when viewed from above.

[0453] F63B shown in FIG. 63 is a 63-63 cross-sectional view of the medicine placement stand 412. As shown in F63B, the recesses 412A, 412B, and 412C have semi-cylindrical bottoms and are the same depth. The same is true for the recesses 412D to 412F. F63B also shows capsules CP1, CP2, and CP3 placed in the recesses 412A, 412B, and 412C, respectively. The capsules CP1 to CP3 are cylindrical with hemispherical ends (bottoms) and have different diameters. As shown in F63B, the capsules CP1 to CP3 are placed in the recesses 412A to 412C by fitting thereinto.

[0454] In this way, the medicine mounting table 412 has a semi-cylindrical recess structure on the mounting surface, which prevents the cylindrical capsule from moving or rolling and allows it to be placed still. In addition, the position of the medicine when photographing can be determined to be the position of the recess structure, making it easy to detect the area of ​​the medicine.

[0455] The medicine mounting table may have a recess structure for an oval tablet. Figure 64 is a diagram showing a medicine mounting table 414 provided with a recess structure for an oval tablet. Note that the same reference numerals are used to designate parts common to Figure 62, and detailed descriptions thereof will be omitted.

[0456] F64A shown in Fig. 64 is a top view of the medicine mounting stand 414. As shown in F64A, a total of six recesses 414A, 414B, 414C, 414D, 414E, and 414F are provided in a 3-row x 2-column arrangement on the mounting surface of the medicine mounting stand 414. Each of the recesses 414A to 414F has a rectangular shape when viewed from above.

[0457] Indentations 414A and 414B are the same size. Indentations 414C and 414D are the same size and relatively smaller than indentations 414A and 414B. Indentations 414E and 414F are the same size and relatively smaller than indentations 414C and 414D.

[0458] Also, F64A shows tablets T61, T62, and T63 placed in recesses 414B, 414D, and 414F, respectively. As shown in F64A, recesses 414B, 414D, and 414F are sized to correspond to tablets T61, T62, and T63, respectively.

[0459] F64B shown in Fig. 64 is a 64-64 cross-sectional view of the medicine placement stand 414. As shown in F64B, the recesses 414A and 414B have flat bottom surfaces. As shown in F64B, the tablet T61 is placed in the recess 414B. The same is true for the tablets T62 and T63.

[0460] In this way, the medicine mounting table 414 has a rectangular parallelepiped recess structure on the mounting surface, which prevents the oval tablet from moving and allows it to be placed still. In addition, the position of the medicine when photographing it can be determined to be the position of the recess structure, making it easy to detect the area of ​​the medicine.

[0461] The shape, number, and arrangement of the recessed structures are not limited to the embodiments shown in Figs. 62 to 64, and may be appropriately combined, enlarged, or reduced. P (See FIG. 48).

[0462] <Other> The medicine identification program can also be provided in a state stored in a non-transitory recording medium such as a CD-ROM (Compact Disk-Read Only Memory).

[0463] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other without departing from the spirit of the present invention. [Explanation of symbols]

[0464] 10. Smartphone 12…Housing 14…Touch panel display 16…Speaker 18…Microphone 20…In-camera 22…Outside camera 24…Light 26…Switch 30…Wireless communication section 32...Telephone section 34…Memory 36...Internal storage 38…External memory unit 40...External input / output section 42…GPS receiver 44...Power supply section 50...Photographic lens 50F…Focus lens 50Z...Zoom lens 54...Image sensor 58...A / D converter 60...Lens drive unit 70...Shooting support equipment 72…Housing 72A…Bottom plate 72B…Side plate 72C…Side plate 72D…Side panel 72E…Side plate 74...Medicine placement table 74A…Reference marker 74B…Reference marker 74C…Reference marker 74D…Reference marker 75…Main light source 76...LED 78…Auxiliary light source 80...Shooting support equipment 82…Housing 82A…Bottom plate 82B…Side plate 84…Main light source 85...LED 86…Auxiliary light source 100...Drug identification device 102...Image acquisition unit 104...Image correction unit 104A…6th trained model 106…Drug detection unit 108... Stamp printing extraction part 108A…First trained model 110…1st drug type recognition unit 110A…Second trained model 112...Candidate output section 114...Determined part 120...Drug identification device 122…Second drug recognition unit 122A…Third trained model 123...Drug-related information storage unit 124...Drug-related information acquisition unit 126...Master image storage unit 127…Master image acquisition unit 128...Template matching section 130...Drug identification device 140...Image processing device 142...Image input unit 144…Database 146...Operation unit 148...Display section 150…CPU 152…RAM 154...ROM 160...Learning device 162...Recognizer 162A…Input layer 162B…Middle layer 162C…Output layer 164...Loss value calculation unit 166...Parameter control section 170...Learning device 172...Recognizer 172A…Input layer 172B…Middle layer 172C…Output layer 174...Error calculation section 176...Parameter control section 200…Drug Identification System 210...Server 212…Communications Department 214...CPU 216…Memory 300…Drug identification device 302…Search character information acquisition unit 304…Search section 306...Barcode identification section 310…Drug Identification System 320…Identification request generation device 322…Personal Identification Information Acquisition Unit 324...Output section 326...Operation unit 328...Display section 330…CPU 332…RAM 340…Electronic medical record system 342...Input section 342A…Identification code reader 344...Display section 346…Database 348…CPU 350…RAM 400…Drug identification device 410…Medicine placement table 410A…dent 410B…dent 410C…dent 410D…dent 410E…Hole 410F…dent 410G…dent 410H…dent 410I…hollow 412…Medicine placement table 412A…dent 412B…dent 412C…dent 412D…dent 412E…Hollow 412F…dent 414…Medicine placement table 414A…dent 414B…hollow 414C…hollow 414D…dent 414E…hollow 414F…dent A C 1. Candidate drug display area A C 2. Candidate drug display area A C 3. Candidate drug display area A C 4. Candidate drug display area A C 5. Candidate drug display area A I …Personal information display area A P …Placement area A S …Selected drug display area B1...Bounding box B2...Bounding box B11…Bounding box B12…Bounding box B13…Bounding box B14…Bounding box B21…Bounding box B22...Bounding box B31…Bounding box B41…Bounding box B42…Bounding box B43…Bounding box B44…Bounding box B51...Rotated rectangle bounding box B52...Rotated rectangle bounding box B53...Rotated rectangle bounding box B B ...Text box BC: Barcode BG…Background B S 1...Shooting button B S 2. Retake button B S 3…Search button B S 4…Button B S 5...Upward transition button B S 6... Downward transition button B S 7...Confirm button BS 8…Button B S 9…Clear button B SL …Correction slider bar C1…Perfect circle C2: Perfect circle CP1: Capsules CP2: Capsules CP3: Capsules C T 1. Candidate drug C T 2. Candidate drug C T 3. Candidate drug C T 4. Candidate drug C T 5. Candidate drug C T 11. Candidate drug C T 12. Candidate drug C T 13. Candidate drug C T 14…Candidate drug C T 15…Candidate drug C T 21...Candidate drug C T 22...Candidate drug C T 23...Candidate drug C T 24…Candidate drug C T 25…Candidate drug C T 26…Candidate drug C T 27…Candidate drug C T 28…Candidate drug C T 29…Candidate drug C T 30…Candidate drug C T 31...Candidate drug C T 32...Candidate drug D1…Screen display D2…Screen display D3…Screen display D4…Screen display D5…Screen display D11…Screen display D12…Screen display D13…Screen display D14…Screen display D21…Screen display D22…Screen display D23…Screen display D24…Screen display D25…Screen display D30…Request for identification of medications brought in by the customer D C 1. Drug type D C 2. Drug type D C 3. Drugs F1…frame F2…Frame F3…Frame F11…Frame F12…Frame F13…Frame I C 1...Photo IC2…Photo IC3…Photo IC4…Image capture I E … Engraved print extraction image I E 1... Engraved print extraction image I E A…Engraving extraction image I E B… Engraved print extraction image I E 13…Engraving extraction image I I 01...Area image I I 02...Area image I I …Area image I I 11…Engraving extraction image I I 12... Engraved print extraction image I I 13…Engraving extraction image I L V…Live view image ILV 2…Live View Image IM1…Marker area IM2: Marker area IM3…Marker area IM41…Marker area IM42…Marker area IM43…Marker area IM44…Marker area IM51…Marker area IM52…Marker area IM53…Marker area INF: Drug supplemental information I R …Area image I R 13...Area image I R 1...Area image I R A…Area image I R B…Area image I S …Standardized image I S 1…Standardized image I S 2. Standardized image IS3: Standardized image I W 11…Synthetic Extraction Image I W 12…Synthetic Extraction Image I W 13…Synthetic Extraction Image I W A…Synthetic Extraction Image I W B: Composite extracted image KB: Software keyboard M1…Marker M2…Marker M3…Marker M4…marker M11…marker M12…Marker M13…Marker M14…Marker MC1: Circular marker MC2: Circular marker MC3: Circular marker MS: Square marker M T …Tap position marker N11…number N12…number N13…number P1~P6…Each process of the drug identification method R D 1. Candidate R D 2. Candidate R D 3. Candidate S1-S25, S31, S41-S46: Each step of the drug identification method S51 to S55: Each step of the training method for the 6th trained model S61 to S65: Each step of the inference method using the sixth trained model SL…Slider SQ1…square SQ2…square ST1…GPS satellite ST2…GPS satellite T1: Tablet T2: Tablet T3: Tablets T11…tablet T12…tablet T13…tablet T21…tablet T22…tablet T23…tablet T24…tablet T31…tablet T32…tablets T41…tablet T42…tablet T43…tablet T51…tablet T52…tablet T53…tablet T61…tablet T62…tablet T63…tablet V1…Vertex V2…Vertex V3…Vertex V4…Vertex

Claims

1. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the region of the drug to be identified in the photographed image to obtain an imprint extraction image without an outer shape of the drug in which the imprint and / or print of the drug to be identified is extracted; a first drug type recognition unit for inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates for the drug type of the drug to be identified; having The imprint extraction unit includes a first trained model that inputs a first image of a drug to which an imprint and / or print is added and outputs a second image without an outline of the drug from which the imprint and / or print of the drug is extracted. Drug identification device.

2. The first drug type recognition unit includes a second trained model that inputs the second image and outputs a drug type corresponding to the imprint and / or printing of the drug. The drug identification device of claim 1 .

3. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the area of ​​the drug to be identified in the photographed image to obtain an imprint extraction image in which the imprint and / or print of the drug to be identified is extracted; a first drug type recognition unit for inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates for the drug type of the drug to be identified; a second drug type recognition unit that inputs at least an area of ​​the drug to be identified in the photographed image and infers the drug type of the drug to be identified; having The imprint extraction unit includes a first trained model that inputs a first image of a medicine to which an imprint and / or print is added and outputs a second image in which the imprint and / or print of the medicine is extracted, The first drug type recognition unit includes a second trained model that inputs the second image and outputs a drug type corresponding to the imprint and / or printing of the drug, The second drug type recognition unit includes a third trained model that inputs the first image and outputs the drug type of the drug, The first drug type recognition unit integrates the inference result of the first drug type recognition unit with the inference result of the second drug type recognition unit to obtain candidates for the drug type of the identification target drug. Drug identification device.

4. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the area of ​​the drug to be identified in the photographed image to obtain an imprint extraction image in which the imprint and / or print of the drug to be identified is extracted; a first drug type recognition unit for inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates for the drug type of the drug to be identified; A medicine-associated information acquisition unit that acquires medicine-associated information including at least one of shapes, sizes, and colors of a plurality of medicines; having The first drug type recognition unit integrates the drug-associated information with the inference result of the first drug type recognition unit to obtain candidates for the drug type of the identification target drug. Drug identification device.

5. The image acquisition unit acquires the captured image in which the identification target drug and the marker are captured, an image correction unit that standardizes a shooting distance and a shooting viewpoint of the captured image based on the marker to obtain a standardized image; The drug detection unit detects a region of the drug to be identified from the standardized image. The drug identification device according to any one of claims 1 to 4.

6. The image acquisition unit acquires the captured image in which a plurality of ArUco markers, a plurality of circular markers, or a plurality of rectangular markers are captured. The drug identification device according to claim 5.

7. The image acquisition unit acquires the captured image captured at a standard shooting distance and shooting viewpoint. The drug identification device according to any one of claims 1 to 6.

8. The image acquisition unit acquires a captured image including a plurality of drugs to be identified, The drug detection unit detects regions of each of the plurality of target drugs, The imprint extraction unit acquires a plurality of imprint extraction images corresponding to the plurality of target drugs, The first drug type recognition unit acquires drug type candidates corresponding to each of the plurality of identification target drugs. A drug identification device according to any one of claims 1 to 7.

9. The first drug type recognition unit is Obtaining a plurality of candidate drug types for the drug to be identified; acquiring a master image of each of the plurality of candidate drug classes; performing template matching between the marking extraction image and the master image; The drug identification device according to any one of claims 1 to 8.

10. A display control unit is provided for displaying on a display at least one image of an image of at least the region of the drug to be identified in the photographed image, the image of the marking and / or printing extraction, and an image of the drug to be identified with the marking and / or printing emphasized; The display control unit further causes the display to selectably display master images of candidates for the drug type of the drug to be identified. The drug identification device according to any one of claims 1 to 9.

11. The display control unit displays at least one image of the region of at least the drug to be identified in the photographed image, the inscription and / or print extraction image, and the image of the drug to be identified with the inscription and / or print emphasized, and a master image of a candidate drug type of the drug to be identified, with the inscription and / or print oriented in the same direction. The drug identification device of claim 10.

12. The display control unit causes the display to display a search window into which a character string can be input; The first drug type recognition unit identifies the drug type of the identification target drug based on the character string entered in the search window. The drug identification device according to claim 10 or 11.

13. A medicine placement table used for capturing the captured image of the medicine identification device according to any one of claims 1 to 12, A placement surface on which the drug to be identified is placed, A plurality of markers are arranged on the placement surface, Each of the plurality of markers is a circular marker or a rectangular marker. Medicine placement table.

14. The placement surface is provided with a recessed structure for placing the drug to be identified. The medicine mounting table according to claim 13.

15. An illumination device used to capture the captured image of the drug identification device according to any one of claims 1 to 12, A plurality of light sources are provided for irradiating the drug to be identified with illumination light from a plurality of different directions. Lighting equipment.

16. An image acquisition step of acquiring an image of the drug to be identified to which the mark and / or print is added; a drug detection step of detecting a region of the drug to be identified from the captured image; an imprint extraction step of processing at least the region of the drug to be identified in the photographed image to obtain an imprint extraction image without the outer shape of the drug in which the imprint and / or print of the drug to be identified is extracted; a first drug type recognition step of inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates of the drug type of the drug to be identified; having The imprint extraction step acquires the imprint extraction image using a first trained model that inputs a first image of a medicine to which an imprint and / or print has been added and outputs a second image without the outline of the medicine from which the imprint and / or print of the medicine has been extracted. Drug Identification Methods.

17. The first drug type recognition step infers the drug type of the drug to be identified using a second trained model that inputs the second image and outputs the drug type corresponding to the imprint and / or printing on the drug. The drug identification method according to claim 16.

18. A program for causing a computer to execute the drug identification method according to claim 16 or 17.

19. A non-transitory computer-readable recording medium having the program according to claim 18 recorded thereon.

20. a learning data collection unit that collects a re-learning data set including a set of an inscription and / or print extraction image of the drug to be identified, which is obtained by extracting the inscription and / or print of the drug to be identified, and information on the correct drug type of the drug to be identified, obtained by the drug identification device according to any one of claims 1 to 12; A re-learning unit that re-learns a second trained model using the collected re-learning dataset; having The second trained model inputs a second image without the outline of a drug from which the drug's mark and / or print is extracted, and outputs a drug type corresponding to the drug's mark and / or print. Learning device.

21. a learning data collection unit that collects a re-learning data set that is a set of a photographed image of a drug to be identified having a mark and / or a print and information on the drug type of the drug to be identified acquired by the drug identification device according to claim 3; A re-learning unit that re-learns the third trained model using the collected re-learning dataset. Learning device.

22. a learning data collection unit that collects a re-learning data set that is a set of a photographed image of a drug to be identified having a mark and / or a print and information on the drug type of the drug to be identified acquired by the drug identification device according to any one of claims 1 to 12; A learning unit that learns a fourth trained model using the collected re-learning dataset; having The fourth trained model inputs a first image of a drug to which an imprint and / or printing is added and outputs the drug type of the drug. Learning device.

23. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the region of the drug to be identified in the photographed image to obtain an imprint extraction image without an outer shape of the drug in which the imprint and / or print of the drug to be identified is extracted; a display control unit that displays on a display at least the imprint extraction image among the image of at least the region of the drug to be identified in the photographed image, the imprint extraction image, and the image of the drug to be identified in which the imprint and / or print is emphasized, with the imprint and / or print oriented upright; having The imprint extraction unit includes a first trained model that inputs a first image of a drug to which an imprint and / or print is added and outputs a second image without an outline of the drug from which the imprint and / or print of the drug is extracted. Drug identification device.

24. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the area of ​​the drug to be identified in the photographed image to obtain an imprint extraction image in which the imprint and / or print of the drug to be identified is extracted; a display control unit that displays at least one image of the region of at least the drug to be identified in the captured image, the image of the extracted mark and / or print, and the image of the drug to be identified with the mark and / or print emphasized on a display with the mark and / or print oriented upright; having The display control unit includes a fifth trained model that inputs a first image of a drug to which an inscription and / or printing has been added and outputs a third image in which the inscription and / or printing of the drug is upright. Drug identification device.

25. An image acquisition unit that acquires an image of a drug to be identified having a mark and / or a print added thereto; A medicine detection unit that detects a region of the medicine to be identified from the captured image; an imprint extraction unit that processes at least the area of ​​the drug to be identified in the photographed image to obtain an imprint extraction image in which the imprint and / or print of the drug to be identified is extracted; a display control unit that displays at least one image of the region of at least the drug to be identified in the captured image, the image of the extracted mark and / or print, and the image of the drug to be identified with the mark and / or print emphasized on a display with the mark and / or print oriented upright; having the display control unit causes the display to display a search window into which character information can be input; a search unit that searches for candidates of drug types of the drug to be identified based on character information entered in the search window; a first drug type recognition unit for inferring a drug type of the drug to be identified by inputting the inscription print extraction image and acquiring candidates for the drug type of the drug to be identified; having The display control unit automatically inputs character information indicating a most likely candidate for the drug type of the identification target drug acquired by the first drug type recognition unit into the search window. Drug identification device.

26. The display control unit displays candidates of drug types of the identification target drug searched for by the search unit.

26. The medication identification device of claim 25.

27. The imprint extraction unit includes a first trained model that inputs a first image of a medicine to which an imprint and / or print is added and outputs a second image in which the imprint and / or print of the medicine is extracted.

27. A drug identification device according to any one of claims 24 to 26.

28. The display control unit acquires a master image of the drug to be identified, the master image being an upright image of the marking and / or printing, and performs a rotational direction comparison between at least one of an image of the region of at least the drug to be identified in the photographed image, an image of the marking and / or printing extraction, and an image of the drug to be identified in which the marking and / or printing is emphasized and the master image by template matching.

28. A drug identification device according to any one of claims 23 to 27.

29. An image acquisition step of acquiring an image of the drug to be identified to which the mark and / or print is added; a drug detection step of detecting a region of the drug to be identified from the captured image; an imprint extraction step of processing at least the region of the drug to be identified in the photographed image to obtain an imprint extraction image without the outer shape of the drug in which the imprint and / or print of the drug to be identified is extracted; a display control process for displaying on a display at least the imprint extraction image among the image of at least the region of the drug to be identified in the photographed image, the imprint extraction image, and the image of the drug to be identified in which the imprint and / or print is emphasized, with the imprint and / or print oriented upright; having The imprint extraction step acquires the imprint extraction image using a first trained model that inputs a first image of a medicine to which an imprint and / or print has been added and outputs a second image without the outline of the medicine from which the imprint and / or print of the medicine has been extracted. Drug Identification Methods.

30. A method for manufacturing an output used for capturing the captured image of the drug identification device according to any one of claims 1 to 12 or 23 to 28, comprising: The output is a personal information display area in which information that identifies an individual is displayed; a placement area for placing the medicine to be identified by the individual; having obtaining information that identifies the individual; a step of arranging and printing the personal information display area and the writing area on a print medium; A method for manufacturing an output object having the above structure.

31. The information identifying the individual includes a barcode. A method for producing an output product according to claim 30.

Images