Medicine identification support system, medicine identification support method, and program

Abstract

To facilitate acquisition and management of an image for identifying medicine in various places and situations.SOLUTION: A medicine identification support system 10 includes: an image recording unit for receiving input of an image of medicine to be identified and recording the image in a storage unit; and an identification result acquisition unit for acquiring a result of identification of medicine included in the image recorded in the recording unit. The medicine identification support system 10 converts the identification result to a bar code or a two-dimensional code, and outputs the code.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for supporting identification of medicines using a computer system. [Background technology]

[0002] International Publication No. 2022 / 163508 (Patent Document 1) discloses a drug identification device. The drug identification device detects a drug region from an image of the drug to be identified and obtains an imprinted print extraction image in which the drug's imprint and / or print is extracted. The drug identification device infers the drug type from the imprinted print extraction image and obtains candidates for the drug type. A trained model is used to infer the drug type, which inputs the imprinted print extraction image and outputs the drug type of the corresponding drug.

[0003] JP 2020-182525 A (Patent Document 2) discloses a drug photographing device that can obtain identification results for drugs brought by patients with a simple configuration. The drug photographing device has a mounting section on which the drug is placed and an upper support section on which a mobile terminal can be installed. The mobile terminal is equipped with an imaging section that photographs the drug placed on the mounting section. The photographed image data is transmitted to an external server, and the external server can perform identification processing of the drug in the image data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 163508 [Patent Document 2] JP 2020-182525 A Summary of the Invention [Problem to be solved by the invention]

[0005] When an individual receives medical services, medical professionals may need to know the medications prescribed to that individual. When medications have been prescribed to an individual from one or more medical institutions, medical professionals will identify the prescribed medication based on the information written on the individual's prescription, medication book or medication packet, or the appearance of the medication itself. This task of identifying medications is called medication identification. Drug identification is performed by qualified personnel such as pharmacists. In recent years, the scope of medical services has expanded, and the locations and situations in which an individual's medications can be identified have become more diverse, such as medical institutions or an individual's home.

[0006] Therefore, the present application discloses a medicine identification support system, a medicine identification support method, and a program that can easily acquire and manage images for identifying medicines in various places or situations. [Means for solving the problem]

[0007] The medicine identification support system according to an embodiment of the present invention includes an individual record unit that generates unique data of an individual to be newly recorded and records the data in a storage unit when a command input for new recording of an individual is received from a user; an image recording unit that receives an input of an image of the drug to be identified and records the image in the storage unit; After the individual is newly recorded, the image recording unit records in the memory unit an image input during a reception period for accepting input of an image of the newly recorded individual's medication in association with the unique data. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of a medicine identification support system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a computer that constitutes the medicine identification support system and the identification system shown in FIG. [Diagram 3] 2 is a flow chart showing an example of the operation of the medicine identification support system shown in FIG. 1. [Figure 4] FIG. 13 is a diagram showing an example of a shooting screen. [Diagram 5]FIG. 13 is a diagram showing an example of an identification screen that displays an identification result and accepts a confirmation input from a user. [Figure 6] FIG. 13 is a diagram showing an example of a group designation screen. [Figure 7] FIG. 13 is a diagram showing another example of the group designation screen. [Figure 8] FIG. 11 is a flow chart showing another operation example of the medicine identification support system. [Figure 9] 13 is a flowchart illustrating an example of an identification process of the identification system. [Figure 10] FIG. 2 is a top view showing a configuration example of the medicine placing table. [Figure 11] 13 is a flowchart illustrating an example of a keystone correction process. [Figure 12] 10 is a flowchart illustrating an example of a color correction process. [Figure 13] 13 is a flowchart showing an example of a classification process for an image of medicine text. [Figure 14] FIG. 13 is a top view showing a modified example of the medicine placing table. [Figure 15] FIG. 13 is a top view showing another modified example of the medicine mounting table. [Figure 16] FIG. 13 is a top view showing another modified example of the medicine mounting table. [Figure 17] FIG. 17 is a diagram showing a modified example of the screen in S4 of FIG. [Figure 18] FIG. 18 is a diagram showing an example of a screen displaying a list of individuals recorded in the storage unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Configuration 1) The medicine identification support system according to an embodiment of the present invention includes an individual record unit that generates unique data of an individual to be newly recorded and records the data in a storage unit when a command input for new recording of an individual is received from a user; an image recording unit that receives an input of an image of the drug to be identified and records the image in the storage unit; After the individual is newly recorded, the image recording unit records in the memory unit an image input during a reception period for accepting input of an image of the newly recorded individual's medication in association with the unique data.

[0010] According to the above configuration 1, an image input during a reception period after a user inputs an instruction to record a new personal record is recorded in association with unique data. The unique data is generated when an instruction to record a new personal record is input. Therefore, an image of an individual's medicine can be recorded by a simple operation of inputting an instruction to record a new personal record by a user and inputting an image of the individual's medicine. In addition, since an image is associated with unique data, it is less likely that an image of an individual's medicine will be mistakenly treated as an image of another individual's medicine or that one image will be treated as an image of multiple individuals. As a result, it becomes easier to obtain and manage images of each individual's medicine for medicine identification.

[0011] For example, according to the medicine identification support system of configuration 1, after the user inputs an instruction to record a new personal record, even before inputting personal information (personal name, date of birth, etc.), the image input during the reception period is recorded in association with the unique data. Therefore, for example, the user can give priority to acquiring images of personal medicines at the place of visit, and input personal information after returning to the pharmacy.

[0012] The instruction input for a new personal record received by the personal record unit may be an input of information instructing a new personal record. The information input in the instruction input for a new personal record does not have to include personal information (personal name, date of birth, etc.).

[0013] For example, the personal record unit may receive an instruction to make a new personal record in a form that does not include information for identifying the individual, generate unique data without using information input by the user when inputting the instruction, and record the unique data in the storage unit. This allows the user to instruct a new personal record without inputting personal information. In this case, the image recording unit may record an image input during the reception period after the instruction to make a new record and before the newly recorded personal information (such as name and date of birth) is input, in association with the unique data of the individual.

[0014] As an example, the image recording unit can record images entered during a reception period for accepting images of an individual's medications before the name of a newly recorded individual is entered, in association with the individual's unique data.

[0015] The unique data generated when recording a new individual may be, for example, data based on the time or place of inputting the instruction for the new record. This allows the unique data to be generated without using information input by the user when inputting the instruction, and the time or place of the new record is indicated in the unique data, making it easy for the user to understand the individual by looking at the unique data. Note that the unique data is not limited to this. The unique data can be, for example, patient information, date and time information, information only about the time, location information, symbols, numbers, initials, characteristics of the house, characteristics of the patient, nicknames, etc., which can be used to easily recall the individual by the user who instructed to record a new individual.

[0016] The acceptance period for accepting the newly recorded image of the personal medicine may be determined according to a user operation. For example, the acceptance period for the image of the personal medicine may be set to a period from when the user performs an operation to specify the start of image input of the personal medicine to when the user performs an operation to specify the end of image input or until a predetermined time has elapsed. In this way, the start and end of the acceptance period may be determined based on a predetermined condition related to the user operation.

[0017] The reception period may be started, for example, on the condition that a user identifies a target individual and starts image input. Identification of the individual may be an instruction input for a new record of an individual, or an input specifying an individual whose unique data has already been recorded. The condition for ending the reception period may be, for example, an input indicating the end of image input of the target individual's medication by the user, or may be a predetermined time having elapsed since the start of the reception period. For example, the reception period may be ended when an image of an individual's medication is input by a user operation.

[0018] (Configuration 2) In the above configuration 1, the medicine identification support system may further include a group determination unit that determines a group to which the image associated with the unique data or the medicine identified in the image belongs based on a group instruction input from a user, and records information indicating the determined group in the storage unit in association with the image or the medicine. This makes it possible to record images of medicines prescribed to an individual or identified medicines in one or more groups. For example, an individual's medicines can be grouped and recorded by medical institution that provided the medicine.

[0019] The group determination unit may classify a plurality of image groups or drug groups associated with one unique data into at least one group based on a group instruction input from a user. The group instruction input from a user may be, for example, an input explicitly specifying a group to which each image or each drug shown in the identification result should belong. Alternatively, the image recording unit may accept a group instruction input from a user when accepting an input of an image during the reception period. For example, together with the input of an image, an input specifying a group for the image may be accepted. Alternatively, an input of an image of an object containing information about the drug, such as a drug bag, a drug notebook, or a prescription, before or after the input of an image of the drug, may be accepted as a group instruction input.

[0020] The group determination unit may classify medicines included in the same image into the same group. As an example, the group determination unit may display medicines included in the same image in a manner that allows a user to collectively designate the medicines as the same group.

[0021] (Configuration 3) In the above configuration 2, the group determination unit may display ungrouped drugs whose group to which they belong is undetermined among the images or drugs identified in the images associated with the unique data, and may receive an input from a user specifying a group to which the ungrouped drugs should belong. This makes it easy for a user to input a group instruction for an image or drug that is not classified into a group.

[0022] In any one of the above configurations 1 to 3, the medicine identification support system may further include a recognition result acquisition unit that provides the image recorded by the image recording unit to a recognition system that executes a recognition process, acquires a recognition result of the medicine included in the image from the recognition system, and records the recognition result in the storage unit. This makes it possible to identify the medicine in the recorded image.

[0023] The identification result acquisition unit may record information on the medicine included in the image indicated by the acquired identification result in the storage unit in association with the unique data associated with the image, thereby making it possible to record the identification result in association with an individual.

[0024] The medicine identification support system may display information about the medicine included in the image shown in the identification result acquired by the identification result acquisition unit to a user together with the image, and may receive an input from the user requesting confirmation of the identification result by the identification system. This allows the user to confirm the identification result while referring to both the information about the medicine identified in the image and the image on which the information is based. This allows the user to efficiently confirm the identification result.

[0025] (Configuration 5) In any of the above configurations 1 to 4, the identification result acquisition unit may provide the image recorded in the storage unit to the identification system in response to an identification instruction from a user to acquire the identification result. This allows the user to control the timing of execution of the identification process. For example, the user may give priority to inputting images of medicines of multiple individuals, and then execute the identification process for the medicines of multiple individuals collectively.

[0026] The classification result acquisition unit may switch between providing the image recorded in the storage unit to the classification system and acquiring the classification result in response to a classification instruction from a user, and providing the image recorded in the storage unit to the classification system and acquiring the classification result in response to recording of the image in the storage unit, based on a user instruction, thereby further improving user convenience.

[0027] The image recording unit may receive from a user a designation of the aspect of the drug in the image along with the image of the drug. The image recording unit may record the aspect of the drug designated by the user in the memory unit in association with the image. The identification result acquisition unit may select an identification process for the drug in the image in the identification system according to the aspect of the drug in the image. For example, the identification result acquisition unit may provide the identification system with information indicating the aspect of the drug in the image along with the image of the drug. The identification result acquisition unit can acquire the identification result of the identification process according to the aspect of the drug in the image.

[0028] The designation of the form of the medicine may, for example, designate whether the subject of the photograph is the medicine itself or text showing information about the medicine. When the subject of the photograph is the medicine itself, the form may be further specified in detail according to the state of the medicine. The form of the medicine to be designated may include, for example, at least one of the medicine itself, text showing information about the medicine, and the medicine code. For the medicine itself, the state of the medicine itself may be further designated, such as an uncovered tablet, a medicine packet, a PTP sheet, etc. For the medicine text, the type of description may be designated, such as a prescription, a medicine bag, or a medicine notebook. For the code, the type of code may be designated, such as GS1, QR code (registered trademark), etc. This makes it possible to perform an appropriate identification process according to the form of the medicine shown in the image. In addition, the identification work of medicines in a wider variety of states becomes easier. Note that an uncovered tablet is a tablet or capsule that is not packed in a medicine packet or the like and is exposed.

[0029] The image recording unit may receive an image capturing operation of the medicine by the user. That is, the image recording unit may record an image obtained by the user's image capturing operation of the medicine in the storage unit. The image recording unit may receive a designation of the form of the medicine from the user along with the image capturing operation by the user. In this case, the image capturing mode may be made switchable according to the designation of the form of the medicine. For example, a screen for accepting the image capturing operation may be switched by switching the image capturing mode.

[0030] The image recording unit may accept a designation of an identification target range in the image when accepting input of the image from a user. This allows information indicating the identification target range in the image to be obtained. The identification result acquisition unit may provide the identification target range together with the image to the identification system. The identification target system can execute a medicine identification process for the identification target range of the image. For example, the image recording unit may accept a designation of the range of one medicine packet as the identification target range.

[0031] The image recording unit may receive an image capturing operation of a medicine placed on a medicine placement table having a marker. That is, the image capturing unit may receive an input of an image of the medicine placed on the medicine placement table having a marker. An image including the medicine and the marker on the medicine placement table is recorded. The image recording unit may detect a marker in an image captured by a camera operated by a user, and instruct the user to perform an image capturing operation based on the position of the marker in the captured image. For example, an instruction regarding at least one of the position and orientation of the camera may be output to the user. The image recording unit may receive a designation of an identification target range from a user based on the position of the marker on the medicine placement table in the image.

[0032] The medicine identification support system may further include a duplicate candidate display unit that displays to a user two or more images with high similarity extracted from the plurality of images based on the medicine identification result in the plurality of images by the identification system as candidates for images in which the same medicine is photographed in duplicate. The extraction of two or more images with high similarity can be performed using, for example, the date and time of each image, the position of the medicine in each image, and information on the identified medicine. For example, the similarity may be calculated by comparing at least one of the date and time of the image included in the identification result, the position of the medicine in the image, and information on the identified medicine between the images. Two or more images whose similarity meets a predetermined condition can be extracted as duplicate candidate images. This extraction process may be performed by the medicine identification support system or the identification system.

[0033] When the identification result acquired by the identification result acquisition unit indicates an image or a drug area in an image that cannot be identified, the image recording unit may display the image or drug area that cannot be identified and accept a user's operation to photograph the drug. This allows the user to be given an opportunity to re-photograph the drug in the image or drug area that cannot be identified. The image recording unit may also accept a user's operation to photograph a range narrower than the range of the image that cannot be identified, or a range limited to the drug area that cannot be identified. For example, the image recording unit may accept a photographing operation as a close-up mode in which the drug is photographed closer than usual. This allows, for example, an image photographed by approaching the drug when there is an image or drug area in which a candidate drug cannot be obtained as an identification result in an image photographed over the entire range in which the drug to be identified is distributed. In this case, the resolution per tablet is increased. Therefore, the identification accuracy is improved. In addition, for example, when a drug area that is recognized as having a drug present is shown on the photographing screen and the user photographs the drug by specifying the drug area, an identification process may be performed on the specified drug area.

[0034] (Configuration 6) A medicine placement table in an embodiment of the present invention has a placement surface on which medicines are placed. A plurality of markers are provided on the placement surface, spaced apart from one another, and each of the plurality of markers includes a combination of two or more reference colors. An image obtained by photographing the medicine placed on the placement surface of the placement table includes the plurality of markers. Therefore, it is possible to perform keystone correction based on the positions of the plurality of markers in the image of the medicine, and color correction based on the combination of the reference colors of each of the plurality of markers. This reduces the effect of the photographing conditions of the medicine on the identification accuracy. As a result, it becomes easier to identify medicines in various places or situations.

[0035] The multiple markers include the same combination of reference colors. That is, the combination of reference colors of one marker is the same as the combination of reference colors of the other markers. The reference colors can be colors that are the reference for the luminance value of each pixel in image processing. The combination of reference colors is not particularly limited, but may be, for example, a combination of three colors: R (red), G (green), and B (blue).

[0036] The medicine placing table is used, for example, to capture an image of the medicine to be recorded by the image recording unit of the medicine identification support system of any one of configurations 1 to 5. In this case, the image recording unit may receive an input of an image of the medicine placed on the placing surface of the placing table, the image including the medicine and a plurality of markers. The identification result acquisition unit may provide the image of the medicine placed on the placing surface of the placing table, the image including the medicine and a plurality of markers, to the identification system.

[0037] The medicine placement table may include an IC tag that holds data that can be read by short-distance wireless communication. The IC tag may hold data for causing a computer constituting the medicine identification support system to execute a predetermined application. This allows an application related to the medicine identification support system to be executed by a user's action of bringing the computer close to the medicine placement table. This makes it easier to identify medicines. The predetermined application may be, for example, an application for causing a computer to operate as the medicine identification support system or an application for executing part of the functions of the medicine identification support system. Alternatively, the predetermined application may be an application for executing a process of acquiring information on a medicine using the GS1 code of the medicine.

[0038] (Configuration 7) An identification system that performs identification processing of a medicine based on an image of the medicine is also included in the embodiment of the present invention. The identification system may include a correction unit that acquires a photographed image of a medicine placed on a placement surface on which a plurality of markers including a combination of two or more reference colors are provided, and generates a corrected image by performing keystone correction based on the positions of the plurality of markers in the image and color correction based on the colors in the image of the combination of the reference colors of each of the plurality of markers on the photographed image, a medicine area extraction unit that extracts a medicine area from the corrected image, and an identification unit that identifies the medicine in the medicine area based on a partial image of the medicine area. In this identification system, keystone correction and color correction are possible using markers included in the photographed image of the medicine. This reduces the influence of the photographing conditions of the medicine on the identification accuracy. As a result, it becomes easy to identify medicines in various places or situations. The medicine identification support system of the above configurations 1 to 5 may further include the identification system.

[0039] In the above configuration 7, the drug region extraction unit of the identification system may extract the drug region using instance segmentation. This makes it possible to efficiently extract the region of each drug in an image in which multiple drugs are overlapped or the edge of a drug is in contact with the edge of another drug, thereby improving the accuracy of identification.

[0040] The program according to the embodiment of the present invention includes a personal record process for generating unique data of a new individual to be recorded and recording the data in a storage unit when a command input for new recording of the individual is received from a user; The image recording process receives an input of an image of the drug to be identified, and records the image in the storage unit. The image recording process includes a process of recording in the memory unit, after a new record of the individual, an image input during a reception period for accepting input of images of the newly recorded individual's medication, in association with the unique data.

[0041] A medicine identification support method according to an embodiment of the present invention includes the following steps executed by a computer: an individual recording step of generating unique data of an individual to be newly recorded and recording the data in a storage unit when a command input for new recording of an individual is received from a user; and an image recording step of receiving an input of an image of the drug to be identified and recording the image in the storage unit. The image recording process includes a process of recording in the memory unit, after a new record of the individual, an image input during a reception period for accepting input of images of the newly recorded individual's medication, in association with the unique data.

[0042] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Embodiment] (System configuration example) FIG. 1 is a diagram showing an example of the configuration of a medicine identification support system 10 according to this embodiment. The medicine identification support system 10 accepts input of an image of an individual's medicine, records the image, and provides it to the identification system. In this way, the medicine identification support system 10 supports the identification of the medicine using the image. The medicine identification support system 10 is connected to a storage unit 6, an input / output device 7, and a camera 8 by wireless or wired connection. The input / output device 7 includes a device for a user to input information, and a device for outputting information to the user. The medicine identification support system 10 is capable of data communication with an identification system 20. The identification system 20 uses an image of the medicine to execute an identification process for identifying the medicine included in the image. The medicine identification support system 10 provides the image of the medicine to the identification system 20, and obtains an identification result from the identification system 20.

[0043] The medicine identification support system 10 includes a personal record unit 1, an image recording unit 2, a recognition result acquisition unit 3, a group determination unit 4, and an overlap candidate display unit 5. The identification system 20 includes a correction unit 21, a medicine region extraction unit 22, and an identification unit .

[0044] The personal record unit 1 receives an instruction input for a new record of an individual from the user, and records a new individual. When an instruction input for a new record is received from the user, the personal record unit 1 generates unique data and records it in the storage unit 6. The unique data is, for example, unique data based on at least one of the time and place of the user's instruction input and is recorded in the storage unit 6. The unique data becomes the individual's initial identifier. The instruction input for a new record of an individual by the user is performed via the input / output device 7. For example, the personal record unit 1 may display a button or link on the screen of the input / output device 7 that accepts an operation of inputting an instruction for a new record of an individual from the user.

[0045] The unique data may be data including information indicating the time or place of the instruction input. The unique data may be, for example, a time stamp indicating the date and time of the instruction input, or a location stamp indicating the place where the instruction input was received. Alternatively, data including both the date and time and the place may be the unique data. The place may be, for example, a real place such as an address or latitude and longitude, or a virtual place such as a logical address in a network or coordinates in a virtual space. The unique data may also be data directly indicating at least one of the time and place of the instruction input, or may be data (for example, a hash value, etc.) obtained by processing the time and place of the instruction input. The individual record unit 1 may receive input of data identifying an individual (such as a name or date of birth) as an instruction input for a new record of an individual. The data identifying an individual may be recorded in association with the unique data.

[0046] The image recording unit 2 accepts input of an image of the drug to be identified. The image recording unit 2 records the input image in the storage unit 6. The image recording unit 2 may, for example, acquire an image of the drug photographed by the camera 8 by the user's operation and record it in the storage unit 6. The image recording unit 2 accepts an operation of the user to photograph the drug via the input / output device 7. In this case, the image recording unit 2 may accept a designation of the state of the drug along with the user's photographing operation. The state of the drug may be recorded in association with the photographed image. In the example of FIG. 1, as the state of the drug to be photographed, any of the following can be designated: a plain tablet drug D placed on the drug placement table 50 or a drug D contained in a drug packet H, a barcode (GS1 code) or two-dimensional code indicating information on the drug, or text indicating information on the drug.

[0047] After a new record of an individual, the image recording unit 2 records in the storage unit 6 an image input during a reception period for receiving input of an image of the individual's medicine in association with the unique data. For example, when a user performs an operation to designate an individual, the reception period is a period from the designation of the individual to the satisfaction of a cancellation condition for the designation. The image input during this reception period is recorded in association with the unique data of the individual. The designation of an individual by the user that triggers the start of the reception period may be, for example, an instruction input for a new record, or an operation to designate unique data or other personal information displayed on the screen. The cancellation condition for the designation of an individual is not particularly limited, but may be, for example, when image input is completed, when another individual is designated or other user's designation cancellation operation is performed, or when a predetermined time has passed. In this way, the image recording unit 2 can determine that an image input during a reception period for an image of a certain individual is an image of the individual's medicine, and record it in association with the unique data of the individual.

[0048] The identification result acquisition unit 3 provides the image recorded by the image recording unit 2 to the identification system 20 that executes the identification process, and acquires the identification result of the drug included in the image from the identification system 20. The identification result is recorded in the memory unit 12 in association with the image. As a result, the identification result is recorded in association with the unique data. The identification result may include information on the drug included in the image. For example, the identification result may include information on drugs that may be included in the image, i.e., information on candidate drugs. Information on multiple candidate drugs may be included in the identification result for one drug in the image.

[0049] The classification result acquisition unit 3 may provide the image recorded in the storage unit 6 to the classification system 20 and acquire the classification result in response to a classification instruction from the user. This allows the user to flexibly respond, for example, by prioritizing the image input (photographing) task and performing classification tasks all at once later. The classification result acquisition unit 3 may be capable of switching, based on a user's instruction, between a mode (KEEP mode) in which the image recorded in the storage unit 6 is provided to the classification system in response to a classification instruction from the user and the classification result is acquired, and a mode (SEND mode) in which the image recorded in the storage unit 6 is provided to the classification system 20 in response to the recording of the image in the storage unit 6 and the classification result is acquired.

[0050] The drug identification support system 10 may display to the user information about the drug included in the image showing the identification result acquired by the identification result acquisition unit 3 together with the image, and may receive from the user an input for confirming the identification result by the identification system 20. For example, candidate drugs may be displayed to the user, and the user may be able to select one of the candidate drugs that he or she determines to be the relevant drug. In other words, the selection of the candidate drug by the user may be received as a confirmation input.

[0051] The group determination unit 4 classifies the images associated with the unique data or each individual's medicine included in the image into at least one group set for the individual based on a group instruction input from the user, and records information indicating the classified group in the storage unit 6. The group determination unit 4 makes it possible to classify and record information on the medicine of the identified individual into any group determined by the user. The group determination unit 4 displays each medicine of the identified individual in a selectable manner, and can accept the selection of a medicine and the group designation of the selected medicine. The group determination unit 4 records information indicating a group to which each of the images or medicines associated with the unique data belongs in the storage unit 6. The information indicating a group may be, for example, information identifying a group such as a group name or a group number, or may be information indicating a division (break) of a group in a plurality of images or medicines associated with the unique data.

[0052] For example, medicines prescribed to one individual may be provided by multiple sources (pharmacies, hospitals, etc.). The group determination unit 4 makes it possible to classify and record the medicines of an individual into groups according to the source. For example, a user can specify a group according to the source for each image associated with unique data or for each medicine after identification based on each image. This makes it possible to distinguish the sources of medicines for an identified individual.

[0053] In addition, the group designation by the user is not limited to the medicine after identification. Even before identification, the group designation input can be received from the user together with the image to be input. For example, when receiving the input of a plurality of images by the user's successive photographing operations, the input of the break of the group or group information may be received from the user between the photographing operations.

[0054] The duplicate candidate display unit 5 displays two or more images with high similarity extracted from the multiple images to the user as candidates for duplicate images based on the identification result of the drug in the multiple images by the identification system 20. When there are two or more images of the same drug, the drug is identified as duplicated by the identification system 20. For example, the drug in the captured image may be blurred and re-captured, or the image may be a little dark and the brightness may be adjusted and re-captured. In this case, multiple images of the same drug are recorded. The duplicate candidate display unit 5 makes it possible to notify the user of images that may be duplicated.

[0055] In this embodiment, the unique data automatically generated when the user inputs an instruction to make a new record for an individual is recorded as the individual's initial identifier, and the image of the individual input during the subsequent reception period is linked to the unique data. This allows the user to start inputting (taking) images of the individual's medicines after instructing a new record, without inputting personal information (such as the individual's name and date of birth). In addition, since the image is associated with the unique data based on the time or place of the instruction input, there is no risk of image confusion or duplication between individuals. In addition, when unique data based on the time or place of the instruction input for the individual record is recorded, the user can easily understand the individual based on the unique data. For example, when the unique data based on the time or place of the instruction input for the individual record is displayed in a state where personal information has not been input, the user can understand the individual based on the time or place indicated by the unique data.

[0056] Medical professionals are busy in hospital wards, pharmacies, and at home medical care sites. In order for medical professionals to know the medications that individuals bring with them, the remaining medications at home, and the medications they regularly keep, it is often necessary to first photograph, i.e., obtain an image of the medication. In the medicine identification support system of this embodiment, a user (e.g., a pharmacist) can prioritize the task of photographing an individual's medications at the site, leaving input of personal information for later. The user can input an image at the site, and then input additional personal information after returning to the pharmacy, thereby recording the personal information in association with the image of the individual.

[0057] (Example of configuration of the identification system 20) In the example of FIG. 1, the identification system 20 includes a correction unit 21, a medicine region extraction unit 22, and an identification unit 23. The correction unit 21 performs keystone correction and color correction on the captured image of the medicine to generate a corrected image. In the example of FIG. 1, an image of a medicine D placed on the placement surface of the medicine placement table 50 may be provided to the identification system 20. A plurality of markers M1 to M4 are provided on the placement surface of the medicine placement table 50. Each of the plurality of markers M1 to M4 includes a combination of two or more reference colors. Therefore, the image of the medicine D placed on the placement surface of the medicine placement table 50 includes the medicine D and the markers M1 to M4. The correction unit 21 performs keystone correction based on the positions of the plurality of markers M1 to M4 included in the image. The keystone correction may be performed, for example, by a calculation that converts each pixel of the image so that one reference color of each of the markers M1 to M4 in the image is placed at a target position. The correction unit 21 performs color correction based on the colors in the image of the combinations of the reference colors of the markers M1 to M4. The color correction may be performed by a calculation that corrects the value of each pixel for each reference color so as to reduce the variation in color (tone value) in the markers M1 to M4 of the image.

[0058] The drug region extraction unit 22 extracts a drug region in the corrected image generated by the correction unit 21. The drug region extraction process may be, for example, a rule-based process that executes a predetermined image process on the image and extracts a region that satisfies a predetermined condition as the drug region. Alternatively, the drug region extraction process may be executed using a trained model. The trained model inputs an image and outputs a drug region. The trained model can be generated, for example, by machine learning using a set of an image and a drug region as training data. The rule-based process can be, for example, a process that recognizes (extracts) features of a target image by comparing the target image with a prerecorded rule (for example, a template).

[0059] The identification unit 23 identifies the medicine in the medicine region based on a partial image of the medicine region. The identification unit 23 may have an imprinting / printing extraction unit that extracts the imprinting or printing of the medicine in the partial image of the medicine region, and a matching unit that compares the image extracted by the imprinting / printing extraction unit with a master image of the medicine in a database and picks out medicines that have a high degree of match as candidate medicines. Extraction of the imprinting or printing of the medicine can also be performed on a rule basis or by using a trained model. The partial image may be, for example, a part of a medicine that has a predetermined characteristic (e.g., an imprint or print).

[0060] A convolutional neural network (CNN) can be applied to the trained model used for extracting the drug region or the engraving or printing. For example, any of the following models can be used for the trained model: object detection, semantic segmentation, or instance segmentation. By using instance segmentation to perform drug region extraction, it is possible to extract a region for each drug even when multiple drugs overlap. For example, SOLO, Mask R-CNN, Detectron, or the like can be used as the instance segmentation model.

[0061] The identification unit 23 outputs candidate drugs contained in the image, i.e., information on candidate drugs, as the identification result. For example, for each drug contained in the image, information on multiple candidate drugs whose degree of match satisfies a condition may be output as the identification result. The identification result may also include the area (position) of the drug in the image, its size, the number of tablets, and other information related to identification.

[0062] 1, the correction unit 21, drug region extraction unit 22, and classification unit 23 of the classification system 20 execute classification processing on an image of the drug itself. The classification system 20 may also execute classification processing on an image of text describing the drug. In this case, the classification system 20 may include a character recognition unit that recognizes characters in the image, and a text classification unit that compares the recognized character string with a master drug name and designates drugs with a high degree of match as candidate drugs.

[0063] The identification system 20 may receive information indicating the state of the drug in the image together with the image to be identified from the identification result acquisition unit 3, and perform an identification process according to the state of the drug. For example, if the state of the drug in the target image is the drug itself, the identification system 20 performs an identification process on the image of the drug itself, and if the state of the drug is text describing the drug, the identification system 20 performs an identification process on the image of the drug text. If the state of the drug itself is specified in more detail as an uncovered tablet, a drug package, etc., an identification process according to each of the detailed drug states, such as an uncovered tablet, a drug package, etc., may be performed.

[0064] The drug identification support system 10 and the identification system 20 can be configured with one or more computers. Each functional unit of the drug identification support system 10 and the identification system 20 can be realized by a computer processor executing a program in a memory. Such programs and non-transitory storage media storing the programs are also included in the embodiments of the present invention. The memory unit 6 can be configured with one or more storages and / or memories accessible to the computer constituting the drug identification support system 10.

[0065] FIG. 2 is a diagram showing an example of the configuration of a computer constituting the medicine identification support system 10 and the identification system 20. In the example of FIG. 2, the medicine identification support system 10 is implemented in a mobile terminal 30. The identification system 20 is implemented in a computer 40 that can communicate with the mobile terminal 30. The mobile terminal 30 has a processor 31, a memory 32, a storage device 33, a touch panel 7a (an example of an input / output device), and a camera 8. The computer 40 has a processor 41, a memory 42, and a storage device 43. The computer 40 may be configured to realize the function of the identification system 20 by edge computing that executes processing at a position close to the mobile terminal 30 on the network. For example, the computer 40 may be a portable edge computer (e.g., an edge PC or an edge server) that can communicate with the mobile terminal 30 without the Internet. This makes it possible to reduce network load and delay and improve security. The computer 40 may also be a computer (e.g., a cloud server) that can communicate with the mobile terminal 30 via a network.

[0066] 2, the medicine identification support system 10 may have an identification result acquisition unit 3 that controls whether to provide the image recorded by the image recording unit 2 to the identification system 20 (for example, whether to use SEND mode or KEEP mode) depending on the communication state between the mobile terminal 30 and the computer 40, and may display the control state. Note that, even in the configuration of FIG. 2, a part of the medicine identification support system 10 may be implemented by the computer 40. A part of the identification system 20 may be implemented by the mobile terminal 30.

[0067] (Example of operation) Fig. 3 is a flow diagram showing an example of the operation of the medicine identification support system 10 shown in Fig. 1. In the example of Fig. 3, the user's operation of selecting "New personal record" on the menu screen G1 displayed by the medicine identification support system 10 in S1 constitutes an input of a new record instruction for an individual. Upon receiving the input of the new record instruction, in S2, the personal record unit 1 records a timestamp indicating the date and time of the input of the new record instruction in the storage unit 6 as unique data for the individual of the new record.

[0068] In S3, a personal information screen G2 of the newly recorded individual is displayed. The personal information screen G2 includes information that identifies the individual, and a user interface for instructing the start of photographing (image input) or personal information input. In this example, the screen G2 indicates that the individual's name has not been entered, and displays timestamp information, which is unique data and indicates the recording date and time, as information that identifies the individual. When the user selects "Photograph" on the screen G2, an acceptance period begins for accepting input of images of this individual's medications.

[0069] In S4, the image recording unit 2 accepts image input from the user. The image recording unit 2 displays a shooting screen for the user to shoot an image of the medicine. The image recording unit 2 enables the user to input a designation of the aspect of the medicine to be photographed. The image recording unit 2 switches the shooting screen according to the aspect of the medicine designated by the user.

[0070] Fig. 4 is a diagram showing an example of a photographing screen that can be switched by a user's specification of the type of medicine. In the example of Fig. 4, photographing screens G3-1 to G3-5 are examples of cases where the type of medicine to be photographed is an uncoated tablet, a medicine packet, a GS1 code, a QR code (registered trademark), and text (OCR). Each photographing screen has a monitor area A1 that displays an image of the photographing range of the camera 8 in real time. In the monitor area A1 of the photographing screens G3-3 and G3-4 of the GS1 code and the QR code (registered trademark), the code included in the image is recognized, and a guide line indicating the code portion is displayed.

[0071] In the text shooting screen G3-5, a medicine description area that may be a text indicating a medicine included in the image is indicated by a guide line. This makes it easy for the user to adjust the position and orientation of the camera 8 so that the area describing the medicine is included in the shooting range while looking at the monitor area A1. For example, a medicine description area is extracted from a character string recognized by character recognition (OCR) processing in an image in the shooting range of the camera 8. The extracted medicine description area is a part of the recognized character string that satisfies a predetermined condition for medicine description. The condition for medicine description can be, for example, at least one of the character spacing of the character string, the number of consecutive characters, and the presence or absence of a numerical value near the end. The extraction process of the medicine description area in the image may be executed by the identification system 290 or may be executed by the medicine identification support system 10. In addition, the image recording unit 2 may allow the user to select a medicine description area in the monitor area A1 and receive a designation of the medicine description area from the user.

[0072] In S4 of Fig. 3, when the image recording unit 2 accepts input of an image of a medicine taken by a user's operation, it records the input image in the storage unit 6 (S5). In this example, the recording of this image marks the end of the period during which images of individual medicines can be input. The image recording unit 2 records the image input in S4 in association with the unique data recorded in S2.

[0073] When the image is recorded in S5, this triggers the identification result acquisition unit 3 to provide the recorded image to the identification system 20 (S6). The image may be provided by transmitting the image data itself, or by a process that enables access to the image (for example, providing access information to the image). The identification system 20 executes a process to identify the drug included in the provided image (S7). The identification result acquisition unit 3 acquires the identification result from the identification system 20 and records it in the storage unit 6. The identification result is recorded in association with the image. The drug identification support system 10 displays an identification screen G4 showing the identification result of the drug in the image to the user, and accepts confirmation from the user (S10).

[0074] FIG. 5 is a diagram showing an example of an identification screen G4 that displays the identification result and accepts confirmation input from the user. In the example of FIG. 5, an image of the identification target and candidate drugs that are candidates for the drug included in the image are displayed at the same time. In the screen G4-1 and the screen G4-3, the entire image of the identification target is displayed, and the drug area is displayed in a visible manner. The display manner of each drug area may be controlled to be different depending on the state of confirmation by the user of the drug identification result. For example, a drug area whose identification result has not been confirmed by the user, a drug area whose identification result has been confirmed by the user, and a drug area whose identification process is incomplete may be displayed in different manners. In the screen G4-2 and the screen G4-4, for one drug included in the image, an image of the drug and information on one or more candidate drugs are displayed in a selectable manner. For example, when a user specifies one drug area in the entire image in the screen G4-1, a screen G4-2 may be displayed that displays an image of the specified drug area and the candidate drugs. The user can input confirmation of the identification result by selecting one of the candidate drugs. As a result, the drug of the identification result is recorded as a confirmed, i.e., identified, drug. That is, the confirmation result by the user is recorded (S11 in FIG. 3). Note that screens G4-1 and G4-2 are examples of classification results for images of the medicine itself, and screens G4-3 and G4-4 are examples of classification results for images of the medicine text.

[0075] As shown in the examples of screens G4-1 and G4-3, the medicine identification support system 10 can display, as the identification result, the area of ​​the identified medicine in the captured image in a visible manner. In this case, in the image, an indication element (for example, B1, B2 in FIG. 5) indicating the area of ​​the identified medicine is displayed superimposed on the image. The indication element may be, for example, an image element having a shape surrounding at least a part of the area of ​​the identified medicine (for example, the medicine itself or the medicine name text). The indication element may be, for example, a polygon. The display mode of the indication element may change depending on the magnification ratio of the displayed image. This can improve the visibility of the medicine. For example, the transparency of the indication element can be adjusted depending on the magnification ratio of the image. As an example, the transparency of the indication element may be increased as the magnification ratio of the image increases. This can prevent the area of ​​the medicine from being enlarged and the overlapping portion of the indication element with the medicine from becoming large, thereby preventing a decrease in visibility. The display mode of the indication element adjusted depending on the magnification ratio is not limited to the transparency. For example, the thickness (width), color, or other attributes of the indication element can be changed depending on the magnification ratio.

[0076] As a specific example, the alpha channel value of the polygon of the instruction element may be changed according to the magnification ratio of the image. In other words, the alpha channel value may be determined according to the magnification ratio. For example, when the entire image is displayed (reduced display), the alpha channel value of the polygon of the instruction element may be set to be large, and when the area of ​​the medicine is enlarged and displayed (enlarged display), the alpha channel value may be set to be smaller than that in the reduced display. This makes it easier to grasp the position of the medicine in the image by the instruction element when reduced, and makes it possible to grasp the details of each medicine (such as the engraving or medicine name text) without being obstructed by the instruction element when enlarged.

[0077] As shown in the example of screen G4-4, the drug identification support system 10 can display a partial image of an area including text recognized as a drug name in an image as an identification result. This allows the user to check the drug name by looking at the captured image of the text. In this case, if the aspect ratio of the area including the text of the drug name does not satisfy a predetermined condition, a partial image of an area with margin in the horizontal direction (left and right direction) of the text recognized as the drug name can be displayed. This makes it possible to display an area including images of other parts of the text of the drug name when only a part of the text of one drug name is extracted as an identification result. This increases the probability that the user can identify the drug name by looking at the image of the text. For example, the part of the drug manufacturer's description "AA Pharmaceutical" in the text of one drug name "Kipresu Granules 4 mg "AA Pharmaceutical"" in G4-3 of FIG. 5 may be extracted as an identification result of one drug. In this case, by displaying an area extended left and right from the text part of "AA Pharmaceutical", the user can easily identify the drug name. As an example, if the aspect ratio of a rectangular area containing text extracted as a candidate drug name is outside a specified range (for example, if the ratio of width to height is less than a specified value), a rectangular area that is extended a specified width to the left and right of the text can be displayed.

[0078] Furthermore, when a partial image including text recognized as a single drug name is displayed, the range of the displayed partial image may be adjustable by a slide operation or the like. This allows the user to easily check the drug name in an image by adjusting the display range when an image of only a portion of the description of a single drug name is displayed as an identification result. In this case, too, when the aspect ratio of the area of ​​the text recognized as a single drug name is outside a predetermined range, an area extended in the left and right directions from the text is displayed, thereby reducing the amount of operation (slide amount) required by the user to check.

[0079] The identification result may indicate that the image or the drug area in the image is unidentifiable. In this case, the image or drug area that is unidentifiable may be displayed, and the process of accepting the image input of S4 again may be executed. In this case, the photographing screen of the re-photographing mode may be displayed. In the re-photographing mode, a user interface may be provided for photographing the drug closer. For example, the user may be allowed to designate the drug area displayed on the photographing screen and press the photographing button when the camera is brought close to the drug and the image is in focus. In this case, the identification result acquiring unit 3 may acquire the identification result for the image of the designated drug area from the identification system 20.

[0080] In S12 of FIG. 3, the medicine identification support system 10 further judges whether or not to continue photographing based on input from the user. If photographing is to be continued, processing of S4 is executed, and if not, processing of S13 is executed. In the example of FIG. 3, the input (photographing) and recording of the image triggers the identification process of the image, the recording of the identification result, and the acceptance of confirmation input. In this case, the user can quickly know and confirm the identification result after photographing the medicine. Note that the judgment of whether to end photographing is not limited to the example of FIG. 3. For example, an instruction input to end photographing (e.g., selection of the back button) may be accepted on the photographing screen displayed in S4.

[0081] 3, the medicine identification support system 10 judges whether or not to execute grouping processing based on an input from the user. If it is judged not to perform grouping (NO in S13), the process returns to the menu screen display in S1.

[0082] In the case of grouping (YES in S13), the group determination unit 4 displays a group designation screen G5 (S14) and accepts designation of a group for each of the personal medicines identified by the user (S15). FIG. 6 is a diagram showing an example of the group designation screen G5. In the example of FIG. 6, a list of the identified personal medicines is displayed on a screen G5-1. When the user performs an operation to start group designation of medicines (screen S5-2), medicines for which a group has not been designated are displayed on a screen G5-3. In this example, the reception screen G5-3 is initially displayed in a state in which all personal medicines for which a group has not been designated are automatically selected. The user can input selection / non-selection for each medicine. The screen G5-3 accepts input of a group number and a group name for the selected medicine. When the user presses the "Group" button to perform a decision operation, the selected medicine is classified into the group with the designated number. This determines the group to which the selected medicine belongs. The classification result is displayed on a screen G5-4.

[0083] FIG. 7 is a diagram showing an example of a group designation acceptance process in the case where a newly identified drug is added when a group for an individual's drugs has already been determined. In the example of FIG. 7, all of the individual's drugs are displayed on the screen G5-6. Group numbers are displayed for drugs for which groups have already been determined. When the user performs an operation to start group designation (screen G5-7), the group determination unit 4 displays ungrouped drugs in a manner that allows selection / non-selection (screen G5-8). The screen G5-8 also allows input of a group number and a group name. In this example, the reception screen G5-3 is initially displayed in a state in which an ungrouped drug "Myonal" is selected and a new group number is input as the group number. When the user presses the "Group" button to perform a decision operation, the selected drug "Myonal" is classified into the group with the specified number (=2). The classification result is displayed on the screen G5-9.

[0084] In this way, when a newly identified drug is added, an ungrouped drug whose group to which it belongs has not been determined is displayed and an instruction is accepted, making it easier for the user to specify a group.

[0085] In the example of Fig. 3, the processes S14 and S15 for accepting group instruction input accept group instruction input for drugs added in the classification result after confirmation of the classification result of one image is completed. This allows group instruction input to be accepted at an appropriate time. Note that the timing for accepting group instruction input is not limited to the example of Fig. 3.

[0086] FIG. 8 is a flow diagram showing another operation example of the medicine identification support system 10. FIG. 8 shows an example in which, after photographing and recording of an image, the medicine of the image is identified in response to a user's instruction. S1 to S3 in FIG. 8 are the same as S1 to S3 in FIG. 3. In S4 in FIG. 8, reception and recording of input of a plurality of images by photographing operation are continuously executed (S5-1 to S5-3). In receiving the input of the plurality of images, the image recording unit 2 may receive the input of an image showing a division of a group. For example, the image recording unit 2 may receive the input of an image showing a division of a medicine provider. For example, an image of a medicine bag of a pharmacy or an entry in a medicine notebook may be received as an image showing a division. The group determination unit 4 may determine a group to which the image belongs based on the input of the image showing the division. This allows the medicine of the image to be classified into groups. For example, image or medicine group determination may be executed so that a medicine identified in an image input before a division image and a medicine identified in an image input after a division image are in different groups.

[0087] In the example of Fig. 8, the start and end of the acceptance period for inputting a plurality of images (S5-1 to 3) is controlled based on instructions from the user. In this example, the start condition of the acceptance period is the user's operation of selecting to take a picture of the personal information screen G2. The end condition of the acceptance period is the user's operation of transitioning from the shooting screen to another screen.

[0088] In the example of FIG. 8, after the user finishes photographing the personal medicines on-site, he / she temporarily closes the medicine identification support system 10. Thereafter, the user returns to the pharmacy, starts up the medicine identification support system 10, displays the menu screen (S1), and selects batch image identification. As a result, the identification result acquisition unit 3 provides the recorded unidentified images to the identification system 20 in a batch, obtains the identification results of those images from the identification system 20, and records them in the storage unit 6 (S6 to S9). The processes of providing each image to the identification system 20, obtaining, recording, and confirming the identification results can be executed in the same manner as S6 to S9 in FIG. 3.

[0089] (Example of identification process) FIG. 9 is a flowchart showing an example of the identification process of the identification system 20. FIG. 9 is an example of the identification process of medicines in an image obtained by photographing medicines placed on the placement surface of the medicine placement table 50. FIG. 10 is a top view showing an example of the configuration of the medicine placement table 50. In the example of FIG. 10, the medicine placement table 50 is in a sheet shape. A plurality of markers M1 to M4 are provided on the placement surface of the medicine placement table 50, spaced apart from each other. The area inside the lines connecting the markers M1 to M4 is the area where the medicine is placed, that is, the area to be photographed. The markers M1 to M4 in FIG. 10 are arranged at four locations corresponding to the corners of a rectangle. The area R1 inside this rectangle is the area where the medicine is placed. Each of the markers M1 to M4 includes a combination of reference colors M11 to M13, M21 to M23, M31 to M33, and M41 to M43. In this example, the combination of reference colors is a combination of red, green, and blue. The combination of reference colors may be the same as the combination of color components represented by pixel values ​​(tone values) in the image data. The number, shape, and arrangement of the markers M1 to M4 are not limited to the example in Fig. 10. For example, the number of markers may be three, or five or more.

[0090] (Example of keystone correction) In FIG. 9, the correction unit 21 performs keystone correction on the captured image of the medicine based on the positions of the markers M1 to M4 in the image (S21). FIG. 11 is a flowchart showing an example of the keystone correction process. In the example of FIG. 11, the correction unit 21 acquires a captured image C1 (S211). The correction unit 21 extracts an area of ​​a first reference color in the captured image C1 (S212). The first reference color is one color (e.g., red) in a combination of reference colors. In S212, the correction unit 21 can extract pixels in the captured image C1 whose pixel values ​​are within a predetermined range as the area of ​​the first reference color. The white area of ​​the image C2 is the area extracted in S212.

[0091] In S213, the correction unit 21 extracts, from the regions extracted in S212, regions whose shapes are close to the shapes of the first reference colors of the markers M1 to M4, the number of which is equal to the number of markers M1 to M4 (four in this example). The white regions in the image C3 are the regions extracted in S213. The extracted regions are regions corresponding to the first reference colors of the markers M1 to M4. That is, the regions of the markers M1 to M4 in the captured image are extracted. If it is not possible to extract all the regions of the markers M1 to M4 in S213, the extraction processes of S212 and S213 may be executed for other reference colors (second or third reference color).

[0092] In S213, the correction unit 21 determines a representative point of each region extracted in S213. The representative point is a point corresponding to the position of the markers M1 to M4 in the photographed image C1. The correction unit 21 transforms the image so that the representative point approaches the target point (S214). As an example, the correction unit 21 acquires the representative point corresponding to the position of the markers M1 to M4 and the coordinates of each target point. The coordinates of the target point may be specified based on the position of the markers M1 to M4 on the mounting surface of the medicine mounting table 50. For example, the positions of the markers M1 to M4 as viewed from the normal direction of the mounting surface may be set as the positions of the target points. The correction unit 21 obtains a transformation matrix that transforms the coordinates of the representative point into the coordinates of the target point. The transformation matrix may be a perspective transformation matrix. The correction unit 21 performs perspective transformation on the entire photographed image C1 using the perspective transformation matrix (S215). The transformed image becomes an image that has been subjected to keystone correction, that is, a keystone corrected image. The perspective transformation matrix can be calculated using, for example, a function of OpenCV. An image C4 in Fig. 11 is an image obtained by performing keystone correction on the captured image C1 and then cropping it.

[0093] (Example of color correction) The correction unit 21 performs color correction on the keystone corrected image (S22 in FIG. 9). FIG. 12 is a flowchart showing an example of the color correction process. In the example of FIG. 12, pixel values ​​(gradation values) of the nth reference color (n=1, 2, 3 in this example) of the markers M1 to M4 in the keystone corrected image that has been keystone corrected are determined (S221). For example, the average value of pixel values ​​of areas corresponding to the nth reference color of each of the markers M1 to M4 in the corrected image may be set as the pixel value of the nth reference color of each of the markers M1 to M4.

[0094] The correction unit 21 calculates correction values ​​(correction parameters) for performing color correction of the nth reference color component on the trapezoid corrected image based on the pixel values ​​of the nth reference color of each of the markers M1 to M4 (S222). For example, the correction values ​​for the pixels of the nth reference color of the markers M1 to M4 may be calculated so that the pixel values ​​of the nth reference color of the markers M1 to M4 are the same or close to each other. The correction values ​​for the pixels in the area inside the line connecting the markers M1 to M4 may be calculated by interpolation, for example, linear interpolation.

[0095] The correction unit 21 corrects the pixel value of the nth reference color component in the keystone corrected image using the correction value calculated in S222 (S223). The processes of S221 to S223 are executed for each reference color. As a result, color correction is applied to the keystone corrected image. That is, a corrected image in which keystone correction and color correction are applied to the captured image is obtained. Areas of the same color that are separated from each other on the placement surface, such as the reference colors of each marker, may have different pixel values ​​in the captured image due to the influence of the shooting environment. By performing color correction based on the combination of the reference colors of each marker, it is possible to correct such fluctuations in pixel values ​​due to the influence of the shooting environment.

[0096] In S23 of FIG. 9, a drug region is extracted in a corrected image in which keystone correction and color correction are performed on the captured image. As an example, the correction unit 21 identifies a drug region in the corrected image using a trained model. The trained model can be generated by learning using, for example, an image and a training data set including a drug region in the image as training data. The training data may include a pen liner and a handwritten or printed character region in addition to the drug. For example, a trained model of instance segmentation can be used as the trained model. This allows each drug region in the input image to be extracted as a drug region.

[0097] The correction unit 21 executes the processes of S24 to S26 for each medicine region extracted in S23. S24 is a process of extracting the marking or printing in the medicine region. For example, a feature image showing the features of the marking or printing is extracted from the medicine region. The extraction of the marking or printing may be performed rule-based by a predetermined image processing such as edge detection or other conversion processing. Alternatively, for example, the marking or printing can be extracted using a trained model. The trained model can be generated by machine learning using a set of an image of the medicine region and the features of the marking or printing extracted from the image as training data. The trained model may be, for example, a trained model of semantic segmentation.

[0098] S25 is a matching process in which the marking or printing extracted in S24 is compared with a master image, and a drug with a high degree of match (score) is determined as a candidate drug. The matching process is, for example, template matching. The master image may be a master image of the marking or printing on the drug. In S26, the correction unit 21 outputs the identification result of the drug in the drug region. The identification result includes information on the candidate drug.

[0099] Fig. 9 is an example of an identification process for an image of a medicine itself. For example, when the form of the target medicine is a plain tablet or a medicine packet, the process of Fig. 9 can be applied to either. The identification process for plain tablets and the identification process for medicine packets may be different. For example, the f extraction process of S23 and the engraving print extraction process of S24 in Fig. 9 may be different for plain tablets and medicine packets.

[0100] (Example of classification processing for images of pharmaceutical text) FIG. 13 is a flowchart showing an example of the identification process for an image of a text of a medicine. In the example of FIG. 13, the identification system 20 executes character recognition process for the image (S31). The character recognition process can be performed using an existing OCR technology. The character recognition process may include preprocessing such as binarization process and edge smoothing process for the image. The identification system 20 concatenates or separates the characters recognized in S31 to generate a character string suitable for medicine identification (S32). For example, a character string to be identified as a medicine is generated by concatenating or separating the characters based on the position and size of the recognized characters. The character string generated in S32 is compared with the master medicine name to determine a candidate medicine that matches the character string highly (S33). In this matching process, a distance (score) indicating the degree of match (degree of difference) between the character string and the master medicine name is calculated. The distance may be calculated based on, for example, the number of insertions, deletions, or substitutions required to transform one character string into the other character string. Information on the candidate medicine determined in S33 is output to the medicine identification support system 10 as an identification result (S34). The identification result may include information identifying the candidate drug as well as the location of the identified drug description in the image (drug description area).

[0101] The character recognition process for the image by the identification system 20 may include a candidate area limiting process that limits a candidate area in the image that is likely to contain a drug name. In this case, character recognition process is performed on the area limited by the candidate area limiting process. The candidate area limiting process may include, for example, at least one of a process of removing ruled lines in a binarized image and a process of removing fine contours that cannot be treated as characters from among contours extracted from a binarized image. This allows OCR processing to be performed on the part of the image that is likely to contain a drug name in the character recognition process. This makes it possible to reduce processing time and improve recognition accuracy.

[0102] The process of removing the ruled line portion may include, for example, a process of extracting edges that are strong in the horizontal direction and edges that are strong in the vertical direction of the image by filtering the binarized image, and a process of removing a masked area using an area including the extracted edges as a mask. The mask may be, for example, a composite mask obtained by combining a horizontal mask inverting edges that are strong in the horizontal direction and a vertical mask inverting edges that are strong in the vertical direction.

[0103] The process of removing fine contours may include, for example, a process of combining adjacent contours in a group of contours in a binarized image, and a process of removing contours that are smaller than a predetermined size from among the contours remaining after the combining process. The process of combining contours may include, for example, a process of combining contours within a predetermined distance from among the group of contours in the white part of an image obtained by inverting the binarized image. The process of removing fine contours can remove noise around the drug name.

[0104] In addition, by executing the above-mentioned process of removing fine contours on the image from which the ruled lines have been removed by the process of removing the ruled lines, it is possible to limit the drug name candidate region more accurately. For example, in an image obtained by inverting the image from which the ruled lines have been removed, adjacent contours may be combined.

[0105] (Modification of the medicine placement table) Fig. 14 is a top view showing a modified example of the medicine placing table. In the example of Fig. 14, grid lines are drawn on the placing surface. The grid lines L1 indicate a plurality of sections in the area inside the lines (imaginary lines) connecting the markers M1 to M4. One section indicated by the grid lines L1 is set to an area large enough for one plain tablet to be placed thereon.

[0106] When photographing both the front and back of a plain tablet, one tablet is placed per section in at least one of the sections indicated by grid line L1. The user places one tablet per section and takes a first photograph. The image recording unit 2 records the first image. Next, the user turns over the tablets in each section, i.e., turns them upside down, and takes a second photograph. The image recording unit 2 records the second image in the memory unit 6 in association with the first image.

[0107] In this way, the image recording unit 2 can accept input of images of both sides of the tablet. In this case, the image recording unit 2 accepts designation of both sides or one side of the plain tablet from the user as the designation of the form of the medicine. When both sides of the plain tablet are designated, the image recording unit 2 notifies the user of an instruction to place one tablet per section and accepts the first photographing. After the first photographing, the image recording unit 2 notifies the user of an instruction to turn over the tablet in each section and take a second photographing, and accepts the second photographing.

[0108] By the first and second photographing, the front and back images of the tablet in each section are associated with each other and stored in the memory unit 6. The identification result acquisition unit 3 provides these two images to the identification system 20 as images of both sides of the tablet. The identification system 20 can perform identification processing using both the front and back images of one tablet. This improves identification accuracy. Also, when a tablet that is prone to rolling is turned over, the position of the tablet is shifted, reducing the possibility of incorrectly identifying the correspondence between the front and back images of the tablet.

[0109] FIG. 15 is a perspective view showing another modified example of the medicine placing table 50. The medicine placing table 50 shown in FIG. 15 is configured to be foldable. FIG. 15(a) shows the medicine placing table 50 in a folded state. FIG. 15(b) and FIG. 15(c) show the medicine placing table 50 in an open state. The medicine placing table 50 has two plate parts 501 and 502 connected by a hinge. In the example of FIG. 15, in the folded state, the placing surfaces 50a of the two plate parts 501 and 502 are on the outside, and the back surfaces 50b of the placing surfaces 50a of the two plate parts 501 and 502 are on the inside. That is, the medicine placing table 50 is configured to be foldable so that the back surfaces 50b are on the inside. The medicine placing table 50 is configured not to be folded in a direction in which the placing surfaces 50a are on the inside. When the medicine mounting table 50 is in an open state, the mounting surfaces 50a of the two plate portions 501 and 502 are arranged on the same plane. The medicine mounting table 50 is used in the open state with the mounting surfaces 50a facing up.

[0110] In the example of FIG. 15, a stopper 51 for maintaining the open state is provided slidably on the back surface 50b of the medicine placement table 50. The stopper 51 is provided slidably along a guide 52 provided on the plate portion 501 and the plate portion 502. The guide 52 may be, for example, a groove or a rib provided on the back surface 50b of the two plate portions 501 and 502. When the medicine placement table 50 is in an open state, the stopper 51 is slidable between a first position where the stopper 51 is disposed across both of the two plate portions 501 and 502, and a second position where the stopper 51 is disposed only on one of the two plate portions 501 and 502. When the stopper 51 is in the first position, the stopper 51 maintains the open state of the medicine placement table 50. That is, the open medicine placement table 50 is prevented from folding by the stopper 51 in the first position. When the stopper 51 is in the second position, the medicine placing table 50 can be folded so that the back surface 50b faces inward.

[0111] By providing the stopper 51 on the medicine placing table 50, the medicine placing table 50 can maintain the open state without folding even when lifted in the open state. Therefore, folding of the medicine placing table 50 prevents medicines from falling off the placing surface 50a. This makes it easy to lift and move the medicine placing table 50 with medicines placed thereon. In addition, the stopper 51 slidably attached to the back surface 50b allows the medicine placing table 50 to be compact while maintaining the open state. For example, the medicine placing table 50 can be made more compact than a configuration using a torque hinge or the like.

[0112] Stopper 51 may be formed of a plate whose longitudinal direction is the sliding direction, for example, as shown in Fig. 15. The material of stopper 51 is not particularly limited, and may be, for example, metal or resin. For example, by forming stopper 51 from stainless steel material (SUS material), it is possible to configure stopper 51 that is thin and has high bending stress.

[0113] The medicine placing table 50 may further include a locking member that locks the stopper 51 at at least one of the first position and the second position. FIG. 16 is a perspective view showing a configuration example of the medicine placing table 50 provided with magnets 53a and 53b as an example of the locking member. In the example of FIG. 16, the stopper 51 is made of a material that is attracted to a magnet. The magnet 53a is arranged at a position where the stopper 51 at the first position overlaps with the stopper 51 when viewed from a direction perpendicular to the placing surface 50a. The stopper 51 is locked at the first position by a force that attracts the stopper 51 to the magnet 53a at the first position. In the example of FIG. 16, the magnet 53b is arranged at a position where the stopper 51 at the second position overlaps with the stopper 51 when viewed from a direction perpendicular to the placing surface 50a. The stopper 51 is locked at the second position by a force that attracts the stopper 51 to the magnet 53b at the second position. The locking member is not limited to a magnet. For example, the locking member may be a projection or a recess that engages with the stopper 51 at least in one of the first position and the second position.

[0114] (Example of IC tag on medicine table) In the example of FIG. 14, the medicine mounting stand 50 includes an IC tag T. The IC tag T is attached to the surface of the medicine mounting stand 50 or embedded inside the medicine mounting stand 50. The IC tag T holds data that can be read by short-range wireless communication. The IC tag T is an RFID tag. The IC tag T may be, for example, an NFC tag capable of communicating according to the Near Field Communication (NFC) standard. The communication standard of the IC tag is not limited to NFC, and may be, for example, Bluetooth (registered trademark) or other RFID standards.

[0115] The IC tag T holds data for a computer to start a specific application. The IC tag T records data for causing a computer constituting the drug identification support system 10 to execute an application related to the drug identification support system 10. The computer constituting the drug identification support system 10 may include a wireless communication unit that reads data from the IC tag T. For example, the mobile terminal 30 shown in FIG. 2 may have a reader that reads data from the IC tag T as the wireless communication unit. The computer executes a specific application using the data read from the IC tag T.

[0116] As an example, the IC tag T may record data for starting an application that realizes at least a part of the functions of the medicine identification support system. In this case, when a computer on which the application is installed approaches the medicine placement table 50, the application can be automatically executed using the data in the IC tag T. For example, data for starting can be recorded in the IC tag T so that the application of the medicine identification support system 10 is automatically started by bringing the computer close to the medicine placement table 50.

[0117] The application executed by the data of the IC tag T may be an application that executes a process of acquiring information about a drug using the drug's GS1 code. As an example, it may be an application that reads the GS1 barcode on the drug's packaging, and uses the GS1 barcode information to enable browsing of information about the drug identified by the GS1 barcode information in a database (for example, package insert information provided by the Pharmaceuticals and Medical Devices Agency (PMDA)).

[0118] The medicine placing table 50 may be foldable. For example, the placing table may be configured to be folded in two. The placing surface of the medicine placing table 50 may be formed of, for example, a material that does not easily reflect light. This makes it difficult to cause erroneous identification of medicines due to disturbance light. The inventors noticed that, for example, the light of the illumination in the shooting environment is concentrated and reflected in a narrow range of the placing surface, causing reflected light to appear in the captured image like halation, which may be identified as a tablet. By forming the placing surface of the medicine placing table 50 from a material that does not easily reflect light, erroneous identification can be suppressed. From the same viewpoint, the color of the placing surface is preferably black. As an example, a sheet of a material that does not easily reflect light may be attached to the placing surface. An example of a material that does not easily reflect light is polycarbonate. The sheet of a material that does not easily reflect light may also be an optical thin film having a reflection suppressing function. The optical thin film may be, for example, an anti-glare film or an anti-reflection film.

[0119] The medicine placement table 50 may be made of, for example, silicone resin. This allows the medicine placement table 50 to be cleaned with alcohol. In addition, the gripping force of the placement surface of the medicine placement table 50 is increased, making the medicine less likely to slip on the placement surface. As an example, the medicine placement table 50 may be made of black silicone rubber. The medicine placement table 50 may be a sheet of black silicone rubber with markers or the like printed on the surface. This allows a compact medicine placement table 50 to be configured from a thin sheet.

[0120] FIG. 17 is a diagram showing a modified example of a screen that accepts image input from a user in S4 of FIG. 2. In the example of FIG. 17, the image recording unit 2 accepts the designation of the identification target range along with the shooting operation from the user. On the screen G3-6, a frame K1 indicating the identification target range is displayed superimposed on the image in the monitor area A1 of the shooting screen. The position and size of the frame K1 can be manipulated by the user. The user can specify the identification target range by manipulating the frame K1 in real time while performing the shooting operation. For example, the user can change the range of the frame K1 by zooming.

[0121] For example, as shown in FIG. 17, a plurality of medicine packets may be placed in the area R1 for placing medicines indicated by markers M1 to M4 on the medicine placing table 50. In such a case, the user can specify the range of one package as the range to be identified. The image recording unit 2 records the coordinates of the range to be identified specified by the user in the memory unit 6 together with the captured image. The identification result acquisition unit 3 provides the image and data indicating the range to be identified to the identification system 20. The identification system 20 is capable of identifying one package of medicines. This allows the processing time to be reduced. Also, the identification results can be recorded separately for each package.

[0122] In the example of FIG. 17, the markers M1 to M4 of the medicine placement table 50 are included in the photographed screen. The image recording unit 2 may detect the positions of the markers M1 to M4 on the photographed screen. The image recording unit 2 may output a photographing instruction to the user based on the positions of the markers M1 to M4. For example, when all the markers M1 to M4 are not included in the photographed screen, a message may be output to urge the user to include all the markers M1 to M4 in the photographed range. Also, when it is determined that the shape of the area R1 inside the markers M1 to M4 is not within the range in which keystone correction is possible, a message may be output to urge the user to change the angle of the camera. Also, the image recording unit 2 may receive, from the user, a designation of an identification target range K1 in the area R1 surrounded by the markers in the photographed image of the camera.

[0123] The image recording unit 2 accepts input of an image obtained by photographing a medicine placed on the medicine placement table 50 including a plurality of markers M1 to M4. The image recording unit 2 may accept input of an image including all of the plurality of markers M1 to M4 as in the above example. As a modified example, the image recording unit 2 may accept input of a photographed image including only one of the plurality of markers M1 to M4. In this modified example, one marker may include a combination of two or more reference colors, or may be only one color. In this case, the correction unit 21 of the identification system 20 may execute size correction of the medicine (medicine area) in the image from the dimensions of one marker included in the photographed image. The correction unit 21 may also perform dimensional correction (for example, keystone correction) based on the shape of one marker included in the photographed image on the photographed image. The correction unit 21 may also determine the identification target area in the photographed image based on the position of one marker included in the photographed image. This makes it possible to obtain a close-up image taken at a position closer to the medicine than a captured image including all of the markers M1 to M4. By identifying the medicine from the close-up image, the medicine can be identified from a captured image in which the markings on the medicine are more clearly visible. This improves the identification accuracy.

[0124] For example, when a single marker in a captured image including only one of a plurality of markers satisfies a close-up photographing condition, the image recording unit 2 may accept and record the captured image as a close-up photographing image. The close-up photographing condition may include, for example, a condition related to at least one of the size or shape of the marker in the captured image. As an example, the close-up photographing condition may include a condition that the size of the marker in the captured image is equal to or larger than a threshold value. The size of the marker may be determined, for example, by the length of the side of the marker. By setting the size of the marker as a condition, an image suitable for correction using one marker is obtained. In addition, the close-up photographing condition may include a condition that the inclination of the marker in the captured image is within an allowable range. The inclination of the marker may be determined, for example, by the inclination of the side of the marker with respect to the horizontal or vertical direction. By setting the inclination of the marker as a condition, an image suitable for drug region determination using one marker is obtained.

[0125] When accepting a shooting operation, the image recording unit 2 may output to the user whether the shooting conditions satisfy the close-up shooting conditions. The image recording unit 2 may also output a shooting instruction to the user based on the form of the marker on the shooting screen. For example, when only one marker out of the multiple markers M1 to M4 is included on the shooting screen and the form of the marker does not satisfy the close-up shooting conditions, a shooting instruction to satisfy the close-up shooting conditions may be output to the user. For example, when the size of the marker is smaller than a threshold, an instruction to move the camera closer to the placement surface may be output. When the inclination of the marker is outside the allowable range, an instruction to move the optical axis of the camera closer to the direction perpendicular to the placement surface may be output.

[0126] The image recording unit 2 may receive a switching operation from a user between an all-marker shooting mode in which all of the markers M1 to M4 are captured in the shooting range, and a close-up shooting mode in which only one of the markers M1 to M4 is captured in the shooting range. Alternatively, the image recording unit 2 may automatically switch between the all-marker shooting mode and the close-up shooting mode depending on the shape of the marker included in the shooting range.

[0127] The correction unit 21 of the identification system 20 may perform size correction of the medicine based on the size of the marker as a dimensional correction based on one marker included in the photographed image. For example, the size of a partial image of the medicine area recognized in the photographed image (e.g., the length of a side of the partial image) can be corrected to a predetermined ratio with respect to the size of the marker in the photographed image (e.g., the length of a side). The correction unit 21 may also perform keystone correction based on the shape of the marker. As the keystone correction, for example, a process of converting the image so that the shape of the marker in the photographed image approaches a target shape (e.g., a positive direction or a rectangle) may be performed. The correction unit 21 may perform dimensional correction based on the shape of a marker of one color (e.g., red) as one marker, for example.

[0128] The correction unit 21 may execute a process of changing the area to be masked depending on the position of the marker as the drug area determination based on the position of one marker. The masking may be, for example, a combination of masking of the periphery of one marker and masking with a line based on the marker as a boundary. The line based on the marker may be, for example, a line extending horizontally and a line extending vertically from the marker. Which side of the line is to be masked may be determined according to the position of the marker in the captured image. The correction unit 21 may determine the masking based on a marker of one color (for example, red) as one marker. Alternatively, the masking may be determined based on a marker of two or more reference colors (for example, red, green, and blue) as one marker.

[0129] FIG. 18 is a diagram showing an example of a screen displaying a list of individuals recorded in the memory unit 6 by the individual recording unit 1. For example, when the user selects "individual list" on the menu screen G1 displayed in S1 of FIG. 2, the individual list screen is displayed. A map is displayed on the screen G1-1 displaying the individual list in FIG. 18. Information indicating an individual is displayed in association with a photographing location on the map. The photographing location is the location where an image of the corresponding individual's medicine was photographed. When the user specifies the photographing location displayed on the map, the screen transitions to the personal information screen G1-2 of that individual.

[0130] The image recorded in the storage unit 6 may include the location where the image was taken as EXIF ​​information. The image in the storage unit 6 is associated with unique data that identifies an individual. Therefore, the information of the individual identified by the unique data (e.g., name, age, etc.) is linked to the location where the image was taken. The medicine identification support system 10 acquires the information of the individual identified by the unique data and the location where the image was taken associated with the unique data. The medicine identification support system 10 displays the information of the individual associated with the location where the image was taken on a map based on the acquired information of the individual and the location where the image was taken. In the example of FIG. 18, the information of the individual displayed on the map includes the name or unique data, the date and time of taking the image, and the number of images taken. In the example of FIG. 18, the image taken at the location where the image was taken is displayed in a form that allows the user to know whether the image has been sent to the identification system 20 or not.

[0131] In this way, by displaying the photographing location on the map in association with the personal information, the user can visually understand where the image of the individual's medicine was photographed. In addition, since the individual is displayed on the map in a selectable manner, the user can access the desired information with an intuitive operation.

[0132] In the example of FIG. 18, personal information is displayed in association with the location where the image was taken. As a variation of this, personal information may be displayed in association with the location of the new record. When a user inputs an instruction to make a new record for an individual, the individual record unit 1 can generate unique data based on the location where the instruction to make the new record was input. In this case, the unique data includes information indicating the location of the new record. The medicine identification support system 10 can display the individual information in association with the location of the new record on a map based on the individual information identified by the unique data in the memory unit 6 and the location of the new record included in the unique data.

[0133] The medicine identification support system 10 may include a package insert output unit that acquires the GS1 code of a medicine specified by a user, and acquires and outputs the package insert of the medicine using the GS1 code. The GS1 code can be acquired, for example, by photographing the GS1 barcode on the medicine packaging with a camera, the identification result of the identification system 20, or by searching the GS1 code based on the medicine name. The medicine package insert using the GS1 code may be acquired, for example, from an external system that provides the medicine package insert corresponding to the GS1 code. Alternatively, the medicine identification support system 10 may acquire data of the package insert associated with the medicine master data held by the identification system 20. The output format of the package insert is not particularly limited, and may be, for example, displayed, transmitted, or recorded.

[0134] (Example of sharing identification results) The medicine identification support system 10 may convert the identification result into a barcode or two-dimensional code and display it on a screen or print it out. Alternatively, the identification system 20 may hold the identification result in a format that can be viewed via a network. The identification system 20 may include a web server that provides data on the identification result. This makes it possible to view the identification result of the identification system 20 from an external web browser.

[0135] The medicine identification support system 10 may have a function of temporarily saving the medicine identification result in text data in a state where it can be pasted, so-called copying to a clipboard, in response to a user's operation. This simplifies the user's operation for using the identification result. For example, when a user performs an operation to specify an individual and instruct output of the identification result, text data showing the medicine identification result of the specified individual may be copied to the clipboard. In this case, text data showing the medicine identified based on an image associated with the unique data of the specified individual may be temporarily saved in a state where it can be pasted. As an example, when a user performs an operation to instruct output of a barcode or two-dimensional code showing the medicine identification result of an individual, the text data showing the medicine identification result of the individual may be copied to the clipboard. In addition, the format of the temporarily saved text data may be selectable by the user.

[0136] The medicine identification support system and the identification system in the above embodiments can be used, for example, as a medicine discrimination support system and a discrimination system for discriminating medicines prescribed to patients. The discrimination in the above embodiments can be discrimination. This enables support for discrimination or discrimination suitable for various situations expected in the medical field. Furthermore, in the medicine identification support system and the identification system, medicines that can be identified include medicines prescribed by doctors as well as OTC medicines (over-the-counter medicines) that can be purchased without a prescription. Furthermore, medicines that can be identified may be medicines brought by patients or medicines returned within the hospital (returned medicines).

[0137] The embodiment of the medicine placing table is not limited to the above example. For example, the medicine placing table may have a wall extending upward outside the medicine placing surface. The wall can prevent the medicine from leaking out of the placing surface. In this case, the marker may be provided on the placing surface or on a part of the wall. For example, the marker may be arranged on the upper surface of the wall at positions corresponding to at least two of the four corners of the placing surface. As an example, the medicine placing table may be formed of a container that has a recess and opens upward. In this case, the bottom surface of the recess becomes the placing surface, and the side wall of the container surrounding the placing surface becomes the wall. In addition, the wall of the medicine placing table may have a support part for supporting a camera that photographs the medicine on the placing table. Thereby, the support part can support the camera at a position suitable for photographing the medicine on the placing table. Note that one marker may include a combination of two or more reference colors as in the above example, or may be only one color (for example, any one of red, blue, and green).

[0138] The above-described embodiment is an example of the present invention, and the present invention is not limited to this example. Furthermore, the configurations and processing functions described in the above-described embodiment can be arbitrarily combined by selecting appropriate ones. [Explanation of symbols]

[0139] 1: personal record unit, 2: image recording unit, 3: recognition result acquisition unit, 4: group determination unit, 5: duplicate candidate display unit, 10: medicine recognition support system, 20: recognition system

Claims

1. an image recording unit that receives an input of an image of the drug to be identified and records the image in a storage unit; an identification result acquisition unit that acquires an identification result of the medicine included in the image recorded in the storage unit, A drug identification support system that converts the identification result into a barcode or a two-dimensional code and outputs it.

2. The drug identification support system according to claim 1, the identification result acquisition unit records the identification result in the storage unit in association with information for identifying an individual; A medicine identification support system that accepts an individual designation from a user, converts the identification result corresponding to information identifying the individual designated by the user into a barcode or two-dimensional code, and outputs the barcode or two-dimensional code.

3. The drug identification support system according to claim 1 or 2, a group determination unit that determines a group to which the image or a drug identified in the image belongs based on a group instruction input from a user, and records information indicating the determined group in the storage unit in association with the image or the drug in the identification result; A medicine identification support system that converts the group identification results into a barcode or a two-dimensional code and outputs the barcode or two-dimensional code.

4. The drug identification support system according to claim 1 or 2, The image recording unit receives an input of an image of the medicine and a designation of a state of the medicine in the image, the identification result acquisition unit acquires an identification result according to an aspect of the medicine in the image, A drug identification support system that converts the identification result according to the state of the drug in the image into a barcode or two-dimensional code and outputs it.

5. The medicine identification support system according to claim 4, The image recording unit receives an input of an image of the text of the medicine to be identified, the identification result acquisition unit acquires an identification result including a drug identified by character recognition processing on the text of the image; A drug identification support system that converts the identification results, including the identified drugs, into a barcode or two-dimensional code and outputs it.

6. The medicine identification support system according to claim 5, A medicine identification support system, wherein the image recording unit accepts input of an image of a prescription, a medicine envelope, or a medicine notebook as an image of the text of the medicine to be identified.

7. 3. The medicine identification support system according to claim 1, wherein the medicine identification support system is implemented in a mobile terminal.

Images