Information processing system

Abstract

An information processing system comprising an imaging unit (14) that captures a plurality of images, and a control unit (13) that acquires the plurality of images and recognizes drugs included in the plurality of images, wherein the control unit (13) determines, if a first drug included in a first image is also included in a second image captured after the first image, that the first drug included in the first image and the first drug included in the second image are the same drug.

Description

Information processing system Cross-reference of related applications

[0001] This application claims priority to Japanese Patent Application No. 2024-180361 (filed on 15 October 2024), the entire disclosure of said application is incorporated herein by reference.

[0002] This disclosure relates to an information processing system, and more particularly to an information processing system that recognizes drugs contained in multiple images.

[0003] Generally, the type and amount of medication required for a patient's treatment is issued as a prescription by the hospital, and the prescription information is managed for each patient. However, when a patient is hospitalized, they may have taken medication prescribed at another hospital where the information is not shared, or medication they purchased themselves. In such cases, the hospital where the patient is admitted has to visually inspect the medication brought by the patient, detect the name of the medication, and digitize the information on the type and amount of medication, which is a time-consuming process. For example, Patent Document 1 discloses a drug detection system that can detect the characteristic quantities of a drug based on an image captured by a camera, thereby reducing the effort required for data entry.

[0004] Japanese Patent Publication No. 2022-187605

[0005] (1) An information processing system according to one embodiment of the present disclosure includes an imaging unit that takes multiple images, and a control unit that acquires the multiple images and recognizes a drug contained in the multiple images, wherein the control unit determines that the first drug contained in the first image and the first drug contained in the second image are the same drug if the second image taken after the first image contains the first drug that was contained in the first image.

[0006] (2) In one embodiment of the present disclosure, in (1), the control unit identifies the position of the first drug using world coordinates in AR space or the orientation and movement amount of the imaging unit.

[0007] (3) In one embodiment of the present disclosure, in (1) or (2), the control unit counts the number of recognized drugs, and after recognizing and counting the number of first drugs included in the first image, does not count the number of first drugs included in the second image.

[0008] (4) In one embodiment of the present disclosure, in any of (1) to (3), an output unit is provided that outputs the plurality of images, and the control unit attaches the same marker to the first drug included in the first image and the first drug included in the second image, and causes the output unit to output the plurality of images including the first drug with the marker attached.

[0009] (5) In one embodiment of the present disclosure, in (4), when the control unit recognizes the second drug included in the first image, it attaches the marker to the second drug included in the first image and causes the output unit to output the plurality of images including the first drug and the second drug with the marker attached.

[0010] (6) In one embodiment of the present disclosure, in (4) or (5), a storage unit is provided that stores the number of first drugs counted in the process by the control unit that recognizes drugs included in the plurality of past images, and the control unit causes the output unit to output the counted number of first drugs and the number of first drugs stored in the storage unit.

[0011] (7) In one embodiment of the present disclosure, in (6), the control unit causes the output unit to output a display indicating that the number of first drugs counted matches the number of first drugs stored in the storage unit.

[0012] Figure 1 is a diagram illustrating an example configuration of an information processing system according to one embodiment of the present disclosure. Figure 2 is a diagram illustrating the process of photographing a drug using a mobile terminal device that constitutes the information processing system. Figure 3 is a diagram illustrating the results of a double-check displayed on the mobile terminal device that constitutes the information processing system. Figure 4 is a diagram illustrating the process of photographing a drug packaged in a single dose using a mobile terminal device that constitutes the information processing system. Figure 5 is a diagram illustrating a table showing the total number. Figure 6 is a diagram illustrating a table for detecting differences in the number of items per dose. Figure 7 is a diagram illustrating the prevention of double counting by the information processing system. Figure 8 is a diagram illustrating the management of the drug's position using coordinates in AR space.

[0013] Hereinafter, an information processing system according to one embodiment of this disclosure will be described with reference to the drawings. Figure 1 shows an example of the configuration of the information processing system according to this embodiment.

[0014] The information processing system according to this embodiment supports the process of detecting the name of a drug from medications brought in by a patient in a hospital, and digitizing information such as the type and quantity of the drug (hereinafter referred to as "drug recognition work"). As shown in Figure 1, the detected information such as the type and quantity of the drug is stored in an electronic medical record system connected to the information processing system via a network, and linked to the patient. The electronic medical record system may be, for example, a database (electronic medical record system) that stores the symptoms and medications of each patient, managed by a computer such as a data server. The electronic medical record system may also be configured to be accessible from the information processing system via another system (drug management system) that has a function to manage medications. The network may be, for example, the Internet. The network may also be configured to include, for example, a LAN (Local Area Network) in part.

[0015] The information processing system comprises a control unit 13 and an imaging unit 14. The information processing system may also comprise a storage unit 12 and an output unit 15, as in this embodiment. The information processing system may also comprise a communication unit 11 for sending and receiving data to and from the electronic medical record via a network. The information processing system may also comprise an input unit for receiving input from the operator. In this embodiment, the input unit is realized by a touch panel display integrated with the output unit 15. The information processing system may consist of multiple devices or a single device. When it consists of a single device, the information processing system may be referred to as an information processing device.

[0016] In this embodiment, the information processing system comprises a portable terminal device used by workers performing tasks on-site, and a computer that performs processing to identify the type and quantity of chemicals, etc.

[0017] The mobile terminal device is a device equipped with at least an imaging function (camera), and may be, but is not limited to, a general-purpose mobile terminal such as a smartphone or tablet. In this embodiment, the mobile terminal device displays the processing results of the information processing system. The mobile terminal device may also have a function to detect location information (e.g., coordinates indicating the position of the mobile terminal device) and orientation information (e.g., the orientation of the mobile terminal device).

[0018] The computer may be a server computer, as in this embodiment, or a portable computer, such as a laptop. In this embodiment, the information processing system consists of a server computer that functions as a communication unit 11, a storage unit 12, and a control unit 13, and a portable terminal device that functions as an imaging unit 14 and an output unit 15. The server computer and the portable terminal device are configured to send and receive various data, including images, for example, via wireless connection. Here, as described above, the information processing system may consist of one or more devices, and as another example, the information processing system (information processing device) may consist of a single smartphone. Therefore, in the following description, when the server computer and the portable terminal device are not specifically distinguished, the information processing system will be considered the main unit of processing.

[0019] The details of the components of the information processing system are described below. The communication unit 11 is composed of one or more communication modules connected to a network. The communication unit 11 may include communication modules that support mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication unit 11 may also include communication modules that support wired or wireless LAN standards.

[0020] The storage unit 12 is one or more memories. The memories are, for example, semiconductor memories, magnetic memories, or optical memories, but are not limited to these and can be any type of memory. The storage unit 12 is, for example, built into an information processing system, but it can also be configured to be accessed externally by the information processing system via any interface.

[0021] The storage unit 12 stores various data used in various calculations performed by the control unit 13. The storage unit 12 may also store the results and intermediate data of various calculations performed by the control unit 13.

[0022] In this embodiment, the storage unit 12 stores, for example, the result of counting the medications brought by the patient by type (number of medications). The counting of medications may be performed using the information processing system according to this embodiment or another system having a similar function, or it may be performed by visual confirmation by an operator. The storage unit 12 may store the number of medications for each patient who brings medications. The number of medications stored in the storage unit 12 may be the total number of medications brought by the patient, or it may be the number for each type of medication (medication name). The information on the number of medications stored in the storage unit 12 may be used in the verification (double-check) process. Verification (double-check) is the process of checking or confirming a single task twice, for example, verifying the accuracy of the result of recognizing the medications brought by the patient in the information processing system, and then recognizing that result in the information processing system. Alternatively, it is the process of verifying the accuracy of the result of visual confirmation of the medications by an operator, and then recognizing that result in the information processing system.

[0023] In this embodiment, the imaging unit 14 is a camera that captures multiple images. The imaging unit 14 may, for example, capture multiple images continuously within a predetermined unit time. The multiple images may also be captured as, for example, a video. The multiple images may be extracted in a manner suitable for processing to recognize medications. Recognition of medications includes recognition of the type and quantity of medication. Recognition of the type of medication may include recognition of the name of the medication. Recognition of the quantity of medications may include recognition of the number of medications, the size of the medications, or the weight of the medications. In the information processing system according to this embodiment, it is not necessary for the operator to take an image of each individual tablet or sheet of medication, as in the prior art. Figure 2 illustrates how medications are photographed using a portable terminal device. The operator can arrange the medications brought by the patient and photograph all of them by moving the portable terminal device horizontally, for example, from left to right. In Figure 2, the operator arranges the medications brought by the patient in a single row horizontally (left to right), but is not limited to this; they may also be arranged in a single column vertically (from the depth direction towards the operator), or in multiple rows both horizontally and vertically. Furthermore, the operator photographs all the medications by moving the mobile terminal device from left to right, but is not limited to this; they may also photograph the medications by moving the mobile terminal device from right to left. The multiple images captured are output from the imaging unit 14 to the control unit 13.

[0024] The output unit 15 outputs multiple images to the operator. The output unit 15 may output images of the subject captured by the imaging unit 14. The output images may be images of the drug captured by the imaging unit 14 with a marker indicating that the drug has been recognized. The output unit 15 is implemented by a display device that displays images. The display device may be various types of displays such as liquid crystal displays, but in this embodiment it is a touch panel display.

[0025] The output unit 15 may output the recognized type and quantity of the drug as the recognition result. The output unit 15 may also output the result of the double check performed by the control unit 13, which will be described later. As a result of the double check, the output unit 15 outputs (displays) the number of drugs that have been counted and the number of drugs that have been counted in the past and stored in the memory unit 12. By referring to the result of the double check, the operator can detect the correct type and quantity of drug and immediately notice any errors.

[0026] The control unit 13 is one or more processors. The processors are, for example, general-purpose processors or dedicated processors specialized for specific processing, but are not limited to these and can be any processor. The control unit 13 controls the overall operation of the information processing system. Here, one or more programs used to control the operation of the information processing system may be stored in the storage unit 12. When a program stored in the storage unit 12 is read by a processor, the processor may function as the control unit 13. In other words, the information processing described below may be executed by the program.

[0027] The following outlines the information processing method performed by the information processing system. First, the mobile terminal device takes a picture of the drug with its camera. The mobile terminal device transmits multiple images it has taken to the server computer. At this time, the mobile terminal device does not need to transmit all of the multiple images it has taken to the server computer; it may select and transmit only the images from the multiple images that are suitable for AI-based drug recognition. Images suitable for drug recognition include, but are not limited to, images with high resolution of the drug, images that are properly in focus on the drug, and images with high contrast of the drug. By transmitting only the images suitable for drug recognition from the multiple images taken to the server computer, the communication line with the server computer can be used effectively, and the processing load on the server computer, such as object recognition, can be reduced. The server computer receives the multiple images that have been transmitted. From the multiple images that have been received, the server computer recognizes the type and number (quantity) of drugs using known object recognition methods. The server computer transmits the recognized type of drug and the number of each type to the electronic medical record. Of these processes, the server computer's processing corresponds to the processing performed by the control unit 13 in this embodiment.

[0028] The control unit 13 acquires multiple images and performs processing to recognize the drugs contained in the multiple images. The object recognition method used for drug recognition may be machine learning. Machine learning is an AI (Artificial Intelligence) technology that enables specific tasks to be performed through training, and is a technology that automatically builds a model that learns a lot of data on a computer and performs tasks such as classification. Machine learning may be one of supervised learning, unsupervised learning, and reinforcement learning, or any combination thereof. In this embodiment, the control unit 13 performs object recognition (recognition of drug type and number) using a pre-trained deep learning model stored in the memory unit 12. Deep learning is a type of neural network with a deep network hierarchy. In machine learning, for example, images of drugs labeled with the correct name of the drug may be loaded as training data.

[0029] Referring to Figure 2, the process by which the control unit 13 recognizes the medication, including the operator's actions, will be explained. In the example in Figure 2, the medication is brought in PTP (Press Through Pack) packaging sheets. The operator arranges the medication brought by the patient on the table. The medication may be arranged not only in a single row horizontally, but also in multiple rows both horizontally and vertically. The operator performs the medication recognition task using an application on a mobile terminal device (hereinafter referred to as "app"). The information processing performed by the control unit 13 is reflected as processing in the app. The operator launches the app and presses the start button in the app. At this time, the start button in the app changes to an end button. If the mobile terminal device is equipped with an audio input unit such as a microphone, instead of pressing the start button, the operator may verbally indicate "start". If the mobile terminal device is equipped with lighting, the lighting may be turned on when the start button is pressed. Whether or not the lighting is turned on may be determined by the surrounding environment (brightness), the brightness or contrast of the captured image. If the mobile terminal device is equipped with an audio output unit such as a speaker, it may output the voice message "Recognition processing has started" when the start button is pressed. The worker moves the camera of the mobile terminal device to a position where it can photograph the medications brought by the worker. As shown in Figure 2, a marker is displayed on the recognized medication, and the type and number of recognized medications are displayed as the recognition result. The displayed type and number of medications may be the cumulative recognition result since the start button of the application was pressed. In other words, if a new medication is photographed by the worker moving the mobile terminal device and changing the camera's shooting range, the recognition result of the new medication is added to the displayed type and number of medications. In the example in Figure 2, the recognition results are displayed in a table format, but are not limited to this. The control unit 13 can attach markers using known augmented reality (AR) techniques, etc. The marker is an image indicating that it has been recognized, and may have a different color, shape, or size for each type of medication. For example, if the color or shape of the marker differs for each medication, it is easier for the worker to recognize that they are different medications. If the size of the marker is changed for each type of drug, a marker appropriate to the size of the drug can be used.For example, if a ring-shaped marker of a size corresponding to the size of the drug is used, the drug becomes easier to see through the opening of the ring. Also, in this embodiment, a marker may be attached to each individual drug tablet, or one marker may be attached to multiple drugs. For example, if multiple drugs of the same type are placed adjacent to each other or clustered in one place, one marker may be attached to all of those drugs of the same type. In this case, the size or shape of the marker may be such that it is clear that it represents the multiple drugs in question. If there are drugs that cannot be recognized among the drugs brought in and photographed by the camera of the portable terminal device, a marker may not be attached to those drugs, or a marker indicating that they could not be recognized may be attached. The operator can treat the drug as recognized by touching the image of the unrecognized drug displayed on the output unit 15. If the operator can visually recognize the type of drug, they can add it to the recognition result using an arbitrary input means (not shown). Specifically, the operator touches the unrecognized drug displayed on the output unit 15 to specify it. If the type of drug in question is one included in the recognition results table (for example, drug A), and the operator touches the corresponding location in the recognition results table (the location of drug A), the control unit 13 updates the number of drugs in the recognition results. If the type of drug in question is one not included in the recognition results table (i.e., an unrecognized type of drug, for example, drug Z), the operator updates the recognition results by adding information about the new drug to the recognition results using any input means. The control unit 13 places a marker on the image of the touched drug to indicate that it has been recognized. Thus, the marker placed based on the operator's visual recognition results and the marker placed based on the results recognized by the information processing system may be displayed in a manner that allows them to be distinguished. The operator checks the display and moves the camera of the portable terminal device to a position where it can photograph the next drug brought in. The operator repeats checking the display and moving the camera of the portable terminal device to a position where it can photograph the next drug brought in until recognition of all drugs is complete.The operator presses the exit button in the app once all medications have been recognized.

[0030] Here, it is important to perform a double-check process in the drug recognition process to confirm that there are no errors in the type and quantity of drugs. By performing the double-check process, the operator can detect the correct type and quantity of drugs and immediately notice any errors. First, the number of drugs counted by the above process (i.e., the process of recognizing the drugs for the first time) using the information processing system according to this embodiment is stored in the storage unit 12. The operator may also store the results of visually confirming the drugs in the storage unit 12 using any input device. The control unit 13 outputs the number of drugs stored in the storage unit 12 as the result of the first recognition and the number of drugs counted from the new (i.e., second) recognition result to the output unit 15. Furthermore, if the number of newly counted drugs matches the number of drugs stored in the storage unit 12, the control unit 13 outputs a display indicating a match (for example, "match") to the output unit 15.

[0031] Referring to Figures 2 and 3, the double-check process performed by the control unit 13, including operator operations, will be explained. In the double-check process, the table in Figure 3 (displaying the double-check results) is used instead of the table in Figure 2 (displaying the types and numbers of recognized drugs).

[0032] The worker arranges the medications brought by the patient to be double-checked on the table. The medications may be arranged in multiple rows, both horizontally and vertically, not just in a single row. The arrangement of the medications may be the same as or different from the arrangement used during the first medication recognition process. The worker launches the app, enters an identification number that can identify the patient, and presses the "Start Double Check" button. At this point, the "Start Double Check" button in the app changes to an "End" button. If the mobile device is equipped with a microphone or other voice input, it may accept a verbal utterance of "Start Double Check" from the worker instead of pressing the "Start" button. If the mobile device is equipped with lighting, the lighting may be turned on when the "Start Double Check" button is pressed. The lighting during the double check may be illuminated in a different manner than during the first medication recognition process. For example, it may be illuminated with different brightness, a different color, or a different frequency than during the first medication recognition process. Performing a double check under different lighting conditions may improve the reliability of the recognition results. The control unit 13 reads the type and number of drugs stored in the memory unit 12 as past (i.e., first) recognition results (registered number in Figure 3), and displays the drug name and registered number as shown in the first and second columns from the left in Figure 3. The operator moves the camera of the portable terminal device to photograph all the drugs lined up. The image captured by the camera, with markers placed on the recognized drugs, and the number of recognized drugs are displayed in the output unit 15. The number of drugs recognized in the double check is displayed in the "Double Check Count" column of the table in Figure 3. If the registered number and the double check count match for the recognized drugs, the control unit 13 displays "Match" as the confirmation result. If the registered number and the double check count do not match for the recognized drugs, the control unit 13 may display "Mismatch" as the confirmation result. During the double-check, if a type of medication not included in the first recognition result is recognized, the medication may be added to the "Medication Name" column in the table in Figure 3, the "Registered Quantity" column may be set to zero, and the number of recognized medications may be displayed in the "Double-Check Quantity" column. The worker checks the display and moves the camera of the mobile terminal device to a position where it can photograph the next medication brought in.The operator repeatedly checks the display and moves the camera of the mobile terminal device to a position where it can photograph the next medication brought in, until all medications have been recognized. Once all medications have been recognized, the operator presses the double-check completion button. The control unit 13 may output (display) to the output unit 15 indicating that the registered number and the double-checked number do not match for the recognized medications, when the operator inputs the termination of the application.

[0033] Furthermore, during the drug recognition process, medications may be brought in already packaged in single doses. Single-dose packaging means that all medications a patient takes in a single dose are placed in one bag. This single-dose packaging can be performed by a pharmacist or a single-dose packaging device. Figure 4 illustrates how a single-dose packaged medication is photographed using a portable terminal device. The same drug recognition process is performed for single-dose packaged medications as for PTP packaging sheets (Figure 2). While Figure 4 shows multiple single-dose packaged medication bags connected together, the bags may be arranged separately. To allow the portable terminal device's camera to photograph the single-dose packaged medication through the bag, the bags used for single-dose packaging should be highly transparent and not reflect light easily.

[0034] The worker arranges their medications on the table. The medications may be arranged in multiple rows, both horizontally and vertically, not just in a single row. The worker launches the app and presses the start button. At this point, the start button in the app changes to an end button. The worker moves the camera of the mobile device to a position where they can photograph the medications. As shown in Figure 4, a marker appears on the recognized medication, and the type and number of recognized medications are displayed. The worker checks the display and moves the camera of the mobile device to a position where they can photograph the next medication. The worker repeats checking the display and moving the camera of the mobile device to a position where they can photograph the next medication until all medications have been recognized. Once all medications have been recognized, the worker presses the end button in the app.

[0035] Here, for the packaged drug, another table (FIG. 5 or FIG. 6) for checking whether there is a change in the content of the drug may be displayed. The other table may be displayed on the output unit 15 together with the table of FIG. 4 or instead of the table of FIG. 4 by the same operation as the process for recognizing the above drug.

[0036] FIG. 5 is an example of a table showing the total number of each drug. The number of tablets in each package is shown, and the cumulative number (total number) of each photographed drug is shown. The number of tablets in a package is the number of each drug contained in the first package (the package arranged on the leftmost side in the example of FIG. 4). Also, since the number of packages of the packaged drug is also shown, the operator can efficiently check whether the total number is the product of the number of tablets in a package and the number of packages to check whether there is a change in the content of the packaged drug.

[0037] FIG. 6 is an example of a table for detecting a difference in the number of tablets in a package. Based on the number of tablets in the first package (the package arranged on the leftmost side in the example of FIG. 4), the number of each drug in the package being checked is shown. Similar to the above double-check process, when the number of all drugs matches, the control unit 13 displays "match" as the confirmation result. Also, when the number of at least one drug does not match, the control unit 13 may display "mismatch" as the confirmation result.

[0038] Also, for the packaged drug, a double-check process is performed in the same manner as in the case of the PTP packaging sheet (FIG. 3).

[0039] The worker arranges the medications brought by the patient on the table. The medications may be arranged in multiple rows, both horizontally and vertically, not just in a single row. The worker launches the app, enters an identification number that can identify the patient, and presses the double-check start button. At this point, the double-check start button in the app changes to an end button. The control unit 13 reads the number of medications stored in the memory unit 12 as past recognition results (registered number in Figure 3) and displays the medication name and registered number as shown in the first and second columns from the left in Figure 3. The worker moves the camera of the mobile terminal device to a position where they can photograph the medications brought by the patient. A marker appears above the recognized medication, and the number of recognized medications is displayed (double-checked number in Figure 3). If the registered number and the double-checked number of the recognized medications match, the control unit 13 displays "Match" as the confirmation result. If the registered number and the double-checked number of the recognized medications do not match, the control unit 13 may display "Mismatch" as the confirmation result. The worker checks the display and moves the camera of the mobile terminal device to a position where they can photograph the next medication brought by the patient. The worker checks the display and moves the camera of the mobile device to a position where it can photograph the next medication until all medications have been recognized. Once all medications have been recognized, the worker presses the exit button in the app.

[0040] Here, it is important to ensure that there is no double counting of medications during the medication recognition process. Double counting refers to the miscounting of the same medication as a different medication when it is photographed multiple times between pressing the start button (or double-check start button) and pressing the end button (or double-check end button). If the same medication is double-counted, the number of medications included in the patient's medication cannot be counted correctly. For example, when moving the portable terminal device in Figure 2, the operator may accidentally move it in the wrong direction (from right to left), move it to reconfirm whether there are any missed recognitions, or move it to start the recognition process again because the operator has forgotten how far they have recognized the medications. In this case, it is possible to acquire an image containing medications that have already been counted. In conventional technology, recognition was performed from a single image, so countermeasures against such double counting were not necessary. In this embodiment, in order to prevent double counting, the control unit 13 identifies and stores the position of the medications in the area photographed by the camera of the portable terminal device between pressing the start button and pressing the end button. Here, one of the smallest units of medication recognized by the control unit 13 is referred to as the first medication. Furthermore, another smallest unit of the drug recognized by the control unit 13 will be referred to as the second drug, the third drug, ... the nth drug. In this embodiment, the smallest unit of the drug is one tablet.

[0041] Figure 7 is a diagram illustrating double-count prevention. The information processing system processes multiple images, with time (t) being t １ At this point, the first image was taken, and the time was t ２ At this point, take the second image. ２ t １ It is a later time. １ From that point on t ２ By this point, the camera's shooting range has changed. The control unit 13 controls t １ After recognizing the first drug contained in the first image taken at that point, ２When the second image taken at that time includes the first drug, the first drug included in the first image and the first drug included in the second image are determined to be the same drug. That is, when the second image taken after the first image includes the first drug that was included in the first image, the control unit 13 determines that the first drug included in the first image and the first drug included in the second image are the same drug. Also, the control unit 13 counts the number of recognized drugs. However, after recognizing and counting the number of the first drugs included in the first image, the control unit 13 does not count the number of the first drugs included in the second image as a new drug. On the other hand, the information processing system is at time t １ at a time later than t ３ If the image taken at that time includes the nth drug that has not been recognized, it is counted as a newly recognized drug.

[0042] Such determination is made possible by the control unit 13 identifying the position of the first drug using world coordinates in AR (Augmented Reality) space. As shown in Figure 8, by managing the position of the first drug in world coordinates in AR space, the control unit 13 can correctly determine that it is the same first drug based on the coordinates, even if the first drug is included in multiple images, thereby preventing double counting. Here, AR space refers to a space that extends reality by overlaying digital information onto the real world using a mobile terminal device. In other words, it is a space that the control unit 13 virtually constructs within the shooting range based on the results captured by the camera. World coordinates refer to coordinates in the coordinate system of AR space. The control unit 13 converts and manages the three-dimensional coordinates indicating the position of the first drug in the real world into world coordinates in AR space. In the example in Figure 8, the coordinate system of AR space is shown as a three-dimensional uvw space. Such conversion and management can be achieved, for example, by using ARCORE®, Google's AR library. Alternatively, the control unit 13 may manage the position of the first drug using the coordinates of the real-world coordinate system and prevent double counting of the first drug in multiple images based on the orientation and movement of the imaging unit 14. The orientation and movement of the imaging unit 14 may be calculated from the attitude information and position information detected by the mobile terminal device, respectively. The control unit 13 identifies and manages the type of recognized drug and its position in the world coordinates of the AR space. This allows the control unit 13 to determine whether the drug in the real space captured by the camera has already been recognized, thus preventing double counting. Specifically, the control unit 13 identifies the location of the drug in the world coordinates of the AR space for the drug included in the image captured by the imaging unit 14. Then, it determines whether the drug at that location has already been recognized. If the drug has already been recognized, it is not counted to avoid double counting. On the other hand, if the drug has not yet been recognized, it is counted as a newly recognized drug, and the recognition result is updated.

[0043] In this case, the control unit 13 may recognize the already recognized first drug as a different drug. Specifically, for example, t １ At that point in time, the first drug, which was recognized as drug A, ２ At this point, the drug may be recognized as drug B. In such cases, the control unit 13 may attach a marker to the drug to indicate that the recognition result has changed. Alternatively, the control unit 13 may update the recognition result to the newly recognized result as the correct result. Here, in the case of a drug recognized by the operator's visual inspection, even if the information processing system recognizes it as a different drug, the visual recognition result may be treated as correct and the original recognition result may be maintained. The world coordinates in AR space and the coordinates in real space may be represented as two-dimensional coordinates.

[0044] Furthermore, the control unit 13 may, by managing the coordinates of the position of the first drug as described above, attach the same marker to the first drug included in the first image and the first drug included in the second image, and output multiple images including the first drug with the marker to the output unit 15. In other words, the control unit 13 may also indicate to the operator that the markers indicate that they are the same drug that has already been recognized. Here, when the control unit 13 recognizes the second drug included in the first image, it attaches a marker to the second drug included in the first image, and outputs multiple images including the first drug and the second drug with the marker to the output unit 15.

[0045] As described above, the information processing system according to this embodiment can appropriately recognize drugs contained in multiple images without double-counting them, thanks to the above configuration. Therefore, the effort of individually photographing each of the multiple drugs without duplication is eliminated, and work efficiency can be improved.

[0046] While embodiments of this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component or process can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated. Embodiments relating to this disclosure can also be realized as programs executed by a processor in the device or as storage media recording such programs. These should also be understood to be included within the scope of this disclosure.

[0047] 11 Communication unit 12 Memory unit 13 Control unit 14 Imaging unit 15 Output unit

Claims

1. An information processing system comprising: an imaging unit that captures multiple images; and a control unit that acquires the multiple images and recognizes a drug contained in the multiple images, wherein the control unit determines that the first drug contained in the first image and the first drug contained in the second image are the same drug if the second image captured after the first image contains the first drug that was contained in the first image.

2. The information processing system according to claim 1, wherein the control unit determines the position of the first drug using world coordinates in AR space or the orientation and movement amount of the imaging unit.

3. The information processing system according to claim 1 or 2, wherein the control unit counts the number of recognized drugs, and after recognizing and counting the number of first drugs included in the first image, does not count the number of first drugs included in the second image.

4. An information processing system according to any one of claims 1 to 3, comprising an output unit that outputs a plurality of images, wherein the control unit attaches the same marker to the first drug included in the first image and the first drug included in the second image, and causes the output unit to output the plurality of images including the first drug with the marker attached.

5. The information processing system according to claim 4, wherein when the control unit recognizes the second drug included in the first image, it attaches the marker to the second drug included in the first image and causes the output unit to output the plurality of images including the first drug and the second drug with the marker attached.

6. The information processing system according to claim 4 or 5, comprising a storage unit that stores the number of first drugs counted in a process by the control unit that recognizes drugs contained in the plurality of past images, wherein the control unit causes the output unit to output the counted number of first drugs and the number of first drugs stored in the storage unit.

7. The information processing system according to claim 6, wherein the control unit causes the output unit to output a display indicating that the counted number of first drugs matches the number of first drugs stored in the storage unit.

Images