Image processing device, image processing method, program, and drug identification device

Abstract

Provided are an image processing device, an image processing method, a program, and a drug identification device capable of contributing to an improvement in the identification accuracy and an improvement in the visibility of identification information in a captured image, for drugs having a shape that is rotationally symmetrical with respect to a longitudinal axis. The image processing device: acquires a drug group image created by imaging a drug group including one or more drugs; identifies a drug region including drug images, which are images of each drug, from the drug group image; identifies the longitudinal direction of each drug image represented in the drug region; and performs image processing for unfolding a cylinder with respect to each drug image in an unfolding direction orthogonal to the identified longitudinal direction, to generate an unfolded image.

Description

Image processing device, image processing method, program, and drug identification device

[0001] The present invention relates to an image processing device, an image processing method, a program, and a medicine identification device.

[0002] There is known a technology for identifying drugs based on photographed images of the drugs to be identified. For example, Patent Literature 1 describes a drug identification system in which a drug is photographed and a computer identifies the drug based on the obtained image data of the drug.

[0003] JP 2023-105994 A

[0004] There is a high demand for identifying drug types by photographing capsules. However, capsules have a cylindrical shape with a longitudinal and lateral direction, and have a rotationally symmetric shape with respect to the longitudinal axis. In photographed images, distortion is greater near one end or the other end of the lateral direction of the capsule compared to the center of the lateral direction of the capsule. When photographed images of the drug are used as input for drug identification, distortion occurring in the photographed images can be a factor in reducing the accuracy of drug type identification.

[0005] Even when an operator visually recognizes the identification information of a capsule in a captured image, it is difficult to see the identification information near one end and the other end of the capsule in the short-side direction. The identification information of a capsule may include characters indicating the type, model, quantity, etc. of the capsule. The characters may include symbols, figures, and symbols. The above-mentioned problem is not limited to capsules, but may also exist in cylindrical tablets on which identification information is attached along the short-side direction. Furthermore, the above-mentioned problem may exist in drugs on which identification information is attached along the long-side direction, such as drugs with identification information attached to the drug surface, such as drugs with identification information attached along the long-side direction that reaches the short-side end, and drugs whose identification information in the captured image is closer to the short-side end due to the orientation during photography. The drug surface is understood to be the side surface of a cylindrical shape.

[0006] The cylindrical tablet referred to here may be a tablet having a shape that includes a cylindrical shape in part. For example, the cylindrical tablet may be a shape that combines a spherical surface at both ends of the longitudinal direction with a cylinder. The bottom surface of the cylinder may be a circle or an ellipse.

[0007] Patent Document 1 describes a specific example of tablet discrimination, and states that discrimination of not only tablets but also capsules can be performed. However, the document does not describe or suggest the above-mentioned problem of distortion in a captured image of a capsule or the like having a rotationally symmetric shape with respect to a longitudinal axis, and does not specifically disclose discrimination of a capsule or the like having a rotationally symmetric shape.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, a program, and a drug identification device that can contribute to improving the identification accuracy and the visibility of identification information in captured images for drugs that have a shape that is rotationally symmetrical about a longitudinal axis.

[0009] The image processing device of the present disclosure includes one or more processors and one or more memories that store instructions to be executed by the one or more processors, and the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug area from the drug group image that contains a drug image, which is an image of each drug, identify the longitudinal direction of the drug image represented in the drug area, and perform image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0010] According to the image processing device of the present disclosure, the longitudinal direction of the medicine image is identified, and image processing is performed to unfold the medicine image into a cylinder in a direction perpendicular to the longitudinal direction, thereby reducing distortion of the identification information at the end of the unfolding direction and improving the identification accuracy for medicines that have a shape that is rotationally symmetrical about the longitudinal axis.

[0011] The image processing method of the present disclosure is an image processing method in which a computer acquires a drug group image generated by photographing a drug group containing one or more drugs, identifies a drug area containing a drug image, which is an image of each drug, from the drug group image, identifies the longitudinal direction of the drug image represented in the drug area, and performs image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0012] The program of the present disclosure is a program that enables a computer to perform the following functions: acquire a drug group image generated by photographing a drug group containing one or more drugs; identify a drug area from the drug group image that contains a drug image, which is an image of each drug; identify the longitudinal direction of the drug image represented in the drug area; and perform image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0013] The drug identification device of the present disclosure includes one or more processors and one or more memories that store instructions to be executed by the one or more processors, and the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug area from the drug group image that contains a drug image, which is an image of each drug, identify the longitudinal direction of the drug image represented in the drug area, and perform image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0014] According to the present invention, the longitudinal direction of the medicine image is identified, and image processing is performed to unfold the medicine image into a cylinder in a direction perpendicular to the longitudinal direction, thereby reducing distortion of the identification information at the end of the unfolding direction and improving the identification accuracy for medicines that have a rotationally symmetric shape with respect to the longitudinal axis.

[0015] FIG. 1 is a diagram illustrating an example of photographing a packaged medicine. FIG. 2 is a schematic diagram of a photographed image showing a specific example of the photographed image. FIG. 3 is a schematic diagram of rotation processing and unfolding processing for the photographed image. FIG. 4 is a schematic diagram of a display screen of a medicine identification result. FIG. 5 is a functional block diagram showing the electrical configuration of a smartphone. FIG. 6 is a functional block diagram showing the functional configuration of a medicine identification device according to an embodiment. FIG. 7 is a flowchart showing the procedure of a medicine identification method according to an embodiment. FIG. 8 is a schematic diagram showing an example of medicine detection. FIG. 9 is a schematic diagram showing another example of medicine detection. FIG. 10 is an explanatory diagram of a capsule and an oval tablet. FIG. 11 is a schematic diagram showing an implementation example of processing target determination. FIG. 12 is an explanatory diagram of rotation processing. FIG. 13 is a schematic diagram of image processing for unfolding a cylinder. FIG. 14 is an explanatory diagram of parameters applied to a capsule in the unfolding processing. FIG. 15 is an explanatory diagram of parameters used in calculations in the unfolding processing. FIG. 16 is an explanatory diagram showing a specific example of calculations applied to the unfolding processing.

[0016] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, identical components are designated by the same reference numerals, and duplicate descriptions will be omitted as appropriate. Furthermore, when multiple components are listed in the following embodiments, it can be interpreted as including at least one of the multiple components.

[0017] [Overview of the drug identification device according to the embodiment] The drug identification device according to the embodiment acquires at least one of a one-side image in which one side of a plurality of drugs is photographed and a other-side image in which the other side is photographed, and generates individual images of each of the plurality of drugs from the one-side image, etc.

[0018] The drug identification device is a device that identifies the drug type of a drug to be identified based on an individual image of the drug to be identified, which is at least one drug among a plurality of drugs, based on a score that indicates the likelihood that the individual images are images of the same drug.

[0019] The multiple medications to be photographed may be individually packaged medications packaged in individual sachets. Any of the multiple medications may be press-through packaged or strip packaged. Any of the multiple medications may be separated from a continuous sheet. Any of the multiple medications may be unpackaged. Note that PTP is an abbreviation for press-through pack.

[0020] In the following description, the marking refers to a groove, which is a recessed area formed on the surface of the medicine as identification information for the medicine. The groove is not limited to one formed by carving the medicine surface, but may be a groove formed by pressing the medicine surface.

[0021] Furthermore, printing refers to identification information of a drug formed by applying edible ink or the like to the drug surface of the drug. The application of edible ink or the like may be contact or non-contact. Note that printing and printing are synonymous. Figure 1 illustrates an arbitrary character string as identification information of the drug. The same applies to Figure 2 and subsequent figures.

[0022] The drug surface of the drug may have a score line without any distinguishing function. An example of a score line is a groove formed on the drug surface. The score line may have a length that reaches one end of the drug surface in the short direction and reaches the other end in the same direction. The score line may be used when dividing the tablet. The drug surface of the drug may be plain, with no markings or printing added.

[0023] The drug identification device is mounted on a portable terminal device, for example. The portable terminal device includes at least one of a mobile phone, a PHS, a smartphone, a PDA, a tablet computer terminal, a notebook personal computer terminal, and a portable game console. Note that PHS is an abbreviation for Personal Handyphone System, and PDA is an abbreviation for Personal Digital Assistant.

[0024] The drug identification device may be mounted on a fixed desktop computer, a workstation, etc., and may be configured in combination with a separate digital camera and display. Below, an example of a drug identification device to which a smartphone is applied will be given.

[0025] 1 is a diagram illustrating an example of photographing a single-dose medication. Using an external camera built into a smartphone 10, an operator photographs a sachet S containing a plurality of single-dose medications by orienting the optical axis of the external camera vertically downward. The sachet S is placed on a mounting table T with its surface F facing vertically upward. The photographed image is stored in the internal memory of the smartphone 10 and is used to identify each medication.

[0026] The operator may use a camera separate from the smartphone 10 to photograph the sachet S in which multiple medications are packaged, and transmit the photographed image to the smartphone 10. The smartphone 10 may store the photographed image transmitted from the camera in its internal memory and use it to identify each medication. The operator described in the embodiment is an example of an operator.

[0027] The mounting table T may be equipped with an illumination device that irradiates illumination light onto the medicine to be photographed. The illumination device may include an annular light source that surrounds the periphery of the sachet S. The illumination device may be built into the mounting table T. The illumination device may be configured to be detachable from the mounting table T.

[0028] The captured image may be generated by capturing an image of the single-dose drug and a marker. The number of markers may be one or more. Examples of markers include an ArUco marker, a circular marker, and a square marker. The captured image may be generated by capturing an image of the single-dose drug and a gray reference color.

[0029] The image may be captured using a standard shooting distance and shooting viewpoint. The shooting distance can be expressed using the distance from the packaged drug to the shooting lens and the focal length of the shooting lens. The shooting viewpoint can be expressed using the angle between the surface on which the marker is printed and the optical axis of the shooting lens.

[0030] When the captured image includes a marker, the captured image may be a standardized image in which the shooting distance and the shooting viewpoint are standardized based on the marker. When the captured image includes a gray area, color correction based on a reference gray color may be performed.

[0031] The multiple drugs contained in a single drug package are not limited to drugs of the same type, but may be drugs of different types. The sachet S may be transparent or translucent. An example of transparency is a state in which the light transmittance is 90% or more. An example of translucency is a state in which the light transmittance is 10% or more but less than 90%.

[0032] The captured image described in the embodiment is an example of a drug group image generated by capturing an image of a drug group containing one or more drugs.

[0033] [Specific example of photographed image] Figure 2 is a schematic diagram of a photographed image showing a specific example of a photographed image. The photographed image PI shown in the figure includes a plurality of individual images DI corresponding to a plurality of medications. Figure 2 also includes an enlarged image of an individual image DI1. Note that the individual image DI described in the embodiment is an example of a medication image.

[0034] The captured image PI is displayed on the touch panel display 14 of the smartphone 10 shown in Fig. 1. The touch panel display 14 may display auxiliary lines SL that serve as markers for coordinates applied to the captured image PI. Fig. 2 illustrates a plurality of auxiliary lines SL extending along two mutually orthogonal axis directions as auxiliary lines of a two-dimensional orthogonal coordinate system.

[0035] The photographed image PI illustrated in Fig. 2 includes individual images DI corresponding to a plurality of types of capsules and individual images DI corresponding to a plurality of types of oval tablets. The photographed image PI may include a round tablet.

[0036] 1 detects one or more individual images DI representing medications from a captured image PI. The smartphone 10 determines whether each individual image DI represents a capsule. Details of the detection of the individual images DI and the determination of capsules will be described later.

[0037] The smartphone 10 identifies the individual drug represented by the individual image DI based on the identification information TI attached to the drug surface of the individual image DI. The identification of the drug may be performed by specifying the drug type of the drug represented by the individual image DI. Note that in FIG. 2, the identification information TI of the individual images DI other than the individual image DI1 is omitted as appropriate.

[0038] 3 is a schematic diagram of the rotation process and the unfolding process for a captured image. The smartphone 10 shown in FIG. 1 detects a drug in the captured image PI and determines whether the detected drug is an individual image DI representing a capsule CA to be unfolded. The capsule CA described in the embodiment is an example of a drug to be unfolded.

[0039] The smartphone 10 performs image processing P1 to rotate an individual image DI representing a capsule CA having an elliptical planar shape in-plane, and then performs image processing P2 to unfold the individual image DI representing the capsule CA into a cylinder in an unfolding direction perpendicular to the longitudinal direction of the individual image DI, thereby generating an unfolded image EI. The smartphone 10 identifies the capsule CA represented by the individual image DI based on the unfolded image EI obtained by unfolding the individual image DI in the unfolding direction.

[0040] In this embodiment, the longitudinal direction of the individual image DI representing the capsule CA is the direction of the line segments connecting the vertices of the curved portions in the individual image DI corresponding to the spherical portions at both ends of the capsule CA. The unfolding direction of the individual image DI representing the capsule CA is the same as the lateral direction of the individual image DI representing the capsule CA.

[0041] The longitudinal direction of the individual image DI representing the capsule CA may intersect with the direction of a line segment connecting vertices of the curved portions in the individual image DI corresponding to each of the two spherical portions of the capsule CA. For example, the longitudinal direction of the individual image DI representing the capsule CA may be the direction of a line segment connecting arbitrary points on the curved portions in the individual image DI corresponding to each of the spherical portions at both ends of the capsule CA.

[0042] The unfolding direction of the individual image DI representing the capsule CA is not strictly perpendicular to the longitudinal direction of the individual image DI representing the capsule CA, but may be a direction substantially perpendicular to the longitudinal direction of the individual image DI representing the capsule CA, which can improve the visibility of the identification information in the unfolded image EI.

[0043] Fig. 4 is a schematic diagram of a display screen of the drug identification result. The captured image PI and the identification result IR are displayed on the touch panel display 14 of the smartphone 10. Fig. 4 illustrates an aspect in which the type of drug represented by the individual image DI specified using the bounding box BB is displayed. Note that the touch panel display 14 is an example of a display device.

[0044] 5 is a functional block diagram showing the electrical configuration of a smartphone 10. The smartphone 10 shown in the figure includes a touch panel display 14, a speaker 16, a microphone 18, an in-camera 20, an out-camera 22, a light 24, and a switch 26.

[0045] The smartphone 10 includes a CPU 28 , a wireless communication unit 30 , a call unit 32 , a memory 34 , an external input / output unit 40 , a GPS receiving unit 42 , and a power supply unit 44 .

[0046] The touch panel display 14 includes a display unit that displays images and the like, and a touch panel unit that is disposed in front of the display unit and accepts touch inputs. A color LCD panel is used for the display unit. Note that CPU is an abbreviation for Central Processing Unit, and LCD is an abbreviation for Liquid Crystal Display.

[0047] The touch panel unit is a capacitive touch panel that is disposed in a planar manner on a light-transmitting substrate body, and that has light-transmitting position detection electrodes and an insulating layer disposed on the position detection electrodes. The touch panel unit generates and outputs two-dimensional position coordinate information corresponding to touch operations by an operator. Examples of touch operations include tapping, double-tapping, flicking, swiping, dragging, pinching in, and pinching out.

[0048] The speaker 16 is an audio output unit that outputs audio during a call and when a video is played. The microphone 18 is an audio input unit that inputs audio during a call and when a video is being captured. The in-camera 20 is an imaging device that captures videos and still images. The out-camera 22 is an imaging device that captures videos and still images. The light 24 is a light source that emits illumination light when capturing images with the out-camera 22. An LED is used as the light 24. LED is an abbreviation for Light Emitting Diode.

[0049] The switch 26 is an input member that receives instructions from an operator. The switch 26 is a push-button switch that turns on when pressed with a finger or the like and turns off when the finger is released due to the restoring force of a spring or the like.

[0050] The CPU 28 is an example of a processor that executes instructions stored in the memory 34. The CPU 28 operates in accordance with the control program and control data stored in the memory 34, and controls all the components of the smartphone 10. The CPU 28 has a mobile communication control function that controls all the components of the communication system to perform voice communication and data communication via the wireless communication unit 30, and an application processing function.

[0051] The CPU 28 also has an image processing function for displaying moving images, still images, text, etc. on the touch panel display 14. This image processing function visually conveys information such as still images, moving images, and text to the operator. The CPU 28 also acquires two-dimensional position coordinate information corresponding to touch operations by the operator from the touch panel portion of the touch panel display 14. The CPU 28 also acquires input signals from the switches 26.

[0052] The hardware structure of the CPU 28 is made up of various processors. Examples of various processors include a CPU, which is a general-purpose processor that executes software and functions as various functional units, and a GPU, which is a processor specialized for image processing. GPU is an abbreviation for Graphics Processing Unit.

[0053] Other examples of various processors include PLDs, which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs, and dedicated electrical circuits, such as ASICs, which are processors with circuit configurations designed specifically to execute specific processes. FPGA is an abbreviation for Field Programmable Gate Array. PLD is an abbreviation for Programmable Logic Device. ASIC is an abbreviation for Application Specific Integrated Circuit.

[0054] A single processing unit may be configured with one of these various types of processors. A single processing unit may be configured with two or more processors of the same type or different types. For example, a single processing unit may be implemented with multiple FPGAs, or a combination of a CPU and an FPGA. A single processing unit may be implemented with a combination of a CPU and a GPU.

[0055] A plurality of functional units may be configured as a single processor. An example of a plurality of functional units configured as a single processor is a computer such as a client or server, where a single processor is configured by applying a combination of one or more CPUs and software, and the single processor operates as a plurality of functional units.

[0056] An example of a system in which multiple functional units are configured as a single processor is a processor that realizes the functions of an entire system including multiple functional units on a single IC chip, as typified by SoC. In this way, the various functional units are configured as a hardware structure using one or more of the above-mentioned various processors. Note that SoC is an abbreviation for System On Chip, and IC is an abbreviation for Integrated Circuit.

[0057] Furthermore, the hardware structure of various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements. Note that an electric circuit may also be called a circuitry.

[0058] The in-camera 20 and the out-camera 22 capture videos and still images in accordance with instructions from the CPU 28. The in-camera 20 and the out-camera 22 have the same internal configuration. Each of the in-camera 20 and the out-camera 22 has a photographing lens, an image sensor, and an image processing unit. Note that the photographing lens, image sensor, and image processing unit are not shown in the figures.

[0059] The imaging element may be a CMOS-type photoelectric conversion element, or may be a CCD or other photoelectric conversion element. The imaging element is provided with an RGB color filter on its light receiving surface. CMOS is an abbreviation for Complementary Metal-Oxide Semiconductor. CCD is an abbreviation for Charge-Coupled Device. In the RGB color filter, R represents red, G represents green, and B represents blue.

[0060] Each of the in-camera 20 and the out-camera 22 receives subject light via a photographing lens and uses an image sensor to receive the subject light. The subject light is imaged on the light-receiving surface of the image sensor. The image sensor converts the subject light imaged on the light-receiving surface into an electrical signal based on the R, G, and B color signals. The image processing unit performs specified processing on the analog image signal output from the image sensor and converts the analog image signal into a digital image signal.

[0061] Each of the in-camera 20 and the out-camera 22 may convert moving image data into compressed moving image data in MPEG format or the like. Each of the in-camera 20 and the out-camera 22 may convert still image data into compressed still image data in JPEG format. MPEG is an abbreviation for Moving Picture Experts Group. JPEG is an abbreviation for Joint Photographic Experts Group.

[0062] The CPU 28 stores the video and still images captured using the in-camera 20 and the out-camera 22 in the memory 34. The CPU 28 may also output the video and still images captured by the in-camera 20 and the out-camera 22 to the outside of the smartphone 10 via the wireless communication unit 30 or the external input / output unit 40.

[0063] Furthermore, the CPU 28 displays the video and still images captured using the in-camera 20 and the out-camera 22 on the touch panel display 14. The CPU 28 may use the video and still images captured using the in-camera 20 and the out-camera 22 in application software.

[0064] The CPU 28 may turn on the light 24 to irradiate the subject with fill light when capturing an image with the outer camera 22. The light 24 may be turned on and off in response to an operator's touch operation on the touch panel display 14 or an operation of the switch 26.

[0065] The wireless communication unit 30 performs wireless communication with a base station device included in the mobile communication network in accordance with instructions from the CPU 28. The smartphone 10 uses wireless communication to send and receive various file data such as audio data and image data, email data, etc., and to receive web data, streaming data, etc. Note that "web" is an abbreviation for World Wide Web.

[0066] The speaker 16 and the microphone 18 are connected to the communication unit 32. The communication unit 32 decodes voice data received using the wireless communication unit 30 and outputs the decoded data from the speaker 16. The communication unit 32 converts the voice of the operator input through the microphone 18 into voice data that can be processed by the CPU 28 and outputs the voice data to the CPU 28.

[0067] The memory 34 stores instructions to be executed by the CPU 28. The memory 34 may be an internal storage unit 36 ​​built into the smartphone 10, or an external storage unit 38 that is detachable from the smartphone 10. The internal storage unit 36 ​​and the external storage unit 38 are realized using known storage media. The external storage unit 38 may be a non-transitory computer-readable storage medium.

[0068] The memory 34 stores the control program of the CPU 28, control data, application software, address data associated with names and telephone numbers of communication partners, data of emails sent and received, web data downloaded by web browsing, downloaded content data, etc. The memory 34 may also temporarily store streaming data, etc.

[0069] The external input / output unit 40 serves as an interface with external devices connected to the smartphone 10. The smartphone 10 is directly or indirectly connected to other external devices by communication or the like via the external input / output unit 40. The external input / output unit 40 transmits data received from external devices to each component within the smartphone 10, and also transmits data within the smartphone 10 to external devices.

[0070] Examples of communication include Universal Serial Bus, IEEE 1394, the Internet, wireless LAN, Bluetooth (registered trademark), RFID, and infrared communication. Universal Serial Bus may be referred to as USB (registered trademark), which is an abbreviation of Universal Serial Bus in English. IEEE is an abbreviation for Institute of Electrical and Electronics Engineers. LAN is an abbreviation for Local Area Network. RFID is an abbreviation for Radio Frequency Identification.

[0071] Examples of external devices include headsets, external chargers, data ports, audio devices, video devices, smartphones, PDAs, personal computers, and earphones.

[0072] The GPS receiver 42 detects the position of the smartphone 10 based on positioning information transmitted from n GPS satellites such as GPS satellite ST1, GPS satellite ST2, and GPS satellite STn.

[0073] The power supply unit 44 is a power supply source that supplies power to each component of the smartphone 10 via a power supply circuit. The power supply unit 44 may include a lithium-ion secondary battery. The power supply unit 44 may include an analog-to-digital conversion circuit that generates a DC voltage from an external AC power source. Note that the electrical circuitry that constitutes the power supply unit 44 is not shown in the figure.

[0074] The smartphone 10 is set to a photography mode in response to an instruction input from an operator using the touch panel display 14 or the like, and captures videos and still images using the in-camera 20 and the out-camera 22 .

[0075] When the smartphone 10 is set to the photographing mode, it enters a photographing standby state and captures a through image using the in-camera 20 or the out-camera 22. The captured through image is displayed on the touch panel display 14 as a live view image.

[0076] The operator can visually check the live view image displayed on the touch panel display 14 to determine the composition, confirm the subject to be photographed, and set the photographing conditions.

[0077] When the smartphone 10 is in a standby state for shooting and receives a shooting instruction in response to an instruction input by an operator operating the touch panel display 14 or the like, the smartphone 10 performs AF control and AE control to capture and store video and still images. Note that AF is an abbreviation for Autofocus, and AE is an abbreviation for Auto Exposure.

[0078] [Functional configuration of drug identification device] Fig. 6 is a functional block diagram showing the functional configuration of the drug identification device according to the embodiment. In the drug identification device 100 shown in Fig. 6, the CPU 28 shown in Fig. 5 executes a program stored in the memory 34 to realize various functions corresponding to various processing units shown in Fig. 6.

[0079] The capsule CA has rotational symmetry with respect to a central axis parallel to the longitudinal direction, and the ends of the individual image DI in the lateral direction are distorted more than the central part in the lateral direction. As a result, when detecting the identification information attached to the medicine surface of the capsule CA, there is a concern that the detection accuracy of the identification information may be reduced due to the distortion in the individual image DI.

[0080] On the other hand, tablets have lower rotational symmetry about a central axis parallel to the longitudinal direction than capsules CA, and it is considered that the possibility of a decrease in the detection accuracy of identification information attached to the drug surface of a tablet is lower than that of capsules CA.

[0081] The drug identification device 100 of the embodiment determines whether the individual image DI detected from the captured image PI represents a capsule CA, and performs image processing to unfold a cylinder on the individual image DI representing the capsule CA.

[0082] This prevents a decrease in visibility of the identification information attached to the medicine surface of the capsule CA at the ends of the shorter side of the capsule CA, and prevents a decrease in detection accuracy of the identification information. The decrease in detection accuracy here may include a decrease in detection accuracy in image processing and a decrease in detection accuracy when an operator visually recognizes the identification information.

[0083] The drug identification device 100 includes a photographed image acquisition unit 102. The photographed image acquisition unit 102 acquires a photographed image PI shown in Fig. 2. The photographed image acquisition unit 102 may include a part or all of a processing unit that generates a photographed image. That is, acquisition of a photographed image may include acquiring raw data in which optical information of a subject is converted into an electrical signal, and generating a photographed image from the acquired raw data.

[0084] The drug identification device 100 includes a drug detection unit 104. The drug detection unit 104 detects individual drugs from the photographed image PI acquired using the photographed image acquisition unit 102, and generates individual images DI representing the individual drugs. The drug detection unit 104 stores the individual images DI in a specified memory.

[0085] The drug detection unit 104 may apply a specified algorithm to detect individual images DI from the captured image PI. A trained learning model may be implemented in the drug detection unit 104. The trained learning model may be trained so that when a captured image PI is input, one or more individual images DI are output. For example, the drug detection unit 104 may attach a bounding box surrounding each drug to the captured image PI to identify an area that will become an individual image DI.

[0086] The individual image DI described in the embodiment is an example of a medicine area including a medicine image, and the medicine represented by the individual image DI described in the embodiment is an example of a medicine image.

[0087] The drug identification device 100 includes a processing target determination unit 106. The processing target determination unit 106 determines whether the drug represented by the individual image DI is a capsule CA. The processing target determination unit 106 may assign additional information, such as the probability that the individual image DI is a capsule CA, to the individual image DI representing a capsule CA. The processing target determination unit 106 may assign additional information, such as the probability that the individual image DI is a tablet, to the individual image DI representing a tablet.

[0088] The drug identification device 100 includes a rotation processing unit 108. The rotation processing unit 108 performs image processing P1, shown in FIG. 3, to perform in-plane rotation on the individual image DI. That is, the rotation processing unit 108 identifies the longitudinal direction of the capsule CA in the individual image DI, and rotates the individual image DI so that the longitudinal direction of the capsule CA is parallel to the axes of a two-dimensional coordinate system applied to the individual image DI. Note that the longitudinal direction of the capsule CA is synonymous with the longitudinal direction of the individual image DI.

[0089] Here, the term "parallel" may include the meaning of two directions that intersect strictly but can be considered to be parallel, and the term "orthogonal" may include the meaning of two directions that are not orthogonal strictly but can be considered to be orthogonal, but can be considered to be orthogonal.

[0090] The drug identification device 100 includes an unfolding processing unit 110. The unfolding processing unit 110 performs image processing P2 to unfold the individual image DI into a cylinder in the unfolding direction, which is the short-side direction of the capsule CA shown in Fig. 3. That is, the unfolding processing unit 110 unfolds the individual image DI in the unfolding direction to generate an unfolded image EI.

[0091] The drug identification device 100 includes a candidate identification unit 112. The candidate identification unit 112 refers to the drug database 114 and identifies a drug candidate represented by the individual image DI to be identified, using the identification information included in the expanded image EI and the form of the individual image DI, etc. The form of the individual image DI may include the shape, color, size, etc. of the individual image DI. The candidate identification unit 112 may be implemented with a trained learning model that realizes the function of identifying drug candidates.

[0092] The drug identification device 100 includes a display screen generation unit 116. The display screen generation unit 116 generates a display screen that displays drug candidates represented by the individual images DI of the identification targets identified by the candidate identification unit 112. The display screen generation unit 116 transmits a display signal representing a display screen including the identification result IR shown in Fig. 4 to the touch panel display 14, and causes the touch panel display 14 to display the display screen including the identification result IR.

[0093] The candidate specifying unit 112 may be removed from the drug identification device 100 to configure an observation support device that supports an operator in observing photographed images of drugs. A display screen generating unit 116 in the observation support device generates a display screen for the unfolded image EI. A touch panel display 14 displays the unfolded image EI. The observation support device aims to improve the visibility of drugs when an operator observes photographed images of drugs.

[0094] [Procedure of Drug Identification Processing Method] Fig. 7 is a flowchart showing the procedure of the drug identification method according to the embodiment. In the photographed image acquisition step S10, the photographed image acquisition unit 102 shown in Fig. 6 acquires the photographed image PI shown in Fig. 2. Once the photographed image PI is acquired in the photographed image acquisition step S10, the process proceeds to the drug detection step S12.

[0095] In the drug detection step S12, the drug detection unit 104 detects individual images DI representing individual drugs from the captured image PI. In the drug detection step S12, the individual images DI are stored in a specified memory. When an individual image DI is detected in the drug detection step S12, the process proceeds to a processing target determination step S14. After the drug detection step S12, it may be determined whether individual images DI for all drugs have been detected.

[0096] In the processing object determination step S14, the processing object determination unit 106 determines whether each individual image DI represents a capsule CA. In the processing object determination step S14, each individual image DI may be assigned additional information representing the determination result.

[0097] In the processing object determination step S14, if it is determined that the individual image DI represents a tablet, the result is No. If the result is No, the process proceeds to the candidate specification step S20. On the other hand, in the processing object determination step S14, if it is determined that the individual image DI represents a capsule CA, the result is Yes. If the result is Yes, the process proceeds to the rotation processing step S16.

[0098] In the rotation processing step S16, the rotation processing unit 108 performs image processing to rotate the individual image DI in-plane. That is, in the rotation processing step S16, the longitudinal direction of the capsule CA represented by the individual image DI is identified, and the individual image DI is rotated in a direction such that the longitudinal direction of the capsule CA is parallel to the axes of a two-dimensional orthogonal coordinate system applied to the image processing. After the process of rotating the individual image DI in the rotation processing step S16 is performed, the process proceeds to the unfolding processing step S18.

[0099] The rotation processing step S16 may include a rotation determination step of determining whether the individual image DI is a rotated rectangle or a non-rotated rectangle. The rotation determination step may be executed before the rotation processing step S16 as a separate step from the rotation processing step S16.

[0100] In the unfolding process step S18, the unfolding processing unit 110 performs image processing to unfold a cylinder on the individual image DI rotated in a specified direction. The individual image DI may be trimmed to fit the planar shape of the capsule CA. After the image processing to unfold a cylinder on the individual image DI is performed in the unfolding process step S18, the process proceeds to a candidate identification step S20.

[0101] The rotation process S16 and the unfolding process S18 are processes performed on each individual image DI determined to represent a capsule CA in the processing target determination process S14. If the rotation angle of the capsule CA is specified, the rotation process S16 may be omitted. Alternatively, the rotation process S16 and the unfolding process S18 may be performed as a single process.

[0102] In the candidate identification step S20, the candidate identification unit 112 refers to the drug database 114 to identify one or more candidate drugs for all individual images DI detected in the drug detection step S12. When one or more candidate drugs have been identified for all individual images DI in the candidate identification step S20, the process proceeds to a display screen generation step S22.

[0103] In the display screen generating step S22, the display screen generating unit 116 generates a display screen including the classification result IR shown in Fig. 4. An example of the display screen is a mode including the captured image PI and classification result IR shown in Fig. 4. Once the display screen is generated in the display screen generating step S22, the process proceeds to a display signal output step S24.

[0104] In the display signal output step S24, the display screen generation unit 116 transmits a display signal representing the display screen to the touch panel display 14. That is, in the display signal output step S24, the classification result IR for each individual image DI is displayed on the touch panel display 14.

[0105] In the display signal output process S24, the recognition result IR may be displayed for each individual image DI. In the display signal output process S24, the recognition results IR for multiple individual images DI may be displayed. An example of displaying the recognition results IR for multiple individual images DI is a display mode in the form of a list. When the recognition result IR for any individual image DI is displayed in the display signal output process S24, the process proceeds to the termination determination process S26. In the termination determination process S26, the drug identification device 100 determines whether or not the recognition process has been performed for all captured images. If it is determined in the termination determination process S26 that the recognition process has not been performed for all captured images, the determination is No. If the determination is No, the process proceeds to the captured image acquisition process S10, and each of the steps from the captured image acquisition process S10 to the termination determination process S26 is repeatedly executed until the determination is Yes in the termination determination process S26.

[0106] On the other hand, in the termination determination step S26, if it is determined that the identification process has been performed on all of the captured images, the determination is Yes. If the determination is Yes, a specified termination process is executed, and the procedure of the medicine identification method is terminated.

[0107] 7 may include processing based on a signal input to the drug identification device by an operator operating an input device. For example, in the photographed image acquisition step S10, an operator viewing a screen displaying multiple photographed images may operate an input device such as a keyboard or mouse to select a photographed image to be processed from the multiple photographed images. The photographed image acquisition unit 102 may acquire an input signal transmitted from the input device and acquire the photographed image to be processed.

[0108] In addition, in the drug detection step S12, an operator viewing the screen on which the photographed image PI is displayed may operate an input device to select an area from the photographed image PI to become an individual image DI. That is, the drug detection unit 104 illustrated in FIG. 6 may acquire an input signal transmitted from the input device and detect an individual image DI in the photographed image PI.

[0109] Each step shown in Figure 7 may acquire information transmitted from an external device of the computer functioning as the drug identification device to realize various functions. For example, the drug detection step S12 may acquire the detection result of the area containing the drug transmitted from a drug detection device that inputs a photographed image and applies a specified algorithm to the photographed image to detect the area containing the drug. The drug identification method whose steps are illustrated in Figure 7 is an example of an image processing method.

[0110] 7 may omit the candidate identification step S20, generate a display screen for displaying the unfolded image EI in a display screen generation step S22, and output a display signal representing the display screen of the unfolded image EI in a display signal output step S24. That is, the drug identification method may be configured as an observation support method whose purpose is to improve the visibility of drugs when an operator observes a photographed image DI of the drug.

[0111] [Specific Example of Drug Detection Processing] Figure 8 is a schematic diagram illustrating an example of drug detection. The figure schematically illustrates an example in which an individual image DI is detected as an unrotated rectangle. The drug detection unit 104 illustrated in Figure 6 may be implemented with a trained learning model. The trained learning model may be generated by performing learning to detect an individual image DI representing a capsule CA, specifying the coordinates of a bounding box BB that circumscribes and surrounds the area of ​​the capsule CA illustrated with the symbol 8A.

[0112] 8 , in the individual image DI representing the capsule CA indicated by the reference symbol 8B, a rectangle RR is defined as a rotated rectangle that circumscribes the area of ​​the capsule CA and whose shorter edges are in contact with the area of ​​the capsule CA, and a bounding box BB is defined as an unrotated rectangle that circumscribes the rectangle RR. The trained learning model may be generated by performing learning that specifies the coordinates of the bounding box BB indicated by the reference symbol 8B.

[0113] The drug detection unit 104 may specify, as the coordinates of the bounding box BB, the coordinates of the vertices of a rectangle representing the bounding box BB. For example, a combination of the coordinates of two vertices connected by a diagonal line, such as the combination of the upper left vertex V1 and the lower right vertex V2 in FIG. 8, may be specified.

[0114] FIG. 9 is a schematic diagram illustrating another example of drug detection. This diagram schematically illustrates an example in which an individual image DI is detected as a rotated rectangle. The drug detection unit 104 illustrated in FIG. 6 may be implemented with a trained learning model. The trained learning model may be generated by performing learning that specifies the coordinates of the center point CO, the width BH, the height BV, and the rotation angle Bθ of the capsule CA included in the individual image DI illustrated in FIG. 9.

[0115] The drug detection unit 104 may specify the coordinates of the center point CO of the bounding box BB, the width BH of the bounding box BB, the height BV of the bounding box BB, and the rotation angle Bθ.

[0116] The drug detection unit 104 may be implemented using an algorithm. For example, the drug detection unit 104 may apply OpenCV to perform edge detection and rectangle detection. OpenCV is an abbreviation for Open Source Computer Vision Library.

[0117] The rectangle detection may be non-rotated rectangle detection as shown in Fig. 8 or rotated rectangle detection as shown in Fig. 9. The individual image DI shown in Fig. 8 is an example of a drug area surrounded by a figure circumscribing the drug image. The individual image DI shown in Fig. 9 is an example of a drug area surrounding the edge of the drug image.

[0118] [Specific example of processing target determination] Fig. 10 is an explanatory diagram of a capsule and an oval tablet. Individual images DI11, DI12, and DI13 shown in Fig. 10 are individual images DI representing capsules CA. The individual images DI11, DI12, and DI13 representing capsules CA have a left half larger than a right half in the longitudinal direction in Fig. 10. That is, individual images DI11 and the like have a midline at the boundary between the right and left halves.

[0119] The processing target determination unit 106 shown in Fig. 6 may detect an individual image DI representing a capsule CA based on whether or not there is a difference in size between the left and right halves in the processing target determination step S14 shown in Fig. 7. The processing target determination unit 106 may also detect an individual image DI representing a capsule CA based on the presence or absence of a midline between the left and right halves.

[0120] The individual images DI12 and DI13 have different colors in the left and right halves, and the processing target determination unit 106 may detect the individual image DI representing the capsule CA based on the difference in color between the left and right halves.

[0121] The individual image DI may include a background region BA, as in the individual images DI11 and DI13. The individual image DI may not include a background region BA, as in the individual image DI12. That is, the individual image DI may be detected as a rectangular region surrounding the drug in the captured image PI, or may be detected as a region having a shape similar to the outline of the drug.

[0122] The individual images DI21 and DI22 shown in Fig. 10 are individual images DI representing tablets. The individual images DI21 and DI22 representing tablets have a front side surface FS and a back side surface BS.

[0123] The individual image DI21 has an inscription on the front side FS, and the back side BS is a plain surface without any inscription etc. The individual image DI22 has the same information printed on both the front side FS and the back side BS.

[0124] The individual image DI21 etc. representing the tablet may be detected as either the front side surface FS or the back side surface BS. Here, the front side surface of the tablet means one side of the tablet, and the back side surface of the tablet means the other side of the tablet.

[0125] The processing target determination unit 106 may be implemented with a trained learning model. The trained learning model may be trained and generated so that, when an individual image DI is input, it outputs information indicating whether the individual image DI is a capsule CA. The information indicating whether the individual image DI is a capsule CA may be a probability that the individual image is a capsule CA.

[0126] A specified algorithm may be applied to the processing object determination unit 106. The processing object determination unit 106 may detect the right half, the left half, and the midline, and determine that the individual image DI represents a capsule CA when the midline is detected.

[0127] The processing target determination unit 106 may detect a difference in size between the right half and the left half, and if the size of the left half is smaller than the size of the right half, determine that the individual image DI represents a capsule CA. The processing target determination unit 106 may detect a difference in size between the right half and the left half, and if the size of the left half is larger than the size of the right half, determine that the individual image DI represents a capsule CA.

[0128] The processing object determination unit 106 may detect a difference between the color of the right half and the color of the left half, and if the color of the right half and the color of the left half are different, may determine that the individual image DI represents a capsule CA. The color difference here may be at least one of a difference in hue, a difference in saturation, and a difference in brightness.

[0129] The processing target determination unit 106 may detect the presence or absence of a dividing line SE, and may determine that an individual image DI represents a tablet if a dividing line SE is detected. For example, an individual image DI21 having a dividing line SE may be determined to be an individual image DI representing a tablet.

[0130] The processing object determination unit 106 may be configured to manually input information indicating whether the individual image DI represents a capsule CA or a tablet by an operator who visually inspects the individual image DI. The processing object determination unit 106 may acquire the information input by the operator, and if the acquired information indicates a capsule CA, determine that the individual image DI represents a capsule CA.

[0131] 11 is a schematic diagram showing an implementation example of the processing target determination, which schematically illustrates an aspect in which the trained learning model LM is implemented in the processing target determination unit 106 shown in FIG.

[0132] The trained learning model LM is an AI that outputs 1 when individual images DI14 and DI15 representing capsules CA are input, and outputs 0 when individual images DI representing drugs other than capsules CA, such as individual images DI24 and DI25 representing tablets, are input. The trained learning model LM can be generated by supervised learning. AI is an abbreviation for Artificial Intelligence.

[0133] [Specific example of rotation processing] Fig. 12 is an explanatory diagram of rotation processing. The rotation processing unit 108 shown in Fig. 6 performs image processing of in-plane rotation on the individual image DI detected as an unrotated rectangle shown in Fig. 8 in the rotation processing step S16 shown in Fig. 7. Fig. 12 schematically illustrates the rotation processing on the individual image DI14 shown in Fig. 11.

[0134] The rotation process for the individual image DI14 applies a two-dimensional Cartesian coordinate system having mutually orthogonal xa and ya axes, and rotates the individual image DI14 in the xaya plane so that the longitudinal direction LD of the capsule CA contained in the individual image DI14 is parallel to the xa axis.

[0135] An upper diagram 1201 in Fig. 12 illustrates the individual image DI14 before the rotation process. A lower diagram 1202 in Fig. 12 illustrates the individual image DI14 after the rotation process. In the rotation process, for each ya-axis position, the sum of the pixel values ​​of multiple pixels aligned parallel to the xa-axis is calculated for the individual image DI14, and the distribution of the sum of pixel values ​​on the xa-axis along the ya-axis is derived. In the upper diagram 1201 in Fig. 12, the distribution of the sum of pixel values ​​on the xa-axis along the ya-axis is illustrated in a graph format for the individual image DI14 after the rotation process.

[0136] In the rotation process, the individual image DI14 is rotated by a specified angle in the xaya plane, and the distribution of the sum of pixel values ​​on the xa axis in the ya axis direction is derived for each rotation angle. In the rotation of the individual image DI14, the capsule CA and the background BA are rotated.

[0137] In the rotation process, the rotation angle of the individual image DI at which the width of the peak in the distribution of the sum of pixel values ​​is shortest is the optimal rotation angle of the individual image DI, and is the rotation angle of the individual image DI at which the longitudinal direction of the capsule CA is parallel to the xa axis.

[0138] In the rotation process, the rotation angle of the individual image DI at which the width of the peak in the distribution of the sum of pixel values ​​on the xa axis for each rotation angle falls within a specified range may be derived as the optimal rotation angle of the individual image DI. That is, the longitudinal direction of the capsule CA may be slightly shifted from being parallel to the xa axis.

[0139] When the individual image DI shown in FIG. 9 is detected as a rotated rectangle, the angle formed between the longitudinal direction LD of the individual image DI and the ya axis is identified, so the rotation process described above may be omitted.

[0140] The xa-axis direction described in the embodiment is an example of the first direction, and the ya-axis direction is an example of the second direction. The sum of the pixel values ​​in the xa-axis described in the embodiment is an example of the sum of the pixel values ​​of multiple pixels along the first direction in the drug region.

[0141] [Specific Example of Unfolding Process] Fig. 13 is a schematic diagram of image processing for unfolding a cylinder. The line segment LS shown in Fig. 13 is an enlarged schematic representation of the entire length of the individual image DI in the ya-axis direction. The ya-axis coordinate ya = r × sin(θ) of the curved portion of the capsule CA corresponds to a position on the line segment LS. The angle θ is expressed in radians. Hereinafter, the angle θ will be expressed in radians.

[0142] 13 corresponds to the total length of the individual image DI in the ya-axis direction in the exfoliated image. In the image processing for exfoliating the cylinder, the position of the individual image DI with coordinate value ya=r×sin(θ) is replaced with the position of coordinate value yb=r×θ in the exfoliated image.

[0143] [Parameters Applied to Expansion Processing] Fig. 14 is an explanatory diagram of parameters applied to capsules in the expansion processing. Each parameter shown in the drawing is understood as a length or angle in the xaya coordinate system applied to the individual image DI.

[0144] 14, W represents the total length of the capsule CA in the longitudinal direction LD. H represents the total length of the capsule CA in the transverse direction SD. θ represents the rotation angle from the xa axis toward the positive direction of the ya axis when an arbitrary point on the curve that is the end of the capsule CA in the longitudinal direction is designated.

[0145] 14 illustrates a case where the curved line at the end of the capsule CA in the longitudinal direction is considered to be a semicircle. The radius r of the semicircle is expressed as r=H / 2, where H is the total length of the capsule CA in the lateral direction.

[0146] Fig. 15 is an explanatory diagram of variables used in calculations in the unfolding process. An XBYB coordinate system, which is a two-dimensional coordinate system having mutually orthogonal XB and YB axes, shown in Fig. 15, is applied to the unfolded image EI after the unfolding process shown in Fig. 2. This figure shows an example in which the origin OB of the XBYB coordinate system is set to the upper left corner of the unfolded image in Fig. 15.

[0147] The XB axis is parallel to the xa axis, and the downward direction in Fig. 15 is the positive direction. The YB axis is parallel to the ya axis, and the direction from left to right in Fig. 15 is the positive direction. The origin oa of the xaya coordinate system applied to the individual image DI is shifted by (π × H) / 4 in the positive direction of the YB axis with respect to the origin OB of the XBYB coordinate system applied to the exfoliated image EI.

[0148] In the unfolding process, any point in the xaya coordinate system represented by coordinate values ​​(xa, ya) is replaced with a point in the XBYB coordinate system represented by coordinate values ​​(XB, YB). Also, in the unfolding process, the length H / 2 in the ya-axis direction in the xaya coordinate system is converted into the length (π×H) / 4 in the YB-axis direction in the XBYB coordinate system.

[0149] 16 is an explanatory diagram showing a specific example of calculations applied to the unfolding process. Equation 1 represents the ya-axis coordinate ya of an arbitrary point in the xaya coordinate system. The ya-axis coordinate ya is expressed as ya = (H / 2) × sin θ. Equation 2 represents the rotation angle θ of Equation 1. The rotation angle θ is expressed as θ = sin -1 It is expressed as {(2×ya) / H}.

[0150] Equation 3 represents the transformation relationship between the coordinate ya before expansion processing and the coordinate yb after expansion processing in the xaya coordinate system. The coordinate yb after expansion processing is expressed as yb=(H×θ) / 2=(H / 2)×sin -1 It is expressed as {(2×ya) / H}.

[0151] Equation 4 represents the transformation relationship between the coordinate yb after expansion processing in the xaya coordinate system and the YB-axis coordinate YB in the XBYB coordinate system. The YB-axis coordinate YB is expressed as YB = {(π × H) / 4} - yb = {(π × H) / 4} - (H / 2) × sin -1 {(2×ya) / H}=(H / 2)×[(π / 2)-sin -1 {(2×ya) / H}].

[0152] Equation 5 represents the relationship between the xa-axis coordinate xa in the xaya coordinate system and the XB-axis coordinate XB in the XBYB coordinate system. The XB-axis coordinate XB is expressed as XB=xa.

[0153] The individual image DI shown in Fig. 3 is converted into an expanded image EI using Equations 4 and 5 shown in Fig. 16 . When an operator visually recognizes the expanded image EI and identifies the drug represented in the individual image DI, the expanded image EI may be displayed on the touch panel display 14 of the smartphone 10 shown in Fig. 4 . In the present embodiment, an expanded image EI is generated by focusing on a relatively large portion of the capsule CA, but the expanded image EI may be generated by focusing on a relatively small portion of the capsule CA. Furthermore, the expanded image EI of the relatively large portion of the capsule CA and the expanded image EI of the relatively small portion may be generated separately.

[0154] A relatively simple algorithm is applied to the image processing for unfolding the cylinder described above. In the image processing for unfolding the cylinder, even when calculations are performed for all pixels that make up the individual image DI, the calculation load is relatively light and is suitable for implementation in a mobile terminal device such as a smartphone 10. The image processing for unfolding the cylinder may be applied in a mode in which calculations are performed for some of the pixels that make up the individual image DI, further reducing the calculation load.

[0155] [Operational Effects of the Drug Identification Device and the Like According to the Embodiment] The drug identification device and drug identification method according to the embodiment can achieve the following operational effects.

[0156] [1] Individual drugs are detected from a captured image PI in which one or more drugs are captured, and an individual image DI is generated. If the drug represented by the individual image DI is a capsule CA, an unfolding process is performed on the individual image DI in the short-side direction SD, and an unfolded image EI is generated. This improves the detection accuracy of identification information near the short-side end of the capsule CA, and can improve the accuracy of drug identification.

[0157] [2] It is determined whether the medicine represented by the individual image DI is a capsule CA to be subjected to the expansion process. This makes it possible to avoid performing the expansion process on a tablet that does not have rotational symmetry in the longitudinal direction.

[0158] [3] The processing object discrimination process for discriminating whether or not the object is a capsule CA may be implemented using a trained learning model, an algorithm, or a manual operation by an operator. This allows various methods suitable for the processing object discrimination process to be applied.

[0159] [4] In the processing object discrimination process to which the algorithm is applied, discrimination is performed based on the shape of the individual image DI in the longitudinal direction, such as the difference in size between one side and the other side in the longitudinal direction of the individual image DI, the presence of a midline in the longitudinal direction of the individual image DI, and the difference in color between one side and the other side in the longitudinal direction of the individual image DI. In this way, discrimination of capsules is performed based on the characteristic shape of the capsules CA.

[0160] [5] When the individual image DI is detected as an unrotated rectangle, the individual image DI is rotated in the xaya plane, the sum of the pixel values ​​in the xa-axis direction is calculated, and the distribution of the sum of the pixel values ​​in the xa-axis direction in the ya-axis direction is calculated. The orientation of the individual image DI in which the width of the distribution peak is smallest is identified as the orientation of the individual image DI in which the longitudinal direction LD of the capsule CA is parallel to the xa-axis direction. This allows a preferable unfolding process to be performed with respect to the transverse direction SD of the capsule CA.

[0161] [6] When the individual image DI is detected as a rotated rectangle, the rotation angle of the longitudinal direction LD of the capsule CA relative to the xa axis is estimated, thereby omitting the rotation process of the individual image DI when the individual image DI is detected as a non-rotated rectangle.

[0162] [7] When an exfoliated picture EI is generated from an individual image DI, an image processing algorithm for exfoliating a cylinder is applied, in which coordinate values ​​applied to the individual image DI are converted to coordinate values ​​applied to the exfoliated picture EI, thereby generating an exfoliated picture EI based on the capsules CA of the individual image DI.

[0163] [Modifications of the embodiment] The following modifications may be applied to the drug identification device and drug identification method according to the above-described embodiment.

[0164] [First Modification] The process of rotating the individual image DI so that the longitudinal direction of the capsule CA contained in the individual image DI is parallel to the ya axis may involve rotating the individual image DI so that the longitudinal direction of the individual image DI is parallel to the xa axis, and the ya axis direction may be the expansion direction of the individual image DI.

[0165] When rotating an individual image DI so that the longitudinal direction of the individual image DI is parallel to the xa axis, the coordinate value of the xa axis, xa = (H / 2) × sin(θ), is replaced with the coordinate value of the XB axis, XB = (H / 2) × θ.

[0166] [Second variant] Images of both sides of the sachet S are obtained, and drug detection, processing target determination, rotation processing, and unfolding processing are performed on each of the images of both sides of the sachet S, and drug identification results can be obtained for each of the images of both sides of the sachet S.

[0167] Note that the image of one side of the sachet described in the embodiment is an example of a first drug group image in which the drugs are photographed from one side, and the image of the other side of the sachet is an example of a first drug group image in which the drugs are photographed from the other side.

[0168] In addition, the unfolded image of an individual image based on a photographed image of one side of the sachet described in the embodiment is an example of a first unfolded image based on a first drug group image, and the unfolded image of an individual image based on a photographed image of the other side of the sachet is an example of a second unfolded image based on a second drug group image.

[0169] [Third Modification] The unfolding processing unit 110 that performs the unfolding process may be implemented with a trained learning model. Using a pair of an individual image DI and an unfolded image EI as learning data, supervised learning is performed to output an unfolded image EI when an individual image DI is input, and the trained learning model is used to generate an unfolded image EI from the individual image DI.

[0170] [Fourth Modification] A trained learning model may be implemented in the candidate identification unit 112 that identifies candidate drugs. The trained learning model implemented in the candidate identification unit 112 may output one or more candidate drugs with scores indicating likelihood as identification results.

[0171] [Example of application to an image processing device] An image processing device may be configured by separating the functions of the captured image acquisition unit 102, drug detection unit 104, processing target determination unit 106, rotation processing unit 108, and expansion processing unit 110 shown in Fig. 6 from the smartphone 10, which is the drug identification device according to the embodiment. The image processing device may include a display signal generation unit 116 shown in the same figure.

[0172] [Application example to a program] A computer functioning as the drug identification device according to the embodiment may be configured with a program that causes the computer to realize various functions of the drug identification device. The program may be stored in a non-transitory computer-readable storage medium. The program may be downloaded and acquired via a communication line from a server device or the like in which the program is stored.

[0173] The image processing for the individual image representing the capsule CA shown in this embodiment is applicable to a cylindrical tablet having identification information displayed on the medication surface. A trained learning model may be applied to the process of determining whether the captured image is a tablet to be processed. The trained learning model may be a trained learning model that determines whether the captured image is a tablet or capsule CA to be processed.

[0174] The above-described embodiments of the present invention may be modified, added, or deleted as appropriate within the scope of the spirit of the present invention. The present invention is not limited to the above-described embodiments, and many modifications may be made by a person skilled in the art within the technical concept of the present invention. Furthermore, the embodiments, modifications, and applications may be implemented in appropriate combinations.

[0175] [Additional Notes] As can be understood from the detailed description of the embodiments above, this specification includes disclosure of various technical ideas including the inventions set forth below.

[0176] [Supplementary Note 1] An image processing device comprising one or more processors and one or more memories storing instructions to be executed by the one or more processors, wherein the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug region containing a drug image that is an image of each drug from the drug group image, identify a longitudinal direction of the drug image represented in the drug region, and perform image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0177] [Appendix 2] In the image processing device of Appendix 1, one or more processors may determine whether a drug image represents a drug to be expanded, and generate an expanded image of the drug image determined to be a drug to be expanded.

[0178] [Supplementary Note 3] In the image processing device of Supplementary Note 1 or Supplementary Note 2, the one or more processors may determine whether the medicine is a capsule or not as the determination of whether the medicine is a target of the expansion process.

[0179] [Supplementary Note 4] In the image processing device of any one of Supplementary Note 1 to Supplementary Note 3, the one or more processors may identify an area surrounding an edge of the medicine image as the medicine area.

[0180] [Supplementary Note 5] In the image processing device of any one of Supplementary Note 1 to Supplementary Note 4, the one or more processors may identify, as the medicine region, a region surrounded by a figure circumscribing the medicine image.

[0181] [Appendix 6] In the image processing device of Appendix 5, one or more processors may identify the longitudinal direction of the drug image based on the distribution in a second direction perpendicular to the first direction of the sum of pixel values ​​of multiple pixels along a first direction in the drug region.

[0182] [Supplementary Note 7] In the image processing device of any one of Supplementary Notes 1 to 6, the one or more processors may send a display signal representing the developed image to a display device.

[0183] [Supplementary Note 8] In the image processing device of any one of Supplementary Note 1 to Supplementary Note 7, the one or more processors may identify the type of drug represented by the developed image and output a signal representing the type of drug.

[0184] [Supplementary Note 9] In the image processing device of Supplementary Note 8, the one or more processors may detect one or more characters included in the unfolded image, and identify the type of drug represented by the unfolded image based on the detected one or more characters.

[0185] [Supplementary Note 10] In the image processing device of any one of Supplementary Note 1 to Supplementary Note 9, the one or more processors may acquire a first medication group image in which the medication is photographed from one side.

[0186] [Appendix 11] In any of the image processing devices of Appendices 1 to 10, one or more processors may acquire a first drug group image in which the drugs are photographed from one side and a second drug group image in which the drugs are photographed from the other side, and generate a first expanded image based on the first drug group image and a second expanded image based on the second drug group image.

[0187] [Appendix 12] An image processing method in which a computer acquires a drug group image generated by photographing a drug group containing one or more drugs, identifies a drug region containing a drug image, which is an image of each drug, from the drug group image, identifies the longitudinal direction of the drug image represented in the drug region, and performs image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0188] [Supplementary Note 13] In the image processing method of Supplementary Note 12, the processing executed by the computer may include processing in response to an operation by an operator operating the computer.

[0189] [Appendix 14] A program that enables a computer to acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug region containing a drug image, which is an image of each drug, from the drug group image, identify the longitudinal direction of the drug image represented in the drug region, and generate an expanded image by performing image processing to expand the drug image into a cylinder in an expansion direction perpendicular to the identified longitudinal direction.

[0190] [Supplementary Note 15] A drug identification device comprising one or more processors and one or more memories for storing instructions to be executed by the one or more processors, wherein the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug region containing a drug image that is an image of each drug from the drug group image, identify a longitudinal direction of the drug image represented in the drug region, and perform image processing to unfold a cylinder on the drug image in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

[0191] [Supplementary Note 16] In the drug identification device of Supplementary Note 15, the one or more processors may identify the drug type of the drug image based on the developed image.

[0192] REFERENCE SIGNS LIST 10 Smartphone 14 Touch panel display 16 Speaker 18 Microphone 20 In-camera 22 Out-camera 24 Light 26 Switch 28 CPU 30 Wireless communication unit 32 Call unit 34 Memory 36 Internal storage unit 38 External storage unit 40 External input / output unit 42 GPS receiving unit 44 Power supply unit 100 Drug identification device 102 Photographed image acquisition unit 104 Drug detection unit 106 Processing target determination unit 108 Rotation processing unit 110 Expansion processing unit 112 Candidate identification unit 114 Drug database 116 Display screen generation unit BA Background area BB Bounding box BS Back side CA Capsule CL Curve CO Center point DI Individual image DI1 Individual image DI11 Individual image DI12 Individual image DI13 Individual image DI14 Individual image DI15 Individual image DI21 Individual image DI22 Individual image DI24 Individual image DI25 Individual image EI Unfolded image F Surface FS Front side IR Recognition result LD Longitudinal direction LM Trained learning model LS Line segment PI Photographed image P1 Image processing P2 Image processing S Sachet SD Short direction SE Secant line SL Auxiliary line T Placement base TI Identification information BH Horizontal width BV Vertical width Bθ Rotation angle V1 Vertex V2 Vertex S1 to S26 Each step of the medicine identification method

Claims

1. An image processing device comprising one or more processors and one or more memories storing instructions to be executed by the one or more processors, wherein the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug region containing a drug image, which is an image of each drug, from the drug group image, identify a longitudinal direction of the drug image represented in the drug region, and perform image processing to unfold the drug image into a cylinder in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

2. The image processing device according to claim 1, wherein the one or more processors determine whether the drug image represents a drug to be expanded, and generate an expanded image of the drug image determined to be a drug to be expanded.

3. The image processing device according to claim 2, wherein the one or more processors determine whether the medicine is a capsule or not as the determination of whether the medicine is a target of expansion processing.

4. The image processing device according to claim 1, wherein the one or more processors identify an area surrounding an edge of the medicine image as the medicine area.

5. The image processing device according to claim 1, wherein the one or more processors identify, as the medication region, an area surrounded by a figure circumscribing the medication image.

6. The image processing device described in claim 5, wherein the one or more processors identify the longitudinal direction of the drug image based on the distribution in a second direction perpendicular to the first direction of the sum of pixel values ​​of multiple pixels along a first direction in the drug region.

7. The image processing device according to claim 1, wherein the one or more processors transmit a display signal representing the developed image to a display device.

8. The image processing device according to claim 1, wherein the one or more processors identify the type of drug represented by the developed image and output a signal representing the type of drug.

9. The image processing device according to claim 8, wherein the one or more processors detect one or more characters included in the unfolded image, and identify the type of medication represented by the unfolded image based on the detected one or more characters.

10. The image processing device of claim 1, wherein the one or more processors acquire a first drug group image in which the drugs are photographed from one side.

11. The image processing device of claim 1, wherein the one or more processors acquire a first drug group image in which the drugs are photographed from one side and a second drug group image in which the drugs are photographed from the other side, and generate a first expanded image based on the first drug group image and a second expanded image based on the second drug group image.

12. An image processing method in which a computer acquires a drug group image generated by photographing a drug group containing one or more drugs, identifies a drug region containing a drug image, which is an image of each drug, from the drug group image, identifies the longitudinal direction of the drug image represented in the drug region, and performs image processing to unfold the drug image into a cylinder in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

13. The image processing method according to claim 12, wherein the processing executed by the computer includes processing in response to an operation by an operator operating the computer.

14. A program that enables a computer to perform the following functions: acquire a drug group image generated by photographing a drug group containing one or more drugs; identify a drug area from the drug group image that contains a drug image, which is an image of each drug; identify the longitudinal direction of the drug image represented in the drug area; and generate an expanded image by performing image processing to expand the drug image into a cylinder in an expansion direction perpendicular to the identified longitudinal direction.

15. A non-transitory computer-readable recording medium on which the program according to claim 14 is recorded.

16. A drug identification device comprising one or more processors and one or more memories storing instructions to be executed by the one or more processors, wherein the one or more processors acquire a drug group image generated by photographing a drug group containing one or more drugs, identify a drug region containing a drug image, which is an image of each drug, from the drug group image, identify a longitudinal direction of the drug image represented in the drug region, and perform image processing to unfold the drug image into a cylinder in an unfolding direction perpendicular to the identified longitudinal direction, thereby generating an unfolded image.

17. The drug identification device according to claim 16, wherein the one or more processors identify the drug type of the drug image based on the developed image.

Images