Electronic device and method for identifying pill by same

Abstract

The present disclosure relates to a method for identifying a pill, the method being performed by at least one processor, wherein the method may comprise the steps of: acquiring information of a first pill corresponding to a first code object identified through a camera; identifying at least one first pill included in a first image acquired through the camera; displaying the number of the at least one first pill included in the first image; identifying at least one first pill included in a second image acquired through the camera; and displaying the total number of pills obtained by summing the number of the at least one first pill included in the first image and the number of the at least one first pill included in the second image.

Description

How to identify electronic devices and their pills

[0001] The present disclosure relates to an electronic device and a method for identifying a pill thereof, and specifically, to a technique for identifying a pill on an image in which the pill is captured.

[0002] Regarding medications, it is crucial that pharmacists prescribe a fixed number of pills and that patients take the prescribed amount. When the number of pills is very large, accurately counting them and classifying them into the prescribed quantity can be time-consuming. When using electronic devices to identify pills, it is important for the user to be aware of the number and location of the pills in real time.

[0003] According to one embodiment of the present disclosure, the technical problem is to identify a pill included in an image and display it in a manner that is easy for the user to recognize.

[0004] According to one embodiment of the present disclosure, the technical problem is to enable a user to easily recognize and classify pills by identifying and displaying pills included in an image in real time.

[0005] According to one embodiment of the present disclosure, the technical problem is to enable the user to easily recognize the total number of pills by summing and displaying the number of identical pills included in a plurality of images.

[0006] The present disclosure may be implemented in various ways, including a method, an apparatus (system) and / or a computer program stored on a computer-readable storage medium, and a computer-readable storage medium on which the computer program is stored.

[0007] A method for identifying pills performed by at least one processor according to various embodiments of the present disclosure comprises: acquiring information of a first pill corresponding to a first code object identified through a camera; identifying at least one first pill included in a first image acquired through the camera; displaying the number of at least one first pill included in the first image; identifying at least one first pill included in a second image acquired through the camera; and displaying a total number of pills obtained by summing the number of at least one first pill included in the first image and the number of at least one first pill included in the second image.

[0008] A method for identifying the pills according to one embodiment may further include the step of identifying at least one first pill included in a third image acquired through the camera; and the step of adding the number of at least one first pill included in the third image to the total number of pills.

[0009] A method for identifying the pill according to one embodiment may further include the step of storing the total number of pills, information of the first pill, the first image, and the second image by matching them with each other.

[0010] According to one embodiment, the storing step may include a step of matching and storing information regarding the time at which the first image was taken and information regarding the time at which the second image was taken.

[0011] A method for identifying pills according to one embodiment may further include: receiving a first user input for selecting an icon to sum pills of the same type; and setting the number of first pills identified in a plurality of images to be summed based on the first user input.

[0012] A method for identifying the pill according to one embodiment may further include: a step of confirming information of a second pill corresponding to a second code object identified through the camera; a step of comparing information of a first pill corresponding to the first code object with information of a second pill corresponding to the second code object; and a step of setting the number of the first pills not to be added further in response to a determination that the information of the first pill corresponding to the first code object and the information of the second pill corresponding to the second code object are different.

[0013] A method for identifying the pill according to one embodiment may further include a step of summing the number of the second pills identified in a plurality of images.

[0014] A method for identifying the pills according to one embodiment may further include: receiving a second user input for re-selecting an icon for summing identical pills; and, based on the second user input, setting the number of the first pills not to be summed further if the information of the pills and the information of the second pills corresponding to the second code object are different.

[0015] A method for identifying the pill according to one embodiment may be configured to display at least one first graphic object by overlaying it on at least one first pill of the first image.

[0016] A computer program stored on a computer-readable recording medium for executing a method for identifying the pill according to one embodiment on a computer is disclosed.

[0017] An electronic device according to various embodiments of the present disclosure comprises: a camera; a display; and at least one processor, wherein the at least one processor is configured to identify information of a first pill corresponding to a first code object identified through the camera, identify at least one first pill included in a first image acquired through the camera, display the number of at least one first pill included in the first image, identify at least one first pill included in a second image acquired through the camera, and display a total number of pills obtained by summing the number of at least one first pill included in the first image and the number of at least one first pill included in the second image.

[0018] According to one embodiment of the present disclosure, pills included in an image can be identified and displayed in a manner that is easy for the user to recognize.

[0019] According to one embodiment of the present disclosure, by identifying and displaying pills included in an image in real time, a user can easily recognize and classify the pills.

[0020] According to one embodiment of the present disclosure, by identifying only a specific area of ​​a pill counting tray and identifying and displaying only the pills located on that specific area, the user can easily recognize only the pills located on that specific area.

[0021] According to one embodiment of the present disclosure, by identifying and displaying the number of pills exceeding the target quantity, the user can easily recognize the pills corresponding to the target quantity.

[0022] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims.

[0023] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0024] FIG. 1 is a schematic diagram illustrating an example of an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 2 is a schematic diagram showing a configuration in which an information processing system and a plurality of user terminals are connected to enable communication between them according to one embodiment of the present disclosure.

[0026] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0027] FIG. 4 is a flowchart of the operation of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 5 is a diagram illustrating an example of a process for performing object recognition according to one embodiment of the present disclosure.

[0029] FIGS. 6, FIGS. 7 and FIGS. 8 are screens of a pill identification application according to one embodiment of the present disclosure.

[0030] FIGS. 9 and FIGS. 10 are drawings illustrating one screen of an application according to one embodiment of the present disclosure.

[0031] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding well-known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0032] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0033] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0034] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0035] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0037] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Optical Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0038] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are used merely to distinguish one component from another, and the essence, order, or sequence of the said component is not limited by such terms.

[0039] Additionally, in the following embodiments, where it is stated that one component is 'connected', 'coupled', or 'joined' to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be 'connected', 'coupled', or 'joined' between each component.

[0040] In the present disclosure, 'each of a plurality of A' may refer to each of all components included in a plurality of A, or each of some components included in a plurality of A.

[0041] Additionally, the terms 'comprises' and / or 'comprising' as used in the following embodiments do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0042] In the present disclosure, artificial intelligence (AI) refers to a technology that mimics human learning, reasoning, and perceptual abilities and implements them on a computer, and may include the concepts of machine learning and symbolic logic. Machine learning (ML) may be an algorithmic technology that classifies or learns the features of input data on its own. AI technology can analyze input data as a machine learning algorithm, learn from the results of the analysis, and make judgments or predictions based on the results of the learning. Furthermore, technologies that mimic the cognitive and judgment functions of the human brain by utilizing machine learning algorithms may also be understood as falling within the category of artificial intelligence. For example, fields of technology such as linguistic understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control may be included.

[0043] In the present disclosure, machine learning may refer to a process of training a neural network model using experience in processing data. Machine learning may refer to computer software improving its own data processing capabilities. A neural network model is constructed by modeling the correlation between data, and such correlation can be expressed by multiple parameters. A neural network model extracts and analyzes features from given data to derive correlations between the data; machine learning can be defined as the process of optimizing the parameters of the neural network model by repeating this process. For example, a neural network model can learn the mapping (correlation) between inputs and outputs for data given as input-output pairs. Alternatively, a neural network model may derive regularities between given data and learn the relationships even when only input data is provided.

[0044] In the present disclosure, an artificial intelligence learning model, a machine learning model, a deep learning model, or a neural network model may be designed to implement the structure of the human brain on a computer and may include a plurality of network nodes having weights that simulate neurons of a human neural network. The plurality of network nodes may have interconnected relationships by simulating the synaptic activity of neurons, in which neurons exchange signals through synapses. In an artificial intelligence learning model, the plurality of network nodes may be located in layers of different depths and exchange data according to convolutional connection relationships. The artificial intelligence learning model may be, for example, an artificial neural network model, a convolutional neural network model, etc. In the present disclosure, the artificial intelligence learning model may be referred to as a deep learning learning model or a deep learning model.

[0045] In the present disclosure, "pill" may refer to a solid formulation having a specific shape. For example, the "pill" may be made of compressed powder or granules and may maintain a formulation form containing one or more active ingredients. Additionally, the pill may be provided in various sizes and shapes. For example, the pill may be in various shapes such as round, oval, square, or cylindrical, or may be in a shape broken from a solid formulation. Additionally, the pill may be in various colors depending on its material and ingredients. Additionally, there may be an imprint, such as a printed material, on one or both sides of the pill. The imprint may include information such as a logo, drug name, dosage, or manufacturer. Additionally, the surface of the pill may have other surface marks, such as scratches or cracks. For example, the pill may be implemented in formulations such as tablets, extended-release tablets, gastro-resistant tablets, triple tablets, orally disintegrating tablets, sugar-coated tablets, troches, vaginal suppositories, enteric-coated pills, capsules, soft capsules, etc., but is not limited thereto. The pill may be packaged in blisters, pouches, sachets, bottles, etc.

[0046] In the present disclosure, the 'recognition result' is information generated by an object recognition model and may include location information, size information, width information, height information, bounding box information, etc. of the recognized object. Additionally, the recognition result may include information generated by the object recognition model based on an image and or an image and prescription data. Additionally, the recognition result may include confidence values ​​and / or category information.

[0047] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0048] FIG. 1 is a schematic diagram illustrating an example of an electronic device according to one embodiment of the present disclosure. As illustrated, the electronic device (100) may include a processor (110), a camera (120), and a display (130).

[0049] A processor (110) according to one embodiment can, for example, execute software to control at least one other component (e.g., a hardware or software component) of an electronic device (100) connected to the processor (110) and can perform various data processing or operations.

[0050] A camera (120) according to one embodiment can capture still images and video. According to one embodiment, the camera (120) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The electronic device (100) can identify at least one pill included in an image captured through a camera (120). Although FIG. 1 is illustrated as identifying pills using one camera (120), it is not limited thereto, and two or more cameras may be used.

[0052] According to one embodiment, the camera (120) can generate an image containing at least one pill. For example, the camera (120) can generate an image containing at least one pill placed on a tray. An image containing at least one pill captured by the camera (120) can be transmitted to a processor (110). The image generated by the camera may be a single still image or a video that is a set of multiple images in a time series.

[0053] A display (130) according to one embodiment can visually provide information to an external (e.g., user) of an electronic device (100). The display (130) may include a touch circuit configured to detect a touch, or a sensor circuit configured to measure the intensity of a force generated by a touch.

[0054] The user can place a tray containing pills to be identified onto the FoV of the camera (120) of the electronic device (100). For example, the user can place the tray containing pills onto the focal length of the camera (120). The user can take an image containing at least one pill placed on the tray.

[0055] According to one embodiment, the processor (110) can generate a recognition result by performing object recognition on an image using an object recognition model. The processor (110) can identify at least one pill included in the image using an object recognition model.

[0056] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) and a plurality of user terminals (210_1, 210_2, 210_3) are connected to enable communication between them according to one embodiment of the present disclosure. As illustrated, the plurality of user terminals (210_1, 210_2, 210_3) may be connected to an information processing system (230) capable of providing a pill identification service through a network (220). Here, the plurality of user terminals (210_1, 210_2, 210_3) may include a user terminal receiving the pill identification service.

[0057] According to one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data associated with providing pill identification services, etc., or one or more distributed computing devices and / or distributed databases based on cloud computing services.

[0058] The pill identification service provided by the information processing system (230) may be provided to the user through a pill identification application installed on each of the multiple user terminals (210_1, 210_2, 210_3). For example, the information processing system (230) may provide information related to pill identification received from the user terminals (210_1, 210_2, 210_3) through the pill identification application, etc., or perform corresponding processing.

[0059] Multiple user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) through a network (220). The network (220) can be configured to enable communication between the multiple user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) may be configured as a wired network such as Ethernet, Power Line Communication, telephone line communication devices and RS-serial communication, a mobile communication network, a Wireless LAN (WLAN), Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited and may include not only a communication method utilizing a communication network (e.g., mobile communication network, wired internet, wireless internet, broadcasting network, satellite network, etc.) that the network (220) may include, but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0060] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto. The user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and running instant messaging applications or web browsers. For example, user terminals may include an AI speaker, a smartphone, a mobile phone, a navigation system, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (Virtual Reality) device, an AR (Augmented Reality) device, a set-top box, etc. Additionally, FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing system (230) through a network (220), but is not limited thereto, and may be configured so that a different number of user terminals communicate with an information processing system (230) through a network (220).

[0061] According to various embodiments, the electronic device (100) that performs the method of identifying a pill may be a user terminal (210_1, 210_2, 210_3).

[0062] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing a pill identification application, etc., and capable of wired / wireless communication, and may include, for example, the mobile phone terminal (210_1), tablet terminal (210_2), PC terminal (210_3) of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data through a network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) through the input / output interface (318).

[0063] The memory (312, 332) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as ROM (read-only memory), a disk drive, an SSD (solid-state drive), or a flash memory. As another example, a permanent mass storage device such as ROM, an SSD, a flash memory, or a disk drive may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. Additionally, the memory (312, 332) may store an operating system and at least one program code (e.g., code for a pill identification application installed and running on the user terminal (210)).

[0064] These software components may be loaded from a computer-readable recording medium separate from memory (312, 332). This separate computer-readable recording medium may include a recording medium that can be directly connected to the user terminal (210) and the information processing system (230), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (312, 332) via a communication module rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (312, 332) based on a computer program installed by files provided through a network (220) by developers or a file distribution system that distributes installation files for the application.

[0065] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a recording device such as memory (312, 332).

[0066] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system). For example, a request or data (e.g., an image containing at least one pill) generated by the processor (314) of the user terminal (210) according to program code stored in a recording device such as memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) can be received by the user terminal (210) through the communication module (336) and the network (220) via the communication module (316) of the user terminal (210).

[0067] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, or a mouse, and the output device may include a device such as a display (e.g., the display (130) of FIG. 1), a speaker, or a haptic feedback device. As another example, the input / output interface (318) may be a means for interfacing with a device in which the configuration or function for performing input and output is integrated into one, such as a touchscreen. For example, when the processor (314) of the user terminal (210) processes instructions of a computer program loaded in memory (312), a service screen configured using information and / or data provided by an information processing system (230) or another user terminal may be displayed on a display through the input / output interface (318). In FIG. 3, the input / output device (320) is depicted as not being included in the user terminal (210), but is not limited thereto and may be configured as a single device with the user terminal (210). Additionally, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that may be connected to the information processing system (230) or included in the information processing system (230). In FIG. 3, the input / output interface (318, 338) is depicted as an element configured separately from the processor (314, 334), but is not limited thereto and may be configured so that the input / output interface (318, 338) is included in the processor (314, 334).

[0068] The user terminal (210) and the information processing system (230) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the prior art components. According to one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. Additionally, the user terminal (210) may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that are generally included in a smartphone, and may be implemented to further include various components such as an accelerometer, a gyroscope, an image sensor, a proximity sensor, a touch sensor, an ambient light sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0069] While a program for a pill identification application, etc. is running, the processor (314) can receive text, images, video, voice and / or actions, etc. that are input or selected through an input device such as a touch screen, keyboard, audio sensor and / or image sensor, camera, microphone, etc. connected to an input / output interface (318), and can store the received text, images, video, voice and / or actions, etc. in memory (312) or provide them to an information processing system (230) through a communication module (316) and a network (220).

[0070] The processor (314) of the user terminal (210) may be configured to manage, process, and / or store information and / or data received from an input / output device (320), another user terminal, an information processing system (230), and / or a plurality of external systems. The information and / or data processed by the processor (314) may be provided to the information processing system (230) through a communication module (316) and a network (220). The processor (314) of the user terminal (210) may transmit information and / or data to the input / output device (320) through an input / output interface (318) to output it. For example, the processor (314) may display the received information and / or data on the screen of the user terminal (210).

[0071] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals (210) and / or a plurality of external systems. The information and / or data processed by the processor (334) may be provided to the user terminals (210) through a communication module (336) and a network (220).

[0072] FIG. 4 is an operation flowchart (400) of an electronic device according to one embodiment of the present disclosure.

[0073] Referring to the operation flowchart (400), the processor (110) of the electronic device (100) according to one embodiment can, in step 410, verify information of a first pill (e.g., a first type of pill) corresponding to a first code object identified through a camera (120). In one example, a user can obtain / verify information of the first pill corresponding to the first code object by scanning the first code object, which allows verification of information of the pill, with the camera (120) of the electronic device (100). The code object may be included, for example, in packaging material contained in the pill (e.g., paper packaging, vinyl packaging, plastic packaging, etc.), a medicine bag corresponding to the pill, a prescription, etc. The code object may be, for example, a one-dimensional code object, a two-dimensional code object, a barcode, a QR code, a data matrix, etc. Information of the pill may include information regarding the name, type, and expiration date of the pill. Alternatively or additionally, the processor (110) can use a pill recognition model (e.g., a pill recognition AI model, etc.) instead of a code object to recognize the type of pill through a combination of the shape, size, color, letters, symbols, etc. of the pill in the image and obtain / confirm information associated with the pill.

[0074] A processor (110) according to one embodiment can identify at least one first pill included in a first image obtained through a camera (120) in step 420. In one example, a user can place at least one first pill on a pill counting tray and photograph the pill counting tray on which at least one first pill is placed through the camera (120) of the electronic device (100). The camera (120) can generate a first image including the pill counting tray on which at least one first pill is placed. The image may be a still image, a video, or a part of a video. The camera (120) can transmit the generated image including the pill counting tray on which a plurality of pills are placed to the processor (110). Although step 420 is shown in FIG. 4 as being performed after step 410, it is not limited thereto. For example, step 420 of counting pills may be performed first, and step 430 of recognizing the type of pill and obtaining associated information may be performed later.

[0075] The processor (110) can identify at least one first pill included in the first image. In one example, the processor (110) can identify at least one first pill in the image using an object recognition model. The processor (110) can identify the number and location of at least one first pill included within the first recognition area. The processor (110) can identify at least one of the size and color of at least one first pill. The object recognition model will be described in detail using FIG. 5.

[0076] A processor (110) according to one embodiment may display the number of at least one first pill included in the first image at step 430. The processor (110) may count the number of at least one first pill included in the first image and display the counted number of at least one first pill through the display (130).

[0077] A processor (110) according to one embodiment can identify at least one first pill included in a second image obtained through a camera (120) in step 440. After capturing the first image, the user can place the first pill of the same type back onto the pill counting tray and generate a second image including the pill counting tray on which at least one first pill is placed through the camera (120).

[0078] In one embodiment, between steps 430 and 440, the processor (110) may further perform the step of identifying at least one second pill included in a fourth image acquired through a camera and indicating the number of at least one second pill included in the fourth image.

[0079] A processor (110) according to one embodiment may, in step 450, display a total number of pills by summing the number of at least one first pill included in the first image and the number of at least one first pill included in the second image. If the number of at least one first pill included in the first image is 16 and the number of at least one first pill included in the second image is 10, the processor (110) may display a total number of pills as 26. To this end, the processor (110) may determine that at least one first pill included in the first image and at least one first pill included in the second image are of the same type.

[0080] Additionally, the processor (110) can identify at least one first pill on a third image obtained through the camera (120) and add the number of at least one first pill included on the third image to the number of previously identified first pills to the total number of pills.

[0081] In this way, the user can photograph the same pill in multiple images and then sum the total number of pills contained in the multiple images. Therefore, the user can easily and accurately determine the total number of pills simply by placing the pills in a pill counting tray and photographing them.

[0082] FIG. 5 is a diagram illustrating an example of a process for performing object recognition according to one embodiment of the present disclosure. The image (510) may be an image of a pill counting tray containing a plurality of pills. For example, the image (510) may be one of images captured through a camera (120) of an electronic device (100).

[0083] The object recognition model (520) can recognize at least one pill included in the image (510) as an object and generate a recognition result (530). For example, the object recognition model (520) may be a neural network model, RetinaNet, YOLO, SSD, R-CNN, etc. The recognition result (530) may include information regarding each object recognized by the object recognition model (520). For example, the recognition result (530) may include location information, size information (width information, height information), category information, confidence value, etc. regarding the bounding box of a specific object.

[0084] According to one embodiment, the object recognition model (520) can learn various pharmaceutical data. Specifically, the object recognition model (520) can learn pharmaceutical data by classifying it by type of pill. For example, the pharmaceutical data may include the name, front image, back image, size information, etc. of each pill. The object recognition model (520) can recognize pill objects within the image (510) based on the learned data. In this case, the category information included in the recognition result (530) may correspond to specific pharmaceutical data (e.g., Tylenol Tablet 500mg (Acetaminophen)).

[0085] According to another embodiment, the object recognition model (520) cannot specify what kind of medicine the pill object in the image (510) is, but can distinguish and recognize different pill shapes included in the image (510). For example, if the image (510) includes a circular pill, a triangular pill, and an elliptical pill, the category information included in the recognition result (530) may correspond to one of the different pill shapes.

[0086] Returning to FIG. 4, a processor (110) according to one embodiment may, in step 450, overlay at least one first graphic object on at least one first pill. A graphic object may be part of a graphical user interface (GUI) that is displayed to allow a user to recognize a pill on an image. The size, color, and shape of the graphic object may be determined based on the identified pill and / or may be changed by user input. At least one first graphic object and at least one first pill may correspond one-to-one. For example, if there are 10 pills within the recognition area of ​​the image, 10 graphic objects may be created. The processor (110) may control at least one first graphic object to be superimposed and displayed on at least one first pill identified on the image.

[0087] FIGS. 6, 7, and 8 are a screen of a pill identification application according to an embodiment of the present disclosure. Referring to FIGS. 6, 7, and 8, a processor (110) of an electronic device (100) may display a screen of an application for pill identification through a display (130). The screen (600) may include an image selection area (610), a pill information display area (620), a pill image display area (630), a pill count display area (640), and a shooting function selection area (650).

[0088] According to one embodiment, the image selection area (610) may be an area for selecting the type of image for pill identification. For example, if the user wants to identify a pill included in a still image (or image), the user may select the snap icon (611), and if the user wants to identify a pill included in a video containing a series of images, the user may select the Live icon (613). Based on the type selected in the image selection area (610), the camera (120) may capture a still image or a video as an image containing at least one pill.

[0089] According to one embodiment, the pill information display area (620) may be an area that displays information about a pill identified through a code object. For example, the information about the identified pill may include the type of pill, expiration date, manufacturing information, etc.

[0090] According to one embodiment, the pill image display area (630) may be an area for displaying an image containing at least one pill (P) received from the camera (120). If a video is selected by user input, the processor (110) may display a real-time video of an image containing at least one pill (P) captured through the camera (120) in the pill image display area (630). If a still image is selected by user input, the processor (110) may display a still image containing at least one pill (P) captured through the camera (120) in the image display area (630).

[0091] According to one embodiment, the pill count display area (640) may be an area that displays the number of identified pills. When the same pill automatic summation function is disabled, only the number of pills identified in the current image is displayed, and when the same pill automatic summation function is enabled, the number of pills identified in the current image and the total sum of pills identified in multiple images during a specific period (e.g., from the time the same pill automatic summation function is enabled, today, the last week, the last month, etc.) may be displayed.

[0092] According to one embodiment, the shooting function selection area (650) may include a button for capturing images or videos. The shooting function selection area (650) may include an icon (651) for summing identical pills. By selecting the icon (651), the user can activate the automatic summing function for identical pills.

[0093] A processor (110) according to one embodiment may display at least one graphic object (GO) overlaid on at least one pill (P) through a display (130). At least one graphic object (GO) may correspond one-to-one with at least one pill (P). As shown in FIG. 6, if there are (or identified) a total of 16 pills in an image, 16 graphic objects may be displayed overlaid on the 16 pills. The shape of at least one graphic object may be one of a circle, an ellipse, and a polygon. For convenience of explanation, the graphic object is described as being circular (i.e., dot shape) in this disclosure, but the shape of the graphic object is not limited thereto.

[0094] FIG. 7 is a screen of an application that captures a first image while the same pill automatic summing function is activated based on user input selecting an icon (651), and displays the number of at least one first pill included in the first image. For example, the user may place 16 first pills in a pill counting tray and capture a first image. Since there are currently 16 pills in the first image, the number of first pills may be displayed as 16 in the pill count display area (640). Additionally, since the same pill automatic summing function is activated, '16' associated with the currently captured first image and a '+' sign indicating that the automatic summing function is performed through additional shooting may be displayed together at the bottom right of the pill count display area (640), so that '16+' is displayed.

[0095] FIG. 8 is a screen of an application in which, with the same pill automatic summing function enabled, a second image is taken after a first image is taken, and the number of at least one first pill included in the second image is displayed. For example, after taking the first image, the user may place 17 identical (or similar) first pills in a pill counting tray and take a second image. Since there are 17 pills in the second image, the number of first pills is 17 and the total sum of the first pills in the first and second images can be displayed as 33 in the pill count display area (640). Additionally, '16+17=33' can be displayed in the pill count display area (640) so that the summing history can be checked. Through this, the user can easily check the number of pills counted in each image and the sum count.

[0096] In the same way, by taking multiple shots (e.g., three or more) of the same pill, the total sum of the pills can be quickly and accurately verified. The counting history of multiple shots of the same pill can be quickly and accurately verified.

[0097] According to one embodiment, if the user wishes to disable the automatic summing function for identical pills, the user may re-select the icon (651) for summing identical pills in the shooting function selection area (650). When the processor (110) receives a second user input for re-selecting the icon (651) for summing identical pills, it may set the number of the first pills not to be summed further based on the second user input.

[0098] According to one embodiment, when a plurality of images of a first pill are captured while the same pill summing function is activated, and the user scans a second code object corresponding to a second pill, the processor (110) may stop automatic summing for the first pill and start automatic summing for the second pill. While performing summing for the first pill, the processor may check the information of the second pill corresponding to the second code object identified through the camera (120). The processor (110) may compare the information of the first pill corresponding to the first code object with the information of the second pill corresponding to the second code object. If the information of the first pill corresponding to the first code object and the information of the second pill corresponding to the second code object are different, the processor (110) may set the number of the first pills not to be summed further. In this case, the processor (110) may set the number of the second pills identified in the plurality of images to be summed. After that, if the user resumes taking multiple images of the first pill, automatic summation can be performed following the existing automatically summed number.

[0099] FIGS. 9 and 10 are screens of a pill identification application according to one embodiment of the present disclosure.

[0100] Referring to FIGS. 9 and 10, a processor according to one embodiment may store, for a first pill, the total number of pills (e.g., 462), information about the first pill (e.g., Dutavan soft capsule (dutasteride 0.5 mg), and a plurality of captured images (e.g., a first image and a second image) by matching them. The processor may also store information regarding the time the first image was taken and information regarding the time the second image was taken by matching them. For the same pill, the automatically aggregated shooting records may be stored by matching the time taken and the captured images. Although not illustrated, the total number of pills of the first pill may be provided along with the aggregate details. The aggregate details may include the images used for aggregation, the time taken for each image, photographer information, and the number of pills included in each image. Additionally, if a user captures images for automatic pill aggregation without recognizing a code object associated with the first pill, the user may select the 'Barcode Scan' button shown in FIG. 10 to [receive] the first It can recognize code objects associated with pills and acquire and store related information.

[0101] The above-described flowchart and description are merely examples and may be implemented differently in some embodiments. For instance, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0102] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0103] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0104] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0105] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0106] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0107] When implemented in software, the techniques described above may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available media accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0108] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disk and disc include CD, laser disc, optical disc, DVD (digital versatile disc), floppy disk, and Blu-ray disc, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0109] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0110] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0111] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

1. A method for identifying pills performed by at least one processor, A step of obtaining information of a first pill corresponding to a first code object identified through a camera; A step of identifying at least one first pill included in a first image acquired through the camera; A step of indicating the number of at least one first pill included in the first image above; A step of identifying at least one first pill included in a second image acquired through the camera; and A step of displaying the total number of pills by summing the number of at least one first pill included in the first image and the number of at least one first pill included in the second image. A method for identifying pills, including 2. In Paragraph 1, A step of identifying at least one first pill included in a third image acquired through the camera; and A step of adding the number of at least one first pill included in the third image to the total number of pills mentioned above. A method for identifying pills that further include 3. In Paragraph 1, A step of storing the total number of pills, information of the first pill, the first image, and the second image by matching them with each other. A method for identifying pills that further include 4. In Paragraph 3, The above-mentioned saving step is, A step of matching and storing information regarding the time the first image was taken and information regarding the time the second image was taken. A method for identifying pills, including 5. In Paragraph 1, A step of receiving a first user input for selecting an icon to sum pills of the same type; and A step of setting the number of first pills identified in a plurality of images based on the first user input above to be summed A method for identifying pills that further include 6. In Paragraph 5, A step of verifying information of a second pill corresponding to a second code object identified through the camera; A step of comparing information of a first pill corresponding to the first code object and information of a second pill corresponding to the second code object; and A step of determining that the information of the first pill corresponding to the first code object and the information of the second pill corresponding to the second code object are different, and setting the number of the first pills not to be added further. A method for identifying pills that further include 7. In Paragraph 6, Step of setting the sum of the number of the second pills identified in multiple images A method for identifying pills that further include 8. In Paragraph 5, A step of receiving a second user input to re-select an icon for summing identical pills; and A step of setting the number of the first pills not to be added up further based on the second user input above. A method for identifying pills that further include 9. In Paragraph 1, A method for identifying a pill by overlaying at least one first graphic object onto at least one first pill of the first image above.

10. A computer program stored on a computer-readable recording medium for executing a method according to any one of paragraphs 1 through 9 on a computer.

11. In an electronic device, camera; Display; and It includes at least one processor, The above-mentioned at least one processor is, Information of a first pill corresponding to a first code object identified through the above camera is obtained, and Identifying at least one first pill included in a first image acquired through the camera, and Indicating the number of at least one first pill included in the first image above, and Identifying at least one first pill included in a second image acquired through the camera, and An electronic device configured to display a total number of pills by summing the number of at least one first pill included in the first image and the number of at least one first pill included in the second image.

Images