System and method for supporting product inspection work

Abstract

A system and method are provided for transmitting recognition results for products included in an inspection list to a server side based on the gestures of an inspector. [Solution] A glasses-type wearable device recognizes products as inspection targets and provides the inspection results to a system server. In the glasses-type wearable device, an imaging unit captures an image of the real world, including the ID codes on the product packaging and the gestures of the inspector. An analysis unit identifies a product in the real world image as a product designated as the current search target from the inspection list based on the scanned value of the ID code. A display unit displays an AR object, including a visual indication to assist in searching for the product, superimposed on the product identified as the search target. A detection unit detects a message corresponding to a gesture made toward the search target. A recognition unit selects an inspection target from the search targets based on the message and recognizes the inspection results.

Description

[Technical Field]

[0001] The present invention relates to a system and method for supporting product inspection (including sorting) required for product delivery from a distribution warehouse to a customer facility via a wholesale office. [Background technology]

[0002] In the field of pharmaceutical logistics, the application of automation and support technologies is progressing for logistics warehouse operations, as centralized management and concentrated investment is suitable. However, the process of delivering products (pharmaceuticals) from logistics warehouses to customer facilities (e.g., pharmacies, hospitals, etc.) via wholesale offices still requires the cumbersome task of sorting products.

[0003] The complexity of sorting medicines is due to the similarities in product names and packaging designs across a wide range of medicines. In addition, the number of pharmacies (including drugstores) across the country exceeds the number of convenience stores, and the similarities in the names of these stores also contribute to the complexity of sorting medicines.

[0004] Under these circumstances, there is a risk that pharmaceutical delivery personnel will send the wrong products (for example, the wrong medicine, the wrong delivery address, etc.) To avoid these risks, pharmaceutical wholesale offices must carry out strict inspections (including sorting) before delivering products, which increases the burden on delivery personnel.

[0005] Conventionally, as a technology for supporting work in logistics warehouses, for example, Patent Document 1 discloses that when picking inventory items, the use of a wearable eyeglass-type terminal that enables AR (Augmented Reality) display eliminates the need to operate handheld tools such as handheld terminals, allowing on-site personnel to be hands-free and thus facilitating on-site inspection work.

[0006] Furthermore, Patent Document 2 discloses that in picking work for inventory items, a work support device that utilizes an AR solution is used to detect optical features (e.g., one-dimensional codes, markers, etc.) contained in paper tags attached to the items or their storage locations using a camera, and then display picking instructions corresponding to the captured optical features. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-48024 [Patent Document 2] Patent No. 6401611 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the glasses-type wearable terminal disclosed in Patent Document 1, it was difficult to recognize the gestures of the inspector as operations on one of multiple physical items whose ID codes had been recognized, and to communicate the inspection intention for that item to the system.

[0009] Furthermore, in the work support device disclosed in Patent Document 2, the optical features (including AR markers) provided on the item or the storage location of the item are used as the basis for recognizing the item or the storage location of the item, so it was difficult to communicate to the system the inspection intention based on the recognition results for each item being inspected, for example, through the gestures of the inspector.

[0010] The present invention has been made to solve these problems, and aims to provide a system and method for supporting the inspection of products (including sorting) required for product delivery, which enables the inspector's recognition results for each product subject to inspection to be transmitted to the system based on the inspector's gestures, etc., and enables the inspector to obtain detailed information by reading and scanning the ID codes (e.g., one-dimensional codes, two-dimensional codes, etc.) of products that appear in the inspector's field of vision, thereby making product management more efficient and accurately automating the recording of the work history of each process, thereby improving traceability. [Means for solving the problem]

[0011] In order to solve the above problems, as one aspect of the present invention, the system is designed to enable an inspector to assist in the inspection (sorting) of products that are included in an inspection list consisting of multiple inspection targets, have packaging containing ID codes, and are to be delivered, and includes the following components: (1) A wearable eyeglass-type device that provides an inspector with a visual view of the real world, recognizes products as inspection targets, and generates an inspection record as the inspector's inspection results for the products. (2) A system server that receives the inspection results provided by the glasses-type wearable device and manages the delivery of the products.

[0012] In addition, the glasses-type wearable device includes the following components: (1-1) An imaging unit that captures an image of the real world, including the ID code on the package and the gestures of the inspector, and generates a real-world image. (1-2) An analysis unit that analyzes the real image and identifies the product included in the real image based on the scanned value of the ID code as the product designated as the current search target from the inspection list. (1-3) A display unit that displays an AR (Augmented Reality) object containing a visual indication to assist in the search of an item, superimposed on the inspector's visual field or on a real-world image of an item that has been identified as a search target. (1-4) a detection unit for detecting a message corresponding to a gesture made toward a search target; and (1-5) A recognition unit that selects the search target as an inspection target based on the message and recognizes the inspection result for the inspection target.

[0013] In this system, the glasses-type wearable device is an AR glass that realizes an augmented reality experience by overlaying AR objects on the visual field of the inspector or on real-world images.

[0014] In this system, the ID code is a one-dimensional or two-dimensional code for identifying the product to be inspected, and corresponds to the product code, serial number, and expiration date.

[0015] In addition, in this system, gestures are (1) Involves maintaining an inspector's finger at a specific location on the package being searched for a period of time equal to or greater than a predetermined threshold time; (2) tracing the inspector's specific finger over a specific area on the package being searched; or (3) The inspector holds the package to be searched in one hand and, while keeping it within the field of view of the eyeglass-type wearable device, brings the distance between the inspector's other hand and the eyeglass-type wearable device within a predetermined threshold distance. It is one of the following.

[0016] In addition, in this system, the detection unit of the glasses-type wearable device recognizes the hand movements of any of the inspectors based on hand tracking technology, which uses a tracking sensor to detect the hand movements of the subject in real time.

[0017] In addition, in this system, the tracking sensor is configured to detect the position of the inspector's hands relative to the real-world image by emitting infrared light and detecting the reflected light to measure the movement of the inspector's hands.

[0018] In addition, in this system, the detection unit of the glasses-type wearable device acquires the three-dimensional shape of one of the inspector's hands and the coordinates of its feature points based on measuring the three-dimensional shape of the object using a 3D scanner.

[0019] In this system, the glasses-type wearable device further includes a communication interface unit that transmits product inspection results to the system server based on a short-range wireless communication standard.

[0020] Furthermore, as one aspect of the present invention, the method is for enabling support for an inspector's inspection (sorting) of products that are included in an inspection list consisting of multiple inspection targets, have packaging containing ID codes, and are to be delivered, and includes the following steps as components: (1) A first step in which a wearable eyeglass-type device that provides an inspector with a visual view of the real world recognizes products as inspection targets and generates an inspection record as the result of the inspector's inspection of the products; and (2) The second step is for the system server to receive the inspection results provided by the glasses-type wearable device and manage the delivery of the product.

[0021] Additionally, the first step comprises the following sub-steps: (1-1) A substep of capturing an image of the real world including the ID code on the package and the gestures of the inspector using an imaging unit of the glasses-type wearable device to generate a real-world image. (1-2) A substep in which an analysis unit of the glasses-type wearable device analyzes the real-world video and identifies the product included in the real-world video based on the scanned value of the ID code as the product designated as the current search target from the inspection list. (1-3) A sub-step of superimposing an AR (Augmented Reality) object containing a visual indication to assist in searching for the product on the display unit of the glasses-type wearable device, onto the product that is included in the visual field of the inspector or in the real image and that has been identified as a search target. (1-4) a sub-step of detecting a message corresponding to a gesture made toward the search target by a detection unit of the glasses-type wearable device; and (1-5) A sub-step of selecting the search target as an inspection target by a recognition unit of the glasses-type wearable device based on the message and recognizing the inspection result for the inspection target.

[0022] The glasses-type wearable device is an AR glass that provides an augmented reality experience by overlaying AR objects on the inspector's visual field or real-world images. [Effects of the Invention]

[0023] According to the present invention, the recognition results for each product subject to inspection can be transmitted to the system side based on the inspector's gestures, etc. Furthermore, according to the present invention, by reading and scanning the ID code (e.g., one-dimensional code, two-dimensional code, etc.) of the product that comes into the inspector's field of vision, detailed information can be obtained, making product management more efficient, and traceability can be improved by correctly automating the recording of the work history of each process. [Brief explanation of the drawings]

[0024] A more detailed understanding of the embodiments disclosed herein can be had from the following description, taken in conjunction with the accompanying drawings, in which:

[0025] [Figure 1]1 is a configuration diagram showing a connection configuration of a system according to an embodiment of the present invention; [Figure 2] 2 is a diagram showing the external configuration of the eyeglass-type wearable device and the medicine package shown in FIG. 1. FIG. [Figure 3] 3 is a configuration diagram showing the internal configuration of a device unit of the glasses-type wearable device shown in FIG. 2. FIG. [Figure 4] FIG. 2 is a configuration diagram showing the internal configuration of the medicine management system server shown in FIG. [Figure 5] 2 is a flowchart showing an inspection process in the system shown in FIG. 1. [Figure 6] 3 is a diagram showing AR display on the display of the glasses-type wearable device shown in FIG. 2 when registering and searching for an inspection target. [Figure 7] 3 is a diagram showing an AR display on the display of the glasses-type wearable device shown in FIG. 2 when the intention to inspect is recognized. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that in multiple drawings, the same reference numerals represent the same components, and the repeated description thereof will be omitted. Also, acronyms such as AR (augmented reality) and MR (mixed reality) will be used below, but will not be explained again unless they are first mentioned.

[0027] [Connection form of this system] FIG. 1 is a configuration diagram showing a connection configuration of a system according to an embodiment of the present invention.

[0028] As shown in Figure 1, in order to assist an inspector 110 at a logistics warehouse in inspecting pharmaceutical packages 120 that are to be delivered from the logistics warehouse to a customer facility via a wholesale office, a glasses-type wearable device 130 worn by the inspector 110 captures and analyzes images of the pharmaceutical package 120 to be inspected and the inspector's 110's own gestures, and then transmits inspection results (inspection record) information for the pharmaceuticals to a pharmaceutical management system server 140 via a communication network 100.

[0029] Here, the eyeglass-type wearable devices 130 worn by the inspectors 110 are connected wirelessly to the drug management system server 140 via the communication network 100. The drug management system server 140, which manages the drug packages 120, is further connected wirelessly or via wire to an external drug-related system (not shown) via the communication network 100.

[0030] The communication network 100 is a network that realizes data communication within a relatively small range based on, for example, a short-range wireless communication standard (e.g., Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.), and may be connected to an external public line network that transmits telephone communications or data communications, or may be connected to a network that includes as part thereof a dedicated line network that provides limited communication between specific systems.

[0031] The glasses-type wearable device 130 is a pair of binocular AR or MR glasses that fuses real and virtual images as a cross-reality. More specifically, the glasses include any of the following: (1) Optical see-through / video see-through binocular AR (Augmented Reality) glasses that realize an augmented reality experience by overlaying AR objects on the visual field of the inspector or real-world images. Or, (2) Binocular MR (Mixed Reality) glasses that overlay virtual images, including AR objects, on real images as mixed reality.

[0032] Furthermore, the medicine management system server 140 manages tasks related to ordering, receiving and shipping, inspection, delivery, billing, etc. of medicines as part of the overall management of the medicine package 120. Before the inspection work, the medicine management system server 140 provides detailed information about the medicine to the glasses-type wearable device 130 via the communication network 100 or a computer-readable storage medium, and after the inspection work, receives inspection result information about the medicine package 120 from the glasses-type wearable device 130 via the communication network 100 or a computer-readable storage medium.

[0033] [Each component of the glasses-type wearable device and pharmaceutical package] FIG. 2 is a diagram showing the external configuration of the eyeglass-type wearable device 130 and the medicine package 120 shown in FIG.

[0034] 2(a), the eyeglass-type wearable device 130 worn by the inspector 110 has a device unit 200, a display unit 210, a sensor unit 220, and a battery unit 230 mounted interchangeably along both sides of a regular eyeglass frame. The device unit 200 is installed on each of the armor parts connecting the rim and temple parts of the eyeglass frame, and incorporates various components for enabling AR objects to be displayed on each optical lens in the eyeglass frame.

[0035] The display units 210 are installed on each rim portion that supports the two lenses fitted to the front side of the eyeglass frame. A coating resin containing fine grooves is applied to the surface of each of these lenses. A portion of the resin-coated lens surface is configured as an organic EL (Electro-Luminescence) display, which refracts the virtual image output from the display units 210 in a complex manner, allowing the virtual image to be brought into the field of view of the inspector 110.

[0036] The sensor unit 220 is installed on each rim and each temple of the eyeglass frame, and has a built-in small camera and a built-in small sensor for generating AR objects. The battery unit 220 is installed on each temple of the eyeglass frame, and has a built-in battery for supplying power to the display unit 200 and the sensor unit 210.

[0037] As shown in FIG. 2(b), the pharmaceutical package 120 to be inspected may be implemented in various ways, such as an outer package for a single pharmaceutical, a shipping box for packaging a fixed number of boxed pharmaceuticals, or a shipping container for loading a fixed number of pharmaceutical boxes. A pharmaceutical identification label 250 is affixed to the surface of the pharmaceutical package 120, which may be implemented in various ways. The pharmaceutical identification label 250 contains various information necessary for inspection and delivery. Examples of information written on the pharmaceutical identification label 250 include the name of the receiving facility (including the order date and time), the name, strength, and quantity of the pharmaceutical, an ID code, the pharmaceutical strength in enlarged form, and the name of the shipping facility (including the scheduled shipping date and time).

[0038] In particular, the ID code is expressed in the form of a one-dimensional code (e.g., a barcode) or a two-dimensional code (e.g., a QR Code (registered trademark)) and serves as an identifier for extracting various pharmaceutical information, such as product code, expiration date, serial number, and quantity, from the device unit 200 or the pharmaceutical management system server 140. More specifically, the product code system conforms to, for example, the Global Trade Item Number (GTIN), and in the case of a GTIN-14 code (sales packaging unit and original packaging unit), it is composed of an index (1 digit) + GS1 (Global Standard 1) company code (7 digits) + product item code (5 digits) + check digit (1 digit). In this case, the form of the one-dimensional or two-dimensional code includes GS1 DataBar limited type, GS1 DataBar double-layer type, a composite symbol thereof (CC-A), or a GS1-128 symbol.

[0039] The GTIN-13 code (prescription packaging unit), which is the GTIN-14 code without the index, is synonymous with the JAN (Japanese Article Number) code. The expiration date, for example, is expressed as a six-digit number, synonymous with the expiration date. The serial number, for example, is expressed as a six-digit code for manufacturing identification. Furthermore, the quantity is not displayed for the prescription packaging unit (the smallest packaging unit manufactured and sold) or the sales packaging unit (the smallest packaging unit sold by wholesalers to medical institutions, etc.), but is displayed as the number of sales packaging units in the original packaging unit (a packaging unit in which multiple sales packaging units are packed by the manufacturer / distributor).

[0040] [Internal structure of the device unit of the glasses-type wearable device] FIG. 3 is a diagram showing the internal configuration of device unit 200 of glasses-type wearable device 130 shown in FIG.

[0041] 3, the device unit 200 of the glasses-type wearable device 130 is configured with various internal components so as to be able to display AR objects on the surface of each lens fitted into the glasses frame. More specifically, in the device unit 200, a control unit 310, a main memory unit 312, a non-volatile memory unit 314, an imaging unit 316, a detection unit 318, a display unit 320, a power supply unit 322, a communication IF (Interface) unit 324, and an auxiliary memory unit 330 are connected to each other via a system bus 300 so as to be able to communicate with each other.

[0042] The auxiliary storage unit 330 contains a video capture unit 340, a video analysis unit 342, an image control unit 344, a code detection unit 346, a gesture detection unit 348, an inspection recognition unit 350, and a drug information extraction unit 352. When the programs held in these components are loaded into the main storage unit 312 based on a call instruction from the control unit 310, each application program constructed by this loading executes various calculation processes that handle the display of AR objects and the processing of various data related to the inspection work.

[0043] Additionally, auxiliary storage unit 330 includes a medicine management data storage unit 360, a gesture extraction data storage unit 362, an inspection result data storage unit 364, and a program data storage unit 366. When the data and programs stored in these components are loaded into main storage unit 312 based on a call instruction from control unit 310, the subsystem constructed by this loading is used to process various calculations that manage data, programs, etc. in the form of files / databases.

[0044] The control unit 310 functions as a central processing unit (CPU) to control the operation of individual system components and perform data calculations, and in particular loads data and programs stored in the auxiliary storage unit 330 into the main storage unit 312 to perform various calculations. The main storage unit 312 functions as a main memory and stores various data and programs input from the imaging unit 316, the detection unit 318, the communication IF unit 324, the auxiliary storage unit 330, etc., as well as computer-executable instructions, based on instructions from the control unit 310, and stores data after calculation processing on these internally and provides the data to the outside via the non-volatile storage unit 314 or the communication IF 324.

[0045] The non-volatile storage unit 314 has a port that can be connected to a computer-readable storage medium, and provides an interface for reading detailed information about the drug package 120 provided by the drug management system server 140 into the computer-readable storage medium attached to this port, and for writing inspection results for the drug package 120. The imaging unit 316 has an RGB camera (e.g., a CCD (Charge Coupled Device) image sensor, etc.) that captures moving images, and provides an interface for outputting images of the real world captured by this RGB camera.

[0046] The detection unit 318 has a distance sensor that detects reflected light from a real object onto which measurement light is irradiated, a depth sensor that detects infrared rays from the real object, a tracking sensor that detects the movement of the real object's hands, a motion sensor that detects the movement of the real object's head, and a 3D scanner that measures the three-dimensional shape of the real object, and provides an interface that outputs the distance, depth, 3D coordinates, orientation of each part, and three-dimensional shape of the real object detected by these sensors.

[0047] The distance sensor is, for example, a LiDAR (Light Detection and Ranging) sensor, which measures the distance to the pharmaceutical package 120 to be inspected or the hand of the inspector 110. The depth sensor is, for example, a TOF (Time of Flight) sensor, which measures the relative position of a virtual AR object to the pharmaceutical package 120 to be inspected. The tracking sensor is, for example, a combination of an infrared LED (Light Emitting Diode) that emits infrared light to measure the movement of the inspector 110's hand, an infrared sensor that detects the reflected light, and a camera, which tracks key points on the inspector's 110 hand over time in real time and measures the movement of the hand. The 3D scanner is, for example, a non-contact laser scanner, which measures the three-dimensional shape of the pharmaceutical package 120 to be inspected or the hand of the inspector 110 and acquires the coordinates of those feature points.

[0048] The motion sensor is an inertial sensor including, for example, an acceleration sensor and an angular velocity sensor, and measures the movement of the head of the inspector 110 wearing the eyeglass-type wearable device 130. The acceleration sensor is, for example, a capacitance accelerometer, and detects the inertial force corresponding to the linear movement of the head of the inspector 110. The angular velocity sensor is, for example, a gyro sensor, and detects the inertial force corresponding to the rotational movement of the head of the inspector 110.

[0049] The display unit 320 outputs display data for the AR object generated by the image control unit 344, and provides an interface for displaying the AR object on a display on the surface of each eyeglass lens connected to the display unit 210. This AR object sequentially displays a list (inspection list) of multiple inspection target candidates corresponding to each scanned value (product code) of the ID code in multiple inspection slips on which the inspection targets are written, in the order in which they are read, or tracks and displays a wireframe that encircles and clarifies the periphery of the inspection target sequentially designated as a search target from the inspection list.

[0050] The power supply unit 322 has a power supply that supplies driving power to built-in components such as the device unit 200, display 210, and sensor unit 220, and adjusts and manages the power supply state to the target components. This power supply is, for example, a small battery including a lithium-ion battery, and has a port that allows charging from an external source. The communication IF unit 324 provides an interface that functions when sending and receiving various data to and from the drug management system server 140 and other internal components.

[0051] More specifically, the video capture unit 340 generates a real-world image from the imaging data provided by the imaging unit 316 regarding the drug identification label 250 affixed to the drug package 120 and the gesture of the inspector 110, and provides this to the video analysis unit 342. The video analysis unit 342 analyzes the real-world image provided by the video capture unit 340. Here, the video analysis unit 342, in combination with the sensor data provided by the detection unit 318, performs object detection and object recognition using computer vision to identify the external characteristics of the drug package 120 and the drug currently being searched for, and, if it detects an ID code candidate in the drug identification label 250 included in the real-world image, provides the image to the code detection unit 346. Furthermore, if the video analysis unit 342, in combination with the sensor data provided by the detection unit 318, detects a gesture candidate of the inspector 110 included in the real-world image, it provides the image to the gesture detection unit 348.

[0052] Here, computer vision uses machine learning, such as deep learning, and improves the accuracy of predictions through a series of iterations until predictions become possible through convolution operations using neural networks for pattern recognition, which is a visualization of machine learning. This allows computer vision to achieve accurate image analysis, and the more it is used on the image being analyzed, the more accurate the pattern recognition becomes.

[0053] The image control unit 344 generates, as AR objects, object representations (including visual presentations that support the search for products) that represent a list of multiple inspection target candidates (inspection list) provided by the code detection unit 346 or a wireframe of the medicine designated as the current search target, and provides these to the display unit 320. Furthermore, the image control unit 344 generates, as AR objects, detailed information of the detected medicine package 120 provided by the medicine information extraction unit 352, and provides this to the display unit 320. The code detection unit 346 reads and scans ID code candidates in multiple medicine packages 120 detected in the field of view of the inspector 110 in the image included in the real video provided by the video analysis unit 342, and provides to the inspection recognition unit 350 a matching result for the medicine being searched for based on the scan values ​​of each of these ID codes.

[0054] The gesture detection unit 348 determines whether or not any gesture candidate of the inspector 110 detected within the field of view of the inspector 110 in the image included in the real video provided by the video analysis unit 342 matches any of the gestures stored in the gesture extraction data storage unit 362, and provides the determination result to the inspection recognition unit 350. Here, the gesture of the inspector 110 is detected by determining the relative coordinates between the knuckles of a specific finger (e.g., the index finger) and other fingers, as well as the relative coordinates indicating the center position of the hand, based on hand tracking technology, and determining the state of each finger and hand.

[0055] The state of each finger and hand is determined based on, for example, any of the following: (1) The index finger is extended in a straight line. (2) All fingers except the index finger are bent. (3) The angle between the vector from the relative coordinate indicating the center position of the wrist (wrist joint) to the third joint at the base of the index finger and the vector from the third joint of the index finger to the first joint closest to the fingertip is within a predetermined threshold angle range; and (4) The index finger is pointing toward one of the multiple candidates for inspection.

[0056] If the gesture determination result provided by the gesture detection unit 348 is approval, the inspection recognition unit 350 extracts a message matching the gesture (e.g., start of inspection, instruction of intention to inspect, completion of inspection, etc.) from the gesture extraction data storage unit 362 based on the gesture of the inspector 110. Furthermore, the inspection recognition unit 350 determines one of the multiple inspection candidate pharmaceutical packages 120 as the search target based on the direction of the pointing finger included in the gesture of the inspector 110 and the three-dimensional shape of the pharmaceutical package 120 located on an extension of that direction, together with the matching results based on each scan value of the ID code within the pharmaceutical package 120, and provides the determination result to the pharmaceutical information extraction unit 352. Furthermore, the inspection recognition unit 350 determines whether the inspection result is approval based on the gesture of the inspector 110, and stores the inspection result information in the inspection result data storage unit 364 or a computer-readable storage medium connected to the non-volatile storage unit 314.

[0057] The determination of whether or not the inspector intends to inspect the medicine package 120 indicated by the index finger of the inspector 110 is to select the medicine package 120 as an object for inspection is made based on, for example, any of the following: (1) The inspection includes having the inspector 110 point at a specific location on the package to be searched for a period of time equal to or greater than a predetermined threshold time. (2) tracing the specific finger of the inspector 110 above a specific location on the package being searched; or (3) The inspector 110 holds the package to be searched in one hand and, while keeping it within the field of view of the eyeglass-type wearable device 130, brings the distance between the inspector 110's other hand and the eyeglass-type wearable device 130 within a predetermined threshold distance.

[0058] When the inspection target determination result provided by the inspection recognition unit 350 is approval, the drug information extraction unit 352 extracts detailed information about the drug package 120 from the drug management data storage unit 360 based on the ID code of the drug package 120 to be inspected, and provides the detailed information to the image control unit 344. Here, the detailed information about the drug package 120 includes, for example, the name of the drug, dosage form, content, quantity, and the name of the wholesale office / customer facility to which the drug is to be delivered.

[0059] The pharmaceutical data storage unit 360 is configured to store detailed information about the pharmaceutical package 120 provided by the pharmaceutical management system server 140 and to extract data according to requests from each component of the device unit 200. Here, the detailed information about the pharmaceutical package 120 includes, for example, management data related to the ordering, receiving and shipping, inspection, delivery, and billing of the pharmaceutical package 120, such as various information such as ID, product code, expiration date, quantity, serial number, ordering, receiving and shipping, inspection, delivery, and billing.

[0060] The gesture extraction data storage unit 362 is configured to store the relative positional relationships of key points on a human body corresponding to typical human gestures and the movement vectors between the key points, and the correspondence between the content of the human's intention, i.e., the message, corresponding to the typical human gesture, and to extract data according to request from each component of the device unit 200.

[0061] The inspection result data storage unit 364 is configured to store inspection result information of the pharmaceutical package 120 by the inspector 110, which is provided by the inspection recognition unit 350, and to extract data in response to requests from each component of the device unit 200. Here, the inspection result information of the pharmaceutical package 120 includes, for example, the ID of the pharmaceutical package 120, the location of the inspection, the date and time of completion of the inspection, the name of the inspector, and various confirmation items (e.g., outer box quality, pharmaceutical product code, package quantity, serial number, manufacturing lot, manufacturing date, expiration date, consumption period, etc.).

[0062] The program data storage unit 366 stores programs for executing a series of processes executed as a transaction or other individual processes, and receives calls to these programs from the control unit 310.

[0063] [Internal configuration of the pharmaceutical management system server] FIG. 4 is a configuration diagram showing the internal configuration of the medicine management system server 140 shown in FIG.

[0064] 4, the medicine management system server 140 is configured similarly to a computer for general server use and supports processing functions related to various tasks, such as ordering, receiving and shipping, inspection, delivery, and billing, for the medicine packages 120, and in particular applies inspection results provided by the glasses-type wearable device 130 to the inspection tasks. More specifically, in the medicine management system server 140, a control unit 410, a main memory unit 412, an input unit 414, an output unit 416, a communication IF 418, and an auxiliary memory unit 420 are communicatively connected to each other via a system bus 400. Of these components, the control unit 410, the main memory unit 414, the communication IF unit 418, and the like basically operate in the same way as the respective components of the glasses-type wearable device 130, and therefore will not be described in detail again.

[0065] The input unit 414 provides an interface for receiving various commands and data (e.g., various master and table data) input by a system operator. The output unit 416 provides an interface for generating output data for displaying various processed data to an operator and output data for printing the data.

[0066] The auxiliary memory unit 420 contains a pharmaceutical management unit 430, an inspection management unit 432, an order management unit 434, an inventory management unit 436, an inspection management unit 438, and a billing management unit 440. When the programs held in these components are loaded into the main memory unit 412 based on a call instruction from the control unit 410, each application program constructed by this loading executes various calculation processes that handle processes such as ordering, inventory management, inspection, delivery, and billing related to pharmaceutical packages 120.

[0067] The auxiliary memory unit 420 also contains a pharmaceutical management data storage unit 450, an inspection management data storage unit 452, an order management data storage unit 454, an inventory management data storage unit 456, a delivery management data storage unit 458, a billing management data storage unit 460, and a program data storage unit 462. When the data and programs stored in these components are loaded into the main memory unit 412 based on a call instruction from the control unit 410, the subsystem constructed by this loading is used to process various operations that manage data, programs, etc. in the form of files / databases.

[0068] More specifically, the pharmaceutical management unit 430 is configured to store general management data for pharmaceuticals provided from an external pharmaceutical-related system in the pharmaceutical management data storage unit 450, and to extract data in response to requests from each internal component from the pharmaceutical management data storage unit 450. The inspection management unit 432 is configured to store management data applied to inspection work on pharmaceutical packages 120 to be received or shipped in the inspection management data storage unit 452, and to extract data in response to requests from each internal component from the inspection management data storage unit 452.

[0069] Furthermore, the inspection management unit 432 provides detailed information about the pharmaceutical package 120 to be inspected to the eyeglass-type wearable device 130 via the communication network 100 or a computer-readable storage medium, regarding the pharmaceutical package 120 before inspection. Here, the detailed information about the pharmaceutical package 120 includes, for example, the ID of the pharmaceutical package 120, the name of the pharmaceutical, dosage form, content, quantity, the name of the wholesale office / customer facility of the delivery destination, the inspection period, the inspection location, the name of the inspector, and inspection confirmation items.

[0070] Furthermore, the inspection management unit 432 acquires inspection result information for the pharmaceutical package 120 after the inspection work from the glasses-type wearable device 130 via the communication network 100 or a computer-readable storage medium. Here, the inspection result information for the pharmaceutical package 120 includes, for example, the ID of the pharmaceutical package 120, the location of the inspection, the date and time of completion of the inspection, the name of the inspector, and various confirmation items (e.g., outer box quality, pharmaceutical product code, package quantity, serial number, manufacturing lot, manufacturing date, expiration date, consumption period, etc.).

[0071] The order management unit 434 is configured to store management data applied to the ordering work of the pharmaceutical packages 120 to be purchased in the order management data storage unit 454, and to extract data in response to requests from each internal component from the delivery management data storage unit 454. The inventory management unit 436 is configured to store management data applied to the receiving / shipping work of the pharmaceutical packages 120 before and after the inspection work in the inventory management data storage unit 456, and to extract data in response to requests from each internal component from the inventory management data storage unit 456.

[0072] The delivery management unit 432 is configured to store management data applied to the delivery work of the pharmaceutical package 120 after inspection is completed in the delivery management data storage unit 458, and to extract data in response to requests from each internal component from the delivery management data storage unit 458. The billing management unit 440 is configured to store management data applied to the billing work of the pharmaceutical package 120 after delivery is completed in the billing management data storage unit 460, and to extract data in response to requests from each internal component from the billing management data storage unit 460.

[0073] The program data storage unit 462 stores programs for executing a series of processes executed as a transaction or other individual processes, and receives calls to these programs from the control unit 410.

[0074] [Inspection process sequence] The inspection process in the above-described system is described below. Fig. 5 is a flowchart showing the inspection process in the system shown in Fig. 1. Fig. 6 is a diagram showing AR displays on the display of glasses-type wearable device 130 when registering and searching for an inspection target. FIG. 7 is a diagram showing an AR display on the display of the glasses-type wearable device 130 when the intention to inspect is recognized.

[0075] As shown in FIG. 5, in step S500, at the logistics warehouse, an inspector 110 puts on the eyeglass-type wearable device 130 on his / her head, starts the system startup of the data unit 200, and transitions various sensors, including the tracking sensor / motion sensor in the sensor unit 220, into an operational state, thereby preparing for the inspection of the pharmaceutical package 120.

[0076] Next, in step S510, the eyeglass-type wearable device 130, which is worn on the head of the inspector 110 and positioned facing the inspection slip placed in the logistics warehouse, captures an image of the real world including the ID code in the inspection slip specified by the pointing gesture of the inspector 110, and generates a real image within the field of view of the inspector 110 as the captured image. The eyeglass-type wearable device 130 also analyzes the real image based on sensor data generated from each built-in sensor.

[0077] For example, as shown in FIG. 6(a), by sequentially scanning each ID code written in a plurality of inspection slips corresponding to each inspection target, a plurality of inspection target candidates are sequentially registered as an inspection list through the glasses-type wearable device 130 based on each scan value (product code) of the ID code. At this time, the glasses-type wearable device 130 recognizes an instruction to scan the ID code by the inspector 110 based on a point-and-scan operation of the inspector's hand on the ID code in the inspection slip. Furthermore, inspection list information for the registration results is displayed as an AR object on the display of the glasses-type wearable device 130. Note that the inspection list may be created in advance based on already stored pharmaceutical information, in addition to reading ID codes from inspection slips.

[0078] Next, in step S520, the eyeglass-type wearable device 130 positioned opposite the pharmaceutical packages 120 placed in the logistics warehouse captures an image of the real world including the pharmaceutical packages 120, and generates a real image within the field of view of the inspector 110 as the captured image. The eyeglass-type wearable device 130 also analyzes the real image based on sensor data generated from each built-in sensor. As a result, the eyeglass-type wearable device 130 determines that an individual pharmaceutical package 120 among the multiple pharmaceutical packages 120 is the current search target based on the scan values ​​of each external appearance feature (shape, size, printing pattern, etc. of the outer box) or ID code of the pharmaceutical packages 120, and displays a wireframe of the pharmaceutical package 120 recognized as the current search target as an AR object on the display.

[0079] For example, as shown in Figure 6(b), when a medicine package 120 is present within the field of view of the real image, a wire frame of the medicine package 120 determined by computer vision to be the current search target is displayed as an AR object on the display of the eyeglass-type wearable device 130. Note that this AR display of the wire frame will be re-displayed as a tracking result even if the current search target temporarily leaves the field of view of the real image and then returns to the field of view of the real image.

[0080] Here, the sensor data includes, for example, distance data for the pharmaceutical package 120, depth data for the pharmaceutical identification marking 250 on the pharmaceutical package 120, scan data for the ID code in the pharmaceutical identification marking 250, tracking data for key points in the hand of the inspector 110, motion data for the head of the inspector 110, and three-dimensional shape data for the pharmaceutical package 120.

[0081] Next, in step S530, the glasses-type wearable device 130 detects a gesture candidate of the inspector 110 included in the real image and extracts a gesture determined to match based on the tracking results of a group of key points on the hand of the inspector 110 included in the gesture candidate in the consecutive images included in the real image. Subsequently, in step S540, the glasses-type wearable device 130 recognizes a message (e.g., start of inspection, instruction of intent to inspect, completion of inspection, etc.) corresponding to the gesture determined to match based on the point-and-scan operation of the inspector 110's hand on the pharmaceutical package 120. An example of point-and-scan is shown in FIG. 7(a).

[0082] Next, in step S550, when the glasses-type wearable device 130 detects ID code candidates in the medicine identification mark 250 on the medicine package 120 designated as an inspection target by the hand of the inspector 110, it performs a read scan on the image included in the real video and determines whether the medicine being searched for matches based on the scanned values ​​of these ID codes. Furthermore, when the glasses-type wearable device 130 displays the medicine identification mark 250 on the medicine package 120 designated as an inspection target in close proximity, it highlights specific parts of the AR object displayed as the inspection list (for example, the name of the customer facility or part of the product image (such as the content) that are easily mistaken) by adding a frame or the like. An example of the display of the AR object is shown in Figure 7(b).

[0083] Next, in step S560, the glasses-type wearable device 130 extracts detailed information about the medicine package 120 to be inspected from its internal components based on the read scan value of the ID code corresponding to the medicine package 120 confirmed as the inspection target, and additionally projects this information on each display located on both glasses lenses. Subsequently, in step S570, the glasses-type wearable device 130 sequentially repeats the above-mentioned steps S520 to S560 for each inspection target in the inspection list, and stores the inspection result information of the medicine package 120 to be inspected in its internal components, and then stores it in a computer-readable storage medium or transmits it to the medicine management system server 140 via the communication network 100.

[0084] [Effects of the embodiment] As described above, according to the embodiment of the present invention, the recognition results of the inspector for each product subject to inspection can be transmitted to the system side based on the inspector's gestures, etc. Furthermore, according to the embodiment of the present invention, by reading and scanning the ID code (e.g., one-dimensional code, two-dimensional code, etc.) of the product that comes into the field of view of the inspector, detailed information can be obtained, making product management more efficient, and traceability can be improved by correctly automating the recording of the work history of each process.

[0085] Although the principles of the present invention have been described above with reference to exemplary embodiments, it should be understood by those skilled in the art that various embodiments that may be modified in configuration and detail can be realized without departing from the spirit and scope of the present invention. That is, the present invention can be implemented in various forms, for example, as a system, an apparatus, a method, a program, or a storage medium. [Explanation of symbols]

[0086] 100 Communication Network 110 Inspector 120 medicine packages 130 Glasses-type wearable devices 140 Pharmaceutical Management System Server 200 device units 210 Display Unit 220 Sensor Unit 230 Battery Unit 250 Pharmaceutical Specific Labeling 300 System Bus 310 Control Unit 312 Main memory 314 Non-volatile memory unit 316 Imaging unit 318 Detection unit 320 Display 322 Power supply section 324 Communication IF section 330 Auxiliary storage unit 340 Videography Department 342 Video Analysis Department 344 Image Control Unit 346 Code Detector 348 Gesture detection unit 350 Inspection and Recognition Department 352 Drug Information Extraction Unit 360 Pharmaceutical Management Data Storage Unit 362 Gesture extraction data storage unit 364 Inspection result data storage unit 366 Program Data Storage Unit 400 System Bus 410 Control Unit 412 Main memory 414 Input section 416 Output section 418 Communication IF Section 420 Auxiliary storage 430 Pharmaceutical Management Department 432 Inspection Management Department 434 Order Management Department 436 Warehousing Management Department 438 Delivery Management Department 440 Billing Control Unit 450 Pharmaceutical Management Data Storage Unit 452 Inspection management data storage unit 454 Order Management Data Storage Unit 456 Intake / Outtake Management Data Storage Unit 458 Delivery result data storage unit 460 Billing Management Data Storage Unit 462 Program data storage unit

Claims

1. A system for supporting an inspector in inspecting a product that is included in an inspection list consisting of a plurality of inspection targets, the product having a package including an ID code and that is to be delivered, the system comprising: a glasses-type wearable device that provides the inspector with a view of the real world, recognizes the product as an inspection target, and generates an inspection record as an inspection result of the product by the inspector, an imaging unit that captures an image of a real world including the ID code on the package and a gesture of the inspector to generate a real image; an analysis unit that analyzes the real image and identifies the product included in the real image based on the scanned value of the ID code as the product designated as the current search target from the inspection list; a display unit that displays an AR (Augmented Reality) object, which includes a visual indication to assist in searching for the product, superimposed on the product that is included in either the visual field of the inspector or the real image and that is identified as the search target; a detection unit for detecting a message corresponding to the gesture toward the search target; a recognition unit that selects the search target as the inspection target based on the message and recognizes an inspection result for the inspection target; a glasses-type wearable device including: a system server that receives the inspection results provided by the glasses-type wearable device and manages delivery of the products; A system comprising:

2. The system according to claim 1, wherein the glasses-type wearable device is an AR glass that realizes an augmented reality feeling by superimposing the AR object on either the visual field of the inspector or the real image.

3. The system according to claim 1 , wherein the ID code is a one-dimensional code or a two-dimensional code for identifying the product to be inspected, and corresponds to the product's code, serial number, and expiration date.

4. The gesture is (1) The method includes having the inspector point at a specific location on the package to be searched for a period of time equal to or longer than a predetermined threshold time. (2) tracing a specific finger of the inspector above a specific portion of the package being searched; or (3) The inspector holds the package to be searched in one hand and brings the package into the field of view of the eyeglass-type wearable device while bringing the distance between the other hand of the inspector and the eyeglass-type wearable device within a predetermined threshold distance.

2. The system according to claim 1, wherein the system is one of the following:

5. The system described in claim 1, characterized in that the detection unit of the glasses-type wearable device recognizes the movement of any of the inspector's hands based on hand tracking technology that detects the hand movements of the subject in real time using a tracking sensor.

6. The system of claim 5, wherein the tracking sensor is configured to detect the position of the inspector's hands relative to the real image by irradiating infrared light and detecting reflected light to measure the movement of the inspector's hands.

7. The system described in claim 5, characterized in that the detection unit of the glasses-type wearable device acquires the three-dimensional shape of one of the inspector's hands and the coordinates of feature points based on measuring the three-dimensional shape of the object using a 3D scanner.

8. The system according to claim 1 , wherein the glasses-type wearable device further includes a communication interface unit that transmits the inspection results of the products to the system server based on a short-range wireless communication standard.

9. A method for supporting an inspector in inspecting a product that is included in an inspection list consisting of a plurality of inspection targets and has a package including an ID code and is to be delivered, the method comprising: a step of recognizing the product as an inspection target using a glasses-type wearable device that provides the inspector with a view of the real world, and generating an inspection record as an inspection result of the product by the inspector, a sub-step of capturing an image of a real world including the ID code on the package and the gesture of the inspector by an imaging unit of the glasses-type wearable device to generate a real image; a sub-step of analyzing the real image by an analysis unit of the glasses-type wearable device and identifying the product included in the real image based on the scanned value of the ID code as the product designated as the current search target from the inspection list; a sub-step of displaying, by a display unit of the glasses-type wearable device, an augmented reality (AR) object including a visual presentation to assist in searching for the product, superimposed on the product that is included in either the visual field of the inspector or the real image and that is identified as the search target; a sub-step of detecting a message corresponding to the gesture made toward the search target by a detection unit of the glasses-type wearable device; a sub-step of selecting the search target as the inspection target based on the message by a recognition unit of the glasses-type wearable device, and recognizing an inspection result for the inspection target; and receiving the inspection result provided by the glasses-type wearable device by a system server and managing delivery of the product; A method comprising:

10. The method according to claim 9, wherein the glasses-type wearable device is an AR glass that realizes an augmented reality by superimposing the AR object on either the visual field of the inspector or the real image.

Images