Image analysis server, object counting method using image analysis server, and object counting system

Abstract

The present application relates to an image analysis server, an object counting method using the image analysis server, and an object counting system. According to an embodiment of the present application, an object counting method using an image analysis server can be provided, including: a step of receiving, by a user terminal, an image including one or more objects; a step of forming, by the image analysis server, a plurality of frames for each of the objects, and retaining only a number of frames corresponding to the objects among the plurality of frames and deleting the remaining frames; and a step of counting, by the image analysis server, the number of retained frames, and transmitting the number of frames corresponding to the objects to the user terminal.

Description

Technical Field

[0001] This invention relates to an image analysis server, an object counting method using the image analysis server, and an object counting system. Background Technology

[0002] As society ages, patients' need to visit hospitals increases, and consequently, the types and quantities of medications required also increase.

[0003] However, manually counting pills is very inconvenient in small pharmacies or hospitals when administering medication to patients or taking inventory. Moreover, when counting pills manually, errors frequently occur, such as administering fewer or more pills than prescribed.

[0004] To address this problem, large pharmacies and hospitals have introduced and used pill counting devices, but these devices are expensive, making them practically unavailable to small pharmacies or hospitals. Summary of the Invention

[0005] Technical problems to be solved

[0006] The embodiments of the present invention are proposed to solve the problems described above, and aim to provide an image analysis server, an object counting method and an object counting system using the image analysis server, which can easily count the number of objects (e.g., pills) without introducing complex and expensive equipment.

[0007] Furthermore, it is desirable to provide an image analysis server, an object counting method using the image analysis server, and an object counting system that can accurately count the number of closely spaced objects (e.g., pills).

[0008] Technical solution

[0009] According to an embodiment of the present invention, an object counting method using an image analysis server can be provided. The method includes: receiving an image including one or more objects through a user terminal; forming multiple bounding boxes for each object through the image analysis server, and retaining only the number of boxes corresponding to the object and deleting the remaining boxes; and counting the number of retained boxes through the image analysis server and transmitting the number of boxes to the user terminal.

[0010] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein the step of forming multiple bounding boxes for each object by the image analysis server, and retaining only the number of boxes corresponding to the object and deleting the remaining boxes in the multiple bounding boxes includes: executing an object recognition deep learning model through a box setting module, and forming multiple bounding boxes for each object.

[0011] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein, after the step of forming multiple boxes for each object, an algorithm for removing a portion of the boxes formed for each object is executed by a first box removal module.

[0012] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein, after an algorithm for removing a portion of the boxes formed in multiple boxes for each object is executed by a first box removal module, a step is executed by a second box removal module to retain only one box for an object and delete the remaining boxes.

[0013] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein the step of retaining only one box for an object and deleting the remaining boxes by the second box removal module includes: a step of setting any box among the retained boxes as a reference box by a reference box setting unit; a step of setting a set box by a set box setting unit, wherein the set box is a collection of multiple boxes overlapping with the reference box; a step of removing the space in the space occupied by the reference box that overlaps with the set box by a comparison space setting unit, and setting the retained space in the reference box as a comparison space; and a step of retaining the box set as the reference box by a box removal unit based on a pill coefficient comparison if the ratio of the comparison space to the space occupied by the reference box is greater than the pill coefficient, and removing the box set as the reference box if the ratio of the comparison space to the space occupied by the reference box is less than the pill coefficient.

[0014] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein the object recognition deep learning model executed by the box setting module is RetinaNet.

[0015] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein the algorithm for removing a portion of the boxes formed by multiple boxes for each object, executed by the first box removal module, is non-maximum suppression (NMS).

[0016] Furthermore, an object counting method utilizing an image analysis server can be provided, wherein the pill coefficients are stored in a database based on the size and shape of the objects, and the pill coefficient judgment module matches the pill coefficients stored in the database based on the size and shape of the objects displayed in the image.

[0017] According to another embodiment of the present invention, an image analysis server can be provided, wherein an image including one or more objects is received from a user terminal, multiple boxes are formed for each object, only a number of boxes corresponding to the object are retained in the multiple boxes and the remaining boxes are deleted, the number of retained boxes is counted, and the number corresponding to the boxes is transmitted to the user terminal.

[0018] Furthermore, an image analysis server can be provided, wherein the image analysis server includes: a bounding box setting module, which executes an object recognition deep learning model and forms multiple bounding boxes for each object; a first bounding box removal module, which is capable of executing an algorithm for removing a portion of the bounding boxes formed for each object; and a second bounding box removal module, which retains only one bounding box for an object and deletes the remaining bounding boxes.

[0019] Furthermore, an image analysis server can be provided, wherein the second box removal module includes: a reference box setting unit, which sets any box among the retained boxes as a reference box; a set box setting unit, which sets a set box, the set box being a collection of multiple boxes overlapping with the reference box; a comparison space setting unit, which removes the space in the space occupied by the reference box that overlaps with the set box, and sets the space retained in the reference box as a comparison space; and a box removal unit based on a pill coefficient comparison, which retains the box set as the reference box when the ratio of the comparison space to the space occupied by the reference box is greater than the pill coefficient, and removes the box set as the reference box when the ratio of the comparison space to the space occupied by the reference box is less than the pill coefficient.

[0020] Furthermore, an image analysis server can be provided, wherein the image analysis server further includes: a database for storing pill coefficients based on the size and shape of the object; and a pill coefficient judgment module for matching the pill coefficients stored in the database according to the size and shape of the object displayed in the image.

[0021] According to another embodiment of the present invention, an object counting system can be provided, the system comprising: a user terminal capable of receiving an image including one or more objects; and an image analysis server capable of forming multiple bounding boxes for each object, and capable of retaining only the number of boxes corresponding to the object among the multiple bounding boxes and deleting the remaining boxes and counting the retained boxes.

[0022] Beneficial effects

[0023] The image analysis server, object counting method, and object counting system according to embodiments of the present invention have the effect of easily counting the number of objects (e.g., pills) without introducing complex and expensive equipment.

[0024] Furthermore, it has the advantage of being able to accurately count the number of closely spaced objects (e.g., pills). Attached Figure Description

[0025] Figure 1 A diagram illustrating an embodiment of the object counting system of the present invention is provided.

[0026] Figure 2 For illustrative purposes only Figure 1 A diagram showing the structure of an image analysis server.

[0027] Figure 3 For illustrative purposes only Figure 2 The second box of the image analysis server removes the sub-components of the module.

[0028] Figure 4 To illustrate the use of Figure 1 The flowchart shows the object counting method of the image analysis server executed by the object counting system.

[0029] Figure 5 To show in more detail Figure 4 The flowchart of step S2 in steps S1 to S3.

[0030] Figure 6 To show in more detail Figure 5 The flowchart of step S36 in steps S32 to S36.

[0031] Figure 7 To conceptually illustrate through Figure 1 The diagram shows the user terminal receiving the object.

[0032] Figure 8 To conceptually illustrate through Figure 1 The image analysis server is used to execute the RetinaNet deep learning model for object recognition and to generate a graph of multiple bounding boxes for a pill.

[0033] Figure 9 To conceptually illustrate through Figure 1 An image analysis server performs non-maximum suppression (NMS) as an algorithm for removing bounding boxes, thereby forming a bounding box graph for a pill (object).

[0034] Figure 10 To conceptually illustrate and aid understanding Figure 6 The diagram shows step S36.

[0035] Figure 11 For illustrative purposes, the use of Figure 1The flowchart illustrates an object counting system that transmits multiple images to an image analysis server and counts one or more objects included in each of the multiple images.

[0036] Figure 12 To illustrate the display in single analysis mode and multiple analysis modes Figure 1 A screenshot of the user terminal screen.

[0037] Figure 13 In multi-analysis mode, this displays the number and type of objects included in multiple images analyzed by an image analysis server. Figure 1 The image shown on the user terminal screen.

[0038] Figure 14 To illustrate the possible placement Figure 1 A diagram of multiple analytical aids for user terminals.

[0039] Figure 15 This is an illustrative example of how to easily implement multiple analysis modes. Figure 14 The diagram shows multiple analytical aids and moving belts. Detailed Implementation

[0040] Figure 1 A diagram illustrating an object counting system 1 according to an embodiment of the present invention is provided.

[0041] Reference Figure 1 The object counting system 1 may include an image analysis server 10, a user terminal 20, and an administrator terminal 30.

[0042] The image analysis server 10, user terminal 20, and administrator terminal 30 can be configured as independent devices and communicate with each other via the communication network 40. Alternatively, the image analysis server 10 and administrator terminal 30 can be configured as a single physical device and communicate directly with each other.

[0043] In this embodiment, the image analysis server 10, user terminal 20, and administrator terminal 30 are described as additional independent devices.

[0044] The object counting system 1 of this embodiment can be understood as a system that can accurately count the number of objects included in an image.

[0045] Specifically, if a user takes a picture of an object through the user terminal 20, the image including the captured object can be transmitted to the image analysis server 10, and the image analysis server 10 can count the number of objects displayed in the image based on a preset algorithm.

[0046] In this embodiment, a pill with a defined shape is used as an example for explanation. When the object photographed by the user terminal 20 is a pill, the object counting system 1 of this embodiment can be understood as a system for counting the number of pills that can be used in pharmacies and hospitals.

[0047] However, the idea of ​​the present invention is not limited to this, and the object can include all objects having a prescribed shape.

[0048] Image analysis server 10 can be understood as a server that receives image data from user terminal 20 and processes the data required to count the number of objects displayed in the image.

[0049] An image can contain objects of the same type with the same size and shape. That is, the image analysis server 10 can count the identical objects contained in an image.

[0050] However, the idea of ​​the present invention is not limited to this. The objects included in an image can be different types of objects with different sizes and shapes. In this case, the image analysis server 10 can also count the different types of objects included in an image.

[0051] User terminal 20 can capture images of objects placed on the object board and display them as images.

[0052] Furthermore, the user terminal 20 is a device that can communicate with the image analysis server 10, and can be a fixed terminal implemented as a mobile terminal or a computing device.

[0053] For example, user terminal 20 may include smartphones, laptops, tablets, wearable devices, computers, etc., that have a camera capable of capturing images of objects. However, user terminal 20 is not limited to these examples and may also be configured as a standalone camera.

[0054] The administrator terminal 30 can be understood as a device capable of updating the functions provided to the user terminal 20 via the image analysis server 10, or inputting specified commands. For example, the administrator terminal 30 may include a smartphone, laptop, tablet, wearable device, computer, etc., capable of communicating with the image analysis server 10.

[0055] Figure 2 For illustrative purposes only Figure 1 A diagram showing the structure of the image analysis server 10. Figure 3 For illustrative purposes only Figure 2 The second frame of the image analysis server 10 removes the sub-components of module 330.

[0056] Reference Figure 2 and Figure 3The image analysis server 10 may include a memory 200, a processor 300, and a communication module 400.

[0057] The processor 300 can perform basic arithmetic, logic, and input / output operations, thereby processing instructions from computer programs. Instructions can be provided to the processor 300 from the memory 200 or the communication module 400. Alternatively, instructions can be provided to the processor 300 through communication channels between the various structural elements constituting the image analysis server 10.

[0058] The processor 300 can perform various functions such as data input / output, data processing, data management, and communication via the communication network 40. These functions require forming multiple boxes for an object, retaining only the number of boxes corresponding to the object within the multiple boxes, and deleting the remaining boxes. The specific structural elements of the processor 300 used to perform this function will be described later.

[0059] Furthermore, the structural elements of processor 300 may include artificial neural networks learned in advance through deep learning. For example, at least one of the structural elements of processor 300 may be an artificial neural network for implementing RetinaNet, which will be described in detail later.

[0060] The memory 200, as a computer-readable recording medium, may include random access memory (RAM), read-only memory (ROM), and permanent mass storage devices such as disk drives.

[0061] The processor 300 can be used to load program code stored in the memory 200 and count objects, or it can be used to determine the type of an object. This program code can be loaded from a separate computer-readable recording medium (such as a DVD, memory card, etc.), or it can be sent from other devices via the communication module 40 and stored in the memory 200.

[0062] Furthermore, the memory 200 is provided with a database 210, which can store multiple boxes formed by objects, and retain only the number of boxes corresponding to the objects and delete the data required for the remaining boxes.

[0063] The communication module 400 can provide the user terminal 20 and the image analysis server 10, or the administrator terminal 30 and the image analysis server 10, with the communication network 40 to communicate with each other.

[0064] The image analysis server 10 may include a frame setting module 310, a first frame removal module 320, a second frame removal module 330, a pill coefficient determination module 340, a counting module 350, and a type determination module 360 ​​as physical structures. Furthermore, the second frame removal module 330 may include a reference frame setting unit 331, a set frame setting unit 332, a comparison space setting unit 333, and a frame removal unit 334 based on pill coefficient comparison, which will be described in detail later.

[0065] Figure 4 To illustrate the use of Figure 1 A flowchart of the object counting method executed by the image analysis server in object counting system 1. Figure 5 To show in more detail Figure 4 The flowchart of step S2 in steps S1 to S3. Figure 6 To show in more detail Figure 5 The flowchart of step S36 in steps S32 to S36. Figure 7 To conceptually illustrate through Figure 1 The user terminal 20 shown is used to receive the image of the object. Figure 8 To conceptually illustrate through Figure 1 The image analysis server 10 is used to execute the RetinaNet object recognition deep learning model and generate a graph with multiple bounding boxes for a pill. Figure 9 To conceptually illustrate through Figure 1 The image analysis server 10 performs non-maximum suppression (NMS) as an algorithm for removing boxes, thereby forming a graph of boxes for a pill (object).

[0066] Reference Figure 4 and Figure 9 The object counting method using an image analysis server may include: step S1 of receiving an image containing one or more objects through a user terminal 20; step S2 of forming multiple boxes for each object through an image analysis server 10, and retaining only the number of boxes corresponding to the object and deleting the remaining boxes; and step S3 of counting the number of retained boxes through an image analysis server 10 and transmitting the number of boxes corresponding to the retained boxes to the user terminal 20.

[0067] An image, which can be counted by the image analysis server 10, may include objects of the same type with the same size and shape, or may include multiple types of objects with different sizes and shapes.

[0068] In this embodiment, an image counted by the image analysis server 10 contains objects of the same type as an example for illustration.

[0069] Furthermore, the above process will be explained in further detail below using a pill as an example.

[0070] First, the step S1 of receiving an image containing one or more objects (e.g., a pill) via the user terminal 20 will be specifically described below.

[0071] Users can place tablets of the same type and size on the object plate 50 (refer to...). Figure 7 (a) of the drug), and to capture images of the tablets via user terminal 20 (see Part 20). Figure 7 (part (b)).

[0072] At this time, the tablets placed on the object plate 50 must not be stacked.

[0073] However, the present invention is not limited thereto. The object counting system 1 may also include the function of alerting users to stacked tablets via the multi-analysis aid 60 or the image analysis server 10, which will be described in detail later.

[0074] The object plate 50 can be a flat plate that can be used to hold tablets, and can be set to a color that contrasts with or differs from the color of the tablets. For example, if the tablets are set to white, the object plate 50 can be set to black.

[0075] Then, the image of the pill captured by the user terminal 20 can be transmitted to the image analysis server 10.

[0076] In this embodiment, the example of a user holding the user terminal 20 to capture images is used for illustration. However, the concept of the present invention is not limited to this. The user terminal 20 can also be placed on the multi-analysis assistive device 60 described later and images can be captured (see reference). Figure 14 This will be explained in detail later.

[0077] Then, step S2, which is performed by the image analysis server 10 to form multiple boxes for each object (e.g., a pill) and retain only the number of boxes corresponding to the object in the multiple boxes and delete the remaining boxes, will be described in detail below.

[0078] Image analysis server 10 can receive images of multiple pills of the same type from user terminal 20.

[0079] Then, multiple frames can be formed for an object through the frame setting module 310 of the image analysis server 10 (step S32).

[0080] For example, the box setting module 310 can be configured to execute an artificial neural network such as RetinaNet as a deep learning model for object recognition. When executing RetinaNet, multiple boxes can be formed for each tablet. However, the object recognition deep learning model that can be executed by the box setting module 310 is not limited to RetinaNet; the box setting module 310 may include more than one executing CenterNet from YOLO.

[0081] When using RetinaNet, the problem caused by a much smaller number of object samples compared to the number of background samples can be solved when learning a neural network in a method that uses bounding boxes to detect objects.

[0082] Specifically, RetinaNet can be considered an ensemble network consisting of a backbone network and two task-specific subnetworks. The backbone network is responsible for computing convolutional feature maps on all input images. The first subnetwork performs object classification by convolutional processing on the results from the backbone, and the second subnetwork performs bounding box estimation by convolutional processing.

[0083] Figure 8 This is a conceptual illustration of RetinaNet, which is used as a deep learning model for object recognition, through the box setting module 310, and a graph that forms multiple boxes B for a pill (object).

[0084] When RetinaNet is executed via the box setting module 310, if the tablets are close together, multiple boxes will be formed for one tablet, resulting in a discrepancy between the number of tablets and the number of boxes. Therefore, in order to accurately count the number of tablets even when they are close together, it is necessary to perform a step of removing a portion of the boxes formed by RetinaNet after executing RetinaNet.

[0085] Specifically, after RetinaNet is executed through the box setting module 310, an algorithm for removing a portion of the boxes formed by multiple boxes for an object can be executed through the first box removal module 320 of the image analysis server 10 (step S34).

[0086] For example, the algorithm executed by the first frame removal module 320 can be non-maximum suppression (NMS). Non-maximum suppression (NMS) can be understood as a non-maximum value suppression algorithm, which compares the current pixel with the surrounding pixels. If the current pixel is the maximum value, it is retained; otherwise, it is removed.

[0087] Figure 9 This is a conceptual illustration of a graph in which a box is formed on a tablet (object) by performing non-maximum suppression (NMS) as an algorithm for removing boxes via the first box removal module 320.

[0088] When the pills remain very close together after non-maximum suppression (NMS), the number of pills and the number of boxes can be different.

[0089] For example, refer to Figure 9 It can be seen that three closely spaced tablets form five frames B1, B2, B3, B4, and B5. In this case, the second frame removal module 330 can perform step S36, which retains only one frame for an object and deletes the remaining frames. The second frame removal module 330 may include a reference frame setting unit 331, a set frame setting unit 332, a comparison space setting unit 333, and a frame removal unit 334 based on tablet coefficient comparison. This structure allows for the execution of step S36 as described below (see...). Figure 6 ).

[0090] Specifically, step S36 may include: step S361 of setting any frame among the retained frames as a reference frame by reference frame setting unit 331; step S362 of setting a collection frame by collection frame setting unit 332, wherein the collection frame is a collection of multiple frames overlapping with the reference frame; step S363 of removing the space in the space occupied by the reference frame that overlaps with the collection frame by comparison space setting unit 333, and setting the retained space in the reference frame as comparison space; and step S364 of frame removal unit 334 based on tablet coefficient comparison, retaining the frame set as the reference frame when the ratio of the comparison space to the space occupied by the reference frame is greater than the tablet coefficient, and removing the frame set as the reference frame when the ratio of the comparison space to the space occupied by the reference frame is less than the tablet coefficient (see reference). Figure 5 ).

[0091] Figure 10 To conceptually illustrate and aid understanding Figure 6The diagram shows step S36.

[0092] Reference Figures 1 to 10 The following is an example of step S36.

[0093] When step S34 is performed by the first box removal module 320, for closely spaced tablets, the number of boxes can be greater than the number of tablets (for example, 5 boxes B1, B2, B3, B4, and B5 can be formed for 3 tablets).

[0094] In this case, the first frame B1, which is any one of the five remaining frames B1, B2, B3, B4, and B5, is set as the reference frame, and the second frame B2, the fourth frame B4, and the fifth frame B5, which are frames that overlap with the first frame B1, are set as set frames.

[0095] Next, remove the space in the first frame B1 that overlaps with the collection frames B2, B4, and B5, and set the remaining space as the comparison space C.

[0096] Subsequently, since the ratio of the space occupied by the comparison space C to the space occupied by the first frame B1, which serves as the reference frame, is greater than the tablet coefficient (comparison space C / space occupied by the reference frame B1) > tablet coefficient), the first frame B1, which is set as the reference frame, can be retained.

[0097] The tablet coefficient represents the space in which the object (tablet) can exist. It can be set differently depending on the size and shape of the object (tablet). The tablet coefficient can be set to a value greater than 0 and less than 1 (for example, the tablet coefficient is 0.85).

[0098] This tablet coefficient can be set by the tablet coefficient judgment module 340 of the image analysis server 10.

[0099] Specifically, pill coefficients based on the size and shape of the object (pill) can be stored in database 210. When an image including the object (pill) is transmitted from user terminal 20 to image analysis server 10, pill coefficient determination module 340 can match the pill coefficients stored in database 210 according to the size and shape of the object (pill), and set different pill coefficients according to the type of object. Conceptually, as the size of the pill increases, the pill coefficient can also increase between 0 and 1.

[0100] Similarly, when the fourth frame B4 is set as the reference frame, since the ratio of the comparison space to the space occupied by the fourth frame B4 as the reference frame is less than the tablet coefficient, the fourth frame B4 set as the reference frame can be removed.

[0101] In this way, after performing steps S361 to S364, even if there are closely adjacent objects, the number of objects and the number of boxes can be the same.

[0102] Then, step S3 can be executed, which counts the remaining boxes through the image analysis server 10 and transmits the number of boxes to the user terminal 20.

[0103] Specifically, the counting module 350 of the image analysis server 10 can count the retained frames and transmit the count to the user terminal 20. The user terminal 20 can display the count or send voice messages to the user via a speaker.

[0104] Furthermore, the types of objects analyzed by the image analysis server 10 and the number of objects counted can be matched and stored in the database 210. Users can query historical data about the types of objects and the number of objects counted through the user terminal 20.

[0105] Through this process, users only need to take pictures of dozens to tens of thousands of pills and transmit them to the image analysis server 10 to count the accurate number of pills and notify the user. As a result, pharmacies or hospitals can save time required to inventory pill stock.

[0106] Furthermore, the processor can be installed on the user terminal 20 as an application or provided as a webpage. Users can automatically receive the number of pills included in the image simply by downloading the application or accessing the webpage to upload the image.

[0107] The following provides a more detailed description of the frame setting module 310, the first frame removal module 320, the second frame removal module 330, the tablet coefficient judgment module 340, and the counting module 350, which are sub-components of the image analysis server 10.

[0108] As described above, the box setting module 310 can execute an object recognition deep learning model and form multiple boxes for each object.

[0109] The first box removal module 320 can execute an algorithm for removing a portion of the boxes from the multiple boxes formed for each object.

[0110] The second box removal module 330 can retain only one box for an object and delete the remaining boxes.

[0111] Specifically, the second frame removal module 330 may include a reference frame setting unit 331, a set frame setting unit 332, a comparison space setting unit 333, and a frame removal unit 334 based on tablet coefficient comparison.

[0112] The reference frame setting unit 331 can set any frame among the retained frames as the reference frame.

[0113] The set frame setting unit 332 can set a set frame, which is a collection of multiple frames that overlap with the reference frame.

[0114] The comparison space setting unit 333 can remove the space in the reference frame that overlaps with the collection frame, and set the space remaining in the reference frame as the comparison space.

[0115] When the ratio of the space occupied by the comparison space and the reference frame is greater than the tablet coefficient, the frame removal unit 334 based on the tablet coefficient comparison can retain the frame set as the reference frame, and when the ratio of the space occupied by the comparison space and the reference frame is less than the tablet coefficient, the frame set as the reference frame can be removed.

[0116] The tablet coefficient determination module 340 can match the tablet coefficients stored in the database 210 based on the size and shape of the objects displayed in the image.

[0117] The counting module 350 can count the number of boxes corresponding to an object and transmit them to the user terminal 20.

[0118] Figure 11 For illustrative purposes, the use of Figure 1 The diagram illustrates a flowchart of an object counting system 1 transmitting multiple images to an image analysis server 10 and counting one or more objects included in each of the multiple images. Figure 12 To illustrate the display in single analysis mode and multiple analysis modes Figure 1 A screenshot of the user terminal screen. Figure 13 In multi-analysis mode, the number and type of objects included in multiple images analyzed by the image analysis server 10 are displayed. Figure 1 The image shown on the screen of user terminal 20.

[0119] In the object counting system 1 of the above embodiment, although the example of transmitting an image to the image analysis server 10 via the user terminal 20 and analyzing multiple objects included in the image using the image analysis server 10 has been described, the following will describe an embodiment of the object counting system 1 that transmits multiple images to the image analysis server 10 and analyzes multiple objects included in the multiple images respectively.

[0120] Before explaining how to use an image analysis server to count objects included in multiple images, refer to... Figure 12 and Figure 13 The screen of user terminal 20 is described as follows.

[0121] The screen of the user terminal 20 may include an image magnification unit 111, a single analysis button 112, a multi-analysis button 113, an image input button 114, a multi-analysis window 115, and a total quantity display unit 119.

[0122] The image magnification unit 111 can display images that the user terminal 20 is capturing or has already captured.

[0123] The multi-analysis window 115 can display multiple images captured by the user terminal 20, and can also display the number of objects corresponding to each image analyzed by the image analysis server 10.

[0124] Furthermore, an image selection window 115a and a quantity display unit 115b can be provided in the multi-analysis window 115. The image selection window 115a can be used to select each image, and the quantity display unit 115b can display the number of each image analyzed by the image analysis server 10. Additionally, a delete button 116 can be provided in the multi-analysis window 115 to delete each image.

[0125] The type display unit 118 can display the type of objects included in the image selected through the image selection window 115a.

[0126] The total quantity display unit 119 can display the sum of objects included in multiple images displayed in the multi-analysis window 115.

[0127] Reference Figures 11 to 13 The method for counting objects included in multiple images using an image analysis server may include: step S10, where the user terminal 20 selects a single analysis mode that can input one image or a multi-analysis mode that can input multiple images; step S20, where, in the case of selecting the multi-analysis mode, the user terminal 20 inputs multiple images including one or more objects and transmits the input multiple images to the image analysis server 10; step S30, where the image analysis server 10 counts the number of objects included in each of the multiple images; and step S40, where the user terminal 20 displays the number of objects included in each of the multiple images.

[0128] First, the step S10, which involves selecting either a single analysis mode that allows input of one image or a multi-analysis mode that allows input of multiple images via the user terminal 20, will be explained in detail below.

[0129] Users can select a single analysis mode or multiple analysis modes through the user terminal.

[0130] Specifically, users can execute a single analysis mode by touching or clicking the single analysis button 112 displayed on the screen of the user terminal 20, and can execute a multi-analysis mode by touching or clicking the multi-analysis button 113.

[0131] In the case of selecting the single analysis mode, it can be understood that the user terminal 20 captures only one image and transmits it to the image analysis server 10, so that only one image is analyzed.

[0132] Furthermore, when the multi-analysis mode is selected, it can be understood that multiple images are captured by the user terminal 20 and transmitted to the image analysis server 10, thereby analyzing all the multiple images.

[0133] Furthermore, when the multi-analysis mode is selected, an input window (not shown in the figure) can be set on the user terminal 20 for selecting the number of images to be captured. In this case, images corresponding to the number of images selected by the user can be captured and generated.

[0134] For example, if five types of pills need to be provided to patient A, the user can enter 5 in the input window, and if five images need to be entered, five images can be transmitted to the image analysis server 10.

[0135] Then, in the case of selecting the multi-analysis mode, the step S20 of inputting multiple images including one or more objects through the user terminal 20 and transmitting the input multiple images to the image analysis server 10 is specifically described below.

[0136] When the user selects the multi-analysis mode, the multi-analysis window 115 can be activated on the screen of the user terminal 20, and multiple captured images can be displayed in the multi-analysis window 115.

[0137] Users can edit multiple images displayed in the multi-analysis window 115. For example, users can touch or click the delete button 116 in the multi-analysis window 115 to delete images that do not need to be analyzed.

[0138] When multiple images containing one or more tablets are input (captured) via user terminal 20, the user can input the type of tablet displayed in the images via user terminal 20. However, the present invention is not limited to this; the type of tablet can be automatically determined by the multi-analysis aid 60 and / or image analysis server 10, as described later. This will be explained in detail later.

[0139] Multiple images input in this manner can be transmitted to the image analysis server 10.

[0140] Next, step S30, which involves counting the number of objects in each of the multiple images using the image analysis server 10, will be explained.

[0141] Specifically, step S30 may include: forming multiple bounding boxes for each object included in multiple images through the image analysis server 10, and retaining only the number of boxes corresponding to the objects in the multiple bounding boxes formed for each image and deleting the remaining boxes; and counting the number of boxes retained in each of the multiple images through the image analysis server 10, and transmitting the number of boxes corresponding to each of the multiple images to the user terminal.

[0142] The method for counting the objects included in each image is the same as in steps S2 and S3 above, and a detailed explanation of this will be provided by steps S2 and S3 above.

[0143] Next, step S40, which involves displaying the number of objects included in each of the multiple images via the user terminal 20, will be explained.

[0144] Specifically, step S40 may include: displaying multiple images in the multi-analysis window 115 of the user terminal 20; displaying the number of objects included in each of the multiple images in the multi-analysis window 115 of the user terminal 20; displaying the types of objects included in each of the multiple images in the type display unit 118 of the user terminal 20; and displaying the sum of objects included in all of the multiple images in the total quantity display unit 119 of the user terminal 20 (see reference). Figure 13 ).

[0145] For example, four images are displayed in the multi-analysis window 115, and the number of pills is displayed on one side (e.g., the bottom) of each image.

[0146] Furthermore, a type display unit 118 can be provided on one side of the multi-analysis window 115, which can display the type of the selected image (e.g., Nexium tablets). At this time, images selected from the multiple images displayed in the multi-analysis window 115 and unselected images can be displayed by different colors.

[0147] On the other hand, the object counting system 1 of this embodiment may also include a multi-analysis auxiliary device 60 and a moving belt 70 for inputting multiple images in the multi-analysis mode of the above step S10.

[0148] Figure 14 To illustrate the possible placement Figure 1 A diagram of the user terminal 20 and the multi-analysis aids 60. Figure 15 This is an illustrative example of how to easily implement multiple analysis modes. Figure 14 The diagram shows the multi-analysis auxiliary device 60 and the moving belt 70.

[0149] Reference Figure 14 and Figure 15The object counting system 1 of this embodiment may also include a multi-analysis auxiliary device 60 and a moving belt 70 that are easy to implement multiple analysis modes.

[0150] The multi-analysis auxiliary device 60 can be understood as a device capable of housing the user terminal 20, and the moving belt 70 can be understood as a device capable of moving multiple object boards 50.

[0151] In settings Figure 14 and Figure 15 In the case of the multi-analysis aid 60 and the moving belt 70 shown, the step of inputting multiple images including one or more objects through the user terminal 20 can be easily implemented in step S20.

[0152] Specifically, the steps of inputting multiple images including one or more objects through the user terminal 20 may include: placing the user terminal 20 on the terminal placement part 67 of the multi-analysis aid 60; placing multiple object plates 50 on which the objects are placed on the moving belt 70; moving the multiple object plates 50 sequentially to the lower end of the user terminal 20 as the moving belt 70 moves; and moving each object plate 50 after it has stayed at the lower end of the user terminal 20 for a predetermined time, thereby moving the multiple object plates 50 at the lower end of the user terminal 20, and the user terminal 20 taking pictures of the objects placed on each object plate 50 and generating multiple images.

[0153] Furthermore, although the above embodiments have described embodiments that only count the number of objects of the same type, the object counting system 1 can also determine different types of objects when using the object board 50 including the type identification label 52.

[0154] Specifically, a type identification label 52 can be provided on one side of the object plate 50, which contains one or more of the following: text, barcode, and specific symbols. The type of the object (pill) can be determined through this type identification label 52.

[0155] For example, a user can place different types of pills on the object board 50 according to the type identification label 52 affixed thereon. The user terminal 20 can capture an image of the type identification label 52 on the object board 50 and generate an image including both the type identification label 52 and the object, or generate an image including both the object and the type identification label 52. The image analysis server 10 then matches and analyzes the type identification label 52 and the object to determine the type and quantity of the object. In this case, the processor 300 of the image analysis server 10 may also include a type determination module 360 ​​capable of determining the type of the type identification label 52.

[0156] In this case, after step S30, which involves counting the number of objects in multiple images using the image analysis server 10, the steps of matching objects and type identification tags 52 and determining the type of objects using the type determination module 360 ​​of the image analysis server 10, and displaying the number and type of objects in multiple images using the user terminal 20, can be performed.

[0157] Specifically, data on the type of an object based on the type identification tag 52 can be stored in the database 210. The type judgment module 360 ​​can receive the object type-related data stored in the database 210 and determine the type of the object.

[0158] For example, if the type identification label 52 is set to symbol 1234, and the type of the object corresponding to symbol 1234 is stored in the database 210 as Nexium Tab, the user can place the Nexium Tab on the object board 50 labeled with symbol 1234, thus easily identifying the type of the object without performing any additional tedious work.

[0159] The physical device capable of determining the type of the aforementioned object will be described in more detail below.

[0160] An object counting system 1 according to an embodiment of the present invention may include: an object board 50, providing space for placing objects, and including one or more type identification tags 52 configured as text, barcodes, and specific symbols; a user terminal 20, capable of photographing the object board 50 and generating an image including one or more objects placed on the object board 50 and an image including the type identification tags 52; and an image analysis server 10, capable of determining the number and type of objects included in the image. The objects and type identification tags 52 may be photographed as a single image or as separate images.

[0161] The object plate 50 may include: a flat placement portion 55 for placing objects; and a type identification label 52 disposed on the outside of the placement portion 55 and configured as one or more of text, barcode and specific symbols.

[0162] Furthermore, the object counting system 1 may also include a multi-analysis auxiliary device 60, which includes a terminal placement part 67. The terminal placement part 67 is separated from the object board 50 by a preset distance and can accommodate a user terminal 20.

[0163] The multi-analysis aid 60 may include: a lower end portion 62, in which an object plate 50 can move; an upper part portion 66, including a terminal mounting portion 67 for placing a user terminal 20; and a side part portion 64 for connecting the lower end portion 62 and the upper part portion 66. The height of the side part portion 64 can be understood as the distance between the object plate 50 and the user terminal 20, and the side part portion 64 may be height-adjustable.

[0164] When using this multi-analysis aid 60, the user terminal 20 can be placed in the terminal placement unit 67 and the object can be photographed to generate an image, so it is easy to photograph the object placed on the object plate 50.

[0165] Furthermore, the multi-analysis aid 60 may include a sensor 69, which can determine the overlap of objects placed on the object plate 50.

[0166] For example, sensor 69 can be disposed on the side portion 64 of the multi-analysis aid 60, with object plate 50 passing over the front surface of sensor 69. Sensor 69 can scan the height of the object placed on object plate 50 as object plate 50 moves. The height of the object can be understood as the length measured vertically from the mounting portion 55 of object plate 50.

[0167] That is, the image captured by the user terminal 20 can be understood as one side surface (top surface) of the object being photographed, and the sensor 69 attached to the multi-analysis auxiliary device 60 can be understood as the other side surface (side surface) of the object being scanned.

[0168] As the object plate 50 passes the front surface of the sensor 69, the sensor 69 can scan all objects placed on the object plate 50. If the number of objects scanned exceeds a specified range, the sensor 69 can notify the user.

[0169] At this time, a speaker (not shown in the figure) connected to the sensor 69 can be provided in the multi-analysis auxiliary device 60 to notify the user by warning sound, or to transmit a signal from the sensor 69 to the user terminal 20 to provide the user with warning sound or warning prompt through the user terminal 20.

[0170] In this case, the user can detect objects placed on the object board 50 and reposition the objects without overlapping them.

[0171] Furthermore, the object counting system 1 may also include a moving belt 70, which is used to place multiple object boards 50 and can move multiple object boards 50 towards the lower end of the user terminal 20.

[0172] The moving belt 70 can form a closed curve. In this case, multiple object plates 50 are placed above the moving belt 70 that forms the closed curve, thereby using the multiple object plates 50 to count the number of pills.

[0173] Furthermore, when multiple object plates 50 are set, the colors of the mounting portions 55 of the multiple object plates 50 can be set differently.

[0174] For example, the object panel 50 where red-series objects are placed can be set to a green-series color, and the placement part 55 of the object panel 50 where white-series objects are placed can be set to a black-series color. In this case, the image analysis server 10 can more easily identify objects by distinguishing between object and background colors.

[0175] The above description, while illustrating specific embodiments of the image analysis server 10, the object counting system 1 including the image analysis server 10, the object counting method using the image analysis server, and the method for counting objects included in multiple images using the image analysis server, is merely an example. The present invention is not limited thereto and should be interpreted as having the widest scope of the basic ideas disclosed in this specification. Those skilled in the art can combine or replace the disclosed embodiments to implement embodiments not specified, without departing from the scope of protection of the present invention. Furthermore, it is clear that those skilled in the art can easily modify or alter the disclosed embodiments based on this specification, and such modifications or alterations also fall within the scope of protection of the present invention.

[0176] Industrial applicability

[0177] The image analysis server, the object counting method using the image analysis server, and the object counting system are applicable to the counting of the number of pills and have industrial applicability.

Claims

1. An object counting method utilizing an image analysis server, wherein, include: The steps of receiving an image containing multiple objects via a user terminal; The steps involve forming multiple bounding boxes for each of the multiple objects using an image analysis server, and retaining only one bounding box among the multiple bounding boxes formed for each of the multiple objects and deleting the remaining bounding boxes; as well as The steps involve using an image analysis server to count the number of retained bounding boxes and transmitting the corresponding number of boxes to the user terminal. Among them, the plurality of objects are tablets. The step of retaining only one box and deleting the remaining boxes from the multiple boxes formed for each of the multiple objects includes: The first box removal module performs the algorithmic steps for removing a portion of a box from a plurality of boxes formed for each of the plurality of objects; and The steps involved in removing only one box from an object using the second box removal module and deleting the remaining boxes are as follows: The algorithm executed by the first box removal module for removing a portion of the boxes within the multiple boxes formed for each object is a non-maximum suppression algorithm. The step of retaining only one box for an object and deleting the remaining boxes using the second box removal module includes: The step of setting any box among the retained boxes as the reference box using the reference box setting unit; The step of setting a collection box through a collection box setting unit, wherein the collection box is a collection of multiple boxes that overlap with the reference box; The steps include: removing the space overlapping with the collection frame from the space occupied by the reference frame through the comparison space setting unit, and setting the remaining space in the reference frame as the comparison space; and The frame removal unit based on the tablet coefficient comparison retains the frame designated as the reference frame when the ratio of the space occupied by the comparison space and the space occupied by the reference frame is greater than the tablet coefficient, and removes the frame designated as the reference frame when the ratio of the space occupied by the comparison space and the space occupied by the reference frame is less than the tablet coefficient.

2. The object counting method using an image analysis server according to claim 1, wherein, The steps of generating multiple bounding boxes for each object using an image analysis server, and retaining only the number of boxes corresponding to the object within the multiple bounding boxes and deleting the remaining boxes include: The object recognition deep learning model is executed through the box setting module, which forms multiple boxes for each object.

3. The object counting method using an image analysis server according to claim 2, wherein, The object recognition deep learning model executed through the frame setting module is RetinaNet.

4. The object counting method using an image analysis server according to claim 1, wherein, The tablet coefficients are stored in a database based on the size and shape of the object. The tablet coefficient judgment module matches the tablet coefficients stored in the database based on the size and shape of the object displayed in the image.

5. An image analysis server, wherein, The image analysis server is configured as follows: Receive an image containing more than one object from the user terminal. For each of the more than one objects, form multiple boxes. In the multiple boxes created for each of the more than one objects, only one box is retained and the remaining boxes are deleted. The number of retained boxes is counted, and the corresponding number of boxes is transmitted to the user terminal. The image analysis server includes: The first box removal module is capable of executing an algorithm to remove a portion of the boxes within multiple boxes formed for each object; and The second box removes modules, keeping only one box for an object and deleting the rest. Wherein, one or more objects are tablets. The algorithm executed by the first box removal module for removing a portion of the boxes formed by each of the multiple boxes for the more than one object is a non-maximum suppression algorithm. The second box removal module includes: The reference frame setting unit sets any frame among the retained frames as the reference frame; A set frame setting unit is used to set a set frame, which is a collection of multiple frames that overlap with the reference frame; The comparison space setting unit removes the space in the reference frame that overlaps with the collection frame, and sets the remaining space in the reference frame as the comparison space; and The frame removal unit based on the tablet coefficient comparison retains the frame set as the reference frame when the ratio of the space occupied by the comparison space and the space occupied by the reference frame is greater than the tablet coefficient, and removes the frame set as the reference frame when the ratio of the space occupied by the comparison space and the space occupied by the reference frame is less than the tablet coefficient.

6. The image analysis server according to claim 5, wherein, The image analysis server also includes: The box setting module executes an object recognition deep learning model to form multiple boxes for each object.

7. The image analysis server according to claim 6, wherein, The image analysis server also includes: A database for storing tablet coefficients based on the size and shape of the object; and The tablet coefficient determination module matches the tablet coefficients stored in the database based on the size and shape of the objects displayed in the image.

8. An object counting system, wherein, include: The user terminal is capable of receiving images that include more than one object; as well as The image analysis server according to any one of claims 5 to 7 forms multiple boxes for each object, and is able to retain only the number of boxes corresponding to the object in the multiple boxes and delete the remaining boxes, and count the retained boxes.

Images