Image processing device and its program

Abstract

To allow assignment of a plurality of connection relationships from one key point to another related key point.SOLUTION: An image processing apparatus includes acquisition means, specifying means, computation means, selection means, and recognition means. The acquisition means acquires an image that shows an object to be recognized. The specifying means specifies two or more kinds of key points set for the object from the acquired image. The computation means calculates evaluation values of combinations in which different types of key points are connected for each combination. The selection means selects two or more patterns of combinations with other key points in descending order of the evaluation values for each key point. The recognition means recognizes the object by connecting combinations of the two or more patterns selected for each of the key points with the other key points.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention relate to an image processing apparatus, its program, and an image processing method.

Background Art

[0002] In recent years, a technique for estimating the posture of a person depicted in an image captured by a camera by processing the image using deep learning methods has been disclosed. For example, "OpenPose" disclosed in Non-Patent Document 1 is a technique for estimating a person's posture by the following first to third steps. The first step estimates the coordinates of so-called key points, which are feature points such as the head, neck, shoulders, elbows, wrists, hands, waist, knees, and ankles of the person depicted in the image. The second step calculates a direction vector representing the connection direction between the key points based on the关联性 of each key point, for example, the head is connected to the neck, the neck is connected to the shoulders, and the shoulders are connected to the elbows. The third step estimates the person's posture by combining each key point and the direction vector between each key point.

[0003] By utilizing this technology, it is possible to estimate the posture of each person even when an image shows two or more people. However, with conventional technology, only one type of direction vector is calculated from one keypoint to other related keypoints, even if there are multiple other related keypoints. For example, suppose person A and person B are shaking hands, so the image shows person A's wrist and hand and person B's wrist, but person B's hand is hidden by person A's hand and not visible. In this case, in the first step, the coordinates of person A's wrist and hand and person B's wrist are estimated as keypoints. In the second step, because there is a relationship between the wrists and hands, a direction vector is calculated, for example, between person A's wrist and person A's hand. However, in this case, since person A's hand has already been linked to a related keypoint, namely the wrist, it cannot be linked to another keypoint of the same type, namely person B's wrist. As a result, it is not possible to estimate the posture of person B from the wrist onward. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Z. Cao et al.,“OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”,arXiv:1812.08008v2 [cs.CV],30 May 2019 [Overview of the project] [Problems that the invention aims to solve]

[0005] The problem that the embodiments of the present invention aim to solve is to provide an image processing apparatus and its program, as well as an image processing method, that can assign multiple connection relationships from one key point to other related key points. [Means for solving the problem]

[0006] In one embodiment, the image processing apparatus comprises an acquisition means, an identification means, a calculation means, a selection means, and a recognition means. The acquisition means acquires an image in which an object to be recognized is displayed. The identification means identifies two or more key points set for the object from the acquired image. The calculation means calculates an evaluation value for each combination of different types of key points that are linked together. The selection means selects two or more combinations of key points with other key points in descending order of evaluation value for each key point. The recognition means recognizes the object by linking the two or more combinations of key points selected for each key point. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a schematic diagram showing the external configuration of a self-service POS terminal according to one embodiment. [Figure 2] Figure 2 is a top view of a shopping basket according to one embodiment. [Figure 3] Figure 3 is a block diagram showing the main circuit configuration of an image processing device in one embodiment. [Figure 4] Figure 4 is a flowchart showing the main steps of information processing performed by the processor of an image processing device according to an image processing program. [Figure 5] Figure 5 is a flowchart showing the specific steps of the object recognition process within the information processing shown in Figure 4. [Figure 6] Figure 6 is a schematic diagram illustrating one of the key features of a shopping basket: the corners. [Figure 7] Figure 7 is a schematic diagram illustrating the midpoint, which is one of the key points of a shopping basket. [Figure 8] Figure 8 is a schematic diagram showing an example of a matrix created in object recognition processing. [Figure 9] Figure 9 is a schematic diagram showing the recognition result of a shopping basket using object recognition processing. [Modes for carrying out the invention]

[0008] The following describes an embodiment of an image processing device capable of assigning multiple connection relationships from one keypoint to other related keypoints, with reference to the drawings. This embodiment illustrates an image processing device that applies "OpenPose" technology to enable the assignment of multiple connection relationships from one key point to other key points, thereby detecting the opening of a shopping basket placed on the basket holder of a self-service POS terminal from an image captured by a camera located near the self-service POS terminal. First, an overview of the self-service POS terminal 10 will be described using Figure 1.

[0009] Figure 1 is a schematic diagram showing the external configuration of the self-service POS terminal 10. The self-service POS terminal 10 comprises a main body 101 and a basket support stand 102 and a bagging stand 103 positioned to the left and right of the main body 101. The basket support stand 102, located to the right of the main body 101, is a stand on which the customer, who is the payer, places the shopping basket 30 containing the purchased items. The bagging stand 103, located to the left of the main body 101, is a stand on which the customer bags the purchased items. The customer places the purchased items into a shopping bag or reusable bag provided on the bagging stand 103. In Figure 1, the bagging stand 103 has a temporary storage stand 104 attached to its upper part via a support column. The temporary storage stand 104 is, for example, a stand for temporarily placing items before putting them into a shopping bag. The temporary storage stand 104 is provided with two holding arms 105 spaced apart to the left and right. Customers can use these holding arms 105 to place shopping bags, reusable bags, etc., on the bagging table 103 with the bags open. Note that the mounting positions of the basket support 102 and the bagging table 103 may be reversed left and right.

[0010] The main unit 101 is equipped with a touch panel 11, a card reader 12, a printer 13, a scanner 14, a hand scanner 15, a cash handling machine 16, a warning light 17, a speaker 18, and the like.

[0011] The touch panel 11 is a display device capable of displaying elements such as characters, symbols, and images on a display. The touch panel 11 is also an input device that uses a sensor to detect the position of a touch operation on the display and processes the display element at that position as input. In the self-service POS terminal 10, various images to assist customer operation are displayed on the touch panel 11.

[0012] The card reader 12 is an input device that reads data recorded on a card medium such as a credit card, electronic money card, or point card. If the card medium is a magnetic card, the card reader 12 is a magnetic card reader. If the card medium is an IC card, the card reader 12 is an IC card reader. The self-service POS terminal 10 may be equipped with either a magnetic card reader or an IC card reader as the card reader 12, or it may be equipped with both. The card reader 12 may also be a card reader / writer that has a function to write data to the card.

[0013] Printer 13 is an output device that prints data related to purchase receipts, credit card slips, etc., onto receipt paper. The receipt paper printed with various data by printer 13 is cut by a cutter and issued from the receipt issuing slot. Printer 13 can be implemented as, for example, a thermal printer or a dot matrix printer.

[0014] Scanner 14 and hand scanner 15 are input devices that read machine-readable codes such as barcodes or two-dimensional codes. Scanner 14 optically reads machine-readable codes held in front of a reading window (glass window). Hand scanner 15 is operated by the customer and optically reads machine-readable codes when brought close to them.

[0015] The cash processor 16 is a device for processing cash. The cash processor 16 has a banknote insertion slot 161, a banknote ejection slot 162, a coin insertion slot 163, and a coin ejection slot 164. The cash processor 16 processes the banknotes inserted into the banknote insertion slot 161. The cash processor 16 ejects the banknotes as change from the banknote ejection slot 162. The cash processor 16 accepts and processes the coins inserted into the coin insertion slot 163. The cash processor 16 ejects the coins as change from the coin ejection slot 164.

[0016] The patrump 17 is for notifying, by display, alerts or the like generated in the self-POS terminal 10. The patrump 17 is attached to the tip of a pole extending in the vertical direction so as to be visible from afar. The patrump 17 includes a light-emitting member and a cylinder. The light-emitting member is rotatably attached coaxially with the cylinder inside the cylinder. When the patrump 17 operates, it can rotate while causing the light-emitting member to emit light.

[0017] The speaker 18 is for notifying, by voice, alerts or the like generated in the self-POS terminal 10. For example, the speaker 18 emits a buzzer sound according to the state of the alert. Alternatively, the speaker 18 emits the voice of a message according to the state of the alert.

[0018] Above the main body 101, a camera 20 as an imaging device is attached. The camera 20 is a monocular camera. The camera 20 photographs the self-POS terminal 10 and the payer operating it. As shown in FIG. 1, the display of the touch panel 11, the card insertion slot of the card reader 12, the receipt issuing port of the printer 13, the reading window of the scanner 14, and the bill insertion port 161, bill ejection port 162, coin insertion port 163, and coin ejection port 164 of the cash processor 16 are arranged on one surface (hereinafter referred to as the front) of the main body 101. Therefore, the payer stands facing the front of the main body 101 and operates the self-POS terminal 10. The camera 20 photographs the self-POS terminal 10 and the payer standing in front of it from above. Accordingly, the payer's head, face, both shoulders, both arms, chest, abdomen, etc. are photographed by the camera 20. Also, the upper surface of the self-POS terminal 10 including the basket receiver 102 and the bagging table 103 arranged on the left and right sides sandwiching the main body 101 is also photographed by the camera 20. Therefore, when the shopping basket 30 is placed on the basket receiver 102, the opening of the shopping basket 30 and the goods stored in the shopping basket 30 are photographed by the camera 20.

[0019] The camera 20 is connected to an image processing device 40. The image processing device 40 has a function of recognizing the shopping basket 30 placed on the basket receiver 102 by processing the image photographed by the camera 20, that is, the camera image. The information of the shopping basket 30 recognized by the image processing device 40 is provided to a monitoring device (not shown). The monitoring device is, for example, a device for monitoring the actions of the payer standing in front of the self-POS terminal 10. Therefore, the recognition function of this shopping basket 30 will be described in detail below.

[0020] First, the outline of the shopping basket 30 according to the present embodiment will be described with reference to FIG. 2. The shopping basket 30 is prepared by the store and all have the same shape. The shopping basket 30 is an example of a container.

[0021] Figure 2 is a top view of the shopping basket 30. The shopping basket 30 has a rectangular bottom wall 31, and four side walls 321, 322, 323, and 324 are erected upward from each edge so as to be slightly inclined outward, thereby forming a storage section 33 with an opening on the top inside. The upper ends 341, 342, 343, and 344 of each side wall 321, 322, 323, and 324 are joined to the upper ends 341, 342, 343, and 344 of the adjacent side walls 321, 322, 323, and 324, respectively, to form a roughly mouth-shaped opening 35. In other words, the opening 35 is composed of upper ends 341, 342, 343, and 344, and four corners: a first corner 361 which is the joint between upper end 341 and upper end 342, a second corner 362 which is the joint between upper end 342 and upper end 343, a third corner 363 which is the joint between upper end 343 and upper end 344, and a fourth corner 364 which is the joint between upper end 344 and upper end 341. The image processing device 40 recognizes the shopping basket 30 placed on the basket support stand 102 by recognizing the opening 35 of the shopping basket 30.

[0022] Although the opening 35 of the shopping basket 30 is usually fitted with a pair of U-shaped handles, in this embodiment, for the sake of explanation, the handles will be ignored.

[0023] Figure 3 is a block diagram showing the main circuit configuration of the image processing device 40. As shown in the figure, the image processing device 40 includes a processor 41, a GPU (Graphics Processing Unit) 42, a ROM (Read Only Memory) 43, a RAM (Random Access Memory) 44, an auxiliary storage device 45, a camera interface 46, a device interface 47, a communication interface 48, and a system transmission line 49. The system transmission line 49 includes an address bus, a data bus, control signal lines, etc. The image processing device 40 constitutes a computer by connecting the processor 41, the GPU 42, the ROM 43, the RAM 44, the auxiliary storage device 45, the camera interface 46, the device interface 47, and the communication interface 48 via the system transmission line 49.

[0024] The processor 41 corresponds to the central part of the computer described above. The processor 41 controls each part in order to realize various functions as an image processing device 40 according to the operating system or application program. The processor 41 is, for example, a CPU (Central Processing Unit).

[0025] The GPU 42 is a computing device that functions as an accelerator to assist the computational processing performed by the processor 41. In the image processing device 40, the GPU 42 is primarily used to perform image processing calculations at high speed.

[0026] ROM 43 and RAM 44 correspond to the main memory portion of the computer described above. ROM 43, a non-volatile memory area, stores the operating system or application programs. ROM 43 may also store data necessary for the processor 41 to perform control operations for various components. RAM 44, a volatile memory area, is used as a work area where data is rewritten as needed. RAM 44 may also store application programs or data necessary for the processor 41 to perform control operations for various components.

[0027] The auxiliary storage device 45 corresponds to the auxiliary storage portion of the computer described above. For example, an EEPROM (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 45. The auxiliary storage device 45 stores data used by the processor 41 or GPU 42 in performing various processes, as well as data created by those processes. The auxiliary storage device 45 may also store the application programs described above.

[0028] The camera interface 46 connects to the camera 20. The camera interface 46 continuously acquires image data captured by the camera 20.

[0029] The device interface 47 connects the input device 51 and the display device 52. The device interface 47 has the function of receiving data signals input from the input device 59 and the function of outputting display data to the display device 52. The input device 51 is a keyboard, pointing device, touch panel, etc. The display device 52 is a display, touch panel, etc.

[0030] The communication interface 48 is a circuit for data communication with the monitoring device. The communication interface 48 can also communicate data with external devices other than the monitoring device.

[0031] The image processing device 40 with this configuration recognizes the shopping basket 30 placed on the basket holder 102 from the camera image using machine learning with a deep neural network (DNN), such as that used in "OpenPose". To this end, the processor 41 has the functions of an acquisition means 411, an identification means 412, a calculation means 413, a selection means 414, a recognition means 415, and an output means 416.

[0032] The acquisition means 411 has the function of acquiring an image in which the object to be recognized is displayed. In this embodiment, the object to be recognized is a shopping basket 30 placed on the basket holder 102. That is, the acquisition means 411 acquires an image in which the shopping basket 30 placed on the basket holder 102 is displayed from the image captured by the camera 20 via the camera interface 46.

[0033] The identification means 412 is a function that identifies two or more key points set for an object from an image acquired by the acquisition means 411. In this embodiment, where the object to be recognized is a shopping basket 30, the key points are defined as the four corners 361, 362, 363, 364 that constitute the opening 35 of the shopping basket 30 as shown in Figure 6, and the midpoints 371, 372, 373, 374 of the four upper ends 341, 342, 343, 344 that constitute the opening 35 as shown in Figure 7. In other words, the identification means 412 identifies the four corners 361, 362, 363, 364 and the four midpoints 371, 372, 373, 374 of the shopping basket 30 from an image of the shopping basket 30. In other words, the identification means 412 identifies multiple key points of two or more types. Such identification means 412 can be realized by a deep learning method that uses the image acquired by the acquisition means 411 as the input image to the DNN used in "OpenPose" and identifies the coordinates that become the peak values ​​(extreme points) of the heatmap as keypoints.

[0034] The calculation means 413 has the function of calculating an evaluation value for each combination of connecting different types of keypoints. When the object to be recognized is a shopping basket 30, the calculation means 413 calculates an evaluation value for each of the 16 combinations of connecting the four corners 361, 362, 363, 364, which are the first type of keypoints, and the four intermediate points 371, 372, 373, 374, which are the second type of keypoints. The combinations between keypoints are obtained by outputting a two-dimensional unit vector vector field (PAF: Part Affinity Field) that can connect pixels between different types of keypoints with straight lines, using the processing of a DNN used in the identification means 412. The evaluation value is the value obtained by dividing the integral value of the PAF between keypoints determined by the heatmap by the path length between those keypoints.

[0035] The selection means 414 is a function that selects two or more combinations of each key point with other key points in descending order of evaluation value. For example, for the first midpoint 371 of the upper end 341, there are four patterns connecting the four key points: the first corner 361 connecting the upper end 341 and the upper end 342, the second corner 362 connecting the upper end 342 and the upper end 343, the third corner 363 connecting the upper end 343 and the upper end 344, and the fourth corner 364 connecting the upper end 344 and the upper end 341. The same applies to the second midpoint 372 of the upper end 342, the third midpoint 373 of the upper end 343, and the fourth midpoint 374 of the upper end 344. The selection means 414 selects two or more patterns from these four patterns in descending order of evaluation value. Such a selection method 414 can be implemented by using the Hungarian algorithm, which solves the assignment problem between keypoints by brute force.

[0036] The recognition means 415 has the function of recognizing an object by linking combinations of two or more other key points selected for each key point. If the object to be recognized is a shopping basket 30, and an opening 35 is formed by linking combinations of two or more other key points selected for each key point, the recognition means 415 determines that it has been able to recognize the shopping basket 30 placed on the basket support stand 102.

[0037] The output means 416 has the function of outputting the recognition result obtained by the recognition means 415. The output means 416 outputs the recognition result to the monitoring device. The output means 416 may also output the recognition result to a device other than the monitoring device. Alternatively, the output means 416 may also display the recognition result on the display device 52.

[0038] The functions of the acquisition means 411, identification means 412, calculation means 413, selection means 414, recognition means 415, and output means 416 are realized by the processor 41 processing the camera image using a deep learning method according to the image processing program. The image processing program is a type of application program stored in the ROM 43 or auxiliary storage device 45. The method of installing the image processing program in the ROM 43 or auxiliary storage device 45 is not particularly limited. The image processing program can be recorded on a removable recording medium, or distributed via network communication and installed in the ROM 43 or auxiliary storage device 45. The recording medium can take any form as long as it can store a program and is readable by the device, such as a CD-ROM or memory card.

[0039] Figures 4 and 5 are flowcharts showing the main steps of the information processing performed by the processor 41 according to the image processing program. By the processor 41 performing this information processing, the image processing device 40 can recognize the shopping basket 30 placed on the basket holder 102.

[0040] The operation of the image processing device 40 will be explained below using the flowcharts in Figures 4 and 5. Note that the processing procedure and content described below are examples only. The processing procedure and content can be modified as appropriate to achieve similar results.

[0041] First, as ACT1, the processor 41 acquires a camera image via the camera interface 46. Once the camera image is acquired, as ACT2, the processor 41 detects the four corners 361, 362, 363, and 364 that make up the opening 35 of the shopping basket 30 placed on the basket holder 102 from the camera image. For example, the processor 41 uses the GPU 42 to determine the probability that each pixel in the camera image is the first corner 361, second corner 362, third corner 363, or fourth corner 364 of the opening 35, and obtains a heatmap showing the distribution of these probabilities.

[0042] Processor 41 determines, as ACT3, whether it was able to detect a heatmap showing the four corners 361, 362, 363, and 364 of the shopping basket 30. If it is not possible to detect a heatmap showing the four corners 361, 362, 363, and 364 of the shopping basket 30, Processor 41 proceeds from ACT3 to ACT13. Processor 41 checks, as ACT13, whether the next camera image has been acquired. If the next camera image has been acquired, Processor 41 returns from ACT13 to ACT1. Processor 41 executes the processing from ACT1 onwards in the same manner as described above. Therefore, Processor 41 continues to acquire camera images until it is able to detect a heatmap showing the four corners 361, 362, 363, and 364 of the shopping basket 30 placed on the basket stand 102 from the camera image.

[0043] If the processor 41 can detect a heatmap from the camera image showing the four corners 361, 362, 363, and 364 of the shopping basket 30 placed on the basket holder 102, the processor 41 proceeds from ACT3 to ACT4. In ACT4, the processor 41 identifies keypoints for each of the four corners 361, 362, 363, and 364. Specifically, the processor 41 identifies the two-dimensional coordinates (Xa,Ya) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the first corner 361 as the keypoint for the first corner 361. Similarly, the processor 41 identifies the two-dimensional coordinates (Xb,Yb) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the second corner 362 as the keypoint for the second corner 362. The processor 41 identifies the two-dimensional coordinates (Xc,Yc) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the third corner 363 as the keypoint for the third corner 363. The processor 41 identifies the two-dimensional coordinates (Xd, Yd) of a pixel corresponding to the peak value of the heatmap showing the probability distribution of the fourth corner 364 as the keypoint of the fourth corner 364.

[0044] Once the key points of the first to fourth corners 361, 362, 363, and 364 are identified, the processor 41 proceeds to ACT5. As ACT5, the processor 41 detects the midpoints 371, 372, 373, and 374 of the four upper ends 341, 342, 343, and 344 that constitute the aperture 35. For example, the processor 41 uses the GPU 42 to determine the probability that each pixel in the camera image is one of the midpoints 371, 372, 373, and 374 of the four upper ends 341, 342, 343, and 344 that constitute the aperture 35, and obtains a heatmap showing the distribution of these probabilities. That is, the processor 41 detects the group of pixels that includes the peak value of the heatmap showing the probability distribution of being the midpoint of the upper end 342 connecting the first corner 361 and the second corner 362 as the second midpoint 372. Similarly, processor 41 detects a group of pixels containing the peak value of a heatmap showing the probability distribution of being the midpoint of the upper end 343 connecting the second corner 362 and the third corner 363 as the third midpoint 373. Processor 41 detects a group of pixels containing the peak value of a heatmap showing the probability distribution of being the midpoint of the upper end 344 connecting the third corner 363 and the fourth corner 364 as the fourth midpoint 374. Processor 41 detects a group of pixels containing the peak value of a heatmap showing the probability distribution of being the midpoint of the upper end 341 connecting the fourth corner 364 and the first corner 361 as the first midpoint 371.

[0045] If the processor 41 can detect the midpoints 371, 372, 373, and 374 of the upper ends 341, 342, 343, and 344, the processor 41 proceeds to ACT6. In ACT6, the processor 41 identifies the peak points of the first to fourth midpoints 371, 372, 373, and 374. Specifically, the processor 41 identifies the two-dimensional coordinates (Xe,Ye) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the first midpoint 371 as the keypoint of the first midpoint 371. Similarly, the processor 41 identifies the two-dimensional coordinates (Xf,Yf) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the second midpoint 372 as the keypoint of the second midpoint 372. The processor 41 identifies the two-dimensional coordinates (Xg,Yg) of the pixel corresponding to the peak value of the heatmap showing the probability distribution of the third midpoint 373 as the keypoint of the third midpoint 373. The processor 41 identifies the two-dimensional coordinates (Xh, Yh) of a pixel corresponding to the peak value of the heatmap showing the probability distribution of the fourth midpoint 374 as the keypoint of the fourth midpoint 374.

[0046] Once the first to fourth midpoints 371, 372, 373, and 374 keypoints are identified, the processor 41 proceeds to ACT7. As ACT7, the processor 41 calculates a PAF, which is a two-dimensional unit vector vector field that can connect pixels between different types of keypoints with straight lines. Specifically, the processor 41 calculates the X and Y components of the direction vector from the first midpoint 371 to the first corner 361, the X and Y components of the direction vector from the first midpoint 371 to the second corner 362, the X and Y components of the direction vector from the first midpoint 371 to the third corner 363, and the X and Y components of the direction vector from the first midpoint 371 to the fourth corner 364. Similarly, the processor 41 calculates the X and Y components of the direction vector from the second midpoint 372 to the first corner 361, the X and Y components of the direction vector from the second midpoint 372 to the second corner 362, the X and Y components of the direction vector from the second midpoint 372 to the third corner 363, and the X and Y components of the direction vector from the second midpoint 372 to the fourth corner 364. The same applies to the third midpoint 373 and the fourth midpoint 374.

[0047] Once the PAF calculation is complete, processor 41 proceeds to ACT8. Processor 41 calculates the evaluation value Aij as ACT8. The evaluation value Aij is the integral of the PAF between the peak point 36i at the i-th (1≦i≦4) corner and the peak point 37j at the j-th (1≦j≦4) midpoint, divided by the path length between the two peak points.

[0048] Once the evaluation value Aij has been calculated, the processor 41 proceeds to ACT9. The processor 41 creates a matrix of evaluation values ​​Aij as ACT9. For example, as shown in Figure 7, the first to fourth midpoints 371, 372, 373, 374 are used as rows, the first to fourth corners 361, 362, 363, 364 are used as columns, and the associated evaluation values ​​Aij are used as the elements of the matrix to create a matrix 50.

[0049] Once matrix 50 has been created, processor 41 proceeds to ACT10. Processor 41 then executes object recognition processing as ACT10.

[0050] Figure 5 is a flowchart showing the specific steps of the object recognition process. Specifically, when the processor 41 enters the object recognition process, it first sets the first counter r to an initial value of "1" as ACT21. The processor 41 also sets the second counter n to an initial value of "1" as ACT22. Both the first counter r and the second counter n are types of add counters formed in the RAM 44.

[0051] The processor 41 searches for the evaluation value Aij described in matrix 50 as ACT23 and selects the evaluation value Aij with the largest value. Then, the processor 41 stores the information of the corner and midpoint that corresponds to the selected evaluation value Aij combination in RAM 44 as ACT24. For example, assuming that the evaluation value A11 in the first column of the first row of matrix 50 is the maximum value, the processor 41 stores the information related to the combination of evaluation value A11, that is, the coordinates (Xe,Ye) of the key point of the first midpoint 371 and the coordinates (Xa,Ya) of the key point of the first corner 361.

[0052] Next, processor 41 excludes the highest evaluation value Aij from the recognition process as ACT25. Processor 41 also excludes row i and column j of the highest evaluation value Aij from the search candidates as ACT26. Processor 41 then increments the second counter n by "1" as ACT27. Processor 41 checks as ACT28 whether the second counter n has exceeded the upper limit N. The upper limit N is the smaller of the number of rows and columns of the matrix 50. In this embodiment, since both the number of rows and columns of the matrix are "4", the upper limit N is "4".

[0053] If the second counter n does not exceed the upper limit N, the processor 41 returns from ACT28 to ACT23. The processor 41 executes the processing from ACT23 onward in the same manner as described above. Therefore, the processor 41 selects the evaluation value Aij with the highest value among the evaluation values ​​Aij, excluding the row i and column j that were excluded from the search candidates. The processor 41 then stores in RAM 44 the information of the corner and midpoint that are the combination of the selected evaluation value Aij. For example, assuming that the evaluation value A22 in the second column of the second row of matrix 50 is the maximum value, the processor 41 stores the information related to the combination of evaluation values ​​A22, that is, the coordinates (Xf, Yf) of the key point of the second midpoint 372 and the coordinates (Xb, Yb) of the key point of the second corner 362.

[0054] Subsequently, processor 41 excludes the evaluation value Aij selected in the processing of ACT23 from the recognition process, and excludes row i and column j of evaluation value Aij from the search candidates. Then processor 41 increments the second counter n.

[0055] If the upper limit N is "4", then at this point the second counter n has not exceeded the upper limit N. Therefore, the processor 41 returns from ACT28 to ACT23. The processor 41 executes the processing from ACT23 onward in the same way as described above. That is, the processor 41 selects the evaluation value Aij with the highest value among the evaluation values ​​Aij excluding the row i and column j that were excluded from the search candidates. The processor 41 then stores in RAM 44 the information of the corner and midpoint that are combinations of the selected evaluation value Aij. For example, assuming that the evaluation value A33 in the third column of the third row of matrix 50 is the maximum value, the processor 41 stores the information related to the combination of evaluation values ​​A33, that is, the coordinates (Xg, Yg) of the key point of the third midpoint 373 and the coordinates (Xc, Yc) of the key point of the third corner 363.

[0056] Subsequently, processor 41 excludes the evaluation value Aij selected in the processing of ACT23 from the recognition process, and excludes row i and column j of evaluation value Aij from the search candidates. Then processor 41 increments the second counter n.

[0057] If the upper limit N is "4", then the second counter n has not yet exceeded the upper limit N at this point. Therefore, the processor 41 returns from ACT28 to ACT23. The processor 41 executes the processing from ACT23 onward in the same manner as described above. That is, the processor 41 selects the evaluation value Aij with the highest value among the evaluation values ​​Aij, excluding the row i and column j that were excluded from the search candidates. The processor 41 then stores in RAM 44 the information of the corners and midpoints that are combinations of the selected evaluation value Aij. For example, assuming that the evaluation value A44 in the fourth column of the fourth row of matrix 50 is the maximum value, the processor 41 stores the information related to the combination of evaluation values ​​A44, that is, the coordinates (Xh, Yh) of the key point of the fourth midpoint 374 and the coordinates (Xd, Yd) of the key point of the fourth corner 364.

[0058] Subsequently, processor 41 excludes the evaluation value Aij selected in the processing of ACT23 from the recognition process, and excludes row i and column j of evaluation value Aij from the search candidates. Then processor 41 increments the second counter n.

[0059] If the upper limit N is "4", the second counter n exceeds the upper limit N. Processor 41 proceeds from ACT28 to ACT29. Processor 41 considers all rows i and columns j of matrix 50 as search candidates in ACT29. Processor 41 then increments the first counter r by "1" in ACT30. Processor 41 checks in ACT31 whether the first counter r has exceeded the upper limit R. The upper limit R can be any integer greater than or equal to "2". In this embodiment, the upper limit N is set to "2".

[0060] If the first counter r does not exceed the upper limit R, the processor 41 returns from ACT31 to ACT22. The processor 41 sets the second counter n to an initial value of "1" as ACT22. Subsequently, the processor 41 repeatedly executes the processes of ACT23 through ACT27 until the second counter n exceeds the upper limit N. However, in ACT23, the maximum value among the evaluation values ​​Aij other than the largest evaluation value Aij that was excluded in the previous process of ACT25 is selected.

[0061] For example, if evaluation value A14 is the maximum value, the processor 41 stores information related to the combination of evaluation values ​​A14, that is, the coordinates (Xe,Ye) of the key point of the first midpoint 371 and the coordinates (Xd,Yd) of the key point of the fourth corner 364.

[0062] Next, for example, if evaluation value A21 is the maximum value, the processor 41 stores information related to the combination of evaluation values ​​A21, that is, the coordinates (Xf, Yf) of the key point of the second midpoint 372 and the coordinates (Xa, Ya) of the key point of the first corner 361.

[0063] Next, if the evaluation value A32 is the maximum value, the processor 41 stores information related to the combination of evaluation values ​​A32, namely the coordinates (Xg, Yg) of the key point of the third midpoint 373 and the coordinates (Xb, Yb) of the key point of the second corner 362.

[0064] Next, if the evaluation value A43 is the maximum value, the processor 41 stores information related to the combination of evaluation values ​​A43, namely the coordinates (Xh, Yh) of the key point of the fourth midpoint 374 and the coordinates (Xc, Yc) of the key point of the third corner 363.

[0065] Thus, when the second counter n exceeds the upper limit N, the processor 41 executes the processes of ACT29 to ACT31. Then, in ACT31, when the first counter r exceeds the upper limit R, the processor 41 proceeds to ACT32. In ACT32, the processor 41 connects the coordinates of key points based on the information stored in the process of ACT24. In other words, in this embodiment, the first to eighth pieces of information shown below are stored in the process of ACT24.

[0066] • First piece of information: Coordinates (Xe,Ye) and coordinates (Xa,Ya) • Second piece of information: Coordinates (Xe,Ye) and coordinates (Xd,Yd) • Third piece of information… Coordinates (Xf, Yf) and coordinates (Xb, Yb) • Fourth piece of information… Coordinates (Xf, Yf) and coordinates (Xa, Yaa) • Fifth piece of information… Coordinates (Xg, Yg) and coordinates (Xc, Yc) • Sixth piece of information… Coordinates (Xg, Yg) and coordinates (Xb, Yb) • Information #7: Coordinates (Xh, Yh) and (Xd, Yd) • Eighth piece of information… Coordinates (Xh, Yh) and coordinates (Xc, Yc) Therefore, as shown as line segment 381 in Figure 9, the processor 41 connects the key point coordinates (Xe,Ye) of the first midpoint 371 with the key point coordinates (Xa,Ya) of the first corner 361. Also, as shown as line segment 382 in Figure 9, the processor 41 connects the key point coordinates (Xe,Ye) of the first midpoint 371 with the key point coordinates (Xd,Yd) of the fourth corner 364.

[0067] Similarly, the processor 41 connects the key point coordinates (Xf, Yf) of the second midpoint 372 with the key point coordinates (Xb, Yb) of the second corner 362, as shown by line segment 383 in Figure 9. The processor 41 also connects the key point coordinates (Xf, Yf) of the second midpoint 372 with the key point coordinates (Xa, Ya) of the first corner 361, as shown by line segment 384 in Figure 9.

[0068] Processor 41 connects the key point coordinates (Xg, Yg) of the third midpoint 373 and the key point coordinates (Xc, Yc) of the third corner 363, as shown by line segment 385 in Figure 9. Processor 41 also connects the key point coordinates (Xg, Yg) of the third midpoint 373 and the key point coordinates (Xb, Yb) of the second corner 362, as shown by line segment 386 in Figure 9.

[0069] The processor 41 connects the key point coordinates (Xh, Yh) of the fourth midpoint 374 and the key point coordinates (Xd, Yd) of the fourth corner 364, as shown by line segment 387 in Figure 9. The processor 41 also connects the key point coordinates (Xh, Yh) of the fourth midpoint 374 and the key point coordinates (Xc, Yc) of the third corner 363, as shown by line segment 388 in Figure 9. Thus, as shown in Figure 9, the area of ​​the opening 35 in the shopping basket 30 is identified by the eight line segments 381, 382, ​​383, 384, 385, 386, 387, and 388.

[0070] Once the key coordinates have been linked, the processor 41 proceeds to ACT 33. The processor 41 determines whether or not it was able to identify the object as ACT 33. The shopping basket 30, which is the object to be recognized, can be identified by recognizing its opening 35 from the camera image. As described above, if the area of ​​the opening 35 is identified, the processor 41 determines that the object has been identified. If the area of ​​the opening 35 cannot be identified, the processor 41 determines that the object could not be identified.

[0071] If the object is identified, the processor 41 proceeds from ACT33 to ACT34. The processor 41 stores information indicating successful recognition as ACT34. For example, the processor 41 sets the judgment flag of the 1-bit data stored in RAM44 to "1". Conversely, if the object cannot be identified, the processor 41 proceeds from ACT33 to ACT35. The processor 41 stores information indicating recognition failure as ACT35. For example, the processor 41 sets the aforementioned judgment flag to "0".

[0072] Once processing ACT34 or ACT35 is complete, processor 41 exits the object recognition process. Return to the explanation of Figure 4. After completing the object recognition process, processor 41 proceeds to ACT11. Processor 41 checks whether or not it successfully recognized the object as ACT11. If the judgment flag is "1", processor 41 determines that it successfully recognized the object. If the judgment flag is "0", processor 41 determines that it failed to recognize the object.

[0073] If the object is successfully recognized, the processor 41 proceeds from ACT11 to ACT12. The processor 41 outputs the recognition result as ACT12. That is, the processor 41 notifies the monitoring device that the shopping basket 30 has been placed on the basket holder 102. Alternatively, the processor 41 displays on the display device 52 that the shopping basket 30 has been placed on the basket holder 102.

[0074] On the other hand, if object recognition fails, the processor 41 proceeds from ACT11 to ACT13. That is, if the camera image was able to detect the four corners 361, 362, 363, and 364 that make up the opening 35 of the shopping basket 30 placed on the basket stand 102, but the area of ​​the opening 35 could not be identified, the processor 41 proceeds to ACT13. The processor 41 checks whether the next camera image has been acquired as ACT13. If the next camera image has been acquired, the processor 41 returns from ACT13 to ACT1. The processor 41 executes the processing from ACT1 onward in the same manner as described above.

[0075] If the next camera image has not been captured in ACT13, the processor 41 proceeds from ACT13 to ACT14. The processor 41 outputs a recognition failure as ACT14. That is, the processor 41 notifies the monitoring device that it failed to recognize the shopping basket 30. Alternatively, the processor 41 displays the failure to recognize the shopping basket 30 on the display device 52.

[0076] This concludes the explanation of the information processing shown in the flowchart in Figure 4. Here, the processor 41 realizes the function of an acquisition means 411 through the processing of ACT1 to ACT3 in Figure 4. The processor 41 realizes the function of a identification means 412 through the processing of ACT4 to ACT6 in Figure 4. The processor 41 realizes the function of an arithmetic means 413 through the processing of ACT7 to ACT8 in Figure 4. The processor 41 realizes the function of a selection means 414 through the processing of ACT9 in Figure 4 and ACT21 to ACT29 in Figure 5. The processor 41 realizes the function of a recognition means 415 through the processing of ACT30 to ACT33 in Figure 5. The processor 41 realizes the function of an output means 416 through the processing of ACT12 to ACT14 in Figure 4.

[0077] As detailed above, this embodiment provides an image processing device 40 capable of assigning multiple connection relationships from one key point to other related key points. Such an image processing device 40 can detect the opening 35 of a shopping basket 30 placed on the basket holder 102 of the self-service POS terminal 10.

[0078] Although one embodiment has been described above, the embodiment is not limited to this.

[0079] For example, in the above embodiment, the shopping basket 30 placed on the basket support stand 102 is recognized by detecting a roughly mouth-shaped opening 35, but the object to be recognized is not limited to this. It is also possible to recognize a container having an opening of a different shape. In short, by assigning two or more connection relationships from one key point to other related key points, it can be applied to the recognition of recognizable objects.

[0080] In the above embodiment, in ACT9, a matrix 50 was created with the first to fourth midpoints 371, 372, 373, and 374 as rows, the first to fourth corners 361, 362, 363, and 364 as columns, and the associated evaluation value Aij as the matrix components. In this regard, a matrix may also be created with the first to fourth corners 361, 362, 363, and 364 as rows and the first to fourth midpoints 371, 372, 373, and 374 as columns.

[0081] In addition, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the invention and its equivalents as described in the claims. The invention described in the original claims of this application is listed below. [1] An image processing apparatus comprising: an acquisition means for acquiring an image showing an object to be recognized; an identification means for identifying two or more key points set for the object from the image; a calculation means for calculating an evaluation value for each combination of different types of key points that connect together; a selection means for selecting two or more combinations of each key point with other key points in descending order of evaluation value; and a recognition means for recognizing the object by connecting the two or more combinations of each key point selected with other key points. [2] An image processing apparatus according to Appendix [1], further comprising output means for outputting the recognition result by the recognition means. [3] The image processing apparatus according to Appendix [1] or [2], wherein the calculation means calculates the evaluation value based on the vector information and path length between the combined key points. [4] The image processing apparatus according to Appendix [1] or [2], wherein the identifying means identifies the four corners and midpoints of the four sides of the opening as key points for an object having a rectangular opening, and the recognition means recognizes the object by recognizing the opening. [5] A program for causing the computer of an image processing device that recognizes an object from an image of the object to be recognized to function as an acquisition means for acquiring the image, an identification means for identifying two or more key points set for the object from the image, a calculation means for calculating an evaluation value for each combination of different types of key points that connect together, a selection means for selecting two or more combinations of each key point with other key points in descending order of evaluation value, and a recognition means for recognizing the object by connecting the two or more combinations of each key point selected for each key point. [6] An image processing device that processes an image of an object to be recognized, identifies two or more key points set for the object from the image, calculates an evaluation value for each combination of different types of key points that connect together, selects two or more combinations of each key point with other key points in descending order of evaluation value, and recognizes the object by connecting the two or more combinations of each key point with other key points that were selected for each key point. [Explanation of symbols]

[0082] 10...Self-service POS terminal, 20...Camera, 30...Shopping basket, 35...Opening, 40...Image processing device, 41...Processor, 42...GPU, 43...ROM, 44...RAM, 45...Auxiliary storage device, 46...Camera interface, 47...Device interface, 48...Communication interface, 49...System transmission path, 51...Input device, 52...Display device, 361~364...First to fourth corners, 371~374...First to fourth midpoints.

Claims

1. An acquisition means for acquiring an image in which the object to be recognized is displayed, A means for identifying two or more key points set for the object from the aforementioned image, A calculation means for calculating an evaluation value for each combination of connecting different types of keypoints, based on the vector information and path length between the keypoints in the combination, A selection means for selecting two or more combinations of each key point with other key points in descending order of evaluation value, A recognition means that recognizes the object by linking two or more combinations of other key points selected for each of the aforementioned key points, An image processing apparatus comprising the following:

2. Output means for outputting the recognition result by the recognition means, The image processing apparatus according to claim 1, further comprising:

3. Acquisition means for acquiring an image in which the object to be recognized is displayed, A means for identifying two or more key points set for the object from the aforementioned image, A calculation means for calculating the evaluation value of each combination of linking different types of keypoints, A selection means for selecting two or more combinations of each key point with other key points in descending order of evaluation value, A recognition means that recognizes the object by linking two or more combinations of other key points selected for each of the aforementioned key points, It is equipped with, The identifying means, for an object having a rectangular opening, identifies the four corners and the midpoints of the four sides of the opening as key points, The recognition means is an image processing device that recognizes the object by recognizing the opening.

4. Output means for outputting the recognition result by the recognition means, The image processing apparatus according to claim 3, further comprising:

5. A computer in an image processing device that recognizes an object from an image in which the object to be recognized is Acquisition means for acquiring the aforementioned image, A means for identifying two or more key points set for the object from the aforementioned image, A calculation means for calculating an evaluation value for each combination of connecting different types of keypoints, based on the vector information and path length between the keypoints in the combination. A selection means for selecting two or more combinations of each key point with other key points in descending order of evaluation value, and Recognition means for recognizing the object by linking two or more combinations of other key points selected for each of the aforementioned key points, A program designed to function as such.

Images