Information processing device and program

Abstract

To display a guide corresponding to a detection result to be displayed even if a user does not draw the guide corresponding to the detection result to be displayed if the guide corresponding to the detection result to be displayed is not displayed in the case that an object is detected from image data and that the guide for displaying the shape and the size of the object is displayed on a display about the detection result whose degree of reliability exceeds a threshold.SOLUTION: An image 32 is displayed on a display, and a guide representing the shape and the size of an object is displayed overlappingly on the image 32 about a detection result whose degree of reliability exceeds a threshold. In the image 32, an additional guide representing the shape and the size of the object represented at a designated position is overlapped on the image 32 on the basis of the designated position and the degree of reliability of each detection result in the case that a position at which the guide is not displayed is designated.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] This invention relates to an information processing device and a program. [Background technology]

[0002] Techniques for detecting objects from image data that represents them are known.

[0003] Patent Document 1 describes a device that recognizes an item from a photograph taken of an item on display, and detects an area where an item may exist among areas where the item was not recognized, based on fixture information related to the fixture on which the item is displayed. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-61158 [Overview of the project] [Problems that the invention aims to solve]

[0005] Incidentally, when an object is detected from image data and the confidence level exceeds a threshold, a guide representing the shape and size of the object may be displayed on the screen. In this case, the guide corresponding to the detection result that should be displayed may not be shown.

[0006] The object of the present invention is to display a guide corresponding to a detection result that should be displayed, without requiring the user to draw the guide corresponding to the detection result that should be displayed, when an object is detected from image data and a guide representing the shape and size of the object is displayed on the display for detection results where the confidence level exceeds a threshold, but the guide corresponding to the detection result that should be displayed is not displayed. [Means for solving the problem]

[0007] The invention according to claim 1 has a processor, and the processor obtains from image data in which a plurality of objects are represented re each detection result of the objects, displays an image based on the image data on a display, and superimposes and displays on the display a guide representing the shape and size of an object for a detection result whose reliability exceeds a threshold value. When a position where the guide is not displayed is specified in the image displayed on the display, For each of the multiple guides that are not displayed corresponding to the detection results where the confidence level is below the threshold, an index is calculated that increases as the specified position and the position of the guide are closer, and increases as the confidence level of the guide is higher. Among the multiple guides, the one with the largest calculated index is selected as an additional guide. it is an information processing apparatus that superimposes and displays on the display on the image.

[0009] Claim 2 In the invention according to, when a specified guide superimposed on the image is specified, the processor does not superimpose and display the specified guide on the image, and the detection result corresponding to the specified guide that is no longer superimposed and displayed on the image is removed from the respective detection results. The information processing apparatus according to claim 1 as described.

[0010] Claim 3 In the invention according to, the size of the additional guide is determined based on the size of one or more guides superimposed and displayed on the image. The information processing apparatus according to claim 1.

[0011] Claim 4 In the invention according to, when a specified guide superimposed on the image is specified, the processor does not superimpose and display the specified guide on the image, and the size of the additional guide is determined based on the size of one or more guides superimposed and displayed on the image after the specified guide is no longer superimposed and displayed on the image. The information processing apparatus according to claim 3 as described.

[0012] Claim 5The invention relating to claims 1 to 1 is that, when a guide superimposed on the image is specified, the processor does not superimpose the specified guide on the image, and causes the display to show an operator for superimposing the specified guide, which is no longer superimposed on the image, on the display. 4 It is an information processing device described in any one of the items.

[0013] Claim 6 The invention relating to claims 1 to 1 is that the processor changes at least one of the position and size of the guide in response to user operation on the guide. 5 It is an information processing device described in any one of the items.

[0014] Claim 7 The invention relating to claims 1 to 1 is that the processor displays on a display a first operator for giving instructions to return the state of the guide to the state before an operation was performed on the guide, and a second operator for giving instructions to redo the operation on the guide, and controls the state of the guide in response to an operation on the first operator or the second operator by the user. 4 It is an information processing device described in any one of the items.

[0015] Claim 8 The invention relating to this invention is that a computer obtains from image data in which multiple objects are represented. re The system receives the detection results for each object, displays an image based on the image data on the display, superimposes a guide representing the shape and size of the object onto the image on the display for detection results where the confidence level exceeds a threshold, and if a position is specified in the image displayed on the display where the guide is not displayed, For each of the multiple guides that are not displayed corresponding to the detection results where the confidence level is below the threshold, an index is calculated that increases as the specified position and the position of the guide are closer, and increases as the confidence level of the guide is higher. Among the multiple guides, the one with the largest calculated index is selected as an additional guide. This is a program for causing the aforementioned image to be superimposed and displayed on the aforementioned display. [Effects of the Invention]

[0016] Claim 1 or 8According to the invention, when an object is detected from image data and a guide representing the shape and size of the object is displayed on the display for detection results where the confidence level exceeds a threshold, if the guide corresponding to the detection result that should be displayed is not displayed, the user can display the guide corresponding to the detection result that should be displayed without having to draw the guide corresponding to the detection result that should be displayed.

[0017] Claim 2 According to the invention, detection results corresponding to guides that are no longer displayed in the image are prevented from being used as object detection results.

[0018] Claim 3 According to the invention, the size of the additional guides is determined without the user having to specify the size of the additional guides.

[0019] Claim 4 According to the invention, it is prevented that the size of another guide is determined based on a guide that is no longer displayed in the image.

[0020] Claim 5 According to the invention, it is possible to make a guide that has disappeared from view reappear.

[0021] Claim 6 According to the invention, the user can change the position and size of the guide.

[0022] Claim 7 According to the invention, the state of the guide can be returned to its original state, or the operation on the guide can be repeated. [Brief explanation of the drawing]

[0023] [Figure 1] This is a block diagram showing the configuration of an image processing device. [Figure 2] This is a block diagram showing the hardware configuration of an image processing device. [Figure 3] This is a diagram showing an image representing an object. [Figure 4] This is a diagram showing an image with guides superimposed on it. [Figure 5] This is a diagram showing an image with guides superimposed on it. [Figure 6] This is a diagram showing the object detection results. [Figure 7] This is a diagram showing an image with guides superimposed on it. [Figure 8] This diagram shows how to hide the guide. [Figure 9] This figure shows the image after the hide operation has been performed. [Figure 10] This diagram shows how to hide the guide. [Figure 11] This is a diagram showing the object detection results. [Figure 12] This is a diagram showing the object detection results. [Figure 13] This diagram shows the operation of adding and displaying a guide. [Figure 14] This figure shows the image after the additional display operation has been performed. [Figure 15] This is a diagram showing the size of the guide. [Figure 16] This is a diagram showing the object detection results. [Figure 17] This is a diagram showing the object detection results. [Figure 18] This diagram shows the operation of the guide. [Figure 19] This diagram shows the operation of the guide. [Figure 20] This diagram shows the operation of the guide. [Figure 21] This diagram shows the operation of the guide. [Figure 22] This diagram shows the operation of the guide. [Figure 23] This diagram shows the operation of the guide. [Figure 24] This diagram shows the operation of the guide. [Figure 25] This diagram shows the operation of the guide. [Modes for carrying out the invention]

[0024] An image processing apparatus according to an embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of the image processing apparatus 10 according to an embodiment.

[0025] In this embodiment, each object is detected from image data representing multiple objects, and the detection result is obtained. The image processing device 10 displays an image based on the image data on a display, and overlays a guide representing the shape and size of each object on the display for the detection result of each object. The image processing device 10 may also overlay a guide representing the shape and size of the object on the display for detection results where the confidence level exceeds a threshold. If a position where no guide is displayed is specified in the image displayed on the display, the image processing device 10 overlays an additional guide representing the shape and size of the object represented at the specified position on the display, based on the specified position and the confidence level of each detection result.

[0026] The object can be any object. An example of an object is various goods displayed on shelves in a store, but the object in the embodiment is not limited to this.

[0027] Known object detection techniques can be used to detect objects from image data. Artificial intelligence (AI) may also be used to detect objects from image data. For example, machine learning such as deep learning may be used. Neural networks such as convolutional neural networks may be used as machine learning techniques.

[0028] The process of detecting objects from image data may be performed by the image processing device 10, or by an external device other than the image processing device 10. If the object detection process is performed by an external device, the image processing device 10 obtains the detection result data from the external device.

[0029] The object detection results include the object's position, shape, size, and the confidence level of the detection result on the image. A guide is a figure with a specific shape and size. For example, a guide can be a rectangular figure. Guides are sometimes referred to as bounding boxes. Multiple detection results may be obtained as candidates for a single object.

[0030] The confidence level of the detection result is the probability that an object exists within the range of a guide such as a bounding box, and can be calculated, for example, using a regression network of artificial intelligence (AI) utilizing machine learning. First, the probability of belonging to each category is calculated using the object's position, size, and category including the background. Examples of categories include beverages, snacks, and instant noodles. For example, if an object could potentially be a beverage, snack, or instant noodles, the probability of it being a beverage is 70%, the probability of it being a snack is 20%, and the probability of it being instant noodles is 10%, and so on, so that the sum of the probabilities of belonging to each category is 100%. The confidence level is defined by the maximum value among the probabilities of belonging to all categories calculated by the bounding box. The confidence level (score) of the bounding box is defined by equation (1) as follows: in a problem with K categories, the confidence level of each category is calculated as c k It is defined by the maximum value of (k=0,1,···,K-1). In the above case, a 70% confidence level is calculated using the 70% probability of belonging to the beverage category that takes the maximum value.

[0031]

number

[0032] The confidence level of the detection result may be calculated by the image processing device 10, or it may be calculated by an external device other than the image processing device 10. If the confidence level of the detection result is calculated by an external device, the image processing device 10 obtains the confidence level data from the external device.

[0033] The image data may be image data generated by the image processing device 10, image data stored in the image processing device 10, or image data generated by an external device other than the image processing device 10. The external device here may be a personal computer (hereinafter referred to as "PC"), a tablet PC, a smartphone, a mobile phone, a camera, or a server.

[0034] For example, the image processing device 10 is a terminal device such as a tablet PC or smartphone equipped with a camera, and generates image data by taking pictures with the camera. The image processing device 10 may download image data from an external device such as a server, or it may receive image data transmitted to the image processing device 10 from an external device.

[0035] As another example, the image processing device 10 may be a server or other device, and image data may be acquired from external devices such as terminal devices or file servers other than the image processing device 10. For example, image data may be generated by taking a picture with a camera installed in a terminal device, and said image data may be transmitted from the terminal device to the image processing device 10.

[0036] The configuration of the image processing device 10 will be described below with reference to Figure 1.

[0037] The image processing device 10 includes, for example, an image acquisition unit 12, an image display unit 14, a detection result acquisition unit 16, a detection result display unit 18, and a detection result correction unit 20.

[0038] The image acquisition unit 12 acquires image data. The image acquisition unit 12 may acquire image data generated by the image processing device 10 itself, or it may acquire image data by receiving image data transmitted to the image processing device 10 from an external device other than the image processing device 10. For example, if the image processing device 10 is a terminal device equipped with a camera, the image acquisition unit 12 acquires image data generated by the camera taking pictures.

[0039] The image display unit 14 displays an image on the display based on the image data acquired by the image acquisition unit 12. For example, if image data is generated by taking a picture with a camera, the image display unit 14 displays an image on the display based on the image data generated by the picture.

[0040] The detection result acquisition unit 16 acquires detection result data for each object obtained from the image data. If the object detection process is performed by the image processing device 10, the detection result acquisition unit 16 acquires the detection result data. If the object detection process is performed by an external device other than the image processing device 10, the detection result acquisition unit 16 acquires the detection result data from the external device.

[0041] When image data is generated by the image processing device 10 and object detection is performed by an external device, the image processing device 10 transmits the image data to the external device, and the detection result acquisition unit 16 acquires detection result data for the image data from the external device.

[0042] The detection result display unit 18 displays on the display a guide representing the shape and size of an object for detection results where the confidence level exceeds a threshold, superimposed on the image representing the object.

[0043] The detection result correction unit 20 corrects the detection results corresponding to the guides displayed on the display. For example, the detection result correction unit 20, upon receiving instructions from the user, deletes a detection result from multiple detection results or adds a detection result to multiple detection results. Specifically, if the user specifies a position in the image displayed on the display where no guide is shown, the detection result correction unit 20 overlays an additional guide representing the shape and size of the object displayed at that specified position onto the image and displays it on the display, based on the specified position and the confidence level of each detection result. The detection result correction unit 20 also adds the detection result corresponding to the additional guide to multiple detection results.

[0044] The hardware configuration of the image processing device 10 will be described below with reference to Figure 2. Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 10.

[0045] As an example, the image processing device 10 is a terminal device such as a tablet PC or smartphone, and includes a camera 22, a communication device 24, a user interface (UI) 26, memory 28, and a processor 30.

[0046] Image data is generated by the camera 22. For example, an object is detected from the image data generated by the camera 22, and the confidence level of the detection result is calculated.

[0047] The communication device 24 includes one or more communication interfaces having a communication chip, communication circuits, etc., and has the function of transmitting information to other devices and receiving information from other devices. The communication device 24 may have wireless communication functions such as short-range wireless communication or Wi-Fi (registered trademark), or it may have wired communication functions.

[0048] UI26 is a user interface and includes a display and an input device. The display is a liquid crystal display or an EL display, etc. The input device is a keyboard, mouse, input keys, or control panel, etc. UI26 may also be a UI such as a touch panel that combines a display and an input device.

[0049] Memory 28 is a device that constitutes one or more storage areas for storing data. Memory 28 is, for example, a hard disk drive (HDD), a solid state drive (SSD), various types of memory (e.g., RAM, DRAM, NVRAM, ROM, etc.), other storage devices (e.g., optical discs, etc.), or a combination thereof.

[0050] The processor 30 controls the operation of each part of the image processing device 10.

[0051] The image acquisition unit 12, image display unit 14, detection result acquisition unit 16, detection result display unit 18, and detection result correction unit 20 are implemented by the processor 30. Specifically, the processor 30 receives detection result data for each object obtained from image data representing multiple objects. The processor 30 displays the image based on the image data on the UI 26 display. For detection results where the confidence level exceeds a threshold, the processor 30 overlays a guide representing the shape and size of the object onto the image and displays it on the display. If the user specifies a position in the image displayed on the display where no guide is shown, the processor 30 overlays an additional guide representing the shape and size of the object represented at that specified position onto the image and displays it on the display, based on the specified position and the confidence level of each detection result.

[0052] The processor 30 may also detect objects from the image data, or calculate the confidence level of the detection result.

[0053] The embodiments will be described in detail below with specific examples.

[0054] Figure 3 shows an image 32 in which multiple objects are represented. The objects α, β, and γ are shown in the image 32. The image 32 may be generated by the image processing device 10, or it may be generated by an external device and transmitted to the image processing device 10.

[0055] For example, consider a scenario where a user photographs multiple objects belonging to the same category using the camera 22 of the image processing device 10. Here, as an example, objects α, β, and γ are plastic bottles containing beverages. By photographing multiple plastic bottles displayed on a store shelf, an image 32 representing these multiple plastic bottles is generated.

[0056] Figure 4 shows the guides and image 32. For example, image 32 is displayed on the UI26 display, and guides α1, β1, and γ1 are superimposed on image 32 and displayed on the display.

[0057] Guides α1, β1, and γ1 are rectangular bounding boxes representing the results of object detection on image 32. Guide α1 is a bounding box that surrounds object α and reflects the shape and size of object α, and is displayed superimposed on image 32 at the position where object α is represented. Guide β1 is a bounding box that surrounds object β and reflects the shape and size of object β, and is displayed superimposed on image 32 at the position where object β is represented. Guide γ1 is a bounding box that surrounds object γ and reflects the shape and size of object γ, and is displayed superimposed on image 32 at the position where object γ is represented. For example, guide α1 may be formed so that object α is inscribed within guide α1, or guide α1 may be formed so that object α is surrounded by guide α1 in a state where it is nearly inscribed. The same applies to guides β1 and γ1.

[0058] The guides α1, β1, and γ1 shown in Figure 4 are guides that represent appropriate detection results. In other words, guide α1 is a bounding box that accurately represents the region occupied by object α, guide β1 is a bounding box that accurately represents the region occupied by object β, and guide γ1 is a bounding box that accurately represents the region occupied by object γ.

[0059] As a result of the process of detecting objects from image 32, detection results corresponding to guides α1, β1, and γ1 may be obtained, as well as other detection results.

[0060] Figures 5 and 6 show examples of detection results obtained from image 32. Figure 5 shows the guide and image 32. Figure 6 shows the detection results.

[0061] By performing an object detection process on image 32, the detection result shown in Figure 6 is obtained. The object detection process may be performed by the processor 30 or by an external device.

[0062] The detection results shown in Figure 6 are explained below. "ID" is information used to identify the detection result, and also information used to identify the guide corresponding to that detection result. "Score" is the confidence level of the detection result. The coordinates "x" and "y" are the coordinates of the guide on image 32. A two-dimensional coordinate system defined by the x and y axes is defined on image 32, and the coordinates "x" and "y" are the coordinates on that two-dimensional coordinate system. "Width" is the width of the guide (length along the x axis). "Height" is the height of the guide (length along the y axis). "Judgement" is information indicating whether the detection result is correct or incorrect. "○" indicates an estimation that the detection result is correct. "×" indicates an estimation that the detection result is incorrect. For example, a threshold is set for the confidence level of the detection result, and "○" is associated with detection results whose confidence level exceeds the threshold, and "×" is associated with detection results whose confidence level is below the threshold. Furthermore, the guide for detection results associated with "○" is displayed superimposed on image 32, while the guide for detection results associated with "×" is not displayed superimposed on image 32.

[0063] In the example shown in Figure 6, detection results a to f are obtained from image 32. Detection results a, c, and d are detection results with a confidence level above the threshold. Detection results b, e, and f are detection results with a confidence level below the threshold.

[0064] Hereinafter, detection results with a confidence level exceeding a threshold will be referred to as "candidate detection results," and the guides corresponding to candidate detection results will be referred to as "candidate guides." Detection results other than candidate detection results will be referred to as "non-candidate detection results," and the guides corresponding to non-candidate detection results will be referred to as "non-candidate guides." Whether or not a detection result is a candidate detection result may be determined based on the confidence level of the detection result and the relationship between the detection result and other detection results (for example, the degree of overlap between the guide corresponding to the detection result and the guide corresponding to the other detection results). Here, as an example, detection results with a confidence level exceeding a threshold will be referred to as "candidate detection results." Furthermore, detection results that should have been detected as candidate detection results among non-candidate detection results will be referred to as "missed detection results," and the guides corresponding to missed detection results will be referred to as "missed detection guides." Detection results that should have been detected as non-candidate detection results among candidate detection results will be referred to as "false detection results," and the guides corresponding to false detection results will be referred to as "false detection guides."

[0065] The detection results may be used to perform a process to identify objects. Candidate detection results are those to which the identification process is applied, while non-candidate detection results are those to which the identification process is not applied. For example, the features of an object represented in the candidate guide are extracted from image 32, and the object is identified by matching these features with the features of an object registered in the database. For example, the type and name of the object may be identified through this matching. In this way, the identification process may not be applied to all detection results, including candidate and non-candidate detection results, but rather only to the candidate detection results among all detection results.

[0066] Figure 5 shows the guides corresponding to each detection result. Guides a to f correspond to detection results a to f, respectively. For example, guide a shown in Figure 5 corresponds to detection result a shown in Figure 6. The same applies to the other guides. Figure 5 displays all detection results.

[0067] A threshold may be set for the reliability of the detection results, and the processor 30 may display a guide on the display superimposed on the image 32 for detection results that exceed that threshold.

[0068] For example, the threshold is set to "80". In this case, the processor 30 displays a guide on the display superimposed on image 32 regarding detection results with a confidence level exceeding "80".

[0069] In the example shown in Figure 6, the confidence levels of each detection result a, c, and d exceed "80". In this case, as shown in Figure 7, the processor 30 superimposes the guides a, c, and d onto the image 32 and displays them on the display.

[0070] In the example shown in Figure 6, detection results a, c, and d are candidate detection results, and detection results b, f, and e are non-candidate detection results. Guides a, c, and d are candidate guides, and guides b, e, and f are non-candidate guides.

[0071] If a guide is specified to be superimposed on image 32, the processor 30 will not superimpose that specified guide onto image 32 and display it on the screen. Furthermore, the processor 30 will exclude the detection result corresponding to that specified guide from all detection results received by the processor 30. For example, the processor 30 will exclude the detection result corresponding to that specified guide from the candidate detection results. This process will be explained below with reference to Figures 8 to 12. Figures 8 to 10 show the guide and image 32. Figures 11 and 12 show the detection results.

[0072] In the example shown in Figure 7, one object is not clearly distinguishable from other objects within the area occupied by guide d, and therefore guide d can be evaluated as not representing the correct detection result for the object. In other words, guide d is a false detection guide, and detection result d is the result of a false detection. In this case, the user may specify guide d and prevent it from being displayed on the screen.

[0073] For example, as indicated by the reference numeral 34 in Figure 8, when the user specifies guide d through an operation such as touch or click, the processor 30 does not superimpose guide d onto image 32 and display it on the display, as shown in Figure 9.

[0074] Furthermore, as shown in Figure 10, an image (for example, an "X" mark) indicating that the guide should be hidden may be attached to and displayed on each guide. When the user presses the image by touching or clicking, the processor 30 prevents the guide with that image from being displayed on the display.

[0075] Figure 11 shows the detection result when guide d is superimposed on image 32 and displayed on the screen, that is, the detection result before the user specifies guide d. Since the confidence level of detection result d exceeds the threshold (e.g., "80"), detection result d is presumed to be a correct detection result, and detection result d is associated with "judgement" which represents "○". In other words, detection result d is treated as a candidate detection result.

[0076] Figure 12 shows the detection results when guide d is not superimposed on image 32, that is, the detection results after the user has specified guide d. Although the confidence level of detection result d exceeds the threshold, since guide d has been specified by the user as a guide to be excluded, the processor 30 excludes detection result d from the candidate detection results by changing the "judgement" associated with detection result d from "○" to "×". As a result, detection result d is treated as a non-candidate detection result. Guide d is also treated as a non-candidate guide.

[0077] The following describes the process of displaying additional guides superimposed on image 32. Based on the position specified by the user on image 32 and the confidence level of each detection result, the processor 30 displays additional guides for the object represented at the specified position on the display, superimposed on image 32. The processor 30 also determines the detection result corresponding to the additional guide as a candidate detection result.

[0078] Specifically, the processor 30 determines additional guides based on the relationship between the position specified by the user on the image 32 and the positions where each detection result is obtained, and the reliability of each detection result. For example, guides corresponding to detection results obtained from positions closer to the position specified by the user, or guides corresponding to detection results with higher reliability, are determined as additional guides.

[0079] The process of determining additional guides will be described in detail below. In the following process, additional guides are selected from the out-of-candidate guides.

[0080] Define the coordinates of the position specified by the user as (x c , y c ). Define the coordinates (for example, center coordinates) of the out-of-candidate guide corresponding to the out-of-candidate detection result having index i as (x i , y i ). Define the reliability of the out-of-candidate detection result having index i as S i . The index corresponds to the ID of the detection result (see FIG. 12, etc.).

[0081] The processor 30 calculates the distance D i according to the following formula (2). The distance D i is a value corresponding to the difference between the coordinates (x c , y c ) of the position specified by the user and the coordinates (x i , y i ) of the out-of-candidate guide having index i.

[0082] ​​​​​​​​​​​​​​​​i This value is obtained by the weighted average with the reciprocal of α. α is a predetermined constant. According to equation (3), the distance D i The smaller the value of R, that is, the closer the user-specified location is to the location of the non-candidate guide, the stronger the R index, which indicates the degree of likelihood. i The value of increases. Also, the higher the confidence of the out-of-candidate detection result corresponding to the out-of-candidate guide, the higher the index R i The value will increase.

[0084]

number

[0085] The processor 30 calculates an index t indicating a missed detection guide according to the following equation (4). The index t is calculated by the index R in the set of indices M that indicate non-candidate guides. i This is the index with the largest value. The non-candidate detection results corresponding to the detection miss guide are the missed detection results. The detection miss guide is an example of an additional guide.

[0086]

number

[0087] The processor 30 may determine the missed detection guides by excluding false positive guides from the list of non-candidate guides. In other words, the processor 30 may determine the missed detection results by excluding false positive results from the list of non-candidate detection results. In this case, the index set M is the index set of non-candidate guides from which the false positive guides have been excluded. By excluding false positive guides from the list of non-candidate guides, the missed detection guides (i.e., additional guides) can be determined with greater accuracy compared to determining the missed detection guides from the list of non-candidate guides that include the false positive guides.

[0088] Referring to Figures 13 and 14, a specific example of the process of displaying leak detection guides (i.e., additional guides) will be described. Figures 13 and 14 show the guides and image 32. Figure 13 shows the result after guide d has been removed, similar to Figure 9.

[0089] As indicated by the symbol 34 in Figure 13, the display position of object γ on image 32 is specified by the user. In this case, the processor 30 calculates an index t indicating a missed detection guide according to equations (2) to (4) above. In the example shown in Figure 12, the non-candidate guides are guides d, b, f, and e, but guide d is treated as a false positive guide and is therefore excluded from the non-candidate guides subject to calculation. As a result, the non-candidate guides subject to calculation are guides b, f, and e.

[0090] Coordinates (x) of the position specified by the user c ,y c ) and the center coordinates (x) of guides b, f, and e respectively i ,y i Based on the above, the respective indicators R of guides b, f, and e i (that is, R b ,R f ,R e ) is calculated.

[0091] Here, the indicator R b ,R f ,R e Among the indicators R f Since f is the maximum value, the processor 30 determines guide f as the leak detection guide and sets detection result f as the leak detection result. As shown in Figure 14, the processor 30 superimposes guide f onto image 32 and displays it on the display.

[0092] The processor 30 may determine the size of the undetected guide based on the size of one or more guides shown in image 32. For example, the processor 30 may calculate the average value of the sizes of one or more guides shown in image 32 and set that average value as the size of the undetected guide. This process will be explained below with reference to Figures 13 and 15. Figure 15 shows the guide and image 32.

[0093] As indicated by the reference numeral 34 in Figure 13, the display position of object γ on image 32 is specified by the user. In this case, guide f is determined to be the detection miss guide.

[0094] Here, we define Nc as the total number of candidate guides (i.e., the total number of guides displayed in image 32), and C as the set of indexes for the candidate guides. The processor 30 calculates the width W and height H of the undetected guides according to the following equations (5) and (6). In equations (5) and (6), i is the index of the candidate guide (i.e., the value corresponding to the ID of the detection result). Width W i is the width of the candidate guide corresponding to the candidate detection result with index i. Height H i This is the height of the candidate guide corresponding to the candidate detection result with index i.

[0095]

number

[0096]

number

[0097] In the example shown in Figure 13, the candidate guides to be calculated are guides a and c. As shown in Figure 15, the width W of guide f, which is a guide that was not detected, is shown. f The width W of guide a is a and guide c Width W c This is the average value, and the height H of the guide f. fThe height H of guide a a and the height H of guide c c This is the average value. Width W f and height H f This is expressed in equation (7) below.

[0098]

number

[0099] Figure 15 shows a guide f. This guide f has a width Wf calculated according to equation (5) and a height Hf calculated according to equation (6).

[0100] The width and height of the detection miss guide are calculated based on the probabilistic assumption that the multiple objects shown in Image 32 belong to the same category and that the size and aspect ratio of each object are approximately the same. Furthermore, if the objects are products displayed on a store shelf, and Image 32 is generated by photographing multiple products displayed on that shelf, the width and height of the detection miss guide are calculated based on the assumption that the multiple products displayed on the same shelf belong to the same category.

[0101] If the image 32 represents objects from multiple different categories, the processor 30 may recognize each category, determine a separate detection gap guide for each category, and calculate the width and height of the detection gap guide for each category.

[0102] Furthermore, the processor 30 calculates the size of the undetected guides by excluding the falsely detected guides from the candidate guides. In other words, as explained with reference to Figures 8 and 9, if a guide to be excluded (e.g., guide d) is specified by the user and excluded from the candidate guides as a falsely detected guide (see Figure 12), the processor 30 determines the size of the undetected guides based on the size of the candidate guides from which the falsely detected guide was excluded. That is, the processor 30 calculates the size of the undetected guides based on the size of one or more guides (e.g., guides a, c) that are displayed superimposed on image 32 after the falsely detected guide (e.g., guide d) is no longer displayed superimposed on image 32. By calculating the size of the undetected guides without using the size of the falsely detected guides, the size of the undetected guides can be calculated with greater accuracy compared to the case where the falsely detected guides are also included in the candidate guides when calculating the size of the undetected guides.

[0103] Figure 16 shows the detection result when guide f is not superimposed on image 32 and displayed on the screen, that is, the detection result before the user specifies object γ. Since the confidence level of detection result f is below the threshold (e.g., "80"), detection result f is presumed to be an incorrect detection result, and a "×" "judgement" is associated with detection result f. In other words, detection result f is treated as a non-candidate detection result.

[0104] Figure 17 shows the detection result when guide f is superimposed on image 32 and displayed on the screen, that is, the detection result after the user has specified object γ. The confidence level of detection result f is below the threshold, but since detection result f has been determined to be a leak detection result, the processor 30 changes the "judgement" associated with detection result f from "×" to "〇", thereby adding detection result f to the candidate detection results as a leak detection result. As a result, detection result f is treated as a candidate detection result. Guide f is treated as a candidate guide. Note that since the confidence level of detection result f is below the threshold, it can be determined that detection result f is a leak detection result by referring to that confidence level.

[0105] The following describes how to operate the guide shown in Image 32, referring to Figures 18 to 23. Figures 18 to 23 show the guide and Image 32.

[0106] The processor 30 changes at least one of the position and size of the guide shown in image 32 in response to user interaction with the guide.

[0107] For example, if a user specifies a location within the area enclosed by the guide (e.g., the center of the guide or its vicinity) and performs an operation to move the guide on image 32, the processor 30 changes the display position of the guide according to that operation. When the display position of the guide is changed, the processor 30 changes the X and Y coordinate values ​​of the guide in the detection result to the values ​​after the display position has been changed.

[0108] In the example shown in Figure 18, the user specifies guide e, as indicated by reference numeral 34. The user moves guide e in front of the display position of object γ on image 32, as shown in Figure 19, and then moves guide e to the display position of object γ, as shown in Figure 20. The processor 30 changes the display position of guide e according to this movement operation. For example, the user changes the display position of guide e by touching and dragging the center or vicinity of guide e. The processor 30 changes the X and Y coordinate values ​​of guide e in the detection result to the values ​​after the display position has been changed.

[0109] Additionally, when the user drags a vertex or edge of the guide, the processor 30 resizes the guide according to the amount of dragging.

[0110] In the example shown in Figure 21, the user specifies a vertex of guide g, as indicated by the reference numeral 34. The user drags the vertex of guide g on image 32, as shown in Figure 22. The processor 30 changes the size of guide g according to the amount of dragging. The user drags the vertex of guide g further, as shown in Figure 23, and the processor 30 further changes the size of guide g according to the amount of dragging.

[0111] The processor 30 may display on the display a first operator for instructing the guide to return to the state before an operation was performed on the guide, and a second operator for instructing the user to redo the operation on the guide. In response to the user's operation on the first operator, the processor 30 returns the guide to the state before an operation was performed on the guide. Also, in response to the user's operation on the second operator, the processor 30 redoes the operation on the guide.

[0112] The operation using the first and second operators will be described below with reference to Figures 24 and 25. Figures 24 and 25 show a terminal device 36. The terminal device 36 is a device such as a tablet PC or smartphone, and is an example of an image processing device 10.

[0113] The display of the terminal device 36 shows image 32. Similar to Figures 7 and 10, objects α, β, and γ are shown in image 32. Additionally, guides a, c, and d are superimposed on image 32.

[0114] Additionally, buttons 38 and 40 are displayed on the screen. Button 38 is an image that instructs the user to return the guide to its state before any operation was performed on it. Button 40 is an image that instructs the user to redo the operation on the guide. Button 38 is an example of an operator or first operator. Button 40 is an example of a second operator.

[0115] For example, if guide c, shown in image 32, is specified by the user, the processor 30 does not superimpose guide c onto image 32 and display it on the screen, as explained with reference to Figures 7 to 12. Furthermore, the processor 30 defines guide c as a false detection guide and defines the detection result c corresponding to guide c as a false detection result. Figure 25 shows image 32 after guide c has been specified by the user. Guide c is not superimposed on image 32.

[0116] If the operation on guide c is incorrect, the user may press (for example, click or touch) button 38. When the user presses button 38, the processor 30 returns the state of guide c to the state it was in before the operation was performed on guide c. Specifically, as shown in Figure 24, the processor 30 displays guide c superimposed on image 32 on the display, sets guide c as a candidate guide, and sets detection result c as a candidate detection result.

[0117] In this state, when the user presses button 40, the processor 30 repeats the process of superimposing guide c onto image 32 and preventing it from being displayed on the display. As a result, as shown in Figure 25, the processor 30 superimposes guide c onto image 32 and prevents it from being displayed on the display, and determines detection result c as a false detection result.

[0118] The above-described embodiment may be applied, for example, to the task of understanding the display status of products displayed on shelves installed in a store (i.e., shelf allocation work). Specifically, the camera of a terminal device such as a tablet PC or smartphone is used to photograph the shelves on which the products are displayed, and each product is detected from the image data generated by the photography. The image processing device 10 receives the detection result data and displays a guide corresponding to each detection result superimposed on the image on the display. If the user specifies a guide that is displayed superimposed on the image, the specified guide is designated as a false detection guide, and the detection result corresponding to that guide is designated as a false detection result. Also, if the user specifies a product for which no guide is displayed, a missing detection guide is calculated and displayed superimposed on the image. The detection result corresponding to that missing detection guide is designated as a missing detection result. In this way, processing such as deleting false detection results from candidate detection results and adding missing detection results to candidate detection results is performed, and the results are used for product identification processing. Through product identification processing, the products represented within the guide (i.e., bounding box) are identified, and shelf allocation work is performed. Here, we have given product identification and shelf allocation as examples of where the embodiment may be applied, but of course, the examples to which the embodiment may be applied are not limited to these.

[0119] The functions of the image processing device 10 described above are realized, for example, through the cooperation of hardware and software. For instance, the processor reads and executes programs stored in the memory of each device, thereby realizing the functions of each device. The programs are stored in memory via a recording medium such as a CD or DVD, or via a communication path such as a network.

[0120] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). Furthermore, the operation of the processor in each of the above embodiments may not be performed by a single processor, but may be performed by multiple processors located in physically separate locations working together. In addition, the order of the processor's operations is not limited to the order described in each of the above embodiments, and may be changed as appropriate. [Explanation of symbols]

[0121] 10 Image processing device, 12 Image acquisition unit, 14 Image display unit, 16 Detection result acquisition unit, 18 Detection result display unit, 20 Detection result correction unit, 22 Camera, 30 Processor, 32 Image.

Claims

1. It has a processor, The aforementioned processor, The system accepts the detection results for each object obtained from image data representing multiple objects. The image based on the aforementioned image data is displayed on the display, and for detection results where the confidence level exceeds a threshold, a guide representing the shape and size of the object is superimposed on the image and displayed on the display. If a position where the guide is not displayed is specified in the image displayed on the display, an index is calculated for each of the multiple guides that are not displayed corresponding to the detection result with a confidence level below the threshold, the index increases as the specified position is closer to the position of the guide and increases as the confidence level of the guide increases, and the guide with the largest calculated index is superimposed on the image and displayed on the display. Information processing device.

2. The aforementioned processor, If a guide is specified to be displayed superimposed on the aforementioned image, the specified guide will not be displayed superimposed on the aforementioned image. The detection results corresponding to the specified guide that are no longer displayed superimposed on the aforementioned image are excluded from each of the detection results. The information processing apparatus according to claim 1.

3. The size of the additional guide is determined based on the size of one or more guides superimposed on the image. The information processing apparatus according to claim 1.

4. The aforementioned processor, If a guide is specified to be displayed superimposed on the aforementioned image, the specified guide will not be displayed superimposed on the aforementioned image. The size of the additional guide is determined based on the size of one or more guides that are displayed superimposed on the image after the designated guide is no longer displayed superimposed on the image. The information processing apparatus according to claim 3.

5. The aforementioned processor, If a guide is specified to be displayed superimposed on the aforementioned image, the specified guide will not be displayed superimposed on the aforementioned image. To display on the display an operator for causing the specified guide, which has disappeared from being superimposed on the aforementioned image, to be superimposed on the aforementioned image again. The information processing apparatus according to any one of claims 1 to 4.

6. The aforementioned processor, In response to user interaction with the guide, change at least one of the guide's position and size. The information processing apparatus according to any one of claims 1 to 5.

7. The aforementioned processor, A first operator for instructing the guide to return to its state before the operation was performed on the guide, and a second operator for instructing the guide to repeat the operation, are displayed on the screen. The state of the guide is controlled in response to the user's operation on the first or second operator. The information processing apparatus according to any one of claims 1 to 4.

8. Computers The system accepts the detection results for each object obtained from image data representing multiple objects. The image based on the aforementioned image data is displayed on the display, and for detection results where the confidence level exceeds a threshold, a guide representing the shape and size of the object is superimposed on the image and displayed on the display. If a position where the guide is not displayed is specified in the image displayed on the display, an index is calculated for each of the multiple guides that are not displayed corresponding to the detection result with a confidence level below the threshold, the index increases as the specified position is closer to the position of the guide and increases as the confidence level of the guide increases, and the guide with the largest calculated index is superimposed on the image and displayed on the display. A program designed to make it work in that way.

Images