Information processing apparatus, control method of the same, and program

Abstract

To provide a user with an environment in which it is possible to efficiently determine whether or not there is an abnormality in a position and a size of a verification part of an object in each of a plurality of images used as learning data.SOLUTION: An information processing apparatus that assists in determining whether information representing a position and a size of a verification part of an object in an image is correct or incorrect, comprises: an acquisition unit for acquiring a plurality of images, and reference frame information representing a position and a size of a reference frame that includes the object and verification frame information representing a position and a size of a verification frame that includes the verification part of the object in each of the plurality of images; a normalization unit for normalizing the size of the reference frame represented by the acquired reference frame information and normalizing the size and the position of the corresponding verification frame according to the normalization; and a display control unit for displaying, for each of the plurality of images, the normalized reference frame at a preset position and superimposing and displaying the normalized verification frame at a relative position according to the normalized position and size for the normalized reference frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program. [Background technology]

[0002] In recent years, many methods have been proposed for processing captured images and detecting objects in the images. In particular, there has been active research into a method for learning the features of objects in an image and recognizing the position and type of the object by using a multi-layered neural network called a deep net (also called a deep neural network or deep learning). Non-Patent Document 1 discloses a method for detecting objects from an image using a deep net.

[0003] To learn the features of an object, a person needs to set correct answer information such as the position and size of the object in the image. This image and correct answer information are called training data. To create a highly accurate recognizer, a large amount of training data needs to be prepared. Patent Document 1 describes a method for acquiring training data with sufficient accuracy by repeating "an operation in which a person adds correct answer information" and "an operation in which the accuracy of the detector is evaluated" until the desired accuracy is reached. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5953151 [Patent Document 2] JP 2019-46095 A [Non-patent literature]

[0005] [Non-Patent Document 1] SSD: Single Shot MultiBox Detector, Wei Liu et al., 2015 [Non-Patent Document 2] Masanari Abe Proposal and comparison of threshold setting methods for the MT method [Non-Patent Document 3] Jiankang Deng et al., “RetinaFace: Single-stage Dense Face Localization in the Wild” 2 May 2019 Summary of the Invention [Problem to be solved by the invention]

[0006] When a user manually prepares training data, there is a possibility that the correct answer information for the position or size may be input in an incorrect state due to operational mistakes or misunderstanding of the definition of the training data. Therefore, the technology described in Patent Document 1 still has a problem that the accuracy of the recognizer decreases if training is performed using training data including incorrect correct answer information.

[0007] Patent Document 2 describes a method that enables a user to efficiently review learning data by selecting and displaying images of learning data with low reliability and correct answer information. However, the method of Patent Document 2 only improves the efficiency of a method for checking a single image, and there is a problem that it takes a long time to check correct answer information for multiple images.

[0008] The present invention has been made in consideration of the above-mentioned problems, and provides a user with an environment in which it is possible to efficiently determine whether or not there is an abnormality in the position and size of a verification portion of an object in each of multiple images used as learning data. [Means for solving the problem]

[0009] In order to solve this problem, for example, an information processing device of the present invention has the following arrangement. An information processing device that supports a determination of whether information representing a position and size of a verification portion of an object in an image is correct, an acquisition means for acquiring a plurality of images, reference frame information representing a position and a size of a reference frame surrounding an object in each of the plurality of images, and verification frame information representing a position and a size of a verification frame surrounding a verification portion of the object; a normalization means for normalizing a size of a reference frame represented by the acquired reference frame information and normalizing a size and a position of a corresponding verification frame according to the normalization; The display control means has a display control means for displaying a normalized reference frame at a predetermined position for each image in the plurality of images, and superimposing a normalized verification frame at a relative position according to the normalized position and size relative to the normalized reference frame. Effect of the Invention

[0010] According to the present invention, a user can efficiently determine whether or not there is an abnormality in the position and size of a verification portion of an object in each of a plurality of images used as learning data. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration according to a first embodiment. [Diagram 2] FIG. 2 is a functional configuration diagram of an information processing device according to the first embodiment. [Diagram 3] 5 is a flowchart showing a processing flow of the information processing device according to the first embodiment. [Figure 4] 5A and 5B are diagrams showing examples of an image and frame information according to the first embodiment. [Diagram 5] 5 is a flowchart showing the flow of normalization processing in the first embodiment. [Figure 6] 5A to 5C are diagrams showing an example of display of frame information of a normalization reference frame and a normalization verification frame and an example of a display transition according to the first embodiment. [Figure 7] FIG. 11 is a functional configuration diagram of an information processing device according to a second embodiment. [Figure 8] 10 is a flowchart showing a processing flow of an information processing device according to a second embodiment. [Figure 9]10 is a flowchart showing the flow of a statistical information calculation process in the second embodiment. [Figure 10] 13A to 13C are diagrams showing an example of frame information of a normalization reference frame and a normalization verification frame, a display example of statistical information, and an example of selection of statistical information according to the second embodiment. [Figure 11] FIG. 13 is a functional configuration diagram of an information processing device according to a third embodiment. [Figure 12] 13 is a flowchart showing the flow of a statistical information calculation process in the third embodiment. [Figure 13] FIG. 13 is a functional configuration diagram of an information processing device according to a fourth embodiment. [Figure 14] 5A and 5B are diagrams illustrating an example of information stored in a frame information storage unit according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] [First embodiment] In this embodiment, a tool for assisting in the verification and correction of a frame that is correct information of the pupil of a person that is input in advance to an image showing the face of the person will be described as an example. The frame of the head of the person, which is correlated with the pupil in position or size, is used as a reference frame (hereinafter, reference frame), and the validity of the verification frame is verified by comparing the relative position and relative size with the frame of the pupil to be verified (hereinafter, verification frame). The correspondence relationship between the input image, the reference frame, and the verification frame will be described later in FIG. 4. In this embodiment, an example in which there is a correlation between the position and the size is illustrated, but it is sufficient if there is a correlation in either one. In addition, a coordinate point indicating only the position of an object without size information may be set, or size information of an object that appears randomly on the image and has no correlation with the position may be compared. In addition, although it is assumed that the verification part is the pupil, this is for convenience, and it may be a face part other than the pupil. <System configuration> An example of the system configuration of an information processing device 100 according to this embodiment is shown in Fig. 1. The information processing device 100 has a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F device 16 as a system configuration.

[0014] The control device 11 controls the entire information processing device 100, and is composed of a CPU and a memory that stores programs executed by the CPU.

[0015] The storage device 12 holds programs and data necessary for the operation of the control device 11, and is typically a hard disk drive or the like.

[0016] The arithmetic unit 13 executes necessary arithmetic processing based on the control from the control device 11.

[0017] The input device 14 is a human interface device or the like, and transmits user operations to the information processing device 100. The input device 14 is composed of a group of input devices such as switches, buttons, keys, a touch panel, and a keyboard.

[0018] The output device 15 is a display or the like, and presents the processing results of the information processing device 100 to the user.

[0019] The I / F device 16 is a wired interface such as a universal serial bus, Ethernet, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth. An imaging device such as a camera can be connected to the I / F device 16. The I / F device 16 also functions as an interface for importing an image captured by the imaging device into the information processing device 100. The I / F device 16 also functions as an interface for transmitting a processing result obtained by the information processing device 100 to the outside. Furthermore, the I / F device 16 also functions as an interface for inputting a program, data, etc. required for the operation of the information processing device 100 into the information processing device 100.

[0020] 2 is a diagram showing the functional configuration of the information processing device 100. The information processing device 100 includes an image storage unit 101, a frame information storage unit 102, a normalization processing unit 103, a display control unit 104, a user operation acquisition unit 105, and a frame information correction unit .

[0021] 1 loads a program stored in the storage device 12 into a memory and executes it. It should be understood that each functional unit constituting the functional configuration diagram in FIG. 2 functions as a result of the control device 11 executing a program.

[0022] The image storage unit 101 stores a plurality of images. The images to be stored may be images captured by a camera or the like, images recorded in a storage device such as a hard disk, or images received via a network such as the Internet. The image storage unit 101 is realized by, for example, the storage device 12.

[0023] The frame information storage unit 102 is linked to each image stored in the image storage unit 101, and stores a table for managing pre-entered frame information. In this embodiment, the frame information is information regarding the presence of a target object (person) in an image, and is information indicating the position and size of a reference frame (typically a circumscribing rectangular frame) that encompasses a target part (face) on the image, as well as the position and size of a frame that encompasses facial features (eyes in this embodiment). The position is the two-dimensional coordinate value of the upper left corner of the frame. The size is a value that represents the horizontal length and vertical length of the frame. Moreover, this frame information storage unit 102 is realized by, for example, the storage device 12.

[0024] FIG. 14 shows an example of a table held by the frame information holding unit 102. The first field of the table is an ID that identifies an image file. It may be an image file name if it identifies the image file. The second field is the size of the image represented by the image file (number of pixels in the horizontal and vertical directions). The third field is the position and size of a reference frame that encompasses the face area of ​​a person in the image. The position of the upper left corner of the image is defined as the origin (0,0), and the horizontal right direction from the origin is defined as the positive direction of the x axis, and the vertical downward direction is defined as the positive direction of the y axis. The position of the reference frame represents the position of the upper left corner of the reference frame, and the size of the reference frame is the horizontal and vertical size (number of pixels) of the reference frame.

[0025] The fourth field of the table is the position and size of the rectangular frame that contains verification frame A (for example, a person's right eye). The definition of the position and size is as explained in the verification frame. The fifth field indicates the correctness confirmation flag for verification frame A in the fourth field, and initially stores "0" indicating unconfirmed.

[0026] The sixth field indicates the position and size of a rectangular frame that encompasses verification frame B (e.g., a person's left eye). The seventh field indicates a correctness confirmation flag for verification frame B in the sixth field, and initially stores "0" indicating unconfirmed.

[0027] The normalization processing unit 103 performs normalization processing on the multiple frames acquired from the frame information storage unit 102. Here, normalization processing refers to conversion processing on the two-dimensional coordinates of the frame. For example, it is a process of converting a certain reference frame so that it has a fixed position and a fixed size on the two-dimensional coordinates of the image. The verification frame is also converted in the same way according to the normalized reference frame. The purpose of the normalization processing is to make it easier to grasp the relative positions and relative sizes of the reference frames and verification frames of each image.

[0028] The display control unit 104 displays the reference frame after normalization by the normalization processing unit 103 (hereinafter referred to as the normalized reference frame), the verification frame after normalization (hereinafter referred to as the normalized verification frame), and the image stored in the image storage unit 101 on the output device 15.

[0029] The user operation acquisition unit 105 acquires user operation information inputted via the input device 14 .

[0030] The frame information correction unit 106 corrects the frame information in accordance with the user operation acquired by the user operation acquisition unit 105 , and stores the corrected frame information in the frame information storage unit 102 .

[0031] Next, an example of the flow of processing by the information processing device 100 according to this embodiment will be described with reference to FIG.

[0032] In S301, the control device 11 refers to the table held in the frame information holding unit 102 and acquires frame information of the reference frame and the verification frame.

[0033] Fig. 4(a) is a diagram showing an example of an image and frame information. In Fig. 4(a), reference symbol 401 is an image including a target object (person), reference symbol 403 is a reference frame corresponding to a target part (head) of the target object, and reference symbols 404 and 405 are verification frames (eyes in this embodiment). Fig. 4(a) also shows another person image 402. This image 402 also shows a reference frame 406 and verification frames 407 and 408. For simplicity, it is assumed that one person is shown in the images 401 and 402.

[0034] In S301, the control device 11 refers to the table (FIG. 14) stored in the frame information storage unit 102, and acquires frame information of the reference frames (reference numerals 403, 406, etc. in FIG. 4) and verification frames (reference numerals 404, 405, 407, 408, etc.) of each image.

[0035] In S302, the normalization processing unit 103 performs normalization processing on the acquired reference frame and verification frame. The flow of the normalization processing is shown in Fig. 5 and will be described. Here, the reference frame 403 in Fig. 4(a) will be described as an example.

[0036] In S501, the normalization processing unit 103 changes the magnification (shrinks or enlarges) of the reference frame and the verification frame so that the width and height of the reference frame 403 become fixed sizes. For example, if the target fixed size is 500 pixels in both the horizontal and vertical directions and the horizontal size of the reference frame 403 is 400 pixels and the vertical size is 300, the normalization processing unit 103 sets the horizontal magnification to 1.25 times (=500 / 400) and the vertical magnification to 1.67 times (=500 / 300). Then, the normalization processing unit 103 changes the position and size of the reference frame according to the determined vertical and horizontal magnifications. For example, if the reference frame of the image ID=0001 in FIG. 14 is the above-mentioned reference frame 403, the normalization processing unit 103 multiplies RX1 and RW including the horizontal components by 1.25 and multiplies RY1 and RH1 including the vertical components by 1.67. Furthermore, the normalization processor 103 also changes the positions and sizes of the verification frames A and B in accordance with the determined vertical and horizontal magnifications.

[0037] In S502, the normalization processing unit 103 translates the coordinates of the normalized reference frame center to the specified position. For example, if the specified position is (x, y) = (500 pixels, 500 pixels) and the coordinates of the center of the reference frame 403 are (x, y) = (300 pixels, 200 pixels), the reference frame 403 is translated by +200 pixels in the x direction and +300 pixels in the y direction. Similarly, the coordinates of the verification frames 404 and 405 are translated.

[0038] In S503, the normalization processing unit 103 holds verification box information in the peripheral region of the reference frame 403. For example, the storage device 12 holds verification box information whose x and y coordinates on the coordinate system are included in the range of 0 to 1000 pixels.

[0039] The normalization processing unit 103 repeats the process of steps S501 to S503 for all the reference frames acquired in S301. In this manner, normalized reference frame information, which is frame information of a plurality of normalized reference frames (hereinafter, normalized reference frames), and normalized verification frame information, which is frame information of a normalized verification frame (hereinafter, normalized verification frame), are obtained.

[0040] Next, with reference to FIG. 4(b), a display example of the normalization reference frame and the normalization verification frame will be described. In FIG. 4(b), reference numeral 410 indicates the normalization reference frame. Even if the size and reference frame of each image are different, the normalization reference frame is the same size, and no deviation occurs. Reference numeral 412 and a plurality of solid-line frames in the normalization reference frame 410 are normalization verification frames. Reference numeral 411 is a frame representing the peripheral area calculated in S503. Since there is a correlation between the positions of the head and the pupil, it can be seen that the normalization verification frame 412 corresponding to the verification frame 408 that does not correctly represent the position of the pupil is in a position that is significantly shifted from the other verification frames. In this way, by superimposing and displaying the normalization reference frames and normalization verification frames of a plurality of images, a plurality of frames can be simultaneously checked and unnatural frames can be identified.

[0041] Returning to the explanation of Fig. 3, in S303, the display control unit 104 controls the output device 15 to display the frame information of the normalization reference frame and normalization verification frame calculated in S302, and the statistical information calculated in S303.

[0042] A display example and a display transition example of frame information of the normalization reference frame and the normalization verification frame will be described with reference to Fig. 6. Reference numeral 601 in Fig. 6(a) is a window displayed on the output device 15. Reference numeral 411 in the window 601 is a frame representing the peripheral area of ​​the normalization reference frame exemplified in Fig. 4(b). In the peripheral area 411, a plurality of normalization verification frames for the normalization reference frame are displayed superimposed on each other.

[0043] In S304, the user operation acquisition unit 105 selects a verification frame according to user input. Here, the user input is received as a selection of a verification frame by operating a pointing device such as a mouse. In FIG. 6(b), reference numeral 602 indicates a mouse cursor, and the user can select a desired verification frame by changing the position of this mouse cursor. In the case of the embodiment, the user selects the normalized verification frame 412, which is unnaturally separated from the other verification frames, in the window 601. Note that when using touch input, the user only needs to touch the normalized verification frame 412, and therefore there is no need to display the mouse cursor.

[0044] In S305, the display control unit 104 receives the verification frame information selected in S305 and transitions the screen from window 601 in Fig. 6(b) to window 603 in Fig. 6(c) which can be edited by the user. At this time, the display control unit 104 refers to the table in Fig. 14 and displays the image 402, reference frame 406, and verification frames 407 and 408 linked to the verification frame 412 selected in S305. In addition, the display control unit 104 arranges and displays a correction button 604 for accepting frame information correction and an OK button 605 for returning to the window 601 in the window 603.

[0045] If the display control unit 104 determines in S306 that the OK button 605 has been pressed, it determines that there is no problem with the frame and skips S307. Furthermore, in order to hide the normalization verification frame for which the OK button has been pressed in S309 described below, the display control unit 104 stores flag information indicating a correct frame as correctness information of the frame in the frame information storage unit 102. For example, the display control unit 104 stores the flag for the corresponding verification frame in the table of FIG. 14 as "1".

[0046] On the other hand, if the display control unit 104 determines in S306 that the Modify button 604 has been pressed, it is determined that there is a problem with the frame, and the process proceeds to S307. In S307, the display control unit 104 transitions to a window 606 in FIG. 6(d) in order to modify the frame, and enables the user to modify the frame information. For example, the position of the verification frame 408 can be modified by continuing to press the center of the verification frame 408 and performing a moving operation (drag operation), and the frame size can be modified by continuing to press the border of the verification frame 408. The position and size of this modified normalized verification frame are subjected to a process reverse to normalization to convert them into a position and size according to the scale of the original image, and the table is then modified.

[0047] 6(d) shows, as an example after correction, that verification frame 408 has been corrected to verification frame 607 in window 606. Frame information regarding the position and size of the corrected verification frame 607 is re-saved in frame information storage unit 102 by frame information correction unit 106 (the table in FIG. 14 is updated).

[0048] In S308, when the display control unit 104 detects pressing of the OK button 605 after correction, the display transitions to window 608 in FIG. 6(e). In addition, in order to hide the normalization verification frame for which the OK button was pressed in S309 described below, a flag indicating that the frame is correct is saved as "1" as the correctness information of the frame. The display control unit 104 hides the verification frame for which the flag is set to "1". As a result, other normalization verification frames that have not been checked become easier to see.

[0049] Next, in S309, the display control unit 104 waits for an instruction input from the user as to whether or not to end the process. When a button (not shown) for instructing to end the series of correction work is pressed or when the correction work for all frames is completed, the display control unit 104 ends this process. Note that when this process is ended, a verification frame whose flag remains at the initial value of "0" is determined to be correct. Then, if the process is ended in S309, the display control unit 104 closes the window 608. If the process is not ended in S309, the display control unit 104 continues to display the window 608 so that the user can confirm and correct the verification frame. Also, if the flag information is set to 1 in S306 or S308, the display control unit 104 hides the corresponding normalization verification frame 412.

[0050] In this embodiment, an example of displaying a rectangular frame as the verification frame has been described, but for example, a polygonal or circular area frame may be set. Also, a coordinate point indicating only the position of an object without size information may be set, or size information of an object that appears randomly on the image and has no correlation with its position may be compared. Furthermore, it may be applied to label information in pixel units. Also, in this embodiment, an example is given of a head frame and a face frame, but a whole body frame and a head frame may be used, or a correspondence between a whole body frame and an arbitrary object held by a person may be used.

[0051] Furthermore, while the above example shows a person, the method can be applied to general objects as well. For example, if we imagine a frame that circumscribes the entire area of ​​a person riding a motorcycle, it is possible to separate a frame that correctly surrounds both the motorcycle and the person from a frame that erroneously surrounds only the motorcycle.

[0052] As described above, the information processing device according to this embodiment simultaneously displays the relative positions of the verification frame (pupils) and the reference frame (head) that is correlated in position and size, thereby enabling the user to efficiently review learning data that is suspected to be erroneous.

[0053] [Second embodiment] In the second embodiment, a configuration for selecting and correcting a normalized validation frame using the distribution of statistical information will be described. Explanations of the same parts as in the first embodiment will be omitted, and only the differences will be described.

[0054] Fig. 7 is a functional configuration diagram of an information processing device 100 in the second embodiment. The difference from Fig. 2 in the first embodiment is that a statistical information calculation unit 107 is added.

[0055] The statistical information calculation unit 107 calculates the relative distance, relative size, and relative angle of the verification frame normalized by the normalization processing unit 103. In addition, the statistical information calculation unit 107 creates graphs such as histograms and scatter diagrams based on the calculated relative distance, relative size, and relative angle.

[0056] The display control unit 104 displays the statistical information calculated by the statistical information calculation unit 107 on the output device 15 .

[0057] An example of the flow of processing by the information processing device 100 according to the second embodiment will be described below with reference to the flowchart of FIG.

[0058] In S801, the statistical information calculation unit 107 calculates statistical information of the normalized verification frame. Details of this statistical information calculation process will be described with reference to the flowchart in FIG.

[0059] In S901, the statistical information calculation unit 107 calculates the distance between the center coordinates of the normalization reference frame and the center coordinates of the normalization verification frame. For example, the Euclidean distance is used as the distance.

[0060] In S902, the statistical information calculation unit 107 calculates the size of the normalized verification frame. For example, the length of the diagonal of the normalized verification frame is set as the size.

[0061] In S903, the statistical information calculation unit 107 calculates the angle of the normalized verification frame. For example, the statistical information calculation unit 107 calculates the angle as the angle of a line between the center coordinates of the normalized reference frame and the center coordinates of the normalized verification frame with respect to the x-axis of the image coordinates, and calculates the cosine similarity based on the angle.

[0062] In S904, the statistical information calculation unit 107 calculates the overlap between the normalization reference frame and the normalization verification frame. For example, the statistical information calculation unit 107 calculates, as the overlap, a ratio (IoU: Intersection over Union) of the area of ​​the intersection (overlap area) of two regions of interest to the area of ​​the union of the two regions.

[0063] In S905, the statistical information calculation unit 107 determines whether the processes from S901 to S904 have been performed for all verification frames.

[0064] If the statistical information calculation unit 107 determines in S905 that there are still verification frames remaining that have not been processed, the process returns to S901, and the process is performed on the next verification frame.

[0065] On the other hand, if the statistical information calculation unit 107 determines in S905 that the processing has been performed on all verification frames, the processing proceeds to S906. In this step S906, the statistical information calculation unit 107 creates a histogram and a scatter diagram based on the calculated relative distance, relative size, and relative angle. The histogram is a histogram of the frequency of verification frames when the horizontal axis is the relative distance, relative size, and relative angle, and is created for the purpose of checking verification frame information that deviates from the distribution of one variable. In addition, the scatter diagram is a scatter diagram of relative distance and relative size, a scatter diagram of relative distance and relative angle, and a scatter diagram of relative size and relative angle, and is created for the purpose of checking frame information that deviates from the distribution of two variables. The distribution of two variables may be displayed as a heat map instead of a scatter diagram.

[0066] Returning to the description of Fig. 8, in S802, the display control unit 104 controls the output device 15 to display the frame information of the normalization reference frame and normalization verification frame calculated in S302, and the statistical information calculated in S801.

[0067] 10(a) to 10(c) show frame information of the normalization reference frame and the normalization verification frame, examples of display of statistical information, and examples of selection of statistical information. Reference numeral 1001 in FIG. 10(a) is a window displayed on the output device 15. Reference numeral 410 in the window 1001 is a normalization reference frame. Reference numerals 412, 1002, 1003, and a solid-line frame in the normalization reference frame 410 are normalization verification frames. Histograms and scatter diagrams of the statistical information calculated in S801 are displayed as shown by reference numerals 1004, 1005, and 1006. Histogram 1004 shows a histogram for distance, and histogram 1005 shows a histogram for size. Also, scatter diagram 1006 is a scatter diagram of size and distance. Here, histograms and scatter diagrams related to angle and overlap are not illustrated, but histograms and scatter diagrams of angle and overlap may be displayed by pressing a button not shown. In addition, the user may be allowed to select the histogram or scatter diagram of the information he or she wishes to display from a pull-down menu (not shown).

[0068] In S803, the display control unit 104 selects a class or region for the distribution of statistical information according to the user's input from the user operation acquisition unit 105. By having the user select a class or region for the distribution of statistical information and limiting the number of displayed verification frames, it becomes easier to check the normalized verification frames. In FIG. 10(b), reference numeral 1007 is a mouse cursor that is linked to a mouse operation. In the illustrated example, the mouse cursor 1007 selects a graph element that represents the largest class of the histogram for distance. In response to this selection, the display control unit 104 transitions the screen from the window 1001 in FIG. 10(b) to the window 1009 in FIG. 10(c). The display control unit 104 allows the user to check which class has been selected by filling in the class selected by the user in the window 1001. In addition, the display control unit 104 displays only the normalized verification frames 412 and 1004 that correspond to the filled-in class in the surrounding area 411, thereby limiting the verification frames to be checked even when many verification frames are displayed.

[0069] Here, an example is shown in which a class of a histogram according to distance is selected, but by selecting the largest class of the size histogram 1005, it is also possible to display only the normalized verification frame 1003 that is larger than the other frames.

[0070] In addition, in the second embodiment, an example is described in which a normalization verification frame is displayed by selecting a rank of statistical information. However, it is also possible to transition to a window 603 in FIG. 6(c) on the screen and prompt the user to check the image and verification frame.

[0071] Furthermore, by selecting an area of ​​the scatter plot 1006 by drawing a circle (not shown) with a mouse or the like, only the normalization display frame included in the circle may be displayed. Also, by specifying the range of the normalization verification frame in 411 of the window 1001 by drawing a circle (not shown), only the normalization verification frame included in the circle may be displayed. Also, in this state, it may be possible to proceed to editing processing as in the first embodiment.

[0072] As described above, in the second embodiment, the distribution of statistical information of the verification frame is visualized, and the distribution class or the group of the normalized verification frame is selected and displayed. This display allows the user to visually identify only the verification frame suspected to be erroneous, facilitating the task of checking the verification frame.

[0073] [Third embodiment] In the third embodiment, a configuration for automatically selecting a normalization verification frame suspected to be erroneous using statistical information will be described. Explanations of the same parts as in the second embodiment will be omitted, and only the differences will be described.

[0074] Fig. 11 is a functional configuration diagram of an information processing device 100 in the third embodiment. The difference from Fig. 7 of the second embodiment is that an error verification frame information determination unit 108 is added.

[0075] The error verification frame information determination unit 108 determines frames with a high probability of error based on statistical information. As the statistical information, one normalized verification frame is assumed to have four vector components, namely, relative distance, relative size, relative angle, and overlapping degree, and the Mahalanobis distance described in Non-Patent Document 2 is calculated. If the Mahalanobis distance exceeds a preset threshold, the normalized verification frame is determined to be one with a high probability of error.

[0076] The flow of the statistical information calculation process according to the third embodiment is shown in Fig. 12. Only the parts that differ from the flow of the statistical information calculation process in Fig. 9 according to the second embodiment will be explained.

[0077] In S905, the statistical information calculation unit 107 determines whether the processes in S901 to S904 have been performed on all verification frames. If the statistical information calculation unit 107 determines in S905 that the processes on all verification frames have been completed, the statistical information calculation unit 107 performs the process in S906, and in S1201 calculates the Mahalanobis distance of the distance, size, angle, and overlap for each verification frame.

[0078] Then, in S1202, the statistical information calculation unit 107 determines whether or not there is a normalized verification frame whose Mahalanobis distance exceeds a threshold. For example, the threshold specified here is set to 1. Next, in S802, the display control unit 104 displays only the normalized verification frame whose Mahalanobis distance exceeds the threshold.

[0079] In the present embodiment, an example in which the threshold value is set in advance has been described, but the threshold value may be changed arbitrarily by the user using an input form (not shown).Furthermore, instead of a single threshold value, multiple threshold values ​​may be set, and a normalization verification frame may be displayed for each region separated by multiple threshold values, and the normalization verification frame may be switched by a button (not shown).

[0080] Moreover, instead of hiding the normalized verification frame that does not exceed the threshold, it may be possible to color-code it to make it easier to see, or the Mahalanobis distance may be displayed near the frame to provide the user with information for making a decision.

[0081] In this embodiment, the normalized verification window is limited using the Mahalanobis distance, but for example, values ​​that are three or more times the standard deviation away from the average value may be set as outliers and used as error verification window candidates. Also, using the median and quartiles, values ​​that are a quarter difference away from the first quartile value may be set as outliers and used as error verification window candidates.

[0082] As described above, according to the third embodiment, outliers in the statistical information of a verification frame are determined by threshold processing from the statistical information of the verification frame. This makes it possible to suggest verification frames suspected to be erroneous to the user, facilitating the task of checking the verification frame.

[0083] [Fourth embodiment] In the fourth embodiment, a configuration will be described in which a reference frame is not a prepared frame, but a frame detected by an object frame detection unit is used to perform normalization processing and select a verification frame. Explanations of the same parts as in the third embodiment will be omitted, and only the differences will be described.

[0084] 13 is a functional configuration diagram of information processing device 100 in the fourth embodiment. The difference is that an object frame detection unit 109 is provided in addition to the configuration of the third embodiment.

[0085] When a pair of an image and a verification frame is input, this object frame detection unit 109 detects a reference frame from the image using, for example, a hierarchical convolutional neural network as shown in Non-Patent Documents 1 and 3. This makes it possible to verify the verification frame against the reference frame without having to prepare the reference frame in advance, thereby saving the effort of inputting the reference frame.

[0086] As a method for verifying the detection frame by object detection unit 109, a configuration may be adopted in which a normalization process is performed on a reference frame prepared in advance and a verification frame detected using the object frame detection unit, and a verification frame is selected.

[0087] The first to fourth embodiments have been described above. In the above embodiments, the verification frame represents the human eye, so there are two verification frames for one reference frame. However, it should be noted that the number of verification frames may be one or more, and there is no particular limit to the number.

[0088] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0089] The disclosure of this specification includes the following information processing device, method, and program. (Item 1) An information processing device that supports a determination of whether information representing a position and size of a verification portion of an object in an image is correct, an acquisition means for acquiring a plurality of images, reference frame information representing a position and a size of a reference frame surrounding an object in each of the plurality of images, and verification frame information representing a position and a size of a verification frame surrounding a verification portion of the object; a normalization means for normalizing a size of a reference frame represented by the acquired reference frame information and normalizing a size and a position of a corresponding verification frame according to the normalization; a display control means for displaying a normalized reference frame at a preset position for each of the plurality of images, and superimposing a normalized verification frame at a relative position corresponding to the normalized position and size with respect to the normalized reference frame; 13. An information processing device comprising: (Item 2) A selection means for selecting the displayed verification frame; editing means for modifying the size and position of the selected verification frame; The display control means displays the verification frame edited by the editing means in a non-display manner. 2. The information processing device according to item 1, (Item 3) 3. The information processing device according to item 2, wherein the plurality of images, the reference frame information of each image, and the verification frame information edited by the editing means are used as learning data. (Item 4) The reference frame is a frame that includes a face of a person in an image, and the verification frame is at least one frame that includes a part that constitutes the face. 4. The information processing device according to any one of items 1 to 3. (Item 5) a calculation means for calculating at least one piece of statistical information representing a relative deviation of a position and a size of each verification frame from the reference frame information and the verification frame information normalized by the normalization means, The display control means Displaying the statistical information calculated by the calculation means as a graph; When an element of the displayed graph is selected by the selection means, only the verification frame belonging to the selected element is displayed. 5. The information processing device according to any one of items 1 to 4. (Item 6) 6. The information processing device according to item 5, wherein the calculation means calculates statistical information from a relative distance, a relative size, or a relative angle between the reference frame and the verification frame. (Item 7) a determination means for determining whether or not an error exists by calculating a value representing a degree of an error in the position and size of the verification frame based on the statistical information calculated by the calculation means and comparing the calculated value with a preset threshold value; The display means displays the verification frame determined by the determination means to contain an error and the corresponding image in an editable manner. 6. The information processing device according to item 5, (Item 8) 8. The information processing device according to item 7, wherein the determination means calculates a Mahalanobis distance as a value representing the degree of error. (Item 9) The apparatus further comprises an object detection means for detecting the target object in an image inputted thereto in order to detect the reference frame, The acquisition means acquires the image acquired by the object detection means and reference frame information for an object in the image. 9. The information processing device according to any one of items 1 to 8. (Item 10) A method for controlling an information processing device that supports a determination of whether information representing a position and a size of a verification portion of an object in an image is correct, comprising: an acquiring step of acquiring a plurality of images, reference frame information representing a position and a size of a reference frame surrounding an object in each of the plurality of images, and verification frame information representing a position and a size of a verification frame surrounding a verification portion of the object; a normalization step of normalizing the size of the reference frame represented by the acquired reference frame information and normalizing the size and position of the corresponding verification frame according to the normalization; a display control step of displaying a normalized reference frame at a preset position for each of the plurality of images, and superimposing a normalized verification frame at a relative position corresponding to the normalized position and size with respect to the normalized reference frame; 13. A method for controlling an information processing apparatus comprising the steps of: (Item 11) A program that, when read and executed by a computer, causes the computer to function as each of the means possessed by the device described in any one of items 1 to 9.

[0090] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0091] 100: information processing device, 11: control device, 12: storage device, 13: arithmetic device, 14: input device, 15: output device, 16: I / F device, 101: image storage unit, 102: frame information storage unit, 103: normalization unit, 104: display control unit, 105: user operation acquisition unit, 106: frame information correction unit

Claims

1. An information processing apparatus for assisting in determining the correctness of information representing the position and size of a verification part of an object in an image, comprising: acquisition means for acquiring a plurality of images, reference frame information representing the position and size of a reference frame including the object in each of the plurality of images, and verification frame information representing the position and size of a verification frame including the verification part of the object; normalization means for normalizing the size of the reference frame represented by the acquired reference frame information and normalizing the size and position of the corresponding verification frame according to the normalization; display control means for displaying the normalized reference frame at a preset position for each image in the plurality of images and superimposing and displaying the normalized verification frame at a relative position corresponding to the normalized position and size with respect to the normalized reference frame; An information processing apparatus characterized by comprising the above.

2. selection means for selecting the displayed verification frame; editing means for correcting the size and position of the selected verification frame, and the display control means makes the verification frame edited by the editing means invisible. The information processing apparatus according to claim 1, characterized by the above.

3. The plurality of images, the reference frame information of each image, and the verification frame information edited by the editing means are used as learning data. The information processing apparatus according to claim 2, characterized by the above.

4. The reference frame is a frame including the face of a person in the image, and the verification frame is at least one frame including a part constituting the face. The information processing apparatus according to claim 1, characterized by the above.

5. calculation means for calculating at least one statistical information representing the relative deviation of the position and size of each verification frame from the normalized reference frame information and the verification frame information by the normalization means; selection means for selecting an element displayed by the display control means; further comprising: the display control means: displays the statistical information calculated by the calculation means as a graph; when an element of the displayed graph is selected by the selection means, only the verification frame belonging to the corresponding element is displayed. The information processing apparatus according to claim 1, characterized by the above.

6. The calculation means calculates statistical information from the relative distance, relative size, or relative angle between the reference frame and the verification frame. The information processing apparatus according to claim 5, characterized by the above.

7. Based on the statistical information calculated by the calculating means, a value representing the degree of error in the position and size of the verification frame is calculated and compared with a preset threshold value, and further comprising determination means for determining whether there is an error. The display control means displays the verification frame determined to have an error by the determination means and the corresponding image in an editable manner. The information processing apparatus according to claim 5, characterized in that.

8. The determination means calculates the Mahalanobis distance as a value representing the degree of error. The information processing apparatus according to claim 7, characterized in that.

9. It further has object detection means for detecting an object in the image in order to input the image and detect the reference frame. The acquisition means acquires the image obtained by the object detection means and the reference frame information for the object in the image. The information processing apparatus according to claim 1, characterized in that.

10. A control method for an information processing apparatus that supports determination of correctness of information representing the position and size of a verification part of an object in an image, comprising: An acquisition step of acquiring a plurality of images, reference frame information representing the position and size of a reference frame including an object in each of the plurality of images, and verification frame information representing the position and size of a verification frame including a verification part of the object; A normalization step of normalizing the size of the reference frame represented by the acquired reference frame information and normalizing the size and position of the corresponding verification frame according to the normalization; For each of the plurality of images, the normalized reference frame is displayed at a preset position, and the normalized verification frame is superimposed and displayed at a relative position corresponding to the normalized position and size with respect to the normalized reference frame. A control method for an information processing apparatus, characterized by comprising.

11. A program for causing a computer to execute each step of the control method according to claim 10 when the computer reads and executes it.

Images