Information processing apparatus, control method therefor, and program

Abstract

An information processing apparatus for assisting in determining correctness of information for indicating a position and a size of a verification site of an object in an image, the apparatus including: an obtaining unit configured to obtain a correctness of information indicating a position and a size of the verification site; the obtaining unit is used for obtaining a plurality of images, reference frame information used for indicating the position and size of a reference frame surrounding an object in the images, and verification frame information used for indicating the position and size of a verification frame surrounding a verification part of the object in the images; the standardization unit is used for standardizing the size of the reference frame indicated by the obtained reference frame information and standardizing the size and the position of the corresponding verification frame according to the standardization; and a display control unit for displaying the standardized reference frame at a preset position for each of the plurality of images, and displaying the standardized verification frame at a relative position corresponding to the standardized reference frame based on the standardized position and size in an overlapping manner.

Description

Technical Field

[0001] The present invention relates to an information processing device, a control method and a program thereof. Background Art

[0002] In recent years, a large number of technologies for processing captured images and detecting objects in the images have been proposed. Among them, in particular, a technology for learning the features of objects in an image and recognizing the position and type of the object using a multi-layer neural network called a "deep network" (or also called a "deep neural network" or "deep learning") has been actively studied. Non-patent document 1 discloses a technology for detecting objects in an image using a deep network.

[0003] In order to train the features of an object, it is necessary for a person to set ground truth information such as the position and size of the object in the image. Such images and ground truth information are called training data. A large amount of training data needs to be prepared to develop an accurate recognizer. Patent document 1 describes a method for obtaining training data that fully ensures accuracy by repeating "the operation of adding ground truth information performed by a person" and "the operation of evaluating the accuracy of the detector" until the desired accuracy is achieved.

[0004] List of existing technologies

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 5953151

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2019-46095

[0008] Non-patent literature

[0009] NPL1: SSD: Single Shot MultiBox Detector, Wei Liu et al., 2015

[0010] NPL2:Abe Masanari,Proposal and Comparison of Threshold Setting Methodin MT Method

[0011] NPL3: Jiankang Deng et al., "RetinaFace: Single-stage Dense FaceLocalisation in the Wild" 2May 2019 Summary of the invention

[0012] Problem that the invention aims to solve

[0013] When a user manually prepares training data, the correct information of the position and size may be incorrectly input due to an operation error or a misunderstanding of the definition of the training data. Therefore, in the case of the technology in Patent Document 1, there is a problem in that when training is performed using training data including incorrect correct information, the accuracy of the recognizer will be reduced.

[0014] Patent document 2 describes a method for selecting images and correct information of low-reliability training data and displaying the selected images and information, thereby enabling a user to efficiently check the training data. However, the method of patent document 2 only improves the efficiency of the method for checking a single image, and there is a problem that it takes time to check the correct information of multiple images.

[0015] The present invention has been made in view of the above problems, and provides a user with an environment in which it is possible to efficiently determine whether there is abnormality in the position and size of a verification part of an object in each of a plurality of images to be used as training data.

[0016] Solutions for solving problems

[0017] In order to solve this problem, for example, an information processing apparatus according to the present invention has the following configuration.

[0018] An information processing device for assisting in determining the correctness of information indicating the position and size of a verification part of an object in an image, the information processing device comprising:

[0019] an obtaining unit configured to obtain a plurality of images, reference frame information indicating a position and a size of a reference frame surrounding an object in the image, and verification frame information indicating a position and a size of a verification frame surrounding a verification portion of the object in the image,

[0020] a standardization unit, configured to standardize the size of the reference frame indicated by the obtained reference frame information, and to standardize the size and position of the corresponding verification frame according to the standardization, and

[0021] A display control unit is used to display a standardized reference frame at a preset position for each of the multiple images, and to display a standardized verification frame in a superimposed manner at a relative position based on the standardized position and size corresponding to the standardized reference frame.

[0022] Effects of the Invention

[0023] According to the present invention, it is possible to provide a user with an environment in which it is possible to efficiently determine whether there is an abnormality in the position and size of a verification portion of an object in each of a plurality of images to be used as training data.

[0024] Other features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0026] Figure 1 is a diagram showing an example of a system configuration according to the first embodiment.

[0027] Figure 2 is a diagram of a functional configuration of an information processing apparatus according to the first embodiment.

[0028] Figure 3 is a flowchart showing the flow of processing performed by the information processing apparatus according to the first embodiment.

[0029] Figure 4A is a diagram showing an example of an image and frame information according to the first embodiment.

[0030] Figure 4B is a diagram showing an example of a normalized reference frame and a normalized verification frame.

[0031] Figure 5 is a flowchart showing the flow of the normalization process according to the first embodiment.

[0032] Fig. 6A : is a diagram showing a display example of frame information of a standardized reference frame and a standardized verification frame according to the first embodiment.

[0033] Figure 6B is a diagram illustrating an example of selection of a verification box based on a user input.

[0034] Figure 6C FIG. 2 is a diagram showing an example of a screen display for accepting an editing operation performed by a user.

[0035] Fig.6D is a diagram showing an example of a verification frame corrected by a user.

[0036] Fig. 6E : is a figure which shows the display example of the frame information after correction.

[0037] Figure 7 is a diagram of a functional configuration of an information processing apparatus according to the second embodiment.

[0038] Figure 8 is a flowchart showing the flow of processing performed by the information processing apparatus according to the second embodiment.

[0039] Fig. 9 is a flowchart showing the flow of statistical information calculation processing according to the second embodiment.

[0040] Fig. 10A : is a diagram showing a display example of frame information of a normalized reference frame and a normalized verification frame, and statistical information according to the second embodiment.

[0041] Fig. 10B is a diagram showing an example of selection of statistical information based on user input.

[0042] Fig. 10C is a diagram showing a display example of frame information when statistical information is selected.

[0043] Fig.11 is a diagram of a functional configuration of an information processing apparatus according to the third embodiment.

[0044] Fig.12 is a flowchart showing the flow of statistical information calculation processing according to the third embodiment.

[0045] Fig.13 is a diagram of a functional configuration of an information processing apparatus according to a fourth embodiment.

[0046] Fig.14 : is a diagram showing an example of information held in a frame information holding unit according to the embodiment. DETAILED DESCRIPTION

[0047] Embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to inventions requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0048] [First embodiment]

[0049] In this embodiment, as an example, a description will be given of a tool for assisting verification and correction of a frame in an image showing a human face, where the frame is the correct information of the human pupil and is input in advance. The validity of the verification frame is verified by defining a frame of the human head related to the pupil in terms of position or size as a reference frame (hereinafter referred to as the "reference frame"), and comparing the relative position and size of the reference frame with the position and size of the frame of the pupil to be verified (hereinafter referred to as the "verification frame"). This will be referred to later. Figure 4A and Figure 4BDescribe the correspondence between the input image, the reference frame, and the verification frame. In this embodiment, an example in which there is a correlation in position and size will be shown, but it is sufficient that there is a correlation in one of the position and size. In addition, a coordinate point indicating only the position of an object without size information may be set, or the size information of an object that appears randomly in the image and has no correlation in position may be compared. In addition, for convenience, the verification part is the pupil, but it may be a part of the face other than the pupil.

[0050] <System Configuration>

[0051] Figure 1 An exemplary system configuration of the information processing apparatus 100 according to the present embodiment is shown. The information processing apparatus 100 includes a control device 11 , a storage device 12 , a calculation device 13 , an input device 14 , an output device 15 , and an I / F device 16 as the system configuration of the information processing apparatus 100 .

[0052] The control device 11 performs overall control of the information processing apparatus 100 , and is constituted by a CPU and a memory storing a program executed by the CPU.

[0053] The storage device 12 holds programs and data necessary for the operation of the control device 11 , and is typically a hard disk drive or the like.

[0054] The calculation device 13 performs necessary calculation processing under the control of the control device 11 .

[0055] The input device 14 is a human interface device or the like, and transmits a user operation to the information processing apparatus 100. The input device 14 is composed of an input device group including a switch, a button, a key, a touch panel, a keyboard, and the like.

[0056] The output device 15 is a display or the like, and presents processing results or the like of the information processing apparatus 100 to the user.

[0057] The I / F device 16 is a wired interface such as a universal serial bus, Ethernet, or an optical cable, or a wireless interface such as Wi-Fi or Bluetooth. An image pickup device such as a camera can be connected to the I / F device 16. In addition, the I / F device 16 is used as an interface for taking an image captured by the image pickup device into the information processing device 100. In addition, the I / F device 16 is also used as an interface for transmitting a processing result obtained by the information processing device 100 to the outside. In addition, the I / F device 16 is also used as an interface for inputting a program and data required for the operation of the information processing device 100 to the information processing device 100.

[0058] Figure 2is a diagram showing a functional configuration of the information processing apparatus 100. The information processing apparatus 100 includes an image holding unit 101, a frame information holding unit 102, a normalization processing unit 103, a display control unit 104, a user operation obtaining unit 105, and a frame information correcting unit 106.

[0059] Notice, Figure 1 The control device 11 shown loads the program stored in the storage device 12 into the memory and executes the program. Figure 2 The functional units of the functional configuration in the embodiment function when the control device 11 executes the program.

[0060] The image holding unit 101 holds a plurality of images. The images held in the image holding unit 101 may be images captured by a camera or the like, images recorded in a storage device such as a hard disk, or images received via a network such as the Internet. For example, the image holding unit 101 is implemented by the storage device 12.

[0061] The frame information holding unit 102 holds a table for managing pre-input frame information in association with the images held in the image holding unit 101. The frame information according to the present embodiment is information related to the presence of an object (person) in each image, information indicating the position and size of a reference frame (typically a circumscribed rectangular frame) surrounding the object part (face) in the image, and the position and size of a frame surrounding a part of the face (eyes in the present embodiment). The position is expressed as a two-dimensional coordinate value of the upper left corner of the frame. The size is expressed as a value representing the horizontal length and vertical length of the frame. In addition, the frame information holding unit 102 is implemented by, for example, the storage device 12.

[0062] Fig.14 An example of a table maintained in the frame information maintaining unit 102 is shown. The first field of the table includes an ID for specifying an image file. When specifying an image file, the image file name can be used. The second field includes the size of the image indicated by the image file (the number of pixels in the horizontal and vertical directions). The third field includes the position and size of a reference frame surrounding the facial area of ​​a person in the image. The position of the upper left corner of the image is defined as the origin (0, 0), the direction horizontally to the right from the origin is defined as the positive direction of the x-axis, and the vertically downward direction is defined as the positive direction of the y-axis. The position of the reference frame refers to the position of the upper left corner of the reference frame, and the size of the reference frame refers to the length (number of pixels) of the reference frame in the horizontal and vertical directions.

[0063] The fourth field of the table includes the position and size of the rectangular frame each including the verification frame A (e.g., the right eye of a person). The definition of the position and size has been given in the description of the verification frame. The fifth field represents the correctness check flag of the verification frame A in the fourth field, and stores '0' indicating "unchecked" in the initial stage.

[0064] The sixth field includes the position and size of rectangular boxes each including the verification box B (eg, the left eye of a person). The seventh field represents a correctness check flag of the verification box B in the sixth field, and stores '0' indicating "unchecked" at the initial stage.

[0065] The standardization processing unit 103 performs standardization processing on the multiple frames obtained by the frame information holding unit 102. Here, the standardization processing refers to the conversion processing of the two-dimensional coordinates of the frame. Examples of such processing include conversion processing performed so that a specific reference frame is located at a fixed position in the two-dimensional coordinate system of the image and has a fixed size. The verification frame is also similarly converted based on the standardized reference frame. The purpose of the standardization processing is to make it easy to grasp the relative position and relative size between the reference frame and the verification frame of each image.

[0066] The display control unit 104 displays the reference frame (hereinafter referred to as the “standardized reference frame”) and the verification frame (hereinafter referred to as the “standardized verification frame”) standardized by the standardization processing unit 103 and the image held in the image holding unit 101 on the output device 15.

[0067] The user operation obtaining unit 105 obtains user operation information input through the input device 14 .

[0068] The frame information correction unit 106 corrects the frame information according to the user operation obtained by the user operation obtaining unit 105 , and stores the corrected frame information in the frame information holding unit 102 .

[0069] Next, we will refer to Figure 3 An example of the flow of processing performed by the information processing apparatus 100 according to the present embodiment is described.

[0070] In S301 , the control device 11 refers to the table held in the frame information holding unit 102 , and obtains frame information of the reference frame and the verification frame.

[0071] Figure 4A is a diagram showing an example of an image and frame information. Figure 4A , reference numeral 401 denotes an image surrounding an object (person), reference numeral 403 denotes a reference frame corresponding to an object part (head) of the object, and reference numerals 404 and 405 denote verification frames (eyes in this embodiment). Figure 4AAlso shown is an image 402 of another person. The image 402 also includes a reference frame 406 and verification frames 407 and 408. Note that for simplicity, the images 401 and 402 each show one person.

[0072] In S301, the control device 11 refers to the table ( Fig.14 ), and obtain the reference frame of the image (e.g., Figure 4A 403 and 406 in the figure) and box information of the verification box (for example, reference numerals 404, 405, 407 and 408).

[0073] In S302 , the normalization processing unit 103 performs normalization processing on the obtained reference frame and verification frame. Figure 5 The flow of the normalization process to be described is shown in FIG. Figure 4A The reference frame 403 in FIG. 4 is described as an example.

[0074] In S501, the standardization processing unit 103 scales (reduces or enlarges) the reference frame and the verification frame so that the width and height of the reference frame 403 reach a fixed size. For example, when the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the fixed size of the object are both 500 pixels, and the size of the reference frame 403 in the horizontal direction and the size in the vertical direction are 400 pixels and 300 pixels, respectively, the standardization processing unit 103 sets the scaling factor in the horizontal direction to 1.25 (=500 / 400) and the scaling factor in the vertical direction to 1.67 (=500 / 300). Then, the standardization processing unit 103 changes the position and size of the reference frame according to the determined scaling factors in the vertical and horizontal directions. For example, in Fig.14 In the case where the reference frame of the image ID=0001 is the reference frame 403, the normalization processing unit 103 scales RX1 and RW having horizontal components by 1.25 times, and scales RY1 and RH1 having vertical components by 1.67 times. In addition, the normalization processing unit 103 also changes the positions and sizes of the verification frames A and B according to the determined scaling factors in the vertical and horizontal directions.

[0075] In S502, the normalization processing unit 103 translates the center coordinates of the normalized reference frame to the designated position. For example, when the designated position is (x, y) = (500 pixels, 500 pixels) and the center coordinates of the reference frame 403 are (x, y) = (300 pixels, 200 pixels), the reference frame 403 is translated by +200 pixels in the x direction and by +300 pixels in the y direction. Similarly, the coordinates of the verification frames 404 and 405 are translated.

[0076] In S503, the normalization processing unit 103 holds the verification frame information in the peripheral area of ​​the reference frame 403. For example, the verification frame information including the x coordinate and the y coordinate on the coordinate system in the range of 0 to 1000 pixels is held in the storage device 12.

[0077] The standardization processing unit 103 repeats the processing of these steps S501 to S503 for all the reference frames obtained in S301. In this way, standardized reference frame information, which is frame information of a plurality of standardized reference frames (hereinafter referred to as "standardized reference frames"), and standardized verification frame information, which is frame information of a standardized verification frame (hereinafter referred to as "standardized verification frame"), are obtained.

[0078] Next, we will refer to Figure 4B Describes the display examples of the standardized reference frame and the standardized verification frame. Figure 4B , reference numeral 410 indicates a standardized reference frame. Even if the sizes of the individual images and the reference frames are different, the standardized reference frame has a consistent size and does not deviate. The frame indicated by reference numeral 412 and the multiple solid-line frames in the standardized reference frame 410 are standardized verification frames. In addition, reference numeral 411 indicates a frame used to represent the surrounding area calculated in S503. The positions of the head and the pupil are related, so it can be seen that the standardized verification frame 412 corresponding to the verification frame 408 that does not correctly represent the pupil position is located at a position that is significantly deviated from the other verification frames. In this way, by superimposing and displaying the standardized reference frames and standardized verification frames of multiple images, multiple frames can be checked at the same time and unnatural frames can be identified.

[0079] Return to reference Figure 3 As described above, in S303, the display control unit 104 performs control to display the frame information of the standardized reference frame and the standardized verification frame calculated in S302 and the statistical information calculated in S303 on the output device 15.

[0080] Will refer to FIG. 6A to FIG. 6E Describes a display example and a display transition example of the frame information of the standardized reference frame and the standardized verification frame. Fig. 6A Reference numeral 601 in the figure denotes a window displayed on the output device 15. Reference numeral 411 in the window 601 denotes a window for representing Figure 4B In the peripheral area 411, a plurality of standardized verification frames corresponding to the standardized reference frame are displayed in a superimposed manner.

[0081] In S304, the user operation obtaining unit 105 selects a verification frame according to an input from the user. Here, the user input is given by an operation performed by a pointing device such as a mouse, and the selection of the verification frame is accepted. Figure 6B 6, reference numeral 602 denotes a mouse cursor, and the user can select an object verification frame by performing an operation of changing the position of the mouse cursor. In the case of the present embodiment, the user selects a standardized verification frame 412 that is unnaturally separated from other verification frames in window 601. Note that in the case of using touch input, the user only needs to touch the standardized verification frame 412, so there is no need to display the mouse cursor.

[0082] In S305, according to the verification frame information selected in S305, the display control unit 104 changes the screen from Figure 6B Window 601 in the Figure 6C The user can edit the window 603 in the display control unit 104. Fig.14 , and displays the image 402, the reference frame 406, and the verification frames 407 and 408 associated with the verification frame 412 selected in S305. In addition, the display control unit 104 arranges and displays a correction button 604 for accepting correction of frame information and an OK button 605 for returning to the window 601 on the window 603.

[0083] If it is determined in S306 that the OK button 605 is pressed, the display control unit 104 determines that there is no problem with the frame and skips S307. In addition, in order to hide the standardized verification frame for which the OK button has been pressed in S309 described later, the display control unit 104 stores flag information indicating "correct frame" as the correctness information of the frame in the frame information holding unit 102. For example, the display control unit 104 Fig.14 The table stores "1" as the flag of the verification box.

[0084] On the other hand, if it is determined in S306 that the correction button 604 is pressed, the display control unit 104 determines that there is a problem with the frame and advances the process to S307. In this S307, in order to correct the frame, the display control unit 104 changes the screen to Fig.6D The window 606 in the image is displayed to enable the user to correct the frame information. A configuration is adopted in which, for example, the position of the verification frame 408 can be corrected by performing a moving operation (drag operation) while continuously pressing the center portion of the verification frame 408, and the frame size can be corrected by continuously pressing the frame line of the verification frame 408. The position and size of the normalized verification frame after correction are processed inversely to the normalization so as to convert the position and size into a position and size based on the ratio of the original image, and then the table is corrected.

[0085] As an example of a corrected box, Fig.6D It indicates that the verification frame 408 has been corrected to the verification frame 607 on the window 606. The frame information correction unit 106 stores the frame information of the position and size of the corrected verification frame 607 in the frame information holding unit 102 again (update Fig.14 in the table).

[0086] In S308, when the pressing of the OK button 605 is detected after the correction, the display control unit 104 changes the screen to Fig. 6E 608 in the display. In addition, in order to hide the standardized verification box for which the OK button has been pressed in S309 described later, the flag indicating "correct box" is set to "1" (which is stored as the correctness information of the box). The display control unit 104 hides the verification box for which the flag is set to "1". Therefore, other standardized verification boxes that have not been checked become easier to view.

[0087] Next, in S309, the display control unit 104 waits for the user to input an instruction on whether to end the processing. When a button (not shown) for indicating the end of a series of correction operations is pressed or the correction operations of all boxes are completed, the display control unit 104 ends the processing. Note that when the processing ends, the verification box whose flag remains at the initial value "0" is judged to be correct. When the processing ends in S309, the display control unit 104 closes the window 608. When the processing does not end in S309, the display control unit 104 continues to display the window 608 so that the user can check and correct the verification box. In addition, when the flag information is 1 in S306 and S308, the display control unit 104 hides the corresponding standardized verification box 412.

[0088] Note that in the present embodiment, an example in which a rectangular frame is displayed as a verification frame has been described, but, for example, a polygonal or circular area frame may be set. In addition, a coordinate point indicating only the position of an object without size information may be set, and the size information of an object that appears randomly on an image and has no correlation in position may also be compared. Furthermore, the present invention may be applied to pixel-level label information. Furthermore, this aspect has been illustrated using a frame for the head and a frame for the face, but a frame for the whole body and a frame for the head may be used, or a relationship between a frame for the whole body and any suitable object held by a person may be used.

[0089] Furthermore, in the above description, a person is used as an example, but the present invention is also applicable to general objects, for example, when a frame circumscribing the entire area of ​​a person on a motorcycle is envisioned, a frame that correctly surrounds both the motorcycle and the person can be separated from a frame that erroneously surrounds only the motorcycle.

[0090] As described above, the information processing device according to this embodiment enables the user to efficiently check suspected incorrect training data by simultaneously displaying the verification frame (pupil) and the relative position of the reference frame (head) related to the verification frame in terms of position and size.

[0091] [Second embodiment]

[0092] In the second embodiment, a configuration of selection and correction of a normalized verification frame using statistical information distribution will be described. Description of the same parts as the first embodiment will be omitted, and only the differences will be described.

[0093] Figure 7 is a diagram of a functional configuration of an information processing device 100 according to the second embodiment. Figure 2 The difference is that a statistical information calculation unit 107 is added.

[0094] The statistical information calculation unit 107 calculates the relative distance, relative size, and relative angle of the verification frame normalized by the normalization processing unit 103. Furthermore, the statistical information calculation unit 107 creates graphs such as a histogram and a scatter diagram based on the calculated relative distance, relative size, and relative angle.

[0095] The display control unit 104 displays the statistical information calculated by the statistical information calculation unit 107 on the output device 15 .

[0096] The following will refer to Figure 8 The flowchart in describes an example of the flow of processing performed by the information processing apparatus 100 according to the second embodiment.

[0097] In S801, the statistical information calculation unit 107 calculates the statistical information of the normalized verification frame. Fig. 9 The flowchart in describes the statistical information calculation process in detail.

[0098] In S901, the statistical information calculation unit 107 calculates the distance between the center coordinates of the normalized reference frame and the center coordinates of the normalized verification frame. For example, the Euclidean distance is used as the distance.

[0099] In S902, the statistical information calculation unit 107 calculates the size of the normalized verification box. For example, the diagonal length of the normalized verification box is used as the size.

[0100] In S903, the statistical information calculation unit 107 calculates the angle of the normalized verification frame. As calculation of the angle, for example, the statistical information calculation unit 107 calculates the angle of the straight line between the center coordinates of the normalized reference frame and the center coordinates of the normalized verification frame relative to the image coordinate x-axis, and calculates the cosine similarity based on the angle.

[0101] In S904, the statistical information calculation unit 107 calculates the degree of overlap between the normalized reference frame and the normalized verification frame. The statistical information calculation unit 107 calculates, for example, the ratio of the area of ​​the intersection (overlapping area) of the two focus regions to the area of ​​the union of the sets of the two regions (IoU: intersection-over-union) as the degree of overlap.

[0102] In S905, the statistical information calculation unit 107 determines whether the processing from S901 to S904 has been performed on all verification frames.

[0103] If it is determined in S905 that there is still a verification frame to be processed, the statistical information calculation unit 107 returns the process to S901 and processes the next verification frame.

[0104] On the other hand, if it is determined in S905 that the statistical information calculation unit 107 has processed all the verification boxes, the process proceeds to S906. In this step S906, the statistical information calculation unit 107 creates a histogram and a scatter plot based on the calculated relative distance, relative size, and relative angle. The histogram is a histogram of the frequency of the verification box when the relative distance, relative size, or relative angle is set as the horizontal axis, and is created to check the verification box information that deviates from the distribution of one variable. In addition, the scatter plot is a scatter plot of relative distance and relative size, a scatter plot of relative distance and relative angle, or a scatter plot of relative size and relative angle, and is created to check the box information that deviates from the distribution of two variables. Instead of a scatter plot, the distribution of the two variables can be expressed as a heat map display.

[0105] return Figure 8 As described above, in S802, the display control unit 104 performs control to display the frame information of the standardized reference frame and the standardized verification frame calculated in S302 and the statistical information calculated in S801 on the output device 15.

[0106] FIG. 10A to FIG. 10C Frame information of a standardized reference frame and a standardized verification frame, a display example of statistical information, and a selection example of statistical information are shown. Fig. 10AThe reference numeral 1001 in the figure represents a window displayed on the output device 15. In the window 1001, the reference numeral 410 represents a standardized reference frame. The frame represented by the reference numerals 412, 1002 and 1003 and the solid line frame in the standardized reference frame 410 are standardized verification frames. As indicated by the reference numerals 1004, 1005 and 1006, a histogram and a scatter diagram of the statistical information calculated in S801 are displayed. The histogram 1004 represents a histogram of distance, and the histogram 1005 represents a histogram of size. In addition, the scatter diagram 1006 is a scatter diagram of size and distance. Here, the histogram or scatter diagram of the angle or overlap is not shown, but the histogram or scatter diagram of the angle or overlap can be displayed by pressing a button (not shown). In addition, a configuration in which the histogram and scatter diagram of the information that the user desires to display can be selected from a pull-down menu (not shown) can also be adopted.

[0107] In S803, the display control unit 104 selects a class or range of distribution of statistical information according to the user input from the user operation obtaining unit 105. By allowing the user to select a class or range of distribution of statistical information and limiting the number of displayed verification boxes, the standardized verification boxes can be easily checked. Fig. 10B In the figure, reference numeral 1007 denotes a mouse cursor that responds to mouse operations. In the illustrated case, the mouse cursor 1007 is selecting the chart element of the largest category in the histogram representing the distance. The display control unit 104 changes the screen from Fig. 10B Window 1001 in the Fig. 10C 1009 in the window 1001. The display control unit 104 fills the category selected by the user on the window 1001 so that it can be checked which category is selected. In addition, even when a large number of verification boxes are displayed, the display control unit 104 can limit the verification boxes to be checked by displaying only the standardized verification boxes 412 and 1004 corresponding to the filled categories in the peripheral area 411.

[0108] Here, an example of selecting a category in the histogram of distance has been described, but it is also possible to display only the standardized verification box 1003 that is larger than other boxes by selecting the largest category in the histogram 1005 of size.

[0109] Furthermore, in the second embodiment, an example has been described in which a standardized verification frame is displayed by selecting a category in statistical information, but, for example, a screen transition to Figure 6C Window 603 in the figure prompts the user to check the configuration of the image and the verification box.

[0110] Furthermore, a configuration may be adopted in which a range in the scatter diagram 1006 is selected by drawing a circle (not shown) by performing a mouse operation or the like, thereby displaying only the standardized display frame included in the circle. Furthermore, a configuration may be adopted in which a region of the standardized verification frame in the peripheral area 411 in the window 1001 is specified by drawing a circle (not shown) or the like, thereby displaying only the standardized verification frame included in the circle. In addition, a configuration may be adopted in which, in this state, editing processing similar to that of the first embodiment can be performed.

[0111] As described above, in the second embodiment, the distribution of the statistical information of the verification frame is visualized, and the category of the distribution or the group of the standardized verification frame is selected and displayed. Due to this display, the user can visually identify only the verification frame suspected of being incorrect, which makes the operation of checking the verification frame easy.

[0112] [Third embodiment]

[0113] In the third embodiment, a configuration for automatically selecting a suspected incorrect standardized verification frame using statistical information will be described. Description of the same parts as in the second embodiment is omitted, and only the differences will be described.

[0114] Fig.11 is a diagram of a functional configuration of an information processing device 100 according to the third embodiment. Figure 7 The difference is that an incorrect verification frame information judging unit 108 is added.

[0115] The incorrect verification frame information judgment unit 108 judges the frame that is most likely to be incorrect based on the statistical information. As the statistical information, it is assumed that a standardized verification frame includes four vector components, namely, relative distance, relative size, relative angle and overlap, and the Mahalanobis distance described in non-patent document 2 is calculated, and if the Mahalanobis distance exceeds a preset threshold, it is judged that the standardized verification frame is likely to be incorrect.

[0116] Fig.12 The flow of the statistical information calculation process according to the third embodiment is shown. Only the flow of the statistical information calculation process according to the second embodiment will be described. Fig. 9 The statistics calculations in the process are handled in different parts.

[0117] In S905, the statistical information calculation unit 107 determines whether the processing of S901 to S904 has been performed on all verification frames. If it is determined in S905 that the processing of all verification frames has been completed, the statistical information calculation unit 107 performs the processing of S906, and in S1201, calculates the Mahalanobis distance of the distance, size, angle, and overlap of each verification frame.

[0118] Then, in S1202, the statistical information calculation unit 107 determines whether there is a normalized verification frame whose Mahalanobis distance exceeds a threshold. For example, the threshold specified here is set to 1. Next, in S802, the display control unit 104 displays only normalized verification frames whose Mahalanobis distance exceeds the threshold.

[0119] Note that in the present embodiment, an example of presetting the threshold has been described, but a configuration in which the threshold can be appropriately changed by the user using an input form (not shown) may also be adopted. In addition, a configuration in which multiple thresholds are set instead of a single threshold and a button (not shown) for displaying a standardized verification frame for each area segmented by multiple thresholds is used for switching.

[0120] Furthermore, instead of hiding the normalized verification frame whose Mahalanobis distance does not exceed the threshold, a different color may be used to make the normalized verification frame easy to see, or information for making a judgment may be given to the user by displaying the Mahalanobis distance near the frame.

[0121] Note that in this embodiment, the Mahalanobis distance is used to limit the standardized verification frame, but for example, a configuration in which a value that deviates from the center (ie, the mean value) by three times the standard deviation or more is defined as an outlier and is considered as a candidate for an incorrect verification frame may also be adopted. In addition, a configuration in which a value that deviates from the first quartile by the quartile difference is defined as an outlier and is considered as a candidate for an incorrect verification frame may also be adopted using the median and quartiles.

[0122] As described above, according to the third embodiment, based on the statistical information of the verification frame, the outlier of the statistical information of the verification frame information is determined by threshold processing. Therefore, the verification frame suspected of being incorrect can be presented to the user, which makes the operation of checking the verification frame easy.

[0123] [Fourth embodiment]

[0124] In the fourth embodiment, a configuration will be described in which a frame detected by an object frame detection unit is used to perform normalization processing and select a verification frame instead of preparing a reference frame in advance. Description of the same parts as in the third embodiment is omitted, and only differences will be described.

[0125] Fig.13 1 is a diagram of a functional configuration of an information processing apparatus 100 according to a fourth embodiment. The difference is that, in addition to the configuration in the third embodiment, an object frame detection unit 109 is provided.

[0126] When a pair of an image and a verification frame is input, the object frame detection unit 109 detects a reference frame in the image using, for example, a hierarchical convolutional neural network such as those shown in non-patent documents 1 and 3. Therefore, without preparing a reference frame in advance, the verification frame can be verified against the reference frame, and the effort required to input the reference frame can be reduced.

[0127] Note that as a method for verifying the detection frame by the object frame detection unit 109 , a configuration may be adopted in which a reference frame prepared in advance and a verification frame detected by the object frame detection unit are normalized and a verification frame is selected.

[0128] The first to fourth embodiments have been described above. In the above embodiments, the eyes of a person are shown in the verification frame, which is an example in which one reference frame is used for two verification frames, but it should be noted that, in particular, it is sufficient for the number of verification frames to be one or more than one, and the number of frames is not particularly limited.

[0129] (Other embodiments)

[0130] The present invention can be implemented by supplying a program for implementing one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read out and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0131] The present invention is not limited to the above-described embodiments, and various changes and modifications may be made within the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are proposed.

[0132] This application claims priority from Japanese Patent Application No. 2022-110587, filed on Jul. 8, 2022, which is hereby incorporated by reference.

Claims

1. An information processing device for assisting in determining the correctness of information indicating the position and size of a verification part of an object in an image, the information processing device comprising: an obtaining unit for obtaining a plurality of images, reference frame information for indicating the position and size of a reference frame surrounding an object in the image, and verification frame information for indicating the position and size of a verification frame surrounding a verification part of the object in the image; A standardization component, used to standardize the size of the reference frame indicated by the obtained reference frame information, and to standardize the size and position of the corresponding verification frame according to the standardization; as well as The display control component is used to display the standardized reference frame at a preset position for each of the multiple images, and display the standardized verification frame in a superimposed manner at a relative position based on the standardized position and size corresponding to the standardized reference frame.

2. The information processing device according to claim 1, characterized in that Also includes: A selection component, used to select the verification box to be displayed; as well as Editing widget, used to correct the size and position of the selected verification box. The display control component hides the verification frame edited by the editing component.

3. The information processing device according to claim 2, characterized in that The plurality of images, reference frame information of the images, and verification frame information edited by the editing means are used as training data.

4. The information processing device according to any one of claims 1 to 3, characterized in that: The reference frame is a frame surrounding a face of a person in an image, and the verification frame is at least one frame surrounding parts constituting the face.

5. The information processing device according to any one of claims 1 to 4, characterized in that: Also includes: a calculation component, configured to calculate at least one statistical information indicating a relative deviation of a position and a size of each verification frame based on the verification frame information and the reference frame information normalized by the normalization component, Wherein, the display control component displaying statistical information calculated by the calculation component as a chart, and When an element of the displayed diagram is selected using the selection means, only the verification boxes belonging to this element are displayed.

6. The information processing device according to claim 5, characterized in that The calculation component calculates statistical information based on a relative distance, relative size or relative angle between the reference frame and the verification frame.

7. The information processing device according to claim 5, characterized in that Also includes: a judging component, configured to calculate a value indicating the degree of incorrectness of the position and size of the verification frame based on the statistical information calculated by the calculating component, and compare the value with a preset threshold value to judge whether there is an incorrectness, The display component displays the verification frame and the corresponding image determined to be incorrect by the determination component in an editable manner.

8. The information processing device according to claim 7, characterized in that The judgment section calculates the Mahalanobis distance as a value indicating the degree of incorrectness.

9. The information processing device according to any one of claims 1 to 8, characterized in that: Also includes an object detection component, configured to receive an image input and detect an object in the image to detect the reference frame, The obtaining unit obtains the image obtained by the object detecting unit and reference frame information for the object in the image.

10. A control method for an information processing device, the information processing device being used to assist in determining the correctness of information indicating the position and size of a verification part of an object in an image, the control method being characterized by comprising: an obtaining step for obtaining a plurality of images, reference frame information for indicating the position and size of a reference frame surrounding an object in the image, and verification frame information for indicating the position and size of a verification frame surrounding a verification part of the object in the image; A standardization step, for standardizing the size of the reference frame indicated by the obtained reference frame information, and standardizing the size and position of the corresponding verification frame according to the standardization; as well as The display control step is used to display the standardized reference frame at a preset position for each of the multiple images, and to display the standardized verification frame in a superimposed manner at a relative position based on the standardized position and size corresponding to the standardized reference frame.

11. A program for causing a computer to execute the steps of the control method according to claim 10 by reading and executing the program on the computer.

Images