Information processing device, image determination method, and program

Abstract

To provide an information processing device that can identify the image of an object within a whole image without limiting an imaging environment.SOLUTION: An information processing device includes: a first imaging unit that images a first imaging range to obtain a first image; a second imaging unit that images a second imaging range other than the first imaging range to obtain a second image; and a determining unit that determines whether a second object image with a feature that has a coincidence level exceeding a determination threshold with respect to the feature of a first object image in the first image is present in the second image or not.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an image determination method, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a method for removing objects reflected parallel to the glass surface from a distant night scene image captured through the glass using a stereo camera. Non-Patent Document 2 discloses a technology for separating the background and the occluding object based on the difference in the relative positions of the background and the occluding object as seen from the camera while slightly moving the camera, and for restoring the background scene as if the occluding object did not exist. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Fumiya Tsurumi, Jun Yamashita, Toru Kaneko, A Study on Removing Reflections from Distant Night Scene Images Scattered Through Glass Using Stereo Images, Image Recognition and Understanding Symposium (MIRU2011), IS3-12:942-947 (https: / / ipsj.ixsq.nii.ac.jp / ej / index.php?action=pages_view_main&active_action=repository_action_common_download&item_id=77776&item_no=1&attribute_id=1&file_no=1&page_id=13&block_id=8) [Non-patent document 2] Tianfan Xue, Michael Rubinstein, Ce Liu, William T. Freeman, A Computational Approach for Obstruction-Free Photography, SIGGRAPH 2015(https: / / sites.google.com / site / obstructionfreephotography / ) Summary of the Invention [Problem to be solved by the invention]

[0004] The techniques disclosed in Non-Patent Documents 1 and 2 require separating a distant view from a foreground view. However, to apply the techniques disclosed in Non-Patent Documents 1 and 2, it is necessary to acquire two or more captured images in which the subject is photographed from a sufficiently distant distance that the distant view and the foreground view can be separated, and the subject is photographed while facing the same direction and moving appropriately. Therefore, if the subject is not photographed from a sufficiently distant distance and is not photographed while moving appropriately in the same direction, the techniques disclosed in Non-Patent Documents 1 and 2 may not be able to separate the distant view from the foreground view. Furthermore, the photographer may be forced to make decisions and perform operations to move appropriately. In other words, when identifying the image of a reflected object using the techniques disclosed in Non-Patent Documents 1 and 2, the shooting environment may be limited. Therefore, one aspect of the present disclosure aims to provide an information processing device, an image determination method, and a program that can identify the image of a reflected object without limiting the shooting environment. [Means for solving the problem]

[0005] An information processing device according to one embodiment of the present disclosure includes a first imaging unit that images a first imaging range to acquire a first image, a second imaging unit that images a second imaging range outside the first imaging range to acquire a second image, and a determination unit that determines whether the second image contains a second object image having features that have a degree of similarity with features of a first object image contained in the first image that exceeds a determination threshold.

[0006] An image judgment method according to one embodiment of the present disclosure includes a step of capturing an image of a first imaging range to obtain a first image, a step of capturing an image of a second imaging range outside the first imaging range to obtain a second image, and a step of judging whether or not the second image contains a second object image having features that have a degree of similarity with features of a first object image contained in the first image that exceeds a judgment threshold.

[0007] A program according to one embodiment of the present disclosure causes a computer to perform the following functions: capturing an image of a first imaging range to obtain a first image; capturing an image of a second imaging range outside the first imaging range to obtain a second image; and determining whether the second image contains a second object image having features that have a degree of similarity with features of a first object image contained in the first image that exceeds a determination threshold. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a front view illustrating an example of an information processing device. [Figure 2] FIG. 2 is a side view taken along an arrow II in FIG. [Figure 3] 10 is a diagram showing an example of a person whose image is captured within a first imaging range and the face of the person reflected on a reflective surface within a second imaging range. FIG. [Figure 4] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of a control unit according to the first embodiment. [Figure 6] 5 is a flowchart illustrating an example of an operation of the information processing device according to the first embodiment. [Figure 7] 7 is a flowchart illustrating an example of the operation of the information processing device following FIG. 6. [Figure 8A] FIG. 2 is a diagram showing an example of a first image. [Figure 8B] FIG. 10 is a diagram showing an example of a second image. [Figure 9] FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of a control unit according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an example of an operation of the information processing device according to the second embodiment. [Figure 12A] FIG. 1 is a diagram showing an example of two facial images with different facial expressions reflected on a reflective surface. [Figure 12B] 10 shows an example of a second image and a second image including images of faces with different expressions. [Figure 13]10A and 10B are diagrams showing an example of a person captured within a first imaging range, an image of the person reflected on a reflective surface within a second imaging range, and a person within the second imaging range. [Figure 14A] 1 shows an example of a first image including a first object image that is an image of a moving person. [Figure 14B] 14B shows an example of a second image captured in which the person shown by the first object image illustrated in FIG. 14A is captured in the second imaging range. [Figure 15] FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit. [Figure 16] FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First embodiment) The configuration of an information processing device 100 according to this embodiment will be described with reference to FIGS. 1 to 9. FIG. 1 is a front view showing an example of the information processing device 100. FIG. 2 is a side view taken along arrow II in FIG. 1. In the following description, the right direction in FIG. 1 will be referred to as the +X direction, the upward direction as the +Y direction, and the forward direction as the +Z direction. In the following description, the +Z direction will be referred to as the front direction. In addition, the -Z direction, which is the opposite direction to the +Z direction, will be referred to as the rear direction.

[0010] The information processing device 100 includes a housing 101, a first imaging unit 102, a display unit 103, a touch panel 104, and a second imaging unit 201. The information processing device 100 is, for example, a smartphone, a tablet, a digital camera, or a video camera.

[0011] The information processing device 100 determines whether or not the object image captured by the first imaging unit 102 is included in the second image acquired by the second imaging unit 201. In other words, the information processing device 100 determines whether or not an object present in the imaging range of one imaging unit is reflected in a reflective object such as glass or a mirror within the imaging range of the other imaging unit.

[0012] The housing 101 has a front surface 105 and a rear surface 202. A first imaging unit 102 and a display unit 103 are arranged on the front surface 105. Furthermore, a second imaging unit 201 is arranged on the rear surface 202.

[0013] The first imaging section 102 captures an image of the first imaging range 211 to obtain a first image 411 (see FIG. 4). The first imaging range 211 is in the direction in which the front 105 faces.

[0014] The second imaging section 201 captures an image of a second imaging range 212 outside the first imaging range 211 to acquire a second image 412 (see FIG. 4). The second imaging range 212 is in the direction in which the back surface 202 faces. In the information processing device 100 illustrated in FIG. 2, the second imaging section 201 captures an image in the opposite direction to that of the first imaging section 102.

[0015] The display unit 103 displays the second image 412. Furthermore, the display unit 103 may display the first image 411. The display unit 103 is a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. Also, for example, the display unit 103 is configured with a touch panel 104 superimposed thereon. That is, the display unit 103 may be a touch panel display having the touch panel 104.

[0016] 3 is a diagram showing an example of a person 301 captured in the first imaging range 211 and the person 301 reflected on a reflective surface 302 in the second imaging range 212. As shown in FIG. 3, when the person 301 is reflected on the reflective surface 302 in the second imaging range 212, the second image 412 acquired by the second imaging unit 201 includes an image of the person 301. For example, when the person 301 operates the information processing device 100 to capture an image of the second imaging range 212, the person 301 who is capturing the image is reflected in the second image 412. Note that FIG. 3 does not show the information processing device 100 being reflected on the reflective surface 302.

[0017] 4 is a block diagram showing an example of the configuration of the information processing device 100. The information processing device 100 includes a first imaging unit 102, a display unit 103, a touch panel 104, a second imaging unit 201, a control unit 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, etc. The first imaging unit 102, the display unit 103, the touch panel 104, and the second imaging unit 201 are as described above, and therefore detailed description thereof will be omitted.

[0018] The control unit 401 executes various processes in accordance with the programs and data stored in the ROM 402. For example, the control unit 401 is realized using a processor such as a CPU (Central Processing Unit).

[0019] The ROM 402 stores a program that operates the information processing device 100. The RAM 403 temporarily stores the program stored in the ROM 402, which is read by the control unit 401, the first image 411, the second image 412, the processing results by the control unit 401, and the like.

[0020] 5 is a block diagram showing an example of the configuration of the control unit 401 according to this embodiment. The control unit 401 includes a first object detection unit 501, a second object detection unit 502, a first feature detection unit 503, a second feature detection unit 504, a determination unit 505, an image processing unit 506, a notification unit 507, etc.

[0021] The first object detection section 501 detects a first object image 511 included in the first image 411. The second object detection section 502 detects a second object image 512 included in the second image 412.

[0022] The first feature detection unit 503 detects a feature amount 513 from the first object image 511. For example, the feature amount 513 indicates a feature amount of a feature point of the first object image 511. The second feature detection unit 504 detects a feature amount 514 from the second object image 512. For example, the feature amount 514 indicates a feature amount of a feature point of the second object image 512.

[0023] The determination unit 505 determines whether or not a second object image 512 having a degree of coincidence 515 exceeding a determination threshold with the features of the first object image 511 is included in the second image 412. Specifically, the determination unit 505 calculates the degree of coincidence 515 based on the result of comparing the feature amount 513 with the feature amount 514.

[0024] When it is determined that the second image 412 includes a second object image 512 having a degree of match 515 with the features of the first object image 511 that exceeds the determination threshold, the image processing unit 506 performs image processing on the second image 412 to hide the second object image 512. Specifically, when it is determined that the second image 412 includes a second object image 512 having a degree of match 515 with the features of the first object image 511 that exceeds the determination threshold, the image processing unit 506 causes the display unit 103 to display an image in which the second object image 512 is hidden in the second image 412.

[0025] When it is determined that the second image 412 includes a second object image 512 having a degree of similarity 515 with the features of the first object image 511 that exceeds the judgment threshold, the notification unit 507 outputs a message indicating that the second image 412 includes a second object image 512 having a degree of similarity 515 with the features of the first object image 511 that exceeds the judgment threshold.

[0026] Fig. 6 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. For example, when the touch panel 104 receives an operation to start imaging, the control unit 401 starts the process of step S601 illustrated in Fig. 6. For example, the control unit 401 may cause an icon for starting imaging to be displayed on the display unit 103. In this case, when the touch panel 104 detects contact at the position of the icon for starting imaging, the control unit 401 determines that the touch panel 104 has received the operation to start imaging, and starts the process of step S601 illustrated in Fig. 6.

[0027] In step S601, the first imaging unit 102 captures an image of the first imaging range 211 to acquire a first image 411. In step S602, the second imaging unit 201 captures an image of the second imaging range 212 to acquire a second image 412. The processing of step S601 and the processing of step S602 may be executed simultaneously. Alternatively, after the first image 411 acquired in step S601 is stored in the RAM 403, the second imaging unit 201 may acquire the second image 412 in step S602.

[0028] In step S603, the first object detection unit 501 detects a first object image 511 from the first image 411. Specifically, the first object detection unit 501 performs image processing such as pattern matching, edge detection, and region division on the first image 411 to detect the first object image 511. For example, the first object detection unit 501 detects a feature according to the type of detection target from the first image 411 to detect the first object image 511 included in the first image 411.

[0029] For example, if the type of detection target is a human face, the first object detection unit 501 detects feature points indicating coordinate values ​​of the edges of the eyes, nose, mouth, etc. from the first image 411, thereby detecting the image of the human face included in the first image 411 as a first object image 511.

[0030] Alternatively, for example, suppose the type of detection target is text. In this case, the first object detection unit 501 detects text from the first image 411 and detects the image of the text included in the first image 411 as the first object image 511. For example, if the first image 411 includes an image of text such as a license plate of a vehicle, a sign, or a sign indicating an address, the first object detection unit 501 can detect the image of the license plate of a vehicle, a sign, or a sign indicating an address included in the first image 411 as the first object image 511 by detecting the text from the first image 411.

[0031] Alternatively, the first object detection unit 501 may perform area division in the first image 411 and detect the first object image 511. In this case, the first object detection unit 501 does not limit the type of the first object image 511, and detects multiple types of object images included in the first image 411.

[0032] In step S604, the first feature detection unit 503 detects a feature amount 513 from the first object image 511 detected in step S603.

[0033] For example, if the type of the first object image 511 is a human face, the first feature detection unit 503 detects, as feature amounts 513, the relative positions between feature points indicating the coordinate values ​​of the ends of the eyes, nose, mouth, etc. contained in the first object image 511, features determined from pixel values, etc.

[0034] Furthermore, for example, when the type of the first object image 511 is a human face, the first feature detection unit 503 may detect a feature indicating a specific facial expression as the feature amount 513. For example, the first feature detection unit 503 detects, as the feature amount 513, an evaluation value of the facial expression determined from the relative positions between feature points.

[0035] Furthermore, for example, when the type of the first object image 511 is a character, the first feature detecting section 503 detects features such as the shape of the character shown by the first object image 511 as the feature amount 513 .

[0036] In step S605, the second object detection unit 502 detects a second object image 512 from the second image 412. Specifically, the second object detection unit 502 performs the same image processing as the first object detection unit 501 on the second image 412 to detect the second object image 512. As a result, the second object detection unit 502 detects, as the second object image 512, an object image of the same type as the first object image 511 included in the second image 412.

[0037] For example, if the type of detection target is a human face, the second object detection unit 502 detects feature points indicating the coordinate values ​​of the ends of the eyes, nose, mouth, etc. from the second image 412, and detects the image of the human face contained in the second image 412 as the second object image 512.

[0038] Furthermore, for example, when the type of detection target is text, the second object detection section 502 detects text from the second image 412, and detects the image of the text included in the second image 412 as the second object image 512.

[0039] For example, if the second image 412 includes an image of characters such as a vehicle license plate, a sign, or a sign indicating an address, the second object detection unit 502 can detect the image of the vehicle license plate, sign, or sign indicating an address included in the second image 412 as the second object image 512 by detecting the characters from the second image 412.

[0040] Alternatively, the second object detection section 502 may perform region division in the second image 412 and detect the second object image 512. In other words, the second object detection section 502 may not limit the type of the second object image 512, and may detect multiple types of object images included in the second image 412.

[0041] In step S606, the second feature detection unit 504 detects a feature amount 514 from the second object image 512 detected in step S605.

[0042] For example, if the type of the second object image 512 is a human face, the second feature detection unit 504 detects, as feature amounts 514, the relative positions of feature points indicating the coordinate values ​​of the ends of the eyes, nose, mouth, etc. contained in the second object image 512, features determined from pixel values, etc.

[0043] Furthermore, for example, when the type of the second object image 512 is a human face, the second feature detection unit 504 may detect a feature indicating a specific facial expression as the feature amount 514. For example, the second feature detection unit 504 detects, as the feature amount 513, an evaluation value of the facial expression determined from the relative positions between feature points.

[0044] Furthermore, for example, if the type of the second object image 512 is text, the second feature detecting section 504 detects the text shown by the second object image 512, the shape of the edge of the area showing the text, and the like as the feature amount 514.

[0045] In step S607, the determination unit 505 calculates the degree of agreement 515 between the feature 513 detected in step S604 and the feature 514 detected in step S606. For example, the determination unit 505 calculates the degree of agreement 515 between the feature 513 and the feature 514 using pattern matching, deep learning, or the like. The information processing device 100 may learn the feature 513 in advance, and then the determination unit 505 may calculate the degree of agreement 515 between the feature 513 and the feature 514 using pattern matching, deep learning, or the like. Alternatively, a server device different from the information processing device 100 may learn the feature 513 in advance. In this case, the information processing device 100 may connect to the server device via a network and acquire the learned feature 513 from the server device. Then, the control unit 401 proceeds to step S701 illustrated in FIG. 7.

[0046] Next, the operation of the information processing device 100 will be further described with reference to FIG.

[0047] In step S701, the determination unit 505 determines whether the degree of match 515 calculated in step S605 illustrated in FIG. 6 exceeds the determination threshold. If the degree of match 515 does not exceed the determination threshold in step S701, the notification unit 507 causes the display unit 103 to display the second image 412 in step S702. Then, the control unit 401 ends the process of determining whether the second object image 512 having a degree of match 515 that exceeds the determination threshold with the features of the first object image 511 is included. On the other hand, if the degree of match 515 exceeds the determination threshold in step S701, the determination unit 505 determines in step S703 that the second object image 512 having a degree of match 515 that exceeds the determination threshold with the features of the first object image 511 is included in the second image 412.

[0048] In step S704, the image processing unit 506 hides the second object image 512 having a degree of similarity 515 with the features of the first object image 511 that exceeds the determination threshold. Specifically, the image processing unit 506 performs image processing on the second image 412 to hide the second object image 512. For example, the image processing unit 506 performs image processing such as stamping to superimpose a predetermined image, filling in a predetermined color, or retouching to make the second object image 512 invisible by changing the brightness and saturation on an area of ​​the second image 412 including the second object image 512. In this way, the image processing unit 506 hides the second object image 512 included in the second image 412. In other words, the image processing unit 506 makes the second object image 512 included in the second image 412 invisible.

[0049] In step S705, notification unit 507 causes display unit 103 to display an image in which second object image 512 included in second image 412 and having a degree of similarity 515 with the feature of first object image 511 that exceeds the determination threshold is not displayed. This enables information processing device 100 to prevent the image of the object present in first imaging range 211 from being displayed on display unit 103. In other words, information processing device 100 can prevent second image 412 including the image of the object present in first imaging range 211 from being output.

[0050] For example, when the feature amount 513 and the feature amount 514 indicate facial features, the information processing device 100 can prevent the second image 412 including an image of the face of a person present in the first imaging range 211 from being output. For example, the information processing device 100 can prevent the second image 412 including an image of the face of the photographer from being output.

[0051] Similarly, when the feature 513 and the feature 514 indicate the characteristics of a specific facial expression, the information processing device 100 can prevent the second image 412 including an image of a face with that expression from being output when the facial expression of the photographer present in the first imaging range 211 is an expression that the photographer does not intend.

[0052] Furthermore, for example, the information processing device 100 can prevent the second image 412 including the second object image 512, such as a vehicle license plate, a signboard, or a sign indicating an address, from being output. This allows the information processing device 100 to prevent the second image 412 including information that can identify an individual or information that can identify a shooting location from being output.

[0053] Furthermore, for example, suppose that a photographer uses a video distribution service to distribute a second image 412 captured from the second imaging range 212 via the Internet through so-called live distribution. In this case, it is difficult for the photographer to perform an editing task to hide the image of an object included in the first imaging range 211 from the second image 412. However, the information processing device 100 performs image processing to hide the second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the determination threshold without any operation by the photographer. This eliminates the need for the photographer to perform an editing task to hide the image of the object included in the first imaging range 211 from the second image 412. Therefore, the information processing device 100 can prevent an image of an inappropriate object that the photographer did not intend to include in the second image 412 without requiring the photographer to perform complicated tasks. As a result, the information processing device 100 allows the photographer to safely use video distribution services, etc., without requiring the photographer to perform complicated tasks.

[0054] In step S706, the notification unit 507 outputs a message indicating that the second image 412 includes a second object image 512 having a degree of match 515 exceeding the determination threshold with the features of the first object image 511. In other words, the notification unit 507 notifies the photographer that an inappropriate object that was not intended by the photographer is reflected in the second image 412.

[0055] For example, the notification unit 507 causes the display unit 103 to display a message indicating that a second object image 512 having a degree of coincidence 515 with the feature of the first object image 511 that exceeds a determination threshold is included. Alternatively, if the information processing device 100 includes a speaker (not shown), the notification unit 507 may output a sound indicating that the second object image 512 is included.

[0056] The information processing device 100 notifies the photographer that an unintended inappropriate object is captured in the image, thereby allowing the photographer to adjust the second imaging range 212 so that the unintended inappropriate object is not captured in the image.

[0057] Fig. 8A is a diagram showing an example of a first image 411. For example, a first object image 511 included in the first image 411 shown in Fig. 8A is an image of the photographer's face. Fig. 8B is a diagram showing an example of a second image 412. The second image 412 shown in Fig. 8B includes a second object image 512, which is an image of the photographer's face, in the portion of the image 801 of the reflecting surface. In other words, the second object image 512 shown in Fig. 8B is an image of the photographer's face reflected in a reflecting surface such as a mirror included in the second imaging range 212.

[0058] For example, suppose that the second object image 512 illustrated in Fig. 8B has a feature with a degree of match 515 that exceeds the determination threshold with respect to the feature amount of the first object image 511 illustrated in Fig. 8A. In this case, the image processing unit 506 performs image processing to hide the second object image 512 included in the image 801 of the reflecting surface. Note that Fig. 8B does not illustrate the information processing device 100 being reflected on the reflecting surface.

[0059] 9 is a diagram showing an example of a display image 901 displayed on the display unit 103. Specifically, the display image 901 includes a mask 902 that fills in an area including the second object image 512, as illustrated in FIG. 8B. For example, the image processing unit 506 fills in the area including the second object image 512, as illustrated in the display image 901. Then, the notification unit 507 causes the display unit 103 to display an image in which the mask 902 is superimposed on the area including the second object image 512, as illustrated in the display image 901. Furthermore, as illustrated in the display image 901, the notification unit 507 may cause the display unit 103 to display an image in which a message 903 saying "Inappropriate object is reflected" is superimposed on the second image 412.

[0060] (Variation) The position where the first imaging unit 102 is arranged is not limited to the front surface 105. For example, the first imaging unit 102 may be arranged on the top surface of the housing 101 facing the +Y direction. Similarly, the position where the second imaging unit 201 is arranged is not limited to the back surface 202. For example, the second imaging unit 201 may be arranged on a side surface of the housing 101 facing the +Z direction.

[0061] Second Embodiment The second embodiment will be described with reference to Figures 10 to 16. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicated explanations will be omitted. Configurations and processes having substantially the same functions as those of the first embodiment will be denoted by the same reference numerals, and explanations will be omitted, and only differences from the first embodiment will be described.

[0062] Fig. 10 is a diagram showing an example of the configuration of a control unit 401 according to this embodiment. The control unit 401 shown in Fig. 10 differs from the control unit 401 shown in Fig. 5 in that the control unit 401 includes a first feature detection unit 1001, a second feature detection unit 1002, and a determination unit 1003 instead of the first feature detection unit 503, the second feature detection unit 504, and the determination unit 505.

[0063] The first feature detection unit 1001 calculates a rate of change 1011 of distance between two or more feature points detected from the first object image 511. The second feature detection unit 1002 calculates a rate of change 1012 of distance between two or more feature points detected from the second object image 512.

[0064] The determination unit 1003 calculates the degree of match 515 based on the result of comparing the movement of the first object image 511 with the movement of the second object image 512. Specifically, the determination unit 1003 calculates the degree of match 515 based on the result of comparing the rate of change 1011 of the distance between two or more feature points detected from the first object image 511 with the rate of change 1012 of the distance between two or more feature points detected from the second object image 512.

[0065] When it is determined that the second image 412 includes a second object image 512 having a degree of coincidence 515 with the movement characteristics of the first object image 511 that exceeds the determination threshold, the notification unit 507 according to the present embodiment outputs a message indicating that the second image 412 includes a second object image 512 having a degree of coincidence 515 with the movement characteristics of the first object image 511 that exceeds the determination threshold. Alternatively, the notification unit 507 according to the present embodiment may cause the display unit 103 to display, on the second image 412, characters, symbols, etc. that indicate the degree of coincidence 515 for the movement of each object image included in the second image 412, superimposed on the second image 412.

[0066] 11 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. The processing in steps S1101 to S1102 is the same as the processing in steps S601 to S602 shown in FIG. 6, and therefore detailed description thereof will be omitted.

[0067] In step S1103, the first object detection unit 501 detects a first object image 511 from the first image 411 acquired in step S1101. For example, the first object detection unit 501 detects an object image of a predetermined type included in the first image 411 as the first object image 511. For example, the predetermined type is a human face, a human body, or the like. Alternatively, for example, the first feature detection unit 1001 may perform region division in the first image 411 and detect the first object image 511. In this case, the first feature detection unit 1001 does not limit the type of the first object image 511, and detects object images of multiple types included in the first image 411.

[0068] In step S1104, the first feature detection unit 1001 detects two or more feature points from the first object image 511. For example, if the type of the first object image 511 is a human face, the first feature detection unit 1001 detects the coordinate values ​​of the edges of the eyes, nose, mouth, etc. included in the first object image 511 as feature points. Alternatively, if the type of the first object image 511 is a human body, the first feature detection unit 1001 may detect the coordinate values ​​of the arms, legs, etc. included in the first object image 511 as feature points. Alternatively, if the type of the first object image 511 is an image of a human body, the first feature detection unit 1001 may detect representative values ​​of the coordinate values ​​of the arms, legs, etc. included in the first object image 511 as feature points.

[0069] In step S1105, the first feature detection unit 1001 calculates the rate of change 1011 of the distance between the feature points detected in step S1104. For example, the first feature detection unit 1001 calculates the rate of change 1011 from the rate of change over time of the length of each side for a polygon having two or more feature points as vertices detected in step S1104. For example, the first feature detection unit 1001 calculates, as the rate of change 1011, a representative value of the rate of change over time of the length of each side for a polygon having two or more feature points as vertices.

[0070] In step S1106, the second object detection unit 502 detects a second object image 512 from the second image 412 acquired in step S1102. In step S1107, the second feature detection unit 1002 detects two or more feature points from the second object image 512. For example, if the type of the second object image 512 is a human face, the second feature detection unit 1002 detects the coordinate values ​​of the edges of the eyes, nose, mouth, etc. included in the second object image 512 as feature points. Also, for example, if the type of the second object image 512 is a human body, the second feature detection unit 1002 detects the coordinate values ​​of the arms, legs, etc. included in the second object image 512 as feature points. Alternatively, if the type of the second object image 512 is a human body image, the second feature detection unit 1002 may detect representative values ​​of the coordinate values ​​of the arms, legs, etc. as feature points.

[0071] In step S1108, the second feature detection unit 1002 calculates the rate of change 1012 of the distance between the feature points detected in step S1107. For example, the second feature detection unit 1002 may calculate the rate of change 1012 from the rate of change over time of the length of each side of a polygon having vertices at two or more feature points detected in step S1107. For example, the second feature detection unit 1002 calculates, as the rate of change 1012, a representative value of the rate of change over time of the length of each side of a polygon having vertices at two or more feature points detected in step S1107.

[0072] In step S1109, the determination unit 505 calculates the degree of match 515 based on the result of comparing the rate of change 1011 calculated in step S1105 with the rate of change 1012 calculated in step S1108. Then, the control unit 401 proceeds to S701 illustrated in FIG.

[0073] Fig. 12A is a diagram showing an example of a person 1205 present in first imaging range 1203, reflected on reflecting surface 1201. Person images 1202a and 1202b shown in Fig. 12A show images of person 1205 with different facial expressions, reflected on reflecting surface 1201 present in second imaging range 212. As shown in Fig. 12A, the facial expression shown by person image 1202a changes to the facial expression shown by person image 1202b.

[0074] 12B shows examples of second images 412a and 412b captured in the second imaging range 1204 shown in FIG. 12A. The second image 412a shown in FIG. 12B includes a second object image 512a showing the face of the person image 1202a. The second image 412b shown in FIG. 12B includes a second object image 512b showing the face of the person image 1202b. The rate of change 1011 of the distance between feature points in the first object image 511, which is the image of the person 1205, is equivalent to the rate of change 1012 of the distance between feature points from the second object image 512a to the second object image 512b. For example, the rate of change 1011 of the distance between feature points of the mouth edges in the first object image 511 is equivalent to the rate of change 1012 of the distance between feature points of the mouth edges from the second object image 512a to the second object image 512b.

[0075] Therefore, the determination unit 505 compares the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points from the second object image 512a to the second object image 512b, and determines that the person 1205 present in the first imaging range 1203 is reflected in the second imaging range 212.

[0076] Furthermore, by comparing the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points in the second object image 512, the information processing device 100 can determine whether or not an object included in the first imaging range 211 is reflected in the second imaging range 212, even if the angle of the first object image 511 and the angle of the second object image 512 are different. For example, by comparing the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points in the second object image 512, the information processing device 100 can identify the image of the photographer's face from the images of multiple objects reflected in the second imaging range 212, even if the angle of the image of the photographer's face, which is the first object image 511, is different from the angle of the image of the photographer's face, which is the second object image 512. In other words, if the information processing device 100 can detect global features from the first image 411 and the second image 412, it can identify the image of the photographer's face reflected in the second imaging range 212 even if the angle of the first object image 511 and the angle of the second object image 512 are different.

[0077] Similarly, if the information processing device 100 can detect global features from the first image 411 and the second image 412, it can identify an object reflected in the second imaging range 212 even if the brightness of the first object image 511 and the brightness of the second object image 512 are different.

[0078] As described above, the information processing device 100 can identify an object that exists in the first imaging range 211 and is reflected in the second imaging range 212 by suppressing the influence of differences in the angle, color tone, etc. of the object image between the first image 411 and the second image 412. In other words, the information processing device 100 can identify an object that exists in the first imaging range 211 and is reflected in the second imaging range 212 by suppressing the influence of differences in the angle, color tone, etc. of the object image between the first image 411 and the second image 412.

[0079] 13 is a diagram showing an example of a person 1301 present in the first imaging range 211, an image of the person 1301 reflected on a reflective surface 1302 in the second imaging range 212, and people 1303 and 1304 present in the second imaging range 212. When the person 1301 shown in FIG. 13 moves, the first imaging unit 102 captures an image of the moving person 1301, and the second imaging unit 201 captures an image of the moving person 1301 reflected on the reflective surface 1302. Similarly, when the people 1303 and 1304 shown in FIG. 13 move, the second imaging unit 201 captures images of the moving people 1303 and 1304. Note that FIG. 13 does not show the information processing device 100 being reflected on the reflective surface 1302.

[0080] 14A shows an example of a first image 411 including a first object image 1411 which is an image of the person 1301 shown in FIG. 13. FIG. 14B shows an example of a second image 412 which is captured with the person 1301 reflected in the second imaging range 212. The second image 412 shown in FIG. 14B includes a second object image 512 which is an image of the person 1301. Furthermore, the second image 412 shown in FIG. 14B includes object images 1401 and 1402 which are images of people present in the second imaging range 212. Note that in FIG. 14B, as in FIG. 13, the information processing device 100 being reflected in the reflective surface 1302 is not shown.

[0081] The change in motion of the first object image 511 illustrated in Fig. 14A is equivalent to the change in motion of the second object image 512 illustrated in Fig. 14B. Therefore, the rate of change 1011 of the distance between feature points in the first object image 511 illustrated in Fig. 14A is equivalent to the rate of change 1012 of the distance between feature points in the second object image 512 illustrated in Fig. 14B.

[0082] For example, the determination unit 505 calculates the degree of match 515 based on the result of comparing the rate of change 1011 of the distance between the feature points of the arms in the first object image 511 with the rate of change 1012 of the distance between the feature points of the arms in the second object image 512. If the degree of match 515 exceeds a determination threshold, the determination unit 505 determines that the second image 412 includes an image of the person represented by the first object image 511.

[0083] 15 is a diagram showing an example of a display image 1501 displayed on a display unit. Specifically, the second object image 512 illustrated in FIG. 14B is not displayed in the display image 1501. Specifically, a mask 1502 is superimposed on an area including the second object image 512 in the display image 1501. Therefore, the second object image 512 is not displayed in the display image 1501. This enables the information processing device 100 to prevent an image including an object image whose degree of coincidence 515 with the movement of the first object image 511 exceeds a determination threshold from being output.

[0084] 16 is a diagram showing an example of a display image 1601 displayed on the display unit 103. Specifically, in the display image 1601, characters indicating the degree of coincidence 515 between the movements of the second object image 512, the object image 1401, and the object image 1402 and the movement of the first object image 511 are superimposed on the second image 412 shown in FIG. 14B. The degree of coincidence 515 between the movements of the first object image 511 and the second object image 512 shown in FIG. 14A is 100%. The degree of coincidence 515 between the movements of the first object image 511 and the object image 1401 shown in FIG. 14A is 60%. The degree of coincidence 515 between the movements of the first object image 511 and the object image 1402 shown in FIG. 14A is 50%.

[0085] In this way, by displaying display image 1601 on display unit 103, information processing device 100 can present degree of coincidence 515 between the movement of an object present in first imaging range 211 and the movement of an object present in second imaging range 212. This allows information processing device 100 to allow the photographer to recognize which of the multiple object images included in second image 412 is an image of an object present in first imaging range 211.

[0086] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment. [Explanation of symbols]

[0087] 100 Information processing device, 101 Housing, 102 First imaging unit, 103 Display unit, 104 Touch panel, 105 Front, 201 Second imaging unit, 202 Rear, 211 First imaging range, 212 Second imaging range, 301 Person, 302 Reflective surface, 401 Control unit, 402 ROM, 403 RAM, 411 First image, 412 Second image, 412a Second image, 412b Second image, 501 First object detection unit, 502 Second object detection unit, 503 First feature detection unit, 504 Second feature detection unit, 505 Determination unit, 506 Image processing unit, 507 Notification unit, 511 First object image, 512 Second object image, 512a Second object image, 512b Second object image, 513 Feature amount, 514 Feature amount, 515 Degree of match, 801 image, 901 display image, 902 mask, 903 message, 1001 first feature detection unit, 1002 second feature detection unit, 1003 determination unit, 1011 change rate, 1012 change rate, 1201 reflecting surface, 1202a image, 1202b image, 1204 first imaging range, 1205 person, 1301 person, 1302 reflecting surface, 1303 person, 1304 person, 1401 object image, 1402 object image, 1501 display image, 1502 mask, 1601 display image

Claims

1. a first imaging unit that captures an image of a first imaging range to obtain a first image; a second imaging unit that captures an image of a second imaging range that is outside the first imaging range and acquires a second image; a determination unit that determines whether or not a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold, the determination unit calculates the degree of coincidence based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image.

2. a notification unit that outputs, when it is determined that the second image includes the second object image, a message indicating that the second image includes the second object image. The information processing device according to claim 1 .

3. an image processing unit that performs image processing on the second image to hide the second object image when it is determined that the second image includes the second object image; 3. The information processing device according to claim 1.

4. further comprising a display unit that displays the second image; when it is determined that the second image includes the second object image, the image processing unit causes the display unit to display an image in which the second image does not include the second object image. The information processing device according to claim 3 .

5. the determination unit calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image; a display unit that displays the second image; a notification unit that displays, on the display unit, information indicating the degree of coincidence for each of the object images superimposed on the second image. The information processing device according to claim 1 .

6. further comprising a housing having a front surface and a back surface; the first imaging range is in a direction in which the front faces, The second imaging range is in the direction in which the back surface faces. The information processing device according to any one of claims 1 to 5.

7. capturing an image of a first imaging range to obtain a first image; capturing an image of a second imaging range outside the first imaging range to acquire a second image; A feature having a degree of match exceeding a determination threshold with a feature of a first object image included in the first image. and determining whether a second object image corresponding to the second object is included in the second image, The image determination method includes a step of calculating the degree of match based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image.

8. On the computer, a function of capturing an image of a first imaging range to acquire a first image; a function of capturing an image of a second imaging range outside the first imaging range to acquire a second image; a function of determining whether or not a second object image having a feature that matches a feature of a first object image included in the first image to a degree exceeding a determination threshold is included in the second image; The determining function is a program that executes a function of calculating the degree of match based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image.

9. a first imaging unit that captures an image of a first imaging range to obtain a first image; a second imaging unit that captures an image of a second imaging range that is outside the first imaging range and acquires a second image; a determination unit that determines whether a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image; a display unit that displays the second image; Notification department and preparation, the determination unit calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image; The notification unit is an information processing device that causes the display unit to display information indicating the degree of coincidence for each of the object images superimposed on the second image.

10. capturing an image of a first imaging range to obtain a first image; capturing an image of a second imaging range outside the first imaging range to acquire a second image; determining whether a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image; displaying the second image on a display unit; the determining step calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image; In the displaying step, information indicating the degree of coincidence for each of the object images is superimposed on the second image and displayed on the display unit.

11. On the computer, a function of capturing an image of a first imaging range to acquire a first image; a function of capturing an image of a second imaging range outside the first imaging range to acquire a second image; a function of determining whether a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image; and a function of displaying the second image on a display unit. the function of determining includes a function of calculating the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image, a program for displaying, in the display function, information indicating the degree of coincidence for each of the object images superimposed on the second image on the display unit;

Images