Image processing device, image processing method, and program

Abstract

To provide a technology that enables easy confirmation of whether camera parameters are appropriate.SOLUTION: An image processing device accepts an input of a size of a target object in a real space, and displays a first image representing a target object having a predetermined three-dimensional shape and the accepted size, superimposed on a first position of the image on the basis of camera parameters. The image processing device accepts selection of a second position in the image, and displays a second image representing the target object at the second position, superimposed on the second position of the image. The appearance of the target object at the first position is different from the appearance of the target object at the second position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to image processing technology. [Background technology]

[0002] One method for monitoring a facility, etc. is to use the video footage from a surveillance camera installed in the facility, etc. Here, the actual size and position of a person or object captured in a surveillance camera video can be calculated using information about the position and attitude of the camera (hereinafter, camera parameters) and the size and position of the person or object captured in the video (image) on the image. With such calculations, for example, if a person of interest (such as the perpetrator of a crime) is captured in the surveillance camera video, it becomes possible to ascertain the height of the person using the surveillance camera video.

[0003] The camera parameters used for the above-mentioned purposes are estimated by, for example, calibration. Non-Patent Document 1 discloses a method of capturing an image of a calibration pattern with a camera and estimating camera parameters (camera rotation and translation) indicating the position and orientation of the camera from the correspondence between the three-dimensional coordinates of the calibration pattern in the real world and the two-dimensional coordinates of the calibration pattern on the captured image.

[0004] In addition, there are cases where estimated camera parameters are obtained and used. For example, camera parameters calculated by previously calibrating a target camera may be obtained, or camera parameters defined based on information such as the position and orientation when the camera was installed may be obtained. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] : Jo Tsuyoshi and Tsuji Saburo, "3D Vision", Kyoritsu Shuppan, pp.79-82, 1998 Summary of the Invention [Problem to be solved by the invention]

[0006] Camera parameters do not necessarily represent the position and orientation of the target camera appropriately. For example, in a method of calculating camera parameters by calibration, camera parameters representing a position and orientation different from the actual position or orientation of the camera may be calculated due to input errors of corresponding points, lens distortion, etc. Even when estimated camera parameters are obtained, it is not known whether the camera parameters are appropriate. For example, the position and orientation of the camera may change over time, and therefore the camera parameters estimated in the past may differ from the current position or orientation of the camera.

[0007] If the camera parameters do not correctly represent the position and orientation of the target camera, there is a problem that an error will occur in the calculation results, for example, when calculating the height of an important person captured in the above-mentioned surveillance camera image.

[0008] The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for easily checking whether or not camera parameters are appropriate. [Means for solving the problem]

[0009] The first image processing device provided by the present invention comprises an input means for accepting input of a movement operation on a first image representing a target object having a predetermined shape and a predetermined size in real space, which is superimposed on an image captured by a camera based on predetermined camera parameters representing the position and attitude of the camera, and a presentation means for presenting the first image representing the target object in an appearance corresponding to its position on the captured image after the movement, based on the camera parameters.

[0010] The second image processing device provided by the present invention comprises a display means for displaying an image captured by a camera, a parameter acquisition means for acquiring camera parameters representing the position and attitude of the camera, an input means for accepting designation of a first position on the captured image, and a presentation means for presenting, at the first position on the captured image, a first image representing the target object on the captured image as seen by the camera determined by the camera parameters when the target object is placed at the second position, based on the camera parameters, a predetermined shape and a predetermined size of the target object in real space, and a second position in real space corresponding to the first position.

[0011] The third image processing device provided by the present invention comprises a first display means for displaying an image captured by a camera, a parameter acquisition means for acquiring camera parameters representing the position and attitude of the camera, an input means for accepting input of a point or a line for the captured image, and a second display means for displaying a first image representing the point or line when mapped onto a plane parallel to the ground surface based on the camera parameters, the position of the point or line on the captured image, and height information of the point or line in real space.

[0012] The first image processing method provided by the present invention is executed by a computer. The image processing method includes an input step of accepting an input of a movement operation on a captured image captured by a camera, the first image representing a target object having a predetermined shape and a predetermined size in real space, superimposed on the captured image based on predetermined camera parameters representing the position and attitude of the camera, and a presentation step of presenting the first image representing the target object in an appearance corresponding to the position on the captured image after the movement based on the camera parameters.

[0013] The second image processing method provided by the present invention is executed by a computer. The image processing method includes a display step of displaying a captured image captured by a camera, a parameter acquisition step of acquiring camera parameters representing a position and an attitude of the camera, an input step of accepting a designation of a first position on the captured image, and a presentation step of presenting, at the first position on the captured image, a first image representing the target object on the captured image that is captured by the camera determined by the camera parameters when the target object is placed at the second position, based on the camera parameters, a predetermined shape and a predetermined size of the target object in real space, and a second position in real space corresponding to the first position.

[0014] The third image processing method provided by the present invention is executed by a computer. The image processing method includes a first display step of displaying a captured image captured by a camera, a parameter acquisition step of acquiring camera parameters representing the position and attitude of the camera, an input step of accepting input of a point or a line for the captured image, and a second display step of displaying a first image representing the point or line when mapped onto a plane parallel to the ground surface based on the camera parameters, the position of the point or line on the captured image, and height information of the point or line in real space.

[0015] The program provided by the present invention causes a computer to operate as the first image processing device, the second image processing device, or the third image processing device provided by the present invention. Effect of the Invention

[0016] According to the present invention, a technique is provided that can easily check whether or not camera parameters are appropriate. [Brief description of the drawings]

[0017] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.

[0018] [Figure 1] 1 is a block diagram illustrating an image processing device according to a first embodiment. [Diagram 2] 11 is a diagram illustrating an example of a state in which the image processing device presents a predetermined object on a captured image. FIG. [Diagram 3] 4 is a flowchart illustrating the flow of processing executed by the image processing apparatus of the first embodiment. [Figure 4] 11 is a diagram illustrating a captured image in which a first image is presented by a presentation unit; FIG. [Diagram 5] FIG. 2 is a block diagram illustrating a hardware configuration of an image processing device. [Figure 6] 1 is a diagram illustrating an example of a first image representing a target object having a planar shape being presented on a captured image. FIG. [Figure 7] FIG. 11 is a block diagram illustrating an image processing device according to a second embodiment. [Figure 8] 13 is a diagram illustrating a captured image into which a line has been input via a second input unit; FIG. [Figure 9] 1 is a diagram illustrating an example of an image showing a dotted line mapped onto a plane representing the earth's surface as viewed from a direction perpendicular to the plane. [Figure 10] FIG. 9(b) illustrates an example image showing the camera position and field of view along with the projection lines shown in FIG. 9(a). [Figure 11] 11 is a flowchart illustrating the flow of processing executed by an image processing apparatus according to a second embodiment. [Figure 12] 11 is a diagram illustrating an example of how error information is presented on a captured image. FIG. [Figure 13] 11 is a diagram illustrating an example of a user moving a target object on a captured image. FIG. [Figure 14] 11A and 11B are diagrams illustrating an example of a target object being moved across a plurality of areas having different heights; [Figure 15] FIG. 11 is a block diagram illustrating an image processing device according to a third embodiment. [Figure 16] 11 is a flowchart illustrating the flow of processing executed by an image processing apparatus according to a third embodiment. [Figure 17] 9(a) is a diagram illustrating an example of a projection line of a target object presented on a captured image onto a plane representing the ground surface described in FIG. 9(a). FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0020] [Embodiment 1] Fig. 1 is a block diagram illustrating an image processing device 2000 according to the first embodiment. In Fig. 1, arrows indicate the flow of information. Furthermore, in Fig. 1, each block indicates a functional configuration, not a hardware configuration.

[0021] The image processing device 2000 includes a display unit 2020 , a parameter acquisition unit 2040 , an input unit 2060 , and a presentation unit 2080 .

[0022] The display unit 2020 displays an image captured by the camera. The parameter acquisition unit 2040 acquires camera parameters that indicate the position and orientation of the camera. The camera parameters may include parameters other than the position and orientation of the camera. The camera parameters other than the position and orientation of the camera will be described later.

[0023] The input unit 2060 accepts the designation of a first position on the captured image. The presentation unit 2080 generates a first image representing a target object on the captured image, which is captured by a camera determined by the camera parameters when the target object is placed at a second position in real space corresponding to the first position. In other words, the first image is an image representing how the target object looks from the viewpoint of the camera determined by the camera parameters. The second position on the real space can be obtained from the camera parameters and height information of the first and second positions. Here, "placing the target object at the second position" means assuming that the target object exists at a position (second position) in real space corresponding to the first position on the captured image. The presentation unit 2080 generates the first image using the camera parameters, a predetermined shape and a predetermined size of the target object in real space, and the second position. Furthermore, the presentation unit 2080 presents the generated first image at the first position on the captured image. The target object is a virtual object having a planar or three-dimensional shape. The predetermined size and the predetermined shape set for the target object are the size and the shape assumed in the real world. These predetermined size and the predetermined shape may be input by the user or may be stored in advance inside or outside the image processing device 2000.

[0024] A specific description will be given with reference to Fig. 2. Fig. 2 is a diagram illustrating an example of a state in which the image processing device 2000 presents a predetermined object on a captured image. In Fig. 2, the predetermined object is a rectangular parallelepiped 20. Fig. 2(a) shows the rectangular parallelepiped 20 viewed from an appropriate angle. As shown in Fig. 2(a), the size of the rectangular parallelepiped 20 is 30 cm in width and length, and 170 cm in height. The rectangular parallelepiped 20 in this example is an object that simplifies the shape and size of an average person.

[0025] 2(b) is a diagram in which the image processing device 2000 presents the rectangular parallelepiped 20 on the captured image 10. A first position 30 represents a first position input to the input unit 2060. The presentation unit 2080 presents a first image 40 at the first position 30. The first image 40 is an image that pseudo-represents the rectangular parallelepiped 20 captured by a camera determined by camera parameters when the rectangular parallelepiped 20 placed at a position corresponding to the first position 30 in the real world is captured by the camera.

[0026] <Processing flow> FIG. 3 is a flowchart illustrating a process flow executed by the image processing device 2000 of the first embodiment. In step S102, the display unit 2020 displays a captured image captured by a camera. In step S104, the input unit 2060 accepts a designation of a first position on the captured image. In step S106, the parameter acquisition unit 2040 acquires camera parameters representing the position and attitude of the camera. In step S108, the presentation unit 2080 generates a first image. As described above, the first image represents a target object on the captured image when it is captured by a camera determined by the camera parameters when placed at a second position. Then, in step S110, the presentation unit 2080 presents the generated first image at a first position on the captured image.

[0027] Note that the process flow shown in Fig. 3 is an example, and the process flow executed by the image processing device 2000 is not limited to the process shown in Fig. 3. For example, the process of acquiring camera parameters (step S106) may be performed prior to the process of accepting input of the first position (step S104).

[0028] <Actions and Effects> According to this embodiment, by viewing the object presented by the presentation unit 2080, the user of the image processing device 2000 can easily check whether the camera parameters properly represent the position, attitude, and the like of the camera (hereinafter, referred to as the real camera) that captured the captured image displayed by the display unit 2020. Hereinafter, a detailed description will be given with reference to FIG.

[0029] Fig. 4 is a diagram illustrating a captured image in which a first image is presented by the presentation unit 2080. Fig. 4(a) is a diagram illustrating a case in which the camera parameters acquired by the parameter acquisition unit 2040 represent a position and orientation that are similar to the position and orientation of the actual camera. On the other hand, Fig. 4(b) is a diagram illustrating a case in which the camera parameters acquired by the parameter acquisition unit 2040 represent a position and orientation that are different from the position and orientation of the actual camera. Note that the target object in Fig. 4 is a rectangular parallelepiped with a height of 170 cm and both vertical and horizontal lengths of 30 cm, similar to the case in Fig. 2.

[0030] The first image presented by the presentation unit 2080 is presented on the captured image as if the target object placed at the location shown in the captured image was captured by a camera installed at the position and orientation represented by the camera parameters. Therefore, when the camera parameters represent a position and orientation that are close to the position and orientation of the actual camera, there is no sense of incongruity in the appearance due to the size or angle when comparing the first image with a person or object shown in the captured image. For example, since the height of the target object is 170 cm, it is considered that the target object and a person are roughly the same height when compared.

[0031] In Fig. 4, the side of the person in the captured image 10 is specified as the first position, so the first image 40 is presented next to the person. In Fig. 4(a), the size of the person and the rectangular parallelepiped represented by the first image 40 are almost the same in any position, so there is no sense of incongruity. Also, in Fig. 4(a), just as people, walls, etc. are shown looking down from diagonally above the front, the rectangular parallelepipeds represented by each first image 40 are also shown looking down from diagonally above the front, so there is no sense of incongruity in the way each rectangular parallelepiped appears depending on the angle.

[0032] In contrast, in FIG. 4(b), the size and angle of the rectangular parallelepiped represented by the first image 40 give an unnatural appearance. For example, the height of the rectangular parallelepiped represented by the first image 40-10 is nearly twice the height of a person, and it cannot be said that the first image 40-10 represents a 170 cm tall object (rectangular parallelepiped 20) placed at the location shown in the captured image 10-2. Also, unlike the rectangular parallelepipeds represented by each of the first images 40 presented in the captured image 10-1, the tops of all the rectangular parallelepipeds are visible in the captured image 10-2, and they appear as if they are being looked down on from close up. In this way, from the size and angle of each rectangular parallelepiped represented by the first image 40, it can be predicted that the depression angle of the line of sight of the camera represented by the camera parameters in FIG. 4(b) is larger than the depression angle of the line of sight of the actual camera.

[0033] As shown in FIG. 4, the user can specify multiple first positions to place multiple target objects within one captured image.

[0034] As described above, according to the image processing device 2000 of the present embodiment, the user using the image processing device 2000 can easily understand whether the camera parameters acquired by the parameter acquisition unit 2040 represent a position and orientation that are similar to the position and orientation of the camera that captured the captured image by comparing the first image presented by the presentation unit 2080 with the captured image. If it is confirmed that the captured image represents a position and orientation that are similar to the position and orientation of the camera that captured the captured image, the user can determine that it is acceptable to use the combination of the camera parameters and the video of the surveillance camera. Conversely, if it is confirmed that the captured image does not represent a position and orientation that are similar to the position and orientation of the camera that captured the captured image, measures can be taken such as re-estimating the camera parameters or correcting the position and orientation of the actual camera.

[0035] The image processing device 2000 of this embodiment will be described in further detail below.

[0036] <Hardware configuration example> Each functional component of the image processing device 2000 may be realized by a hardware component (e.g., a hardwired electronic circuit, etc.) that realizes each functional component, or may be realized by a combination of hardware components and software components (e.g., a combination of an electronic circuit and a program that controls it, etc.).

[0037] FIG. 5 is a block diagram illustrating a hardware configuration of the image processing device 2000. The image processing device 2000 includes a bus 1020, a processor 1040, a memory 1060, a storage 1080, and an input / output interface 1100. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage 1080, and the input / output interface 1100 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection. The processor 1040 is an arithmetic processing device such as a central processing unit (CPU) or a graphics processing unit (GPU). The memory 1060 is a memory such as a random access memory (RAM) or a read only memory (ROM). The storage 1080 is a storage device such as a hard disk, a solid state drive (SSD), or a memory card. The storage 1080 may also be a memory such as a RAM or a ROM. The input / output interface 1100 is an input / output interface for the image processing device 2000 to transmit and receive data between an input device, an external device, etc. For example, the image processing device 2000 acquires a captured image, a first position, etc. via the input / output interface 1100. Also, for example, the image processing device 2000 outputs a captured image presenting the first image via the input / output interface.

[0038] The storage 1080 stores a program for realizing the functions of the image processing device 2000. Specifically, the storage 1080 stores program modules for respectively realizing the functions of the display unit 2020, the parameter acquisition unit 2040, the input unit 2060, and the presentation unit 2080. The processor 1040 executes each of these program modules to respectively realize the functions of the display unit 2020, the parameter acquisition unit 2040, the input unit 2060, and the presentation unit 2080. Here, when executing each of the above modules, the processor 1040 may read these modules onto the memory 1060 and then execute them, or may execute them without reading them onto the memory 1060.

[0039] The hardware configuration of the image processing device 2000 is not limited to the configuration shown in Fig. 5. For example, each program module may be stored in the memory 1060. In this case, the image processing device 2000 does not need to include the storage 1080.

[0040] <Camera parameter details> As described above, the camera parameters may include parameters other than the position and orientation of the camera. For example, the camera parameters include internal parameters that represent the internal characteristics of the camera, such as focal length, lens distortion, or the coordinates of the image center. The position and orientation of the camera are external parameters that represent the external characteristics of the camera. The camera parameters can be calculated by associating two-dimensional coordinates on the captured image with three-dimensional coordinates in real space.

[0041] By using the camera parameters, two-dimensional coordinates on the captured image and three-dimensional coordinates in real space can be converted to each other. However, only two-dimensional coordinates on the captured image do not uniquely determine three-dimensional coordinates in real space corresponding to the two-dimensional coordinates. In order to uniquely determine three-dimensional coordinates in real space corresponding to the two-dimensional coordinates on the captured image, for example, it is necessary to specify one of the x-coordinate, y-coordinate, or z-coordinate of the three-dimensional coordinates. In the image processing device 2000 of this embodiment, the height information (z-coordinate) of the second position in real space is specified to uniquely determine the second position in real space corresponding to the first position on the captured image. Note that in this embodiment, the origin in real space is set on the ground surface directly below the camera, and the x-coordinate and y-coordinate are set in the horizontal and vertical directions parallel to the ground surface, and the z-coordinate is set in the direction perpendicular to the ground surface. Here, the technology of converting coordinates on an image and coordinates in real space to each other using camera parameters is a known technology, and is described in, for example, Non-Patent Document 1. Therefore, further detailed description of this technology is omitted.

[0042] There are various methods for the parameter acquisition unit 2040 to acquire the camera parameters. For example, the parameter acquisition unit 2040 receives the camera parameters transmitted from an external device. In addition, for example, the parameter acquisition unit 2040 accepts manual input of the camera parameters. Furthermore, for example, the parameter acquisition unit 2040 reads out the camera parameters from a storage device that stores the camera parameters.

[0043] <Display section 2020 details> The display unit 2020 displays the captured image on a display screen such as a display. Here, this display screen may be a stationary display, or may be a portable display provided in a mobile terminal or the like.

[0044] <Details on how to enter the first position> The input unit 2060 can accept the designation of the first position by various methods capable of identifying a position on the captured image. For example, the input unit 2060 accepts an operation (such as a click operation) of designating an arbitrary position on the captured image by an input device such as a mouse. Furthermore, when the captured image is displayed on a touch panel, the input unit 2060 accepts a touch input or the like to an arbitrary position on the captured image. Furthermore, the input unit 2060 may accept an input of coordinates representing a position on the captured image.

[0045] <Details of the target object> The target object is, for example, an object having a predetermined size and shape in real space. For example, information defining the predetermined target object, "a rectangular parallelepiped with a height of 170 cm and length and width of 30 cm," as described above, is stored in advance inside or outside the image processing device 2000. In this case, the presentation unit 2080 uses this predetermined target object.

[0046] The image processing device 2000 may also have a function of accepting input of information defining a target object. In this case, information representing both the shape and size of the target object in real space may be accepted, or information representing only either the shape or the size may be accepted. In the latter case, for example, the shape of the target object is previously determined to be a rectangular parallelepiped, and the size (length, width, and height) is specified by the user.

[0047] The shape of the target object is not limited to a rectangular parallelepiped. For example, the target object may be a cone or a sphere. The target object may also be an avatar or other object that represents the shape of a person or an animal.

[0048] The target object may also be planar. FIG. 6 is a diagram illustrating an example of a state in which a first image 40 representing a planar target object is presented on the captured image 10. In this case, for example, the user specifies the vertical and horizontal lengths of the plane. If the camera parameters appropriately represent the position and orientation of the real camera, the first image presented by the presentation unit 2080 will be parallel to the ground surface. The user can easily check whether the camera parameters appropriately represent the position and orientation of the real camera by comparing the first image 40 presented by the presentation unit 2080 with the ground surface shown in the captured image 10 and checking whether the plane shown by the first image 40 is parallel to the ground surface. In addition, since the size of the plane is specified, the user can easily check whether the camera parameters appropriately represent the position and orientation of the real camera by comparing the appearance and size on the image with an object of a known size shown in the captured image and checking for any incongruity.

[0049] <Details of Presentation Section 2080> As described above, the presentation unit 2080 generates an image representing the target object on the captured image when the target object is placed at the second position and captured by the camera determined by the camera parameters. For example, the presentation unit 2080 performs the following process.

[0050] First, the presentation unit 2080 calculates a second position in real space corresponding to a first position on the target image. As described above, the first position (two-dimensional coordinates) on the target image alone does not uniquely determine the second position (three-dimensional coordinates) in real space corresponding to the first position. Therefore, the presentation unit 2080 acquires information representing the height of the second position (z coordinate of the second position). For example, the height information of the second position represents the height of the ground surface in real space (z=0). By identifying the height information of the second position in this way, the position in real space corresponding to the first position on the target image is uniquely determined. Here, the presentation unit 2080 calculates the three-dimensional coordinates of the second position by using the two-dimensional coordinates representing the first position, the height information of the second position, and the camera parameters. As described above, by using these pieces of information, the two-dimensional coordinates on the captured image can be converted into three-dimensional coordinates in real space. The height information of the second position may be given to the presentation unit 2080 in advance, or may be input from outside. Alternatively, the height information of the second position may be set as a different height for each of a plurality of regions in the target image.

[0051] Next, the presentation unit 2080 generates a first image representing the target object to be presented on the captured image. For example, when the target object is a rectangular parallelepiped or a pyramid, the presentation unit 2080 calculates the coordinates of each vertex of the target object to be presented on the captured image in order to generate the first image. Specifically, the presentation unit 2080 converts the three-dimensional coordinates of each vertex when the target object is placed at a second position in real space into two-dimensional coordinates of each vertex on the captured image using the camera parameters. Then, the presentation unit 2080 generates the first image by connecting each vertex with a straight line or the like.

[0052] The angle of the target object placed in the real space is arbitrary. For example, the presentation unit 2080 considers that the target object is placed at the second position such that the horizontal side of the target object is parallel to the x-axis, the vertical side is parallel to the y-axis, and the height side is parallel to the z-axis in the xyz space representing the real space. The directions of these sides may be predetermined, or may be specified by the user. For example, in the captured image 10 of FIG. 6, when a flat target object is used, it is possible to more easily determine whether the target object is parallel to the ground surface by aligning the vertical side with a line on the ground surface. In addition, for example, when a grid of a predetermined width is drawn on the target object in the real space, the size of the tile can be measured by aligning the vertical side of the target object with a line of a tile on the floor surface whose size is known. The determination can be more easily performed by checking the size. Therefore, for example, the image processing device 2000 allows the user to rotate the target object presented on the captured image 10 using a mouse or the like. For example, when a target object or its periphery presented on the captured image 10 is dragged with a mouse or the like, the image processing device 2000 determines the direction in which the target object is rotated according to the direction of the drag. For example, the rotation direction when the target object is dragged to the left is set to clockwise, and the rotation direction when the target object is dragged to the right is set to counterclockwise. The image processing device 2000 also determines the angle by which the target object is rotated according to the distance of the drag. In this case, the relationship between the drag distance and the rotation angle is defined in advance. Then, the image processing device 2000 rotates the target object in the determined direction and angle with a straight line (for example, a straight line parallel to the z-axis) passing through the second position as the rotation axis. Then, the user makes the vertical side of the target object follow a line on the ground surface, and then compares the target object on the captured image 10 with the ground surface. Note that the second position is not limited to a point inside the target object, and may be outside.

[0053] Furthermore, the presentation unit 2080 may accept an operation to move the target object on the captured image 10. For example, the user moves the target object on the captured image 10 by an operation such as "dragging the captured image 10 with the right button of the mouse". At this time, the input unit 2060 repeatedly acquires the position of the moving mouse pointer as the above-mentioned first position. For example, this acquisition is performed at a predetermined time interval. The presentation unit 2080 presents a first image 40 that is newly generated at a first position on the captured image 10 newly acquired by the input unit 2060 based on the first position, the camera parameters acquired fixedly, and the height information of the second position. Furthermore, the presentation unit 2080 deletes the first image 40 that was presented at the first position acquired before this first position from the captured image 10. In this way, from the user's viewpoint, the target object appears to be moving in the space shown in the captured image 10.

[0054] FIG. 13 is a diagram illustrating an example of a user moving a target object on the captured image 10. In FIG. 13, a trajectory 170 represents a trajectory along which the user moves the target object. The first positions 30-1 to 30-5 represent positions on the trajectory 170, respectively. The first images 40-1 to 40-5 represent the first images 40 presented at the first positions 30-1 to 30-5, respectively. Here, the first images 40 drawn with dotted lines represent the first images 40 that have already disappeared from the captured image 10, and the first images 40 drawn with solid lines represent the first images 40 that are currently presented. In FIG. 13, since the currently designated first position 30 is the first position 30-5, the first image 40-5 is presented, and the first images 40-1 to 40-4 have disappeared.

[0055] For example, as shown in FIG. 13, the user checks whether there is anything strange about the appearance of the target object by moving the target object so as to pass beside a person or the like appearing on the captured image 10. In the case of FIG. 13, if there is no strangeness in the size or orientation of the target object even when the target object is moved beside any person, it is considered that the camera parameters acquired by the parameter acquisition unit 2040 represent a position and orientation that are close to the position and orientation of the camera that captured the captured image 10. By providing such a movement operation, the user can easily verify whether there is anything strange about the appearance of the target object for various positions on the captured image 10. In particular, by providing the appearance of the target object by moving continuously, the legitimacy and strangeness according to human vision are more emphasized, which functions effectively for verification.

[0056] Also, for example, as shown in Fig. 14, in a captured image showing an area with steps such as stairs, height information may be set for each area with steps. In this case, by moving a target object on the image as shown by the trajectory in Fig. 14, a user can easily verify whether or not there is something strange about the way the target object appears seamlessly including the steps.

[0057] [Embodiment 2] Fig. 7 is a block diagram illustrating an image processing device 2000 according to the second embodiment. In Fig. 7, arrows indicate the flow of information. Furthermore, in Fig. 7, each block indicates a functional configuration, not a hardware configuration.

[0058] The image processing device 2000 of the second embodiment includes a display unit 2020, a parameter acquisition unit 2040, a second input unit 2100, and a second display unit 2120. The functions of the display unit 2020 and the parameter acquisition unit 2040 of the present embodiment are similar to the functions of the display unit 2020 and the parameter acquisition unit 2040 described in the first embodiment, respectively.

[0059] The second input unit 2100 accepts input of a point or a line for the captured image displayed by the display unit 2020. The second display unit 2120 displays an image representing the point or line when it is mapped on a plane parallel to the ground surface based on the camera parameters, the position of the input point or line on the captured image, and height information of the input point or line in real space. In other words, the second display unit 2120 displays an image in which the point or line assumed to be present within the field of view of the camera that captured the input captured image is mapped onto a plane parallel to the ground surface, assuming that the point or line is present within the field of view of the camera that captured the captured image. The second display unit 2120 may display on the same display as the display on which the captured image is displayed by the display unit 2020, or may display on a different display.

[0060] Height information of the input points or lines in real space may be provided in advance to the second display unit 2120, or may be input together with these points or lines to the second input unit 2100. When height information of the input points or lines in real space is provided in advance to the second display unit 2120, this height information is set to the height of the ground in real space (for example, height information (z coordinate) = 0).

[0061] As described above, the second display unit 2120 maps a point or a line on the captured image onto a plane parallel to the ground surface in the real space. First, a method of mapping a point will be described. The second display unit 2120 converts the two-dimensional coordinates of a point on the captured image into three-dimensional coordinates in the real space. As described above, the three-dimensional coordinates in the real space corresponding to the two-dimensional coordinates on the captured image are not uniquely determined. Therefore, the second display unit 2120 uses the height information of the input point. Specifically, the height information of the input point in the real space is assumed to be given height information. This allows the second display unit 2120 to uniquely convert the two-dimensional coordinates on the captured image into three-dimensional coordinates in the real space. The position of the input point on the plane parallel to the ground surface in the real space is represented by the horizontal and vertical coordinates (x and y coordinates excluding the z coordinate indicating the height) of the calculated three-dimensional coordinates.

[0062] As described in the first embodiment, the technology for calculating three-dimensional coordinates in real space corresponding to the two-dimensional coordinates based on the camera parameters, the two-dimensional coordinates of a point on a captured image, and height information of the point in real space is a known technology, and therefore a detailed description of this technology will be omitted.

[0063] The principle of the process of mapping a line input on a captured image onto a plane parallel to the ground surface in real space is the same as the principle of the process of mapping points described above. For example, the second display unit 2120 maps two or more points (e.g., points at both ends) on the input line onto a plane parallel to the ground surface in real space. Then, the second display unit 2120 connects these mapped points with a line such as a straight line. In this way, the line input on the captured image is mapped onto a plane parallel to the ground surface in real space.

[0064] A method of using the image processing device 2000 according to the second embodiment will be described below.

[0065] For example, a user of the image processing device 2000 inputs a line tracing a pattern in the real world or a boundary between a wall and the ground surface to the second input unit 2100. FIG. 8 is a diagram illustrating a captured image 10 to which a line has been input via the second input unit 2100. A dotted line 90 represents the line input to the second input unit 2100. A pattern 100 is a line drawn on the ground surface in the real world shown in the captured image. Patterns 100-1 and 100-2 are lines parallel to each other in the real world. A boundary 110 is a boundary between a wall and the ground surface in the real world shown in the captured image. Boundary 110-1 and boundary 110-2 intersect perpendicularly to each other in the real world.

[0066] The second display unit 2120 maps the dotted line 90 onto a plane parallel to the ground surface. The second display unit 2120 then displays the dotted line 90 mapped onto the plane parallel to the ground surface as viewed from a direction perpendicular to the plane. FIG. 9 is a diagram illustrating an example of an image showing the dotted line 90 mapped onto a plane representing the ground surface as viewed from a direction perpendicular to the plane. Here, FIG. 9(a) is a diagram showing a case where the camera parameters represent a position and orientation that are approximate to the position and orientation of the real camera. As described above, in the real world (at a location shown in the captured image), the patterns 100-1 and 100-2 are lines drawn parallel to each other. Therefore, in FIG. 9(a) where the camera parameters represent a position and orientation that are approximate to the position and orientation of the real camera, the projection line 120-1 obtained by mapping the dotted line 90-1 onto the plane representing the ground surface and the projection line 120-2 obtained by mapping the dotted line 90-2 onto the plane representing the ground surface are parallel or nearly parallel to each other. As described above, in the real world (at the location shown in the captured image), the boundaries 110-3 and 110-4 intersect perpendicularly to each other. Therefore, in FIG. 9(a), the projection line 120-3 obtained by mapping the dotted line 90-3 onto the plane representing the ground surface and the projection line 120-4 obtained by mapping the dotted line 90-4 onto the plane representing the ground surface intersect with each other perpendicularly or at an angle close to perpendicular.

[0067] 9B shows a case where the camera parameters represent a position and orientation different from the position and orientation of the actual camera. In this case, the projection lines 120-1 and 120-2 are not parallel or nearly parallel, and the projection lines 120-3 and 120-4 are not perpendicular or nearly perpendicular.

[0068] In this way, a user of the captured image shown in FIG. 8 can use patterns 100 and boundaries 110, whose relationships in the real world are known or easy to predict, and by looking at the results displayed by the second display unit 2120, easily confirm whether the camera parameters appropriately represent the position and attitude of the real camera.

[0069] The method of using the pattern on the ground surface is not limited to the above-mentioned method. For example, a method of inputting a plurality of points on the pattern 100-1 and checking whether or not these points are arranged on a straight line is conceivable.

[0070] Furthermore, the image processing device 2000 of this embodiment may map and present the target object presented on the captured image in the first embodiment on the plane. FIG. 17 is a diagram illustrating a projection line 180 of a target object presented on a captured image on the plane representing the ground surface described in FIG. 9(a). FIG. 17(a) shows a case in which a projection line 180 of a target object that is stationary is presented when a first image representing the target object is presented on the captured image (e.g., FIG. 2(b)). On the other hand, FIG. 17(b) shows a case in which a projection line 180 of a target object is moved in accordance with the movement of the target object on the captured image when an operation to move the target object is performed (e.g., FIG. 13). Here, a trajectory 190 represents the trajectory of the movement of the projection line 180.

[0071] Furthermore, if an object (such as a manhole) whose original shape is known is captured on the ground surface of the captured image, a line tracing that shape may be input to the second input unit 2100. If the camera parameters represent a position and orientation that are close to the position and orientation of the actual camera, the shape of the line displayed by the second display unit 2120 represents a shape that is close to the original shape of the traced object. For example, if a line is input to trace a manhole captured in the captured image, the shape of the line displayed by the second display unit 2120 will be a perfect circle or a shape close to a perfect circle. On the other hand, if the camera parameters represent a position and orientation different from the position and orientation of the actual camera, the shape of the line presented by the second display unit 2120 will be a shape different from a perfect circle (for example, an ellipse).

[0072] The second display unit 2120 may also display the camera position and field of view on the image together with points and lines mapped onto a plane parallel to the ground surface. Fig. 10 is a diagram illustrating an example of an image in which the camera position and field of view are displayed together with the projection line shown in Fig. 9(a). In Fig. 10, camera position 150 represents the position of the camera, and field of view 160 represents the field of view of the camera.

[0073] A system setter or the like who uses the image processing device 2000 of the second embodiment checks whether the camera parameters properly represent the position and attitude of the real camera by looking at the positional relationship of the points and lines mapped on a plane parallel to the ground surface. Here, as shown in Fig. 10, when the camera position and field of view are presented together with the points and lines mapped on a plane parallel to the ground surface, the system setter or the like can further grasp the positional relationship between the mapped points and lines and the camera position and field of view. Therefore, the system setter or the like can more easily and accurately check whether the camera parameters properly represent the position and attitude of the real camera.

[0074] <Processing flow> Fig. 11 is a flowchart illustrating a process flow executed by the image processing device 2000 of the second embodiment. The processes performed in steps S102 and S106 are similar to those performed in steps S102 and S106 in Fig. 3. In step S202, the second input unit 2100 accepts input of a point or a line for the captured image displayed by the display unit 2020. In step S204, the second display unit 2120 displays an image representing the point or line when mapped onto a plane parallel to the ground surface.

[0075] <Actions and Effects> According to the image processing device 2000 of this embodiment, a user can input lines or the like whose original shape and positional relationship are easy to identify into the captured image, and by checking whether the lines or the like displayed by the second display unit 2120 satisfy the original shape and positional relationship, the user can easily check whether the camera parameters appropriately represent the position and attitude of the actual camera.

[0076] [Embodiment 3] Fig. 15 is a block diagram illustrating an image processing device 3000 according to the third embodiment. In Fig. 15, arrows indicate the flow of information. Furthermore, in Fig. 15, each block indicates a functional configuration, not a hardware configuration.

[0077] The image processing device 3000 of the third embodiment has an input unit 3020 and a presentation unit 3040. The input unit 3020 accepts an input of an operation to move a first image presented on a captured image captured by a camera. The first image is an image in which a target object having a predetermined shape and a predetermined size in real space is superimposed on the captured image based on predetermined camera parameters representing the position and attitude of the camera. For example, if the position on the captured image where the first image is presented is set as position A, this first image corresponds to the first image presented by the presentation unit 2080 when position A is specified as the first position in the image processing device 2000 of the first embodiment. Note that the target object in the third embodiment is the same as the target object described in the first embodiment. Also, the predetermined camera parameters in the third embodiment are the same as the camera parameters described in the first embodiment.

[0078] The presentation unit 3040 presents the first image representing the target object in an appearance corresponding to the position on the captured image after the movement based on the above camera parameters. The method by which the presentation unit 3040 presents the first image corresponding to the moved target object is similar to the "method by which the presentation unit 2080 presents the first image 40 corresponding to the target object moved on the captured image 10" described in the first embodiment.

[0079] The hardware configuration of the image processing device 3000 is similar to that of the image processing device 2000 .

[0080] <Processing flow> 16 is a flowchart illustrating a process flow executed by the image processing device 3000 of the third embodiment. In step S302, the input unit 3020 accepts an input of a movement operation for the first image superimposed on the captured image. In step S304, the presentation unit 3040 presents the first image representing the target object in an appearance corresponding to the position on the captured image after the movement based on the camera parameters.

[0081] It should be noted that the process flow shown in FIG. 16 is merely an example, and the process flow executed by the image processing device 3000 is not limited to the flow shown in FIG.

[0082] <Actions and Effects> 13 and 14, the user can easily check whether there is anything unusual about the appearance of the target object by moving the target object so as to pass beside a person or the like appearing in the captured image 10. In particular, by presenting the appearance of the target object through continuous movement, the legitimacy or unusualness as perceived by human vision is further emphasized, which functions effectively for verification.

[0083] [Variations] The image processing device 2000 may have the following functions. The image processing device 2000 having the following functions is referred to as the image processing device 2000 of Modification 1. Note that the image processing device 2000 of Modification 1 may or may not have the functions of the image processing device 2000 of the above-mentioned first and second embodiments.

[0084] As described above, the camera parameters are estimated using a method in which a calibration pattern or an equivalent is photographed with a camera, and the camera parameters are estimated based on the correspondence between the three-dimensional coordinates of the calibration pattern in the real world and the two-dimensional coordinates of the calibration pattern in the captured image (Non-Patent Document 1). Specifically, the camera parameters are calculated so as to reduce the error (reprojection error) between the two-dimensional coordinates obtained when the three-dimensional coordinates of the calibration pattern in the real world are projected onto the captured image using the estimated camera parameters, and the two-dimensional coordinates of the calibration pattern shown in the captured image. For example, there is a method of calculating the estimated values ​​of the camera parameters so as to minimize the sum of squares of the error.

[0085] Usually, when a system setter or the like who operates the image processing device 2000 performs the task of estimating camera parameters using the above-mentioned calibration, the system setter or the like sees only the camera parameters, which are the estimated results, and does not see the above-mentioned errors, which are the process. However, it is considered that the accuracy of the estimation of the camera parameters can be improved by showing the above-mentioned errors, which are the process, to the system setter or the like. For example, when positions with large errors are concentrated toward the edge of the captured image, it is considered that the errors are large due to input errors of corresponding points or lens distortion. In such a case, the accuracy of the camera parameters can be improved by changing the way of selecting the calibration patterns so that the calibration patterns captured at positions within a predetermined distance from the edge of the image are not used for estimating the camera parameters.

[0086] Therefore, the image processing device 2000 presents the above-mentioned error for the calibration pattern captured at each position around the position on the captured image corresponding to the position, for each position where the calibration pattern was captured. Fig. 12 is a diagram illustrating a state in which information indicating the error (error information 140) is presented on the captured image. In Fig. 12, a person is used to obtain the calibration pattern. Specifically, a line 130 connecting the feet and head of a person standing almost upright is used as the calibration pattern. The error information 140 presented next to the line 130 indicates the reprojection error corresponding to the line 130.

[0087] The image processing device 2000 may map the calibration pattern on the Earth's surface by the method described in the second embodiment, and display the error in association with the calibration pattern mapped on the Earth's surface.

[0088] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0089] Below, examples of reference forms are given. 1. An input means for receiving an input of a movement operation on a captured image captured by a camera with respect to a first image representing a target object having a predetermined shape and a predetermined size in real space, the first image being superimposed on the captured image based on predetermined camera parameters representing the position and orientation of the camera; a presentation means for presenting a first image showing the target object in a manner that corresponds to the position on the captured image after the movement based on the camera parameters; An image processing device comprising: 2. the input means accepts the movement operation by repeatedly accepting designation of a first position on the captured image; The image processing device described in 1., when a certain first position is specified, generates a first image representing the target object in the captured image as seen by the camera determined by the camera parameters when the target object is placed at the second position, based on the camera parameters, a specified shape and specified size of the target object in real space, and a second position in real space corresponding to the first position, and presents the first image at the first position on the captured image. 3. The presentation means is Acquire height information of the second position; The image processing device according to 2., wherein the second position is calculated based on the camera parameters, the first position, and height information of the second position. 4. The image processing device according to claim 3, wherein the presentation means acquires information indicating the height of the ground surface in real space as the height information of the second position. 5. The image processing device according to claim 3, wherein the presentation means obtains different height information for each of a plurality of regions on the captured image as the height information of the second position. 6. An image processing device according to any one of 1 to 5, wherein the target object has a planar shape. 7. A second input means for receiving an input of a point or a line on the captured image; a second display means for displaying a second image representing the point or line when mapped onto a plane parallel to the ground surface, based on the camera parameters, the position of the point or line on the captured image, and height information of the point or line in real space; 7. The image processing device according to any one of 1 to 6, comprising: 8. An input means for receiving a designation of a first position on a captured image; a presentation means for presenting, at a first position on the captured image, a first image representing the target object in the captured image captured by a camera determined by the camera parameters when the target object is placed at the second position, based on predetermined camera parameters representing a position and orientation of a camera, a predetermined shape and a predetermined size of the target object in real space, and a second position in real space corresponding to the first position; An image processing device comprising: 9. The input means accepts designation of a plurality of first positions; 8. The image processing device according to claim 8, wherein the presentation means presents first images representing a plurality of target objects corresponding to the plurality of first positions at the corresponding first positions on the captured image. 10. The input means repeatedly accepts the designation of the first position. The image processing device described in 8. or 9., wherein the presentation means, when a first position is specified, generates a first image representing the target object placed at a second position in real space corresponding to the first position, and presents the first image at the first position on the captured image. 11. An input means for receiving an input of a point or a line on an image captured by a camera; a display means for displaying a first image representing the point or line when mapped onto a plane parallel to the ground surface, based on predetermined camera parameters representing a position and attitude of a camera, the position of the point or line on the captured image, and height information of the point or line in real space; An image processing device comprising: 12. A computer-implemented image processing method comprising: an input step of receiving an input of a movement operation on a captured image captured by a camera with respect to a first image representing a target object having a predetermined shape and a predetermined size in real space, the target object being superimposed on the captured image based on predetermined camera parameters representing a position and orientation of the camera; a presentation step of presenting a first image showing the target object in a manner corresponding to the position on the captured image after the movement based on the camera parameters; An image processing method comprising the steps of: 13. The input step includes accepting the movement operation by repeatedly accepting designation of a first position on the captured image; The image processing method described in 12., wherein the presentation step, when a certain first position is specified, generates a first image representing the target object in the captured image as seen by a camera determined by the camera parameters when the target object is placed at the second position based on the camera parameters, a specified shape and specified size of the target object in real space, and a second position in real space corresponding to the first position, and presents the first image at the first position on the captured image. 14. The presenting step comprises: Acquire height information of the second position; 14. The image processing method according to claim 13, further comprising calculating the second position based on the camera parameters, the first position, and height information of the second position. 15. The image processing method according to claim 14, wherein the presentation step acquires information indicating the height of the ground surface in real space as the height information of the second position. 16. The image processing method according to claim 14, wherein the presentation step obtains different height information for each of a plurality of regions on the captured image as the height information of the second position. 17. An image processing method according to any one of claims 12 to 16, wherein the target object has a planar shape. 18. A second input step of accepting an input of a point or a line on the captured image; a second display step of displaying a second image representing the point or line when mapped onto a plane parallel to the ground surface based on the camera parameters, the positions of the point or line on the captured image, and height information of the point or line in real space; 18. The image processing method according to any one of 12 to 17, further comprising: 19. A computer-implemented image processing method comprising: an input step of accepting a designation of a first position on a captured image; a presentation step of presenting, at a first position on the captured image, a first image representing the target object in the captured image captured by the camera determined by the camera parameters when the target object is placed at the second position, based on predetermined camera parameters representing a position and orientation of a camera, a predetermined shape and a predetermined size of the target object in real space, and a second position in real space corresponding to the first position; An image processing method comprising the steps of: 20. The input step includes accepting designation of a plurality of first positions; 19. The image processing method according to claim 19, wherein the presenting step presents first images representing a plurality of target objects corresponding to the plurality of first positions at the corresponding first positions on the captured image. 21. The input step repeatedly receives the designation of the first position. The image processing method described in 19. or 20., wherein the presentation step, when a certain first position is specified, generates a first image representing the target object placed at a second position in real space corresponding to the first position, and presents the first image at the first position on the captured image. 22. A computer-implemented image processing method comprising: an input step of accepting an input of a point or a line on a captured image photographed by a camera; a display step of displaying a first image representing the point or line when mapped onto a plane parallel to the ground surface, based on predetermined camera parameters representing a position and attitude of a camera, a position of the point or line on the captured image, and height information of the point or line in real space; An image processing method comprising the steps of: 23. A program for causing a computer to operate as an image processing device according to any one of 1 to 11.

[0090] This application claims priority based on Japanese Patent Application No. 2014-191480, filed September 19, 2014, and Japanese Patent Application No. 2014-257137, filed December 19, 2014, the disclosures of which are incorporated herein in their entirety.

Claims

1. a first input means for receiving an input of a size of a target object in real space; a presentation means for superimposing and displaying a first image representing the target object having a predetermined three-dimensional shape and the size at a first position on the image based on camera parameters; a second input means for accepting a selection of a second position in the image; Equipped with The presentation means displays a second image representing the target object at the second position in a superimposed manner on the second position of the image, a view of the target object at the first location differs from a view of the target object at the second location; Image processing device.

2. The first image and the second image have different sizes. The image processing device according to claim 1 .

3. the first image shows a target object located in front of the target object shown in the second image in a real space; the first image is larger than the second image; The image processing device according to claim 2 .

4. The camera parameters include camera parameters representing a position and an orientation of a camera. The image processing device according to claim 1 .

5. The target object is a rectangular parallelepiped. The image processing device according to claim 1 .

6. Accepting input of the size of the target object in real space; based on the camera parameters, superimposing a first image representing the target object having a predetermined three-dimensional shape and the size at a first position of the image; Accepting a selection of a second location in the image; a second image representing the target object at the second position is displayed in a superimposed manner on the second position of the image; a view of the target object at the first location differs from a view of the target object at the second location; Image processing methods.

7. the first image shows a target object located in front of the target object shown in the second image in a real space; the first image is larger than the second image; The image processing method according to claim 6.

8. receiving an input of a size of a target object in real space; a process of superimposing and displaying a first image representing the target object having a predetermined three-dimensional shape and the size at a first position on the image based on camera parameters; accepting a selection of a second location in the image; a process of superimposing and displaying a second image representing the target object at the second position on the image at the second position; A program for causing a computer to execute the following: a view of the target object at the first location differs from a view of the target object at the second location; program.

9. the first image shows a target object located in front of the target object shown in the second image in a real space; the first image is larger than the second image; The program according to claim 8.

Images