Information processing device, 3D measuring device, and information processing system

Abstract

To appropriately delete a deletion target area included in 3D information.SOLUTION: An information processing device 20 includes a three-dimensional information acquisition unit 28A, a posture information acquisition unit 28B, a determination unit 28D, and a deletion unit 28E. The three-dimensional information acquisition unit 28A acquires three-dimensional information obtained by photographing a photographing area by a photographing unit 14. The posture information acquisition unit 28B acquires posture information of the photographing unit 14 when photographing three-dimensional information. The determination unit 28D determines an area to be deleted included in the three-dimensional information on the basis of the posture information. The deletion unit 28E deletes the deletion target area included in the three-dimensional information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , , , , , , , , , , , , ,

[0005]

[0001] The present invention relates to an information processing apparatus, a three-dimensional measurement apparatus, and an information processing system.

Background Art

[0002] In recent years, three-dimensional information of a three-dimensional space, which is a real space, has been acquired using a laser scanner, LiDAR, or the like. Such three-dimensional information is used in industries such as construction, facility maintenance, and real estate for measuring the dimensions of facilities and formulating renovation plans. For acquiring three-dimensional information, for example, a photographing device that photographs a 360° photographing area in all directions, up, down, left, and right, with a single shutter operation is used. However, in the prior art, since a wide range such as 360° can be photographed at once, unnecessary objects such as a photographer may be captured in the acquired three-dimensional information.

[0003] Therefore, in order to obtain three-dimensional information from which the captured unnecessary objects are removed, a plurality of photographed images having different photographing ranges are acquired by performing a plurality of photographings using a photographing device fixed to a tripod, and the plurality of photographed images are combined so as not to include a specific subject, thereby generating a composite image of a 360° photographing range.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, in order to obtain one piece of three-dimensional information, it is necessary to perform a plurality of photographings using a photographing device fixed to a tripod, and depending on the photographing environment, it may be difficult to appropriately remove a deletion target area including unnecessary objects included in the three-dimensional information.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, a three-dimensional measurement apparatus, and an information processing system capable of appropriately removing a deletion target area included in three-dimensional information.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the information processing device includes a 3D information acquisition unit that acquires 3D information obtained by imaging the imaging area by the imaging unit, and an orientation information acquisition unit that acquires orientation information of the imaging unit at the time of imaging the 3D information, A reception unit that receives operation instructions from the user, and reference setting information that associates the reference deletion area included in the shooting area with the reference posture information of the shooting unit, The system includes a determination unit that determines a region to be deleted from the 3D information based on the posture information, and a deletion unit that deletes the region to be deleted from the 3D information. The reference posture information is information representing the reference posture of the imaging unit, and the reference deletion area is information representing the area to be deleted included in the 3D information when the imaging unit is captured in the posture represented by the reference posture information, and the reference deletion area is received by the reception unit. [Effects of the Invention]

[0007] The present invention has the effect of being able to appropriately delete the region to be deleted included in 3D information. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram of an example of an information processing system. [Figure 2] Figure 2 is a block diagram representing the functional configuration of the information processing system. [Figure 3A] Figure 3A is an explanatory diagram illustrating an example of how the imaging device is used. [Figure 3B] Figure 3B is a schematic diagram showing an example of 3D information captured under the usage conditions shown in Figure 3A. [Figure 4A] Figure 4A is an explanatory diagram illustrating an example of how the imaging device is used. [Figure 4B] Figure 4B is a schematic diagram showing an example of 3D information captured under the usage conditions shown in Figure 4A. [Figure 5A] Figure 5A is an explanatory diagram illustrating an example of how to determine the area to be deleted. [Figure 5B] Figure 5B is an explanatory diagram illustrating an example of how to determine the area to be deleted. [Figure 6A] Figure 6A is an explanatory diagram illustrating an example of how to determine the area to be deleted. [Figure 6B] Figure 6B is an explanatory diagram illustrating an example of how to determine the area to be deleted. [Figure 7]Figure 7 is a flowchart showing an example of an information processing flow. [Figure 8] Figure 8 is a block diagram showing the functional configuration of the information processing system. [Figure 9] Figure 9 is a flowchart showing an example of an information processing flow. [Figure 10] Figure 10 is a block diagram showing the functional configuration of an information processing system. [Figure 11] Figure 11 is a flowchart showing an example of an information processing flow. [Figure 12] Figure 12 is a hardware configuration diagram. [Modes for carrying out the invention]

[0009] Embodiments of the information processing device, 3D measuring device, and information processing system will be described in detail below with reference to the attached drawings.

[0010] (First Embodiment) Figure 1 is a schematic diagram of an example of the information processing system 1 according to this embodiment.

[0011] The information processing system 1 comprises a camera 10 and an information processing device 20. The camera 10 and the information processing device 20 are connected in a communication manner.

[0012] The imaging device 10 is a device for obtaining three-dimensional information of a three-dimensional space, which is the imaging area S. The imaging device 10 obtains three-dimensional information by imaging. Imaging means that the imaging device 10 obtains three-dimensional information of the imaging area S. The imaging device 10 performs imaging and obtains three-dimensional information by being operated by a user. The user who operates the imaging device 10 will be referred to as the photographer.

[0013] The imaging device 10 is, for example, a camera using the ToF (Time-of-Flight) method, a stereo camera, or the like. The ToF method is a method of irradiating an infrared ray to a measurement target object and obtaining the distance from the time until the reflected light returns. A stereo camera is a camera that obtains depth information as distance information using the distance between two cameras and the parallax information of the images obtained by each of the two cameras.

[0014] The imaging device 10 may be any device that can obtain three-dimensional information, such as a LiDAR (Light Detection and Ranging), a system using photogrammetry, or the like. Further, the imaging device 10 may be a smartphone equipped with a laser scanner, a LiDAR, or the like.

[0015] The three-dimensional information is information representing a three-dimensional space that is the imaging area S. The file format of the three-dimensional information is not limited. The three-dimensional information is represented, for example, by shape data representing the three-dimensional shape of a three-dimensional object included in the imaging area S. The three-dimensional information is, for example, a file in a point cloud format representing the three-dimensional space by discrete points, a file in a polygon mesh format representing the three-dimensional space by vertices and faces, or the like. A file in the point cloud format may be referred to as a depth map, a distance image, or the like.

[0016] Examples of the file in the point cloud format include files represented by extensions such as ".xyz", ".e57", ".ply", etc. Examples of the file in the polygon mesh format include files represented by extensions such as ".obj", ".fbx", ".stl", etc.

[0017] FIG. 1 shows an example of the imaging device 10 equipped with a camera using the ToF method. The imaging device 10 includes a photographing unit 14, an attitude detection unit 15, an operation unit 16, a control unit 17, and a display unit 18.

[0018] The imaging unit 14 is a mechanism that obtains three-dimensional information, etc., through imaging. For example, the imaging unit 14 includes an image acquisition unit 11, a distance information acquisition unit 13, and a projection unit 12. The image acquisition unit 11, the distance information acquisition unit 13, and the projection unit 12 operate synchronously.

[0019] The image capture unit 11 is a mechanism that obtains captured image data of a 360° shooting area S by taking pictures. The captured image data will be referred to as the captured image below. For example, the image capture unit 11 includes a fisheye lens 11B, a fisheye lens 11b, an image sensor 11A that receives light focused by the fisheye lens 11B, and an image sensor 11a that receives light focused by the fisheye lens 11b. The captured image obtained by the image capture unit 11 may be a monochrome captured image or an RGB color captured image. In this embodiment, the case where the captured image is an RGB color captured image will be described as an example.

[0020] The projection unit 12 and the distance information acquisition unit 13 are mechanisms for obtaining three-dimensional information of the shooting area S. For example, the projection unit 12 includes a light source unit 12A and a light source unit 12a that emit laser light used for measuring distance, and wide-angle lenses 12B and 12b that project the laser light emitted from each of the light source units 12A and 12a toward the shooting area S.

[0021] The distance information acquisition unit 13 includes wide-angle lenses 13B and 13b, and distance sensors 13A and 13a. Distance sensors 13A and 13a are, for example, ToF sensors. Distance sensor 13A receives reflected light from the laser beam emitted from the projection unit 12 via the wide-angle lens 13B. Distance sensor 13a receives reflected light from the laser beam emitted from the projection unit 12 via the wide-angle lens 13b.

[0022] The operation unit 16 is a switch for receiving operation instructions from the photographer. For example, the operation unit 16 is used to input a shooting instruction signal from the photographer. The control unit 17 controls the shooting device 10. When the photographer operates the operation unit 16 and a shooting instruction signal is input, the control unit 17 controls the image acquisition unit 11, the distance information acquisition unit 13, and the projection unit 12 to be driven synchronously to perform shooting. As a result, one shooting is performed with one input of a shooting instruction signal, and a captured image and 3D information of the shooting area S are obtained. The display unit 18 is a display that shows various information.

[0023] The imaging device 10 is configured to be portable by the photographer. Therefore, by the photographer holding the imaging device 10 with their hand or the like and operating the control unit 16 once, the imaging device 10 acquires a captured image and 3D information through imaging.

[0024] The posture detection unit 15 is a sensor that detects posture information of the imaging unit 14. The posture detection unit 15 detects at least one of the rotation angles with respect to each of the mutually orthogonal directions of the x, y, and z axes as posture information. The y-axis direction is, for example, the direction along the vertical direction. The x-axis and z-axis directions are, for example, mutually orthogonal directions along a horizontal plane perpendicular to the vertical direction. In other words, the posture information is represented by the rotation angles of the imaging unit 14 with respect to each of the x-axis and z-axis directions that form the horizontal plane, and the y-axis direction which is the vertical direction.

[0025] Note that the posture information does not need to include information on the rotation angles for all axes: x-axis, y-axis, and z-axis. For example, it may include information on the rotation angle with the x-axis as the axis of rotation and the rotation angle with the z-axis as the axis of rotation. Also, the rotation angle may be the relative rotation angle with respect to the x-axis and z-axis from the reference posture of the imaging device 10 (for example, the imaging device placed on a horizontal surface). In this case, the posture information does not include information on the rotation angle with respect to the y-axis as the axis of rotation, but when the photographer holds the imaging device 10 and takes an image, it is easier to fix the rotation angle with respect to the y-axis (vertical direction) as the axis of rotation. On the other hand, it is difficult for the photographer to stabilize the rotation angles with respect to the x-axis and z-axis as the axes of rotation, and the rotation angles with respect to the x-axis and z-axis as the axes of rotation become unstable. Therefore, the posture information only needs to include information related to the rotation angles with respect to the x-axis and z-axis as the axes of rotation.

[0026] Another example of attitude information is that it may be the relative rotation angle in the vertical direction with respect to the optical axis direction of the wide-angle lens 13B or 13b.

[0027] For the attitude detection unit 15, for example, an acceleration sensor such as an IMU (Inertial Measurement Unit), a GPS (Global Positioning System), a gyroscope, a geomagnetic sensor, etc., may be used.

[0028] Here, if an IMU is used as the attitude detection unit 15, the attitude detection unit 15 can detect the attitude of the imaging device with respect to the vertical direction (gravity direction), and thus attitude information corresponding to the rotation angles with the x-axis and z-axis directions as the rotation axes can be obtained. Furthermore, if the attitude detection unit 15 includes a gyro sensor, GPS, geomagnetic sensor, etc. in addition to the IMU, the attitude detection unit can detect the rotation angle with the y-axis direction as the rotation axis, so more accurate attitude information can be obtained.

[0029] The imaging unit 14 is mounted on the imaging device 10. Therefore, in the following explanation, the posture information of the imaging unit 14 may be referred to as the posture information of the imaging device 10. In other words, the posture information of the imaging unit 14 and the posture information of the imaging device 10 have the same meaning.

[0030] The control unit 17 of the imaging device 10 transmits the 3D information obtained by imaging, the captured image, and the orientation information of the imaging unit 14 at the time of capturing the 3D information to the information processing device 20.

[0031] The information processing device 20 is an information processing device that performs processes such as deleting areas to be deleted from the 3D information obtained by the imaging unit 14. Details of the areas to be deleted will be described later. The information processing device 20 is, for example, a smartphone, a tablet terminal, a personal computer, etc.

[0032] Figure 2 is a block diagram showing the functional configuration of the information processing system 1 of this embodiment.

[0033] The imaging device 10 includes an imaging unit 14, an attitude detection unit 15, an operation unit 16, a display unit 18, a communication unit 19, and a control unit 17. The imaging unit 14, attitude detection unit 15, operation unit 16, display unit 18, and communication unit 19 are communicated with the control unit 17 via a bus or the like. The communication unit 19 communicates with the information processing device 20 via a network NW or the like.

[0034] The control unit 17 includes a 3D information output unit 17A, an image capture output unit 17B, an attitude information output unit 17C, and an output control unit 17D. The 3D information output unit 17A, the image capture output unit 17B, the attitude information output unit 17C, and the output control unit 17D are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may also be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may also be implemented by using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0035] The 3D information output unit 17A outputs 3D information obtained by the imaging unit 14 imaging area S to the information processing device 20. The captured image output unit 17B outputs the captured image obtained by the imaging unit 14 imaging area S to the information processing device 20. The posture information output unit 17C outputs the posture information of the imaging unit 14 detected by the posture detection unit 15 to the information processing device 20. Specifically, for each imaging, the 3D information output unit 17A, the captured image output unit 17B, and the posture information output unit 17C output imaging information to the information processing device 20, which includes the 3D information and captured image obtained by the imaging, the posture information of the imaging unit 14 at the time of imaging, and imaging time information representing the timing of imaging. In other words, the control unit 17 transmits imaging information to the information processing device 20 via the communication unit 19 each time imaging is performed.

[0036] The information processing device 20 comprises a communication unit 22, a UI (user interface) unit 24, a storage unit 26, and a control unit 28. The communication unit 22, the UI unit 24, the storage unit 26, and the control unit 28 are communicated with each other.

[0037] The communication unit 22 communicates with an external information processing device via a network NW or the like. In this embodiment, the communication unit 22 communicates with the imaging device 10. The UI unit 24 includes a display unit 24A and an operation unit 24B. The display unit 24A is a display that shows various kinds of information. The operation unit 24B receives operation instructions from the user. The operation unit 24B is, for example, a keyboard, a pointing device, a mouse, etc. The display unit 24A and the operation unit 24B may be an integrated touch panel. The storage unit 26 stores various kinds of information.

[0038] The control unit 28 includes a 3D information acquisition unit 28A, a posture information acquisition unit 28B, a captured image acquisition unit 28C, a determination unit 28D, a deletion unit 28E, a storage control unit 28F, and a display control unit 28G. The 3D information acquisition unit 28A, posture information acquisition unit 28B, captured image acquisition unit 28C, determination unit 28D, deletion unit 28E, storage control unit 28F, and display control unit 28G are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be implemented by a dedicated IC or other processor, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0039] At least one of the components included in the information processing device 20 may be mounted on an external information processing device, such as a server, connected via a network NW.

[0040] The 3D information acquisition unit 28A acquires 3D information obtained by imaging the imaging area S by the imaging unit 14. The 3D information acquisition unit 28A acquires 3D information by reading the 3D information contained in the imaging information received from the imaging device 10 via the communication unit 22.

[0041] The posture information acquisition unit 28B acquires the posture information of the imaging unit 14 at the time of imaging of the 3D information acquired by the 3D information acquisition unit 28A. The posture information acquisition unit 28B acquires posture information by reading the posture information contained in the imaging information received from the imaging device 10 via the communication unit 22.

[0042] The image acquisition unit 28C acquires the captured image of the shooting area S. The image acquisition unit 28C acquires the captured image by reading the captured image included in the shooting information received from the shooting device 10 via the communication unit 22.

[0043] The determination unit 28D determines the region to be deleted from the 3D information acquired by the 3D information acquisition unit 28A, based on the posture information acquired by the posture information acquisition unit 28B. The deletion unit 28E deletes the region to be deleted from the 3D information, as determined by the determination unit 28D.

[0044] The area to be deleted is information indicating an area representing unwanted objects other than the subject being photographed, which are included in the shooting area S. Unwanted objects include, for example, the photographer operating the shooting device 10. In this embodiment, the form in which the unwanted object is the photographer will be described as an example.

[0045] The area to be deleted can be any information that indicates an area representing unwanted objects included in the imaging area S. For example, the area to be deleted may be information represented by a range of directions relative to the imaging unit 14, a range of the field of view of the imaging unit 14, or directions corresponding to the rotation angles of the x, y, and z axes from the imaging unit 14. Alternatively, if the 3D information 30 is a depth image, the area to be deleted may be information represented by the coordinates of one or more pixels included in the depth image.

[0046] Here, we will explain how the photographer used the camera 10.

[0047] As described above, the imaging device 10 is configured to be portable by the photographer. Therefore, the photographer holds the imaging device 10 with their hand or the like and operates the control unit 16 to input an imaging instruction signal. Alternatively, a rod-shaped member can be attached to the imaging device 10, and the photographer can hold the rod-shaped member to adjust the imaging device 10, which is supported by the rod-shaped member, to a desired position before inputting the imaging instruction signal. The rod-shaped member is sometimes referred to as a selfie stick. Upon receiving the input of the imaging instruction signal, the imaging device 10 captures 3D information and images.

[0048] When the imaging device 10 is held directly by the photographer or via a rod-shaped member or the like while imaging is performed, unwanted objects such as the photographer may appear in the 3D image.

[0049] Figure 3A is an explanatory diagram illustrating an example of the usage of the imaging device 10. Figure 3A shows an example of imaging the imaging area S in which the target object OB is located. Figure 3A also shows target object OB1 and target object OB2 as the target object OB. Figure 3B is a schematic diagram showing an example of 3D information 30 captured in the usage situation shown in Figure 3A.

[0050] For example, photographer B attaches the rod-shaped member 9 to the imaging device 10 and performs imaging. By gripping the rod-shaped member 9, photographer B adjusts the imaging device 10, which is supported by the rod-shaped member 9, to a desired position, and inputs an imaging instruction signal for the imaging area S by operating the rod-shaped member 9 or the control unit 16 directly. Upon input of the imaging instruction signal, the imaging device 10 images the imaging area S and obtains three-dimensional information 30 and an image.

[0051] As shown in Figure 3A, objects located in the blind spot BS of the imaging device 10 in the imaging area S are not included in the 3D information 30, as shown in Figure 3B. In the example shown in Figures 3A and 3B, one example is the case where a part Ba of the photographer B's arm is located within the blind spot BS. In this case, as shown in Figure 3B, the 3D information 30 does not include 3D information representing the part Ba of the photographer B's arm.

[0052] However, objects that are outside the blind spot BS of the imaging device 10 at the time of imaging will be captured in the 3D information 30. In the example shown in Figure 3A, the area of ​​the photographer B's arm other than a portion Ba is outside the blind spot BS. Therefore, as shown in Figure 3B, the 3D information 30 includes 3D information representing the area of ​​the photographer B other than the portion Ba. In this case, the 3D information 30 includes not only the object to be photographed OB, but also 3D information representing an object that is not the object to be photographed, i.e., an unwanted object, which is a portion of the photographer B.

[0053] One method for removing photographer B, an unwanted object captured in 3D information 30, is to detect photographer B in the 3D information 30 and remove the area containing the detected photographer B. Detecting photographer B can be done by matching the 3D information corresponding to photographer B with a predetermined model, or by using image recognition on a 2D image created from the 3D information. However, as the area containing photographer B in the 3D information 30 decreases, the amount of information about photographer B in the 3D information 30 decreases, which can make detecting photographer B difficult.

[0054] Figure 4A is an explanatory diagram illustrating another example of the usage of the imaging device 10. Figure 4B is a schematic diagram showing an example of 3D information 30 captured under the usage conditions shown in Figure 4A.

[0055] In the example shown in Figure 4A, a larger area of ​​photographer B's body is within the blind spot BS of the imaging device 10 compared to the example shown in Figure 3A. In this case, as shown in Figure 4B, the 3D information 30 contains less of photographer B's image compared to the 3D information 30 shown in Figure 3B. The less of photographer B's image is included in the 3D information 30, the more difficult it becomes to identify the unwanted area of ​​photographer B included in the 3D information 30. Therefore, it becomes difficult to detect photographer B's area, and difficult to remove photographer B from the 3D information 30.

[0056] Returning to Figure 2, we continue the explanation.

[0057] Therefore, the determination unit 28D of this embodiment determines the region to be deleted included in the 3D information 30 based on the orientation information of the imaging unit 14 at the time of capturing the 3D information 30.

[0058] In this embodiment, the determination unit 28D determines the region to be deleted as the region obtained by correcting the reference deletion region included in the reference setting information based on the posture information.

[0059] Reference setting information is information that associates the reference posture information of the imaging unit 14 with the reference deletion area included in the imaging area S. In this embodiment, the storage unit 26 stores the reference setting information in advance.

[0060] Reference posture information refers to information that represents the standard posture of the imaging unit 14. Similar to posture information, reference posture information is represented by the rotation angles with respect to the x, y, and z axes.

[0061] The reference deletion area is information representing the area to be deleted, included in the 3D information 30 captured by the imaging unit 14 in the posture represented by the reference posture information. The reference deletion area is predetermined depending on the type of unwanted object to be deleted. For example, let's assume the unwanted object is the photographer B. In this case, depending on the usage of the imaging unit 14, the estimated range in which photographer B would be captured when the imaging unit 14 photographs the imaging area S in the posture represented by the reference posture information is predetermined as the reference deletion area.

[0062] The determination unit 28D reads the reference setting information stored in the storage unit 26 and uses it for the process of determining the area to be deleted.

[0063] In this embodiment, the memory unit 26 is described as having one set of reference setting information pre-stored in it. However, the memory unit 26 may have multiple sets of reference setting information pre-stored in it. In this case, for example, the display control unit 28G displays the multiple sets of reference setting information stored in the memory unit 26 on the display unit 24A so that they can be selected. When one set of reference setting information is selected by an operation instruction from the user via the operation unit 24B, the determination unit 28D can then use the selected set of reference setting information for determining the area to be deleted.

[0064] The control unit 28 may store in advance reference setting information set by the user in the storage unit 26. Alternatively, the control unit 28 may acquire the usage status and shooting conditions of the shooting unit 14 from the shooting device 10, create reference setting information in advance using this acquired information, and store it in advance in the storage unit 26.

[0065] The determination unit 28D determines the reference deletion area associated with the reference posture information as the area to be deleted if the reference posture information included in the reference setting information read from the storage unit 26 matches the posture information acquired by the posture information acquisition unit 28B.

[0066] As described above, the basic posture information and the posture information are each represented by at least one of the rotation angles with respect to the x, y, and z axes, respectively. Therefore, when the basic posture information and the posture information match, it means that the rotation angles represented by the basic posture information and the posture information are the same. In other words, when the basic posture information and the posture information match, it means that the posture of the imaging device 10 represented by the basic posture information and the posture of the imaging device 10 represented by the posture information are the same posture.

[0067] Here, for accuracy, it is preferable that the basic posture information and posture information include rotation angles with the x, y, and z axes as the rotation axes, respectively. When photographer B grasps the imaging device 10 as shown in Figure 3A, the reference posture information and posture information should include information on rotation angles with the x-axis and z-axis as the rotation axes.

[0068] Figures 5A and 5B are explanatory diagrams illustrating an example of determining the region E2 to be deleted. Figure 5A shows an example of the usage of the imaging device 10. Figure 5A also shows an example where the reference posture information and the posture information of the imaging device 10 at the time of imaging match. Figure 5B is a schematic diagram showing an example of 3D information 30 obtained by removing the region E2 to be deleted from the 3D information 30 captured under the usage conditions shown in Figure 5A.

[0069] As shown in Figure 5A, if the reference posture information matches the posture information acquired by the posture information acquisition unit 28B, the determination unit 28D determines the reference deletion area E1 as the deletion target area E2. Then, the deletion unit 28E deletes the deletion target area E2 included in the 3D information 30. Therefore, as shown in Figure 5B, the 3D information 30 after the deletion process by the deletion unit 28E will be information that does not include the photographer B included in the shooting area S.

[0070] The deletion process performed by the deletion unit 28E means invalidating or setting to zero the 3D information corresponding to the deletion target area E2 included in the 3D information 30. For example, consider the case where the 3D information 30 is a distance image in which the distance from the imaging device 10 is defined for each pixel. In this case, the deletion unit 28E deletes the deletion target area E2 from the 3D information 30 by performing a process such as setting the pixel values ​​within the range of the deletion target area E2 included in the 3D information 30 to zero or invalid. Alternatively, the deletion unit 28E may delete the deletion target area E2 by assigning a label indicating that it is invalid to the pixels within the range of the deletion target area E2 included in the 3D information 30.

[0071] Therefore, the 3D information 30 from which the deletion target area E2 has been removed by the deletion unit 28E will be information that does not include the photographer B, as shown in Figure 5B. In other words, the 3D information 30 from which the deletion target area E2 has been removed will be 3D information that does not include unwanted objects such as the photographer B.

[0072] On the other hand, if the reference posture information and the posture information acquired by the posture information acquisition unit 28B do not match, the determination unit 28D determines the area to be deleted as the area to be deleted E2, which is the corrected reference deletion area E1 according to the difference between the reference posture information and the posture information.

[0073] As described above, the imaging device 10 is used by being held by the photographer B. Therefore, the posture of the imaging device 10 during imaging will vary from the posture represented by the reference posture information. For example, the angle of photographer B's wrist during imaging may not be stable, and the posture of the imaging device 10 held by photographer B will vary during imaging. The posture of the imaging device 10 held by photographer B is unstable and prone to fluctuations with respect to rotational directions with the x-axis and z-axis as the axes of rotation. In this case, the posture information representing the posture of the imaging device 10 during imaging will be different from the reference posture information.

[0074] Therefore, if the reference posture information and the posture information acquired by the posture information acquisition unit 28B do not match, the determination unit 28D determines the area to be deleted as the area to be deleted E2, which is the corrected reference deletion area E1 according to the difference between the reference posture information and the posture information.

[0075] Figure 6A is an explanatory diagram illustrating an example of how the deletion target area E2 is determined. Figure 6A shows an example of the usage status of the imaging device 10. Figure 6A also shows an example of a case where the reference posture information and the posture information of the imaging unit 14 of the imaging device 10 at the time of imaging do not match.

[0076] Figure 6A shows an example of a posture in which the imaging device 10, held by photographer B, has a rotation angle with the x-axis as the axis of rotation. As shown above, the posture of the imaging device 10 held by photographer B is prone to fluctuations in the rotation angle with the x-axis as the axis of rotation. On the other hand, the rotation angle with the y-axis (vertical direction) as the axis of rotation is less prone to fluctuations, as mentioned above. In the following explanation, posture information and reference posture information will be described using examples where the rotation angles are with the x-axis, y-axis, and z-axis as the axes of rotation, but posture information and reference posture information only need to include information on at least one of these rotation angles or quantities related to rotation angles, and more preferably, they need to include information on the rotation angles with the x-axis and z-axis as the axes of rotation, respectively.

[0077] The determination unit 28D calculates the difference in rotation angles for the x, y, and z axes between the basic posture information and the posture information, which are represented by rotation angles for the x, y, and z axes, respectively. The determination unit 28D then corrects the reference deletion region E1 so as to cancel out the difference in rotation angles for the x, y, and z axes between the basic posture information and the posture information.

[0078] Specifically, Figure 6A shows an example where the imaging device 10 is tilted +10° with respect to the y-axis in the circumferential direction of the x-axis, with the x-axis as the axis of rotation, relative to the posture represented by the reference posture information. In this case, the determination unit 28D corrects the reference deletion area E1 corresponding to the reference posture information so that it becomes a region tilted -10° with respect to the y-axis in the circumferential direction of the x-axis, with the x-axis as the axis of rotation. Then, the corrected reference deletion area E1 is determined as the deletion target area E2.

[0079] In this way, the determination unit 28D determines the area to be deleted as the area E2, which is the corrected area of ​​the reference deletion area E1, according to the difference between the reference posture information and the posture information.

[0080] The deletion unit 28E then deletes the deletion target area E2 included in the 3D information 30. Therefore, similar to the 3D information 30 shown in Figure 5B, the 3D information 30 after the deletion process by the deletion unit 28E becomes information that does not include the photographer B included in the shooting area S. In other words, the 3D information 30 from which the deletion target area E2 has been deleted becomes 3D information without the photographer B being captured.

[0081] Figure 6B is an explanatory diagram illustrating an example of how the deletion target area E2 is determined. Figure 6B shows an example of the usage status of the imaging device 10. Figure 6B also shows an example of a case where the reference posture information and the posture information of the imaging unit 14 of the imaging device 10 at the time of imaging do not match. Furthermore, Figure 6B shows an example of a state in which the imaging device 10 is tilted at -10° with respect to the y-axis in the circumferential direction of the x-axis, with the x-axis as the axis of rotation, relative to the posture represented by the reference posture information.

[0082] In this case, the determination unit 28D corrects the reference deletion region E1 corresponding to the reference posture information so that it becomes a region tilted by +10° with respect to the y axis in the circumferential direction of the x axis, with the x axis as the axis of rotation. Then, the corrected reference deletion region E1 is determined as the deletion target region E2.

[0083] In this way, the determination unit 28D determines the area to be deleted as the area E2, which is the corrected area of ​​the reference deletion area E1, according to the difference between the reference posture information and the posture information.

[0084] The deletion unit 28E then deletes the deletion target area E2 included in the 3D information 30. Therefore, similar to the 3D information 30 shown in Figure 5B, the 3D information 30 after the deletion process by the deletion unit 28E becomes information that does not include the photographer B included in the shooting area S. In other words, the 3D information 30 from which the deletion target area E2 has been deleted becomes 3D information without the photographer B being captured.

[0085] Returning to Figure 2, the explanation continues. The memory control unit 28F stores information related to the deleted 3D information 30 in the memory unit 26. For example, the memory control unit 28F stores the 3D information 30, posture information, captured image, and capture time information included in the capture information acquired from the shooting device 10 in the memory unit 26, associating them with each other. The memory control unit 28F may further assign an identifiable shooting ID or the like to the captured 3D information 30 and store it in the memory unit 26 as well.

[0086] Furthermore, the memory control unit 28F may store information related to the deleted 3D information 30 from the deletion target area E2 in an external information processing device or storage device connected to the information processing device 20 via the network NW.

[0087] The display control unit 28G displays an image containing information related to the 3D information 30 on the display unit 24A. More specifically, the display control unit 28G displays an image on the display unit 24A that includes at least one of the following: a captured image of the shooting area S represented by the 3D information 30, information related to the deletion target area E2 determined by the determination unit 28D, and information related to the 3D information 30 from which the deletion target area E2 has been deleted.

[0088] For example, the display control unit 28G displays an image on the display unit 24A that includes the deletion target area E2 in a visible manner. Specifically, the display control unit 28G converts the deleted 3D information 30 into a 2D image using known coordinate transformation processing, etc., and displays it on the display unit 24A. At that time, the display control unit 28G displays the deletion target area E2 superimposed on the image in a visible manner. By displaying the deletion target area E2 in a visible manner within the image, it is possible to easily provide the user with the ability to confirm whether or not the deletion process was appropriate.

[0089] Furthermore, the display control unit 28G may identify the deletion target area E2 in the captured image taken at the same time as the 3D information 30, and display the image with the identified deletion target area E2 removed on the display unit 24A. Known techniques can be used to identify the deletion target area E2 in the captured image.

[0090] By displaying the above image on the display unit 24A, the display control unit 28G can provide the user with information regarding the deletion target area E2 in a verifiable manner. Furthermore, the display control unit 28G can provide the user with the 3D information 30 from which the deletion target area E2 has been removed in a verifiable manner.

[0091] The display control unit 28G may also display the above image on the display unit 18 provided on the shooting device 10. In this case, the display control unit 28G transmits the above image to the shooting device 10. The control unit 17 of the shooting device 10, upon receiving the image, should then display the received image on the display unit 18.

[0092] Next, the flow of information processing performed by the information processing device 20 of this embodiment will be described.

[0093] Figure 7 is a flowchart showing an example of the information processing flow performed by the information processing device 20 of this embodiment.

[0094] The 3D information acquisition unit 28A and the attitude information acquisition unit 28B acquire 3D information 30 obtained by imaging the imaging area S by the imaging unit 14 and attitude information of the imaging unit 14 at the time of imaging the 3D information 30 (step S100). The 3D information acquisition unit 28A and the attitude information acquisition unit 28B acquire 3D information 30 and attitude information by reading the 3D information 30 and attitude information contained in the imaging information received from the imaging device 10 via the communication unit 22.

[0095] The determination unit 28D reads the basic setting information from the storage unit 26 (step S102). As described above, the basic setting information is information that associates the reference deletion area E1 with the basic posture information.

[0096] Next, the determination unit 28D determines whether the attitude information acquired in step S100 matches the reference attitude information included in the reference setting information read in step S102 (step S104). The determination unit 28D makes the determination in step S104 by determining whether the rotation angles for the x, y, and z axes represented by the basic attitude information and the attitude information match.

[0097] If it is determined that the reference posture information and the posture information match (step S104, Yes), the process proceeds to step S106. In step S106, the determination unit 28D determines that the reference deletion area E1 included in the reference setting information read in step S102 is the deletion target area E2 (step 106). Then, the process proceeds to step S112, which will be described later.

[0098] On the other hand, if it is determined that the reference posture information and the posture information do not match (step S104: No), the process proceeds to step S108. In step S108, the determination unit 28D corrects the reference deletion area E1 according to the difference between the reference posture information and the posture information (step S108). The determination unit 28D calculates the difference in rotation angles for the x, y, and z axes between the basic posture information and the posture information, which are represented by rotation angles for the x, y, and z axes, respectively. The determination unit 28D then corrects the posture of the reference deletion area E1 so as to cancel out the difference in rotation angles for the x, y, and z axes between the basic posture information and the posture information.

[0099] Then, the determination unit 28D determines the reference deletion area E1 corrected in step S108 as the deletion target area E2 (step S110). Then, the process proceeds to step S112.

[0100] In step S112, the deletion unit 28E deletes the deletion target area E2 included in the 3D information 30 acquired in step S110 (step S112). The process in step S112 deletes the deletion target area E2, which includes unwanted objects such as photographer B, from the 3D information 30. The 3D information 30 from which the deletion target area E2 has been deleted becomes 3D information without unwanted objects such as photographer B.

[0101] The memory control unit 28F stores in the memory unit 26 the information related to the 3D information 30 from which the deletion target area E2 was deleted by the deletion process in step S112 (step S114). The display control unit 28G displays an image containing the information related to the 3D information 30 from which the deletion target area E2 was deleted by the deletion process in step S112 on the display unit 24A (step S116).

[0102] Next, the control unit 28 determines whether or not to terminate the process (step S118).

[0103] For example, consider a scenario where the imaging device 10 sequentially transmits imaging information, including the 3D information 30, to the imaging device 10 each time it obtains 3D information 30 through imaging. Then, consider a scenario where the control unit 28 executes the processing described in steps S100 to S116 each time it receives a new piece of imaging information from the imaging device 10. In this case, the control unit 28 only needs to decide to terminate processing if it does not receive the next piece of imaging information from the imaging device 10 within a predetermined period since the previous piece of imaging information was received.

[0104] The control unit 28 may also store one or more pieces of imaging information received from the imaging device 10 in chronological order in the storage unit 26. The control unit 28 may then sequentially read the imaging information stored in the storage unit 26 one by one and perform the processing in steps S100 to S116. In this case, the control unit 28 determines whether the processing in steps S100 to S116 has been performed for all three-dimensional information 30 included in the one or more pieces of imaging information stored in the storage unit 26. If the control unit 28 determines that the processing has been performed, it makes an affirmative decision in step S118. If there is three-dimensional information 30 that has not yet been processed stored in the storage unit 26, the control unit 28 should make a negative decision in step S118.

[0105] If the decision in step S118 is negative (step PS118: No), the process returns to step S100. If the decision in step S118 is positive (step S118: Yes), this routine terminates.

[0106] As described above, the information processing device 20 of this embodiment comprises a 3D information acquisition unit 28A, a posture information acquisition unit 28B, a determination unit 28D, and a deletion unit 28E. The 3D information acquisition unit 28A acquires 3D information 30 obtained by imaging the imaging area S by the imaging unit 14. The posture information acquisition unit 28B acquires the posture information of the imaging unit 14 at the time of imaging the 3D information 30. The determination unit 28D determines the deletion target area E2 included in the 3D information 30 based on the posture information. The deletion unit 28E deletes the deletion target area E2 included in the 3D information 30.

[0107] Here, the posture of the imaging unit 14 may change depending on how the imaging device 10, which is equipped with the imaging unit 14, is held by the photographer B. In particular, with a handheld imaging device 10 that is held by the photographer B, the angle of the photographer B's wrist during shooting is not stable, causing the posture of the imaging unit 14 to change during shooting.

[0108] Conventional technology discloses a method in which the area to be deleted is fixed regardless of changes in the orientation of the imaging unit 14, and this fixed area to be deleted is deleted from the 3D information 30. However, when the area to be deleted is fixed regardless of changes in the orientation of the imaging unit 14, depending on the imaging conditions, unwanted objects such as the photographer B may be captured in areas other than the area to be deleted. For this reason, if the area to be deleted is fixed regardless of changes in orientation, it may not be possible to properly delete the area to be deleted in which unwanted objects have been captured. Also, depending on the imaging conditions, the subject OB may be captured within the fixed area to be deleted, and the subject OB may be deleted.

[0109] On the other hand, the information processing device 20 of this embodiment determines the region E2 to be deleted from the 3D information 30 based on the orientation information of the imaging unit 14 at the time of imaging, and deletes the region E2 from the 3D information 30. In other words, the information processing device 20 of this embodiment determines the region E2 to be deleted according to the orientation information of the imaging unit 14 at the time of imaging of the 3D information 30.

[0110] Therefore, even if the posture of the imaging unit 14 changes or fluctuates during shooting, the information processing device 20 of this embodiment can appropriately delete the deletion target area E2 included in the 3D information 30.

[0111] Therefore, the information processing device 20 of this embodiment can appropriately delete the deletion target area E2 included in the three-dimensional information 30.

[0112] (modified version) In the above embodiment, a configuration in which the imaging device 10 and the information processing device 20 are configured as separate components was described as an example. However, the imaging device 10 and the information processing device 20 may be configured as an integrated unit. For example, each functional unit included in the control unit 28 of the information processing device 20 may be mounted on the control unit 17 of the imaging device 10.

[0113] In more detail, a device comprising the information processing device 20 of the above embodiment, an imaging unit 14, and a posture detection unit 15 may be configured as a three-dimensional measuring device.

[0114] (Second embodiment) In the above embodiment, a configuration in which reference setting information including the reference deletion area E1 is pre-stored in the storage unit 26 was described as an example. In this embodiment, a configuration in which the reference deletion area E1 is set by user operation instructions will be described as an example.

[0115] In this embodiment, the same reference numerals are used for components similar to those in the above embodiment, and detailed descriptions are omitted.

[0116] Figure 8 is a block diagram showing the functional configuration of the information processing system 1B of this embodiment.

[0117] Information processing system 1B comprises a camera 10, an information processing device 21, and an information terminal 40. The camera 10, the information processing device 21, and the information terminal 40 are connected to each other via a network NW or the like.

[0118] The information terminal 40 is an information processing device managed by a user. The information terminal 40 is, for example, a smartphone or a tablet device. The user who manages the information terminal 40 may be the same as or different from the photographer B of the camera 10. In this embodiment, the case where the user who manages the information terminal 40 is photographer B will be described as an example.

[0119] The information terminal 40 comprises an operation unit 42, a display unit 44, a communication unit 46, and a control unit 48. The operation unit 42 receives operation instructions from the photographer B. The operation unit 42 is, for example, a keyboard, a pointing device, a mouse, etc. The display unit 44 is a display that shows various kinds of information. The operation unit 42 and the display unit 44 may be an integrated touch panel. The communication unit 46 communicates with the imaging device 10 and the information processing device 23 via the network NW. The control unit 48 performs information processing in the information terminal 40. The control unit 48 is composed of one or more processors.

[0120] The information processing device 21 comprises a communication unit 22, a UI unit 24, a storage unit 26, and a control unit 27. The communication unit 22, the UI unit 24, the storage unit 26, and the control unit 27 are communicated to each other via a bus or the like. The information processing device 21 is the same as the information processing device 20 of the above embodiment, except that it includes a control unit 27 instead of a control unit 28.

[0121] The control unit 27 includes a 3D information acquisition unit 28A, a posture information acquisition unit 28B, a captured image acquisition unit 28C, a determination unit 28D, a deletion unit 28E, a storage control unit 28F, a display control unit 28G, a reception unit 27I, and an output control unit 27H. The control unit 27 is the same as the control unit 28 in the above embodiment, except that it further includes a reception unit 27I and an output control unit 27H.

[0122] The reception unit 27I receives operation instructions from the user. In this embodiment, the reception unit 27I receives various operation instructions input by the photographer B through the operation of the operation unit 42 of the information terminal 40, via the communication unit 22.

[0123] The output control unit 27H controls the imaging device 10 to acquire three-dimensional information 30 for correction by imaging. The three-dimensional information 30 for correction refers to the three-dimensional information 30 used for correcting the reference deletion region E1.

[0124] For example, the output control unit 27H transmits an instruction signal to the imaging device 10 to instruct it to acquire the 3D information 30 for correction by imaging. The control unit 17 of the imaging device 10 displays information representing the instruction to image the 3D information 30 for correction on the display unit 18. The photographer B, after viewing the information displayed on the display unit 18, adjusts the posture of the imaging device 10 to be the same as or similar to the posture to be imaged, and inputs an image instruction signal by operating the operation unit 16.

[0125] When the operator operates the control unit 16 and inputs a shooting instruction signal, the control unit 17 controls the image acquisition unit 11, the distance information acquisition unit 13, and the projection unit 12 to synchronously drive and perform shooting. As a result, the shooting device 10 obtains correction 3D information 30. The control unit 17 of the shooting device 10 transmits correction shooting information, which includes the 3D information 30 obtained by shooting and a correction flag indicating that it is correction 3D information 30, to the information processing device 21. Alternatively, the control unit 17 may transmit correction shooting information, which includes the 3D information 30 obtained by shooting, the correction flag, posture information, and the captured image, to the information processing device 21.

[0126] The imaging device 10 may be configured to switch between multiple modes, including a correction imaging mode for correction-oriented 3D information 30 and a normal imaging mode for 3D information 30 that is not for correction. The photographer B may then switch from the imaging mode to the correction imaging mode by operating the operation unit 16, adjust the orientation of the imaging device 10 to be the same as or similar to the orientation to be photographed, and input an imaging instruction signal by operating the operation unit 16. When imaging is performed in the correction imaging mode, the control unit 17 should then transmit the correction imaging information, including the 3D information 30 obtained by imaging and a correction flag, to the information processing device 21.

[0127] Furthermore, the imaging device 10 may transmit the 3D information 30 obtained from the first image taken after power is supplied to each part of the imaging device 10 by the operator B's operation of the power button, etc., to the information processing device 21 as correction 3D information 30. In this case, the imaging device 10 only needs to transmit the correction imaging information, which includes the 3D information 30 obtained from the first image taken and a correction flag, to the information processing device 21.

[0128] Furthermore, when the imaging device 10 captures multiple 3D information 30 in a time series, it may transmit any one of the multiple 3D information 30 to the information processing device 21 as correction 3D information 30. In this case, the imaging device 10 only needs to transmit correction imaging information, including the arbitrary one 3D information 30 and a correction flag, to the information processing device 21.

[0129] When the output control unit 27H of the information processing device 21 acquires correction imaging information including a correction flag from the imaging device 10, it reads the reference deletion area E1 included in the reference setting information stored in the storage unit 26. Then, the output control unit 27H transmits the correction 3D information 30 included in the correction imaging information and the reference deletion area E1 read from the storage unit 26 to the information terminal 40.

[0130] The control unit 48 of the information terminal 40 displays the correction image, which includes the correction 3D information 30 and the reference deletion area E1 received from the information processing device 21, on the display unit 44.

[0131] For example, the control unit 48 of the information terminal 40 displays a correction image on the display unit 44, which is a distance image that is the 3D information 30 for correction superimposed with a reference deletion area E1. Alternatively, for example, the control unit 48 of the information terminal 40 may convert the 3D information 30 for correction into a 2D display image using a known method, and then display a correction image on the display unit 44, which is the display image with the reference deletion area E1 superimposed on it. Furthermore, the output control unit 27H of the information processing device 21 may transmit the captured image taken when the 3D information 30 for correction was captured to the information terminal 40. In this case, the control unit 48 of the information terminal 40 may display a correction image on the display unit 44, which is the captured image with the reference deletion area E1 superimposed on it.

[0132] Photographer B, operating the information terminal 40, corrects at least a portion of the position, size, and range of the reference deletion area E1 so that it becomes the desired reference deletion area E1 by operating the operation unit 42 while viewing the correction image displayed on the operation unit 42. Photographer B corrects the reference deletion area E1 using any input method, such as mouse operation (an example of the operation unit 42), position on the touch panel (an example of the operation unit 42), or numerical input representing coordinates using the keyboard (an example of the operation unit 42).

[0133] The control unit 48 then transmits information representing the corrected reference deletion area E1, which was input by the photographer B through the operation instructions of the operation unit 42, to the information processing device 21.

[0134] Furthermore, if photographer B uses the reference deletion area E1 displayed on the display unit 44 without correction, they only need to input an instruction to use the displayed reference deletion area E1 as the corrected reference deletion area E1. In this case, the control unit 48 should transmit the displayed reference deletion area E1 to the information processing device 21 as information representing the corrected reference deletion area E1.

[0135] The receiving unit 27I of the information processing device 21 receives the corrected reference deletion area E1 input by the operator B through the operation instructions of the operation unit 42. The receiving unit 27I overwrites the reference deletion area E1 stored in the storage unit 26 with the received corrected reference deletion area E1 and stores it. Therefore, the determination unit 28D uses the reference deletion area E1 input by the operator B through the operation instructions to determine the deletion target area E2 in the same manner as in the above embodiment.

[0136] In other words, the determination unit 28D can determine the deletion target area E2 included in the 3D information 30 in the same manner as in the above embodiment, based on the posture information acquired by the posture information acquisition unit 28B and the reference deletion area E1 received by the reception unit 27I from the operation unit 42 of the information terminal 40.

[0137] Next, an example of the information processing flow executed by the information processing device 21 of this embodiment will be described.

[0138] Figure 9 is a flowchart showing an example of the information processing flow performed by the information processing device 21 of this embodiment.

[0139] The output control unit 27H acquires correction imaging information, including correction 3D information 30 and correction flags, from the imaging device 10 (step S200). The output control unit 27H acquires correction 3D information 30 from the imaging device 10, for example, by controlling the imaging device 10 to acquire the correction 3D information 30 by imaging.

[0140] Next, the output control unit 27H reads the reference deletion area E1 included in the reference setting information stored in the memory unit 26 (step S202). Then, the output control unit 27H transmits the correction 3D information 30 acquired in step S200 and the reference deletion area E1 read in step S202 to the information terminal 40 (step S204).

[0141] The control unit 48 of the information terminal 40 displays a correction image, including the correction 3D information 30 and the reference deletion area E1 received from the information processing device 21, on the display unit 44. The photographer B, operating the information terminal 40, corrects at least a portion of the position, size, and range of the displayed reference deletion area E1 by operating the operation unit 42 while viewing the correction image displayed on the operation unit 42, so that it becomes the desired reference deletion area E1. The control unit 48 of the information terminal 40 transmits information representing the corrected reference deletion area E1, input by the photographer B's operation instructions for the operation unit 42, to the information processing device 21.

[0142] The receiving unit 27I of the information processing device 21 receives the corrected reference deletion area E1 input by the photographer B via the operation instructions of the operation unit 42 from the information terminal 40 (step S206). The receiving unit 27I overwrites the reference deletion area E1 stored in the storage unit 26 with the corrected reference deletion area E1 received in step S206 and stores it (step S208).

[0143] Then, the control unit 27 of the information processing device 21 executes the deletion process, which is represented by steps S100 to S118 (see Figure 7) of the above embodiment (step S210), and terminates this routine.

[0144] As described above, the information processing device 21 of this embodiment includes a reception unit 27I that receives operation instructions from the user. The determination unit 28D determines the deletion target area E2 included in the 3D information 30 based on the posture information and the reference setting information including the reference deletion area E1 received by the reception unit 27I.

[0145] In other words, in this embodiment, the determination unit 28D determines the deletion target area E2 of the 3D information 30 based on the reference deletion area E1 corrected by the operator B's instructions and the orientation information of the imaging unit 14 at the time of capturing the newly captured 3D information 30. Then, the deletion unit 28E executes a deletion process to delete the determined deletion target area E2 from the 3D information 30.

[0146] Here, the positional relationship between the imaging device 10 and the photographer B will differ depending on the size of the photographer B, the height of the imaging device 10 held by the photographer B relative to the photographer B, etc. On the other hand, the information processing device 21 of this embodiment determines the deletion target area E2 using the reference deletion area E1 set by the user. Then, the deletion unit 28E executes a deletion process to delete the determined deletion target area E2 from the 3D information 30.

[0147] Therefore, in addition to the effects of the above embodiment, the information processing device 21 of this embodiment can more appropriately delete the deletion target area E2 from the three-dimensional information 30.

[0148] Furthermore, the information processing device 21 of this embodiment corrects the reference deletion area E1 by receiving operation instructions from the user. Therefore, even if the photographer B of the camera 10 changes to a different photographer B, or if the shooting conditions of the camera 10 change, the information processing device 21 can use the appropriately corrected reference deletion area E1 for processing.

[0149] Furthermore, the information processing device 21 of this embodiment uses the reference deletion area E1 corrected by user operation instructions to determine the deletion target area E2 and delete the deletion target area E2 from the 3D information 30, thereby enabling the deletion process to be performed quickly and appropriately.

[0150] (Third embodiment) In the second embodiment described above, a configuration was described as in which the reference deletion area E1 included in the reference setting information pre-stored in the storage unit 26 is transmitted to the information terminal 40, and the user performs an operation input to correct the reference deletion area E1 using the reference deletion area E1.

[0151] This embodiment describes a configuration in which the information processing device 20 sets reference setting information.

[0152] In this embodiment, the same reference numerals are used for components similar to those in the above embodiment, and detailed descriptions are omitted.

[0153] Figure 10 is a block diagram showing the functional configuration of the information processing system 1C of this embodiment.

[0154] The information processing system 1C comprises a camera 10, an information processing device 23, and an information terminal 40. The camera 10, the information processing device 23, and the information terminal 40 are communicated with each other via a network NW or the like. The camera 10 and the information terminal 40 are the same as in the above embodiment.

[0155] The information processing device 23 comprises a communication unit 22, a UI unit 24, a storage unit 26, and a control unit 29. The communication unit 22, the UI unit 24, the storage unit 26, and the control unit 29 are communicated to each other via a bus or the like. The information processing device 23 is the same as the information processing device 21 of the above embodiment, except that it includes a control unit 29 instead of a control unit 27.

[0156] The control unit 29 includes a 3D information acquisition unit 28A, a posture information acquisition unit 28B, a captured image acquisition unit 28C, a determination unit 28D, a deletion unit 28E, a storage control unit 28F, a display control unit 28G, a reception unit 27I, an output control unit 27H, and a setting unit 29J. The control unit 29 is the same as the control unit 27 of the second embodiment described above, except that it further includes a setting unit 29J.

[0157] The setting unit 29J sets reference setting information based on the positional relationship between the photographer B and the imaging unit 14 in the 3D information 30. For example, the setting unit 29J sets reference setting information using at least one of the 3D information for correction 30 and the image captured simultaneously, and the 3D information for correction 30.

[0158] In detail, in this embodiment, the imaging device 10 transmits corrected imaging information to the information processing device 23, which includes corrective 3D information 30 obtained by imaging, an image obtained by the same imaging as the 3D information 30, posture information, and a correction flag. The 3D information acquisition unit 28A, posture information acquisition unit 28B, and image acquisition unit 28C of the information processing device 23 receive the corrected imaging information from the imaging device 10 and acquire the corrective 3D information 30, posture information, and image, respectively, that are included in the corrected imaging information.

[0159] The setting unit 29J uses at least one of the correction 3D information 30 and the captured image, which are included in the correction shooting information, to determine the positional relationship between the shooting unit 14 and the photographer B at the time of shooting the 3D information 30. The setting unit 29J uses at least one of the correction 3D information 30 and the captured image to determine the positional relationship between the photographer B and the shooting unit 14 in the 3D information 30 or the captured image, for example, by semantic segmentation processing using deep learning.

[0160] The setting unit 29J then uses the identified positional relationship to set the area including photographer B included in the 3D information 30 for correction as the reference deletion area E1. The setting unit 29J then uses the posture information of the shooting unit 14 at the time of shooting the 3D information 30 as reference posture information and stores it in the storage unit 26 as reference setting information in association with the set reference deletion area E1.

[0161] Through these processes, the setting unit 29J sets reference setting information that associates the reference deletion area E1 with the reference posture information and stores it in the storage unit 26. Then, the control unit 29 performs the same processing as in the second embodiment. That is, as in the second embodiment, the photographer B corrects the reference deletion area E1, and the deletion process is performed using the corrected reference deletion area E1. Therefore, in addition to the effects of the above embodiment, the information processing device 23 of this embodiment can delete the deletion target area E2 even more appropriately.

[0162] Furthermore, when correcting the reference deletion area E1, the control unit 48 of the information terminal 40 displays the correction image, including the 3D correction information 30 and the reference deletion area E1 received from the information processing device 23, on the display unit 44, similar to the second embodiment described above. Therefore, photographer B can easily confirm which area has been set as the reference deletion area E1 by the information processing device 23 by looking at the display unit 44. In addition, photographer B can correct the displayed reference deletion area E1 by operating the operation unit 42 as necessary, similar to the second embodiment described above.

[0163] Next, an example of the information processing flow performed by the information processing device 23 of this embodiment will be described.

[0164] Figure 11 is a flowchart showing an example of the information processing flow performed by the information processing device 23 of this embodiment.

[0165] The output control unit 27H acquires correction information, including correction flags, 3D information 30 for correction, captured images, and correction information from the imaging device 10 (step S300).

[0166] Next, the setting unit 29J uses at least one of the 3D information 30 acquired in step S200 and the captured image to determine the positional relationship between photographer B and the imaging unit 14. Then, based on the determined positional relationship, the setting unit 29J sets reference setting information that associates the reference deletion area E1 with the reference posture information and stores it in the storage unit 26 (step S302).

[0167] Next, the output control unit 27H reads the reference deletion area E1 included in the reference setting information stored in the storage unit 26 in step S302 (step S304). That is, the output control unit 27H reads the reference deletion area E1 set by the setting unit 29J.

[0168] Then, the output control unit 27H transmits the correction 3D information 30 acquired in step S300 and the reference deletion area E1 read in step S304 to the information terminal 40 (step S306).

[0169] The control unit 48 of the information terminal 40 displays the correction image, including the correction 3D information 30 and the reference deletion area E1 received from the information processing device 23, on the display unit 44. The photographer B, who is operating the information terminal 40, corrects at least a part of the position, size, and range of the displayed reference deletion area E1 by operating the operation unit 42 while viewing the correction image displayed on the operation unit 42, so that it becomes the desired reference deletion area E1. The control unit 48 of the information terminal 40 transmits information representing the corrected reference deletion area E1, which was input by the photographer B's operation instructions for the operation unit 42, to the information processing device 23.

[0170] The receiving unit 27I of the information processing device 23 receives the corrected reference deletion area E1 input by the photographer B via the operation instructions of the operation unit 42 from the information terminal 40 (step S308). The receiving unit 27I overwrites the reference deletion area E1 stored in the storage unit 26 with the corrected reference deletion area E1 received in step S308 and stores it (step S310).

[0171] Then, the control unit 29 of the information processing device 23 executes the deletion process, which is represented by steps S100 to S118 (see Figure 7) of the above embodiment (step S312), and terminates this routine.

[0172] As described above, the information processing device 23 of this embodiment includes a setting unit 29J. The setting unit 29J sets reference setting information based on the positional relationship between the photographer B and the imaging unit 14 of the 3D information 30. The determination unit 28D determines the deletion target area E2 included in the 3D information 30 based on the set reference setting information and the posture information.

[0173] Therefore, in addition to the effects of the above embodiment, the information processing device 23 of this embodiment can determine the deletion target area E2 using more accurate reference setting information.

[0174] Next, the hardware configuration of the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment will be described.

[0175] Figure 12 is a hardware configuration diagram of an example of the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment.

[0176] The imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment each include a control device such as a CPU 90A, a storage device such as a ROM (Read Only Memory) 90B and a RAM (Random Access Memory) 90C, an HDD (Hard Disk Drive), an I / F 90D for communication via a network, and a bus 90E for connecting each part.

[0177] The programs executed by the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment are provided pre-loaded into a ROM 90B or the like.

[0178] The programs executed by the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment may be configured to be provided as a computer program product by recording them in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), flexible disk (FD), CD-R (Compact Disk Recordable), or DVD (Digital Versatile Disk).

[0179] Furthermore, the programs executed by the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. Alternatively, the programs executed by the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment may be provided or distributed via a network such as the Internet.

[0180] The programs executed by the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment can cause the computer to function as the respective components of the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment. This computer can read programs from a storage medium readable by the computer onto the main memory and execute them using the CPU 90A.

[0181] Furthermore, the imaging device 10, information terminal 40, information processing device 20, information processing device 21, and information processing device 23 of the above embodiment may be implemented as virtual machines operating on a cloud system.

[0182] Although embodiments and modifications of the present invention have been described above, these embodiments and modifications are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0183] 10. Imaging device 20, 21, 23 Information Processing Equipment 27I Reception Department 28A 3D information acquisition section 28B Attitude information acquisition unit 28C Image acquisition unit 28D Decision section 28E Deleted section 28F Memory Control Unit 28G Display Control Unit 29J Setting section 40 Information terminals [Prior art documents] [Patent Documents]

[0184] [Patent Document 1] Japanese Patent Publication No. 2020-16767

Claims

1. A 3D information acquisition unit acquires 3D information obtained by imaging the imaging area by the imaging unit, A posture information acquisition unit that acquires posture information of the imaging unit when the three-dimensional information is captured, A reception unit that receives operational instructions from the user, A reference setting information that associates a reference deletion area included in the imaging area with the reference posture information of the imaging unit, and a determination unit that determines the deletion target area included in the three-dimensional information based on the posture information, A deletion unit that deletes the area to be deleted included in the three-dimensional information, Equipped with, The aforementioned reference posture information is information that represents the reference posture of the imaging unit, The aforementioned reference deletion region is information representing the region to be deleted included in the three-dimensional information when the imaging unit is captured in the posture represented by the reference posture information. The aforementioned criteria deletion area is received by the reception unit. Information processing device.

2. The aforementioned determination unit, The deletion target area is determined by correcting the reference deletion area according to the difference between the reference posture information and the posture information. The information processing apparatus according to claim 1.

3. The system includes a setting unit that sets the reference setting information based on the positional relationship between the photographer and the imaging unit of the three-dimensional information, The aforementioned determination unit, Based on the set reference setting information and the posture information, the region to be deleted included in the three-dimensional information is determined. The information processing apparatus according to claim 1 or claim 2.

4. The system includes an image acquisition unit that acquires an image of the shooting area from the aforementioned shooting unit, The aforementioned setting unit is, Based on the positional relationship identified using at least one of the three-dimensional information and the captured image, the reference setting information is set. The information processing apparatus according to claim 3.

5. A display control unit that displays an image containing information related to the three-dimensional information on the display unit. An information processing apparatus according to any one of claims 1 to 4, comprising

6. The display control unit, The image of the shooting region represented by the three-dimensional information, The information relating to the determined area to be deleted, The information relating to the three-dimensional information from which the area to be deleted has been removed, Displaying the image which includes at least one of the following: The information processing apparatus according to claim 5.

7. A storage control unit stores information related to the three-dimensional information from which the area to be deleted has been deleted in the storage unit. An information processing apparatus according to any one of claims 1 to 6, comprising:

8. An information processing device according to any one of claims 1 to 7, The aforementioned imaging unit, A posture detection unit that detects the posture of the aforementioned imaging unit, A three-dimensional measuring device equipped with the following features.

9. An information processing device according to any one of claims 1 to 7, A photography device comprising: a photography unit; a three-dimensional information output unit that outputs the three-dimensional information obtained by photography by the photography unit to the information processing device; a posture detection unit that detects the posture of the photography unit; and a posture information output unit that outputs the posture information representing the posture detected by the posture detection unit to the information processing device. An information processing system equipped with the following features.

Images