Data processing device, data processing method, and program

Abstract

A data processing device (102) comprises a processor and a memory. The processor uses the memory to generate point cloud data using a plurality of first images (S204), generates a second image using the point cloud data (S205), receives designation of a first point by a user with respect to the second image (S206, S312), selects a third image corresponding to the first point from the plurality of first images (S206, S313), receives designation of a second point by the user with respect to the third image (S206, S315), and identifies a first corresponding point included in the point cloud data corresponding to the second point (S208).

Description

Data processing device, data processing method and program

[0001] The present disclosure relates to a data processing device, a data processing method, and a program.

[0002] There is a known technique for measuring the distance between any two points using three-dimensional data. In this case, a user specifies any point on a two-dimensional image based on the three-dimensional data. For example, Patent Literature 1 discloses a technique for accepting input of a position on an image.

[0003] International Publication No. 2018 / 030386

[0004] When selecting an arbitrary point from a two-dimensional image based on such three-dimensional data, it is desirable for the user to be able to easily select the desired point.

[0005] Therefore, an object of the present disclosure is to provide a data processing device, a data processing method, or a program that allows a user to easily select a desired point from a two-dimensional image based on three-dimensional data.

[0006] A data processing device according to one aspect of the present disclosure includes a processor and a memory, wherein the processor uses the memory to generate point cloud data using a plurality of first images, generate a second image using the point cloud data, accept a user's designation of a first point for the second image, select a third image from the plurality of first images that corresponds to the first point, accept a user's designation of a second point for the third image, and identify a first corresponding point included in the point cloud data that corresponds to the second point.

[0007] The present disclosure can provide a data processing device, a data processing method, or a program that allows a user to easily select a desired point from a two-dimensional image based on three-dimensional data.

[0008] FIG. 1 is a block diagram of a data processing system according to an embodiment. FIG. 2 is a sequence diagram showing the flow of dimension measurement processing according to an embodiment. FIG. 3 is a sequence diagram showing measurement point selection processing according to an embodiment. FIG. 4 is a diagram showing an example of a rendered image according to an embodiment. FIG. 5 is a diagram showing an example of selection of measurement point A according to an embodiment. FIG. 6 is a diagram showing an example of selection of measurement point B according to an embodiment. FIG. 7 is a flowchart of processing in a data processing device according to an embodiment. FIG. 8 is a diagram showing example information of each 3D point of a 3D model according to an embodiment. FIG. 9 is a diagram showing example information of each pixel included in a rendered image according to an embodiment. FIG. 10 is a flowchart of generation processing of a rendered image according to an embodiment. FIG. 11 is a diagram showing an example of a viewpoint and multiple points of a rendered image according to an embodiment. FIG. 12 is a diagram showing an example of a viewpoint and multiple points of a captured image according to an embodiment. FIG. 13 is a diagram showing an example of a viewpoint and multiple points of a rendered image according to an embodiment. FIG. 14 is a diagram showing an example of a viewpoint and multiple points of a rendered image according to an embodiment. FIG. 15 is a flowchart of measurement point acquisition processing according to an embodiment.

[0009] A data processing device according to one aspect of the present disclosure includes a processor and a memory, wherein the processor uses the memory to generate point cloud data using a plurality of first images, generate a second image using the point cloud data, accept a user's designation of a first point for the second image, select a third image from the plurality of first images that corresponds to the first point, accept a user's designation of a second point for the third image, and identify a first corresponding point included in the point cloud data that corresponds to the second point.

[0010] This allows a user to select points using a third image that has higher accuracy than a second image based on point cloud data. Therefore, a user can easily select desired points from a two-dimensional image based on three-dimensional data. Furthermore, improving the accuracy of the selected second points improves the accuracy of the first corresponding points identified from the point cloud data. Therefore, for example, the accuracy of distance measurement using the first corresponding points can be improved.

[0011] For example, in generating the second image, the average and standard deviation of the positions of a plurality of third points included in the point cloud data, which are located on a light ray corresponding to a pixel included in the second image, may be calculated, and information on the pixel may be generated using a fifth point included in a plurality of fourth points obtained by excluding points located at positions that are at least the standard deviation away from the average from the plurality of third points. This allows noise and the like to be removed, thereby improving the accuracy of the second image based on the point cloud data.

[0012] For example, in generating the second image, among a plurality of third points included in the point cloud data that are located on a light ray corresponding to a pixel included in the second image, the third point on the first image used to generate the third point may be preferentially used to generate information about the pixel, the third point being closest to the center of the first image, thereby improving the accuracy of the second image based on the point cloud data.

[0013] For example, in generating the second image, information on the pixel may be generated by preferentially using a third point that is closest to the viewpoint of the second image among a plurality of third points included in the point cloud data and that are located on a light ray corresponding to the pixel included in the second image, thereby improving the accuracy of the second image based on the point cloud data.

[0014] For example, the plurality of first images may be a plurality of images taken by an RGBD camera, which allows the data processing device to easily generate point cloud data from the first images.

[0015] For example, the third image may be a first image among the plurality of first images, the first image having the closest distance between the center of the first image and the first point. This allows the data processing device to select an appropriate third image from the plurality of first images. This allows the user to easily select the second point from the third image.

[0016] For example, the third image may be an image from the plurality of first images that was used to generate the first point. This allows the data processing device to select an appropriate third image from the plurality of first images, thereby allowing the user to easily select the second point from the third image.

[0017] For example, in identifying the first corresponding point, a point included in the point cloud data generated using information on a pixel corresponding to the second point in the third image may be identified as the first corresponding point. This allows the data processing device to easily and accurately identify the first corresponding point from multiple points included in the point cloud data.

[0018] For example, the processor may further accept a sixth point specified by the user for the second image, select a fourth image from the plurality of first images corresponding to the sixth point, accept a seventh point specified by the user for the fourth image, identify a second corresponding point included in the point cloud data corresponding to the seventh point, and measure the distance between the first corresponding point and the second corresponding point. This allows the data processing device to improve the accuracy of the first corresponding point and the second corresponding point, thereby improving the accuracy of distance measurement.

[0019] For example, the processor may further determine a viewpoint of each of the first images using the first images, thereby enabling the data processing device to determine the positional relationship between the first images and generate point cloud data over a wider range using the results of the first images.

[0020] A data processing method according to one aspect of the present disclosure generates point cloud data using a plurality of first images, generates a second image using the point cloud data, accepts a user's designation of a first point for the second image, selects a third image from the plurality of first images that corresponds to the first point, accepts the user's designation of a second point for the third image, and identifies a first corresponding point included in the point cloud data that corresponds to the second point.

[0021] This allows a user to select points using a third image that has higher accuracy than a second image based on point cloud data. Therefore, a user can easily select desired points from a two-dimensional image based on three-dimensional data. Furthermore, improving the accuracy of the selected second points improves the accuracy of the first corresponding points identified from the point cloud data. Therefore, for example, the accuracy of distance measurement using the first corresponding points can be improved.

[0022] A program according to one aspect of the present disclosure causes a computer to execute the data processing method.

[0023] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0024] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0025] (Embodiment) [Configuration of Data Processing System] The configuration of a data processing system according to this embodiment will be described. FIG. 1 is a block diagram showing an example configuration of a data processing system 100 according to this embodiment. This data processing system 100 generates three-dimensional data from a plurality of captured images, for example. The data processing system 100 displays a two-dimensional image based on the three-dimensional data and accepts designation of two points by a user on the displayed two-dimensional image. The data processing system 100 measures the distance between the two designated points using the three-dimensional data and displays the measured distance to the user.

[0026] Here, the measurement target may be any facility. For example, the data processing system 100 may be used to measure the layout of an area where a system kitchen or the like is to be installed before construction, to measure the arrangement of items on shelves in a warehouse, or to measure electrical equipment.

[0027] As shown in FIG. 1, the data processing system 100 includes an imaging device 101 , a data processing device 102 , and a terminal device 103 .

[0028] The image capturing device 101 is, for example, an RGBD camera, and generates a captured image. The captured image includes, for example, an RGB image that indicates color information of a captured space for each pixel, and a distance image that indicates distance information of the captured space for each pixel.

[0029] The data processing device 102 is, for example, a server, and is capable of communicating with the imaging device 101 and the terminal device 103. The data processing device 102 includes a communication unit 111, a storage unit 112, and an image processing unit 113.

[0030] The communication unit 111 is a communication circuit that performs data communication with the imaging device 101 and the terminal device 103. Note that the communication method between the devices is not particularly limited. For example, any method such as wired communication or wireless communication may be used. Furthermore, these communications may be performed directly between the devices, or may be performed indirectly via another communication device, a server, or the like.

[0031] The storage unit 112 stores various data. For example, the storage unit 112 stores a plurality of captured images acquired from the imaging device 101 and camera parameters of the imaging device 101. The camera parameters include, for example, the focal length of the imaging device 101, the image center, and a distortion coefficient. The image center is, for example, the two-dimensional coordinates of the center of the captured image captured by the imaging device 101.

[0032] The storage unit 112 may store various calculation results such as a three-dimensional model (point cloud data) generated by the image processing unit 113. The storage unit 112 may also store, for example, a control program for causing a processing circuit (for example, a processor) to execute processing by each processing unit included in the data processing device 102. The storage unit 112 is realized, for example, by a hard disk drive (HDD) or a flash memory.

[0033] The image processing unit 113 includes a three-dimensional model generation unit 131, a rendering image generation unit 132, a measurement point selection unit 133, and a dimension measurement unit 134. For example, the functions of the image processing unit 113 are realized by a processing circuit (e.g., a processor) included in the data processing device 102 executing a program stored in the storage unit 112. Note that some or all of the functions of the image processing unit 113 may be realized by a dedicated circuit.

[0034] The three-dimensional model generation unit 131 generates a three-dimensional model from multiple captured images. The rendering image generation unit 132 generates a rendering image, which is a two-dimensional image, from the three-dimensional model. The measurement point selection unit 133 selects measurement points based on a user operation. The dimension measurement unit 134 measures the dimension (distance) between two measurement points specified by the user.

[0035] The terminal device 103 is a tablet terminal or a smartphone carried by a user, etc. The terminal device 103 includes a communication unit 121, a display unit 122, and an input unit 123.

[0036] The communication unit 121 is a communication circuit that performs data communication with the data processing device 102. The display unit 122 is a display that displays rendering images and the like generated by the data processing device 102. The input unit 123 is a user interface that acquires user operations, and is, for example, a touch panel, or a keyboard and mouse.

[0037] Although an example in which the data processing system 100 is configured with three devices is shown here, the system may be configured with two or a single device. For example, a tablet terminal or smartphone carried by a user may include all of the functions of the processing units shown in FIG. 1 . In this case, the communication units 111 and 121 are not necessarily required. Furthermore, when the data processing system 100 is configured with multiple devices, some of the functions of one device shown in FIG. 1 may be included in another device. For example, some of the functions of the data processing device 102 may be included in the imaging device 101 or the terminal device 103.

[0038] [Outline of Dimension Measurement Processing] Next, an outline of the dimension measurement processing by the data processing system 100 will be described. FIG. 2 is a sequence diagram showing the flow of the dimension measurement processing by the data processing system 100. First, the imaging device 101 generates multiple captured images by capturing images of a subject from multiple viewpoints (S101). For example, a user uses the imaging device 101 to capture multiple captured images from different viewpoints by capturing images of a subject while changing the viewpoint (camera position and attitude (capture direction)). Note that the multiple captured images may be multiple still images or multiple frames included in a moving image. Furthermore, the multiple captured images may be captured by multiple imaging devices. The multiple imaging devices may be fixed cameras.

[0039] The communication unit 111 of the data processing device 102 acquires a plurality of captured images and stores the acquired captured images in the storage unit 112. As described above, each captured image includes an RGB image and a distance image.

[0040] Next, the three-dimensional model generating unit 131 generates a three-dimensional model (e.g., point cloud data) of the subject using the acquired multiple captured images (S102). Next, the rendering image generating unit 132 generates a rendering image, which is a two-dimensional image of the three-dimensional model viewed from a predetermined viewpoint (S103).

[0041] Next, a process for selecting measurement point 1 (S104) and a process for selecting measurement point 2 (S105) are performed. Fig. 3 is a sequence diagram of the process for selecting measurement points (S104 or S105). As shown in Fig. 3, first, a rendering image generated by the data processing device 102 is sent to the terminal device 103. The terminal device 103 displays the received rendering image (S111).

[0042] 4 is a diagram showing an example of a rendering image 200. Here, the object 201 to be measured is an arbitrary piece of equipment. The rendering image 200 is a two-dimensional image of a three-dimensional model of the object 201 viewed from a predetermined direction (in this example, a direction facing the equipment).

[0043] Next, the user selects measurement point A on the displayed rendering image (S112). Fig. 5 is a diagram showing an example of selecting measurement point A. As shown in Fig. 5, measurement point A is selected by the user selecting (e.g., tapping) a desired position on rendering image 200.

[0044] The terminal device 103 sends information indicating the selected measurement point A to the data processing device 102. The measurement point selection unit 133 of the data processing device 102 selects a corresponding image corresponding to the measurement point A from the multiple captured images stored in the storage unit 112 (S113). For example, the corresponding image is a captured image that includes the measurement point A.

[0045] The data processing device 102 sends the corresponding image to the terminal device 103. The terminal device 103 displays the corresponding image (S114). The user selects measurement point B on the displayed corresponding image (S115). FIG. 6 is a diagram showing an example of selecting measurement point B. As shown in FIG. 6, measurement point B is selected by the user selecting (e.g., tapping) a desired position on the corresponding image 210. In other words, the user selects the same position as measurement point A selected on the rendering image 200 as measurement point B on the corresponding image 210 again. The terminal device 103 sends information indicating the selected measurement point B to the data processing device 102. Measurement point B is measurement point 1 in step S104 and measurement point 2 in step S105.

[0046] Note that the corresponding image 210 is an image selected from a plurality of captured images, and is not necessarily an image from the same viewpoint as the rendering image 200. In other words, in this example, the corresponding image 210 is not necessarily an image viewed from a direction directly facing the equipment.

[0047] Here, a rendering image is an image generated from a three-dimensional model (point cloud data) and has lower accuracy than a captured image. Therefore, it is difficult for a user to accurately select desired points from a rendering image. In this embodiment, when selecting measurement points, a captured image with high accuracy is displayed as a corresponding image, and the user can select measurement points from the corresponding image, thereby enabling the user to easily select desired points.

[0048] The processes shown in FIG. 3 are performed in steps S104 and S105, whereby measurement points 1 and 2 are selected.

[0049] 2, next, the dimension measurement unit 134 of the data processing device 102 measures the dimension (distance) between measurement point 1 and measurement point 2 (S106), and sends information indicating the obtained dimension to the terminal device 103. The terminal device 103 displays the received dimension (S107).

[0050] [Processing in Data Processing Device 102] Next, the above-mentioned dimension measurement processing will be described in detail as an explanation of the processing in the data processing device 102. FIG.

[0051] First, the data processing device 102 acquires a plurality of captured images from the imaging device 101 (S201). Next, the three-dimensional model generation unit 131 estimates the position and orientation (hereinafter also referred to as the camera position and orientation) of the imaging device 101 when each of the plurality of captured images was captured (S202).

[0052] For example, the three-dimensional model generation unit 131 may estimate the camera position and orientation using Visual-SLAM (Simultaneous Localization and Mapping), Structure-From-Motion, ICP (Iterative Closest Point), or the like.

[0053] Specifically, the 3D model generation unit 131 performs feature point matching processing on multiple captured images. That is, the 3D model generation unit 131 extracts features from multiple frames and extracts sets of similar points that are similar between the multiple frames from the extracted feature points. Next, the 3D model generation unit 131 estimates the camera position and orientation using the extracted sets of similar points. The estimated camera position and orientation for each captured image are stored in the storage unit 112.

[0054] Next, the three-dimensional model generation unit 131 performs plane estimation using the marker 202, and converts the coordinate system of the camera position and orientation of each captured image into the coordinate system of the equipment (S203). As shown in FIG. 4, the subject 201 includes the marker 202. The marker 202 is a plate that includes a predetermined pattern and is arranged parallel to the subject 201 (equipment), and the plane of the marker 202 (i.e., the plane of the equipment) can be estimated by image analysis of the pattern. Furthermore, the coordinate system of the equipment is a coordinate system based on the plane of the marker 202, and is, for example, a coordinate system in which the plane of the marker 202 coincides with the xz plane.

[0055] Next, the 3D model generation unit 131 generates a 3D model (point cloud data) using the multiple range images included in the multiple captured images and the camera position and orientation (S204). Specifically, the 3D model generation unit 131 places 3D points at positions corresponding to the distances of each pixel indicated in the range images, using the camera position and orientation as a reference. This process is performed for each of the multiple captured images to generate a 3D model.

[0056] The method of generating the 3D model is not limited to the above. For example, the 3D model generation device may generate a 3D model using multiple RGB images and the camera positions and orientations of each RGB image, without using a distance image. For example, the 3D model generation unit 131 may generate a 3D model using a technique such as MVS (multi-view stereo) that uses feature point matching. In this case, the image capture device 101 does not need to generate a distance image. In other words, the image capture device 101 may be, for example, an RGB camera.

[0057] Furthermore, when generating a three-dimensional model, information on each three-dimensional point of the three-dimensional model is generated and stored in the storage unit 112. FIG. 8 is a diagram showing an example of information on each three-dimensional point of the three-dimensional model. As shown in FIG. 8, the information on each three-dimensional point includes a three-dimensional position 301, color information 302, image information 303, and a distance 304 from the image center. The three-dimensional position 301 indicates the three-dimensional position of the three-dimensional point, for example, three-dimensional coordinates (e.g., x, y, z coordinates). The color information 302 indicates color information of the three-dimensional point (e.g., R value, G value, B value). The image information 303 includes information indicating the captured image used to generate the three-dimensional point and information indicating the pixel in the captured image used to generate the three-dimensional point (e.g., x, y coordinates). The distance 304 from the image center indicates the distance between the center of the captured image indicated by the image information and the pixel indicated by the image information. Note that information indicating the position of the center of the image is acquired in advance from the image capturing device 101 as a camera parameter, for example, and stored in the storage unit 112 .

[0058] In this way, in addition to the point cloud data (3D position 301 and color information 302), image information 303 indicating the image and pixels used to generate the 3D point, and the distance 304 of the 3D point from the image center are managed for each 3D point. In other words, the 3D position 301 and image information 303 shown in Fig. 8 indicate the correspondence between the 3D point and the pixels in the captured image used to generate the 3D point.

[0059] Next, the rendering image generating unit 132 generates a rendering image from the three-dimensional model (S205). Details of this process will be described later.

[0060] Furthermore, information about each pixel included in the rendering image is generated and stored in the storage unit 112. FIG. 9 is a diagram showing an example of information about each pixel included in the rendering image. As shown in FIG. 9, the information about each pixel in the rendering image includes a pixel position 311, color information 312, point information 313, and image information 314. The pixel position 311 indicates the position of the pixel (e.g., x, y coordinates). The color information 312 indicates color information about the pixel (e.g., R value, G value, B value). The point information 313 is information indicating a three-dimensional point used to generate the pixel among multiple three-dimensional points included in the three-dimensional model, and indicates, for example, the three-dimensional coordinates (e.g., x, y, z coordinates) of the three-dimensional point used to generate the pixel. The image information 314 indicates the captured image used to generate the three-dimensional point indicated by the point information 313.

[0061] In this way, in addition to the image data of the rendering image (pixel position 311 and color information 312), for each pixel of the rendering image, point information 313 indicating the three-dimensional point used to generate the pixel and image information 314 indicating the captured image used to generate the three-dimensional point are managed.

[0062] Next, the measurement point selection unit 133 acquires information indicating measurement point 1 and information indicating measurement point 2 designated by the user (S206, S207). Details of this process will be described later.

[0063] Next, the dimension measurement unit 134 calculates the distance (dimension) between measurement point 1 and measurement point 2 (S208). Specifically, the dimension measurement unit 134 uses the three-dimensional position 301 and image information 303 shown in FIG. 8 to determine corresponding point 1, which is a three-dimensional point corresponding to the position (pixel position) of measurement point 1 in the corresponding image, and acquires the three-dimensional coordinates of corresponding point 1. Similarly, the dimension measurement unit 134 determines corresponding point 2, which is a three-dimensional point corresponding to the position (pixel position) of measurement point 2 in the corresponding image, and acquires the three-dimensional coordinates of corresponding point 2. Next, the dimension measurement unit 134 calculates the distance between corresponding point 1 and corresponding point 2 (the difference in three-dimensional coordinates), thereby calculating the distance between measurement point 1 and measurement point 2.

[0064] [Rendering Image Generation Process] Next, the rendering image generation process (S205) shown in FIG. 7 will be described in detail. FIG. 10 is a flowchart of the rendering image generation process (S205) shown in FIG. 7. First, the rendering image generation unit 132 determines a viewpoint of the rendering image to be generated (S301). For example, based on the marker 202 included in the three-dimensional model, the rendering image generation unit 132 determines a viewpoint from which the entire object is included in the rendering image while facing the marker 202 directly. Note that the method of determining the viewpoint of the rendering image is not limited to this. For example, a viewer for the three-dimensional model may be displayed on the terminal device 103, and the viewpoint of the rendering image may be determined based on a user operation.

[0065] Next, the rendering image generating unit 132 converts the coordinate system of the three-dimensional model into a coordinate system based on the determined viewpoint (S302). Next, the rendering image generating unit 132 performs the processes of steps S303 to S306 on each pixel included in the rendering image.

[0066] First, the rendering image generating unit 132 determines, from among multiple points (three-dimensional points) included in the three-dimensional model, multiple first points that exist on a ray corresponding to a target pixel that is a pixel to be processed (S303). A ray is a line that connects the camera (image capturing device 101) and a pixel on the image plane.

[0067] Next, the rendering image generation unit 132 calculates the average and standard deviation of the positions of the plurality of first points in the ray direction (S304). Next, the rendering image generation unit 132 determines a plurality of third points from the plurality of first points, excluding second points that are at least one standard deviation away from the average (S305).

[0068] FIG. 11 is a diagram for explaining the processing of steps S303 to S305, and shows an example of a viewpoint and multiple points in a rendering image. For example, in the example shown in FIG. 11, point P0, multiple points P1, and multiple points P2 exist on a ray corresponding to a target pixel. For example, point P0 is a point caused by noise, the multiple points P1 correspond to the foreground, and the multiple points P2 correspond to the background. In this case, the average of the positions of point P0, the multiple points P1, and the multiple points P2 in the optical axis direction is calculated. Furthermore, of point P0, the multiple points P1, and the multiple points P2, point P0 and the multiple points P2 that are further away from the average by more than the standard deviation are excluded, and only the multiple points P1 remain as the third point.

[0069] In this way, by using the mean and standard deviation to exclude points far from the mean position, noise and the like can be eliminated. Also, it is possible to extract only a number of points corresponding to the foreground that are suitable for generating a rendered image. This improves the accuracy of the rendered image.

[0070] Next, the rendering image generation unit 132 determines the fourth point, among the multiple third points, that is closest to the center of the captured image, and generates information about the target pixel using the fourth point (S306). Here, the point closest to the center of the captured image is the point that is the shortest distance between the position of the pixel corresponding to the third point in the captured image used to generate the third point and the center of the captured image. This distance is also associated with each three-dimensional point as the distance 304 from the image center shown in FIG. 8. Therefore, the rendering image generation unit 132 can use this information to determine the fourth point that is closest to the center of the captured image.

[0071] 12 and 13 are diagrams for explaining the processing of step S306. FIG. 12 is a diagram showing an example of a viewpoint and a plurality of points in a captured image. FIG. 13 is a diagram showing an example of a viewpoint and a plurality of points in a rendered image. In the example shown in FIG. 12, in generating a three-dimensional model, points A0, A1, and A2 are generated from captured image A, and points B0, B1, and B2 are generated from captured image B. Also, as shown in FIG. 13, in the rendered image, points B0 and A1 exist on ray 1 corresponding to a certain pixel, and points A2 and B2 exist on ray 2 corresponding to another pixel.

[0072] In the processing for light ray 1, the distance DA1 between point A1 and the image center CA of captured image A is shorter than the distance DB0 between point B0 and the image center CB of captured image B. Therefore, as shown in Figure 13, of points A1 and B0 on ​​light ray 1, point A1 is determined to be the fourth point closest to the image center of the captured image.

[0073] In the processing for light ray 2, the distance DA2 between point A2 and the image center CA of captured image A is shorter than the distance DB2 between point B2 and the image center CB of captured image B. Therefore, as shown in Figure 13, of points A2 and B2 on light ray 2, point A2 is determined to be the fourth point closest to the image center of the captured image.

[0074] Here, points closer to the image center are more likely to be accurate than points farther from the image center, so generating a rendering image using points closer to the image center can improve the accuracy of the rendering image.

[0075] Although the example described here is one in which the fourth point is determined based on the distance from the image center, the rendering image generation unit 132 may determine the fourth point based on the distance between the viewpoint of the rendering image and multiple third points. Fig. 14 is a diagram for explaining this process, showing an example of the viewpoint and multiple points of the rendering image. As shown in Fig. 14, for example, the rendering image generation unit 132 determines that point P3, which is closest to the viewpoint of the rendering image among the multiple third points on the light ray, is the fourth point.

[0076] Note that the above-described method based on the distance from the image center and the method based on the distance from the viewpoint of the rendered image may be combined. For example, the rendering image generation unit 132 calculates, for each third point, a first evaluation value based on the distance from the image center and a second evaluation value based on the distance from the viewpoint of the rendered image. For example, the shorter the distance from the image center, the higher the first evaluation value is set, and the shorter the distance from the viewpoint of the rendered image, the higher the second evaluation value is set. The rendering image generation unit 132 may calculate a third evaluation value by adding the first evaluation value and the second evaluation value, and determine the third point with the highest third evaluation value among the multiple third points as the fourth point.

[0077] The rendering image generation unit 132 may also calculate a fourth evaluation value based on the distance from the average in the process of excluding the second points described above (steps S303 to S305). For example, the shorter the distance from the average, the higher the fourth evaluation value is set. The rendering image generation unit 132 may calculate a third evaluation value by adding the first evaluation value, the second evaluation value, and the fourth evaluation value, and determine the third point with the highest third evaluation value among the multiple third points as the fourth point.

[0078] Furthermore, among the above-mentioned process of excluding the second point (steps S303 to S305), the process based on the distance to the image center, and the process based on the distance to the viewpoint of the rendering image, only one, any two, or all of these may be performed. Furthermore, criteria other than these may also be added.

[0079] A rendering image is generated by performing the processes of steps S303 to S306 for each pixel. Also, the information shown in FIG.

[0080] [Corresponding Image Selection Process] Next, the measurement point acquisition process (S206 or S207) shown in Fig. 7 will be described in detail. Fig. 15 is a flowchart of the measurement point acquisition process (S206 or S207) shown in Fig. 7. First, the measurement point selection unit 133 displays the rendering image on the terminal device 103 (S311). Next, the measurement point selection unit 133 acquires (receives) information indicating measurement point A designated based on a user operation on the rendering image from the terminal device 103 (Yes in S312).

[0081] Next, the measurement point selection unit 133 determines a corresponding image corresponding to measurement point A from the multiple captured images. For example, the measurement point selection unit 133 determines an image including measurement point A from the multiple captured images as a corresponding image. Specifically, the measurement point selection unit 133 selects the captured image used to generate measurement point A in the rendering image as the corresponding image. For example, the measurement point selection unit 133 uses pixel position 311 and image information 314 shown in FIG. 9 to determine the captured image used to generate the three-dimensional point used to generate the pixel of measurement point A.

[0082] As described in the rendering image generation process described above, the generation of each pixel of the rendering image involves at least one of the following processes: excluding the second point (steps S303 to S305), processing based on the distance from the image center, and processing based on the distance from the viewpoint of the rendering image. This allows the measurement point selection unit 133 to select an appropriate corresponding image. Specifically, the processing of excluding the second point (steps S303 to S305) prevents images containing noise, etc. from being displayed as the corresponding image. Furthermore, the processing based on the distance from the image center allows a captured image in which measurement point A is close to the image center to be displayed as the corresponding image. This allows the user to easily select corresponding point B that corresponds to measurement point A in the corresponding image.

[0083] Next, the measurement point selection unit 133 displays the corresponding image on the terminal device 103 (S314). Next, the measurement point selection unit 133 acquires (receives) information indicating measurement point B designated based on an operation on the corresponding image by the user from the terminal device 103 (Yes in S315). Measurement point B is measurement point 1 in step S206 and measurement point 2 in step S207.

[0084] Although the example described here is one in which the three-dimensional points used to generate the rendering image are selected based on the corresponding image selection process in generating the rendering image, the corresponding image may be selected independently of the determination of the points to be used to generate the rendering image. For example, even if the processing based on the distance from the image center is not performed in generating the rendering image, the measurement point selection unit 133 may perform the same processing as the processing based on the distance from the image center described above in selecting the corresponding image, and select the captured image in which the corresponding point A is closest to (e.g., closest to) the image center as the corresponding image.

[0085] [Summary] As described above, the data processing device 102 according to one aspect of the present disclosure includes a processor (e.g., the image processing unit 113) and a memory (e.g., the storage unit 112). The processor uses the memory to generate point cloud data using a plurality of first images (e.g., a plurality of captured images) (e.g., S204 in FIG. 7), generate a second image (e.g., a rendered image) using the point cloud data (e.g., S205 in FIG. 7), accept a user's designation of a first point (e.g., measurement point A) on the second image (e.g., S206 in FIG. 7 and S312 in FIG. 15), select a third image (e.g., a corresponding image) corresponding to the first point from the plurality of first images (e.g., S206 in FIG. 7 and S313 in FIG. 15), accept a user's designation of a second point (e.g., measurement point B) on the third image (e.g., S206 in FIG. 7 and S315 in FIG. 15), and identify a first corresponding point included in the point cloud data corresponding to the second point (e.g., S208 in FIG. 7).

[0086] This allows a user to select points using a third image that has higher accuracy than a second image based on point cloud data. Therefore, a user can easily select desired points from a two-dimensional image based on three-dimensional data. Furthermore, improving the accuracy of the selected second points improves the accuracy of the first corresponding points identified from the point cloud data. Therefore, for example, the accuracy of distance measurement using the first corresponding points can be improved.

[0087] For example, in generating the second image (e.g., S205 in FIG. 7), the data processing device 102 calculates the average and standard deviation of the positions of multiple third points included in the point cloud data that are located on rays corresponding to pixels included in the second image (e.g., S303 and S304 in FIG. 10), and generates pixel information using a fifth point included in multiple fourth points that are obtained by excluding points located at positions that are more than the standard deviation away from the average of the multiple third points (e.g., S305 in FIG. 10). This makes it possible to eliminate noise, etc., thereby improving the accuracy of the second image based on the point cloud data.

[0088] For example, when generating the second image (e.g., S205 in FIG. 7), the data processing device 102 generates pixel information by preferentially using, among multiple third points included in the point cloud data that are located on rays corresponding to pixels included in the second image, third points on the first image used to generate the third points whose positions are closest to the center of the first image (e.g., S306 in FIG. 10). This can improve the accuracy of the second image based on the point cloud data.

[0089] For example, in generating the second image (e.g., S205 in FIG. 7), the data processing device 102 generates pixel information by preferentially using third points that are located on rays corresponding to pixels included in the second image and are included in the point cloud data and that are closest to the viewpoint of the second image (e.g., FIG. 14). This can improve the accuracy of the second image based on the point cloud data.

[0090] For example, the plurality of first images are a plurality of images captured by an RGBD camera, which allows the data processing device 102 to easily generate point cloud data from the first images.

[0091] For example, the third image is the first image among the plurality of first images whose center is closest to the first point. This allows the data processing device 102 to select an appropriate third image from the plurality of first images. This allows the user to easily select the second point from the third image.

[0092] For example, the third image is an image used to generate the first point from among the plurality of first images. This allows the data processing device 102 to select an appropriate third image from the plurality of first images. This allows the user to easily select the second point from the third image.

[0093] For example, in identifying the first corresponding point (e.g., S208 in FIG. 7 ), the data processing device 102 identifies a point included in the point cloud data that is generated using information on a pixel corresponding to the second point in the third image as the first corresponding point. This allows the data processing device to easily and accurately identify the first corresponding point from multiple points included in the point cloud data.

[0094] For example, the processor further accepts a sixth point specified by the user for the second image (e.g., S207 in FIG. 7 and S312 in FIG. 15), selects a fourth image corresponding to the sixth point from the plurality of first images (e.g., S207 in FIG. 7 and S313 in FIG. 15), accepts a seventh point specified by the user for the fourth image (e.g., S207 in FIG. 7 and S315 in FIG. 15), identifies a second corresponding point included in the point cloud data corresponding to the seventh point, and measures the distance between the first corresponding point and the second corresponding point (e.g., S208 in FIG. 7). In this manner, the data processing device 102 can improve the accuracy of the first corresponding point and the second corresponding point, thereby improving the accuracy of distance measurement.

[0095] For example, the processor further determines the viewpoint of each of the first images using the first images (e.g., S202 in FIG. 7 ). This allows the data processing device 102 to determine the positional relationship between the first images, and therefore allows point cloud data over a wider range to be generated using the results of the first images.

[0096] Although the data processing system, data processing device, etc. according to the embodiment and modified examples of the present disclosure have been described above, the present disclosure is not limited to these embodiments.

[0097] Furthermore, each processing unit included in the data processing system and data processing device according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0098] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0099] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0100] The present disclosure may also be realized as a data processing method or a dimension measurement method executed by a data processing system, a data processing device, or the like.

[0101] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0102] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0103] While the data processing system and data processing device according to one or more aspects have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0104] The present disclosure is applicable to data processing systems and data processing devices.

[0105] REFERENCE SIGNS LIST 100 Data processing system 101 Imaging device 102 Data processing device 103 Terminal device 111 Communication unit 112 Storage unit 113 Image processing unit 121 Communication unit 122 Display unit 123 Input unit 131 Three-dimensional model generation unit 132 Rendered image generation unit 133 Measurement point selection unit 134 Dimension measurement unit 200 Rendered image 201 Subject 202 Marker 210 Corresponding image 301 Three-dimensional position 302 Color information 303 Image information 304 Distance from image center 311 Pixel position 312 Color information 313 Point information 314 Image information

Claims

1. A data processing device comprising: a processor; and a memory, wherein the processor uses the memory to generate point cloud data using a plurality of first images, generate a second image using the point cloud data, accept a first point specified by a user for the second image, select a third image from the plurality of first images that corresponds to the first point, accept a second point specified by the user for the third image, and identify a first corresponding point included in the point cloud data that corresponds to the second point.

2. A data processing device as described in claim 1, wherein in generating the second image, the average and standard deviation of the positions of a plurality of third points included in the point cloud data that are located on a light ray corresponding to a pixel included in the second image are calculated, and information on the pixel is generated using a fifth point included in a plurality of fourth points obtained by excluding points that are located from the plurality of third points at a distance of more than the standard deviation from the average.

3. A data processing device as described in claim 1, wherein, in generating the second image, information on the pixel is generated by preferentially using, among a plurality of third points contained in the point cloud data that are located on a light ray corresponding to a pixel contained in the second image, a third point whose position on the first image used to generate the third point is closest to the center of the first image.

4. A data processing device as described in claim 1, wherein, in generating the second image, information on the pixel is generated by preferentially using a third point that is closest to the viewpoint of the second image out of multiple third points included in the point cloud data that are located on a light ray corresponding to the pixel included in the second image.

5. The data processing device according to claim 1, wherein the plurality of first images are a plurality of images taken by an RGBD camera.

6. A data processing device according to claim 1, wherein the third image is the first image among the plurality of first images, the first image having the closest distance between the center of the first image and the first point.

7. A data processing device according to claim 1, wherein the third image is an image from among the plurality of first images that was used to generate the first point.

8. A data processing device according to claim 1, wherein in identifying the first corresponding point, a point included in the point cloud data generated using information on a pixel corresponding to the second point in the third image is identified as the first corresponding point.

9. A data processing device as described in claim 1, wherein the processor further: accepts the user's designation of a sixth point for the second image; selects a fourth image from the plurality of first images that corresponds to the sixth point; accepts the user's designation of a seventh point for the fourth image; identifies a second corresponding point included in the point cloud data that corresponds to the seventh point; and measures the distance between the first corresponding point and the second corresponding point.

10. The data processing apparatus of claim 1, wherein said processor is further configured to use said plurality of first images to determine a viewpoint for each of said plurality of first images.

11. A data processing method comprising: generating point cloud data using a plurality of first images; generating a second image using the point cloud data; accepting a user's designation of a first point for the second image; selecting a third image from the plurality of first images that corresponds to the first point; accepting a user's designation of a second point for the third image; and identifying a first corresponding point included in the point cloud data that corresponds to the second point.

12. A program for causing a computer to execute the data processing method according to claim 11.

Images