Image processing device and image processing program

Abstract

This image processing device comprises a window frame detection unit and a window region processing unit. The window frame detection unit detects a window frame from a three-dimensional point group in the real world by using a deep learning model. The window region processing unit processes a window region inside the window frame on the basis of the three-dimensional point group of the window frame.

Description

Image processing device and image processing program

[0001] The present invention relates to an image processing device and an image processing program.

[0002] Various efforts have been proposed to build a communication stage that eliminates locational and spatial constraints by collecting information from the real world using sensors and constructing a metaverse world.

[0003] "Launch of the 'TENGUN Ogijima Project' to Promote Regional Co-creation," online, accessed May 9, 2024, on the Internet (https: / / group.ntt / jp / newsrelease / 2022 / 11 / 15 / 221115b.html); "About Metaverse SHIZUOKA," online, accessed May 9, 2024, on the Internet (https: / / www.pref.shizuoka.jp / kensei / pr / 1059727.html);

[0004] The data (3D point cloud) collected from the real world by sensors includes windows of various buildings. Therefore, if the collected data is used as is, some of it may pose a privacy issue for residents.

[0005] Therefore, it becomes necessary to detect windows individually and then replace the original data with other data.

[0006] The present invention has been made in light of the above circumstances, and its object is to provide an image processing device and an image processing program for processing window areas contained in a three-dimensional point cloud in the real world.

[0007] One aspect of the present invention is an image processing device. The image processing device includes a window frame detection unit and a window region processing unit. The window frame detection unit detects a window frame from a 3D point cloud of the real world using a deep learning model. The window region processing unit processes a window region inside the window frame based on the 3D point cloud of the window frame.

[0008] One aspect of the present invention is an image processing program that causes a computer having a processor and a storage device to execute at least some of the functions of at least the window frame detection unit and the window region processing unit of the image processing device described above.

[0009] According to the present invention, an image processing device and an image processing program are provided for processing window regions included in a three-dimensional point cloud of the real world.

[0010] FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to an embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the image processing apparatus according to an embodiment. FIG. 3 is a flowchart illustrating an example of the operation of a window frame detection unit of the image processing apparatus according to an embodiment. FIG. 4 is a flowchart illustrating an example of the operation of a window region processing unit of the image processing apparatus according to an embodiment. FIG. 5 is a diagram illustrating an example of a 3D point cloud of an input window frame and a plane of the detected window frame. FIG. 6 is a diagram illustrating an example of a plane of the window frame and a 2D point cloud of the generated window frame (2D display and 3D display). FIG. 7 is a diagram illustrating how a binary image representing the window frame and the non-window frame is generated from the 2D point cloud of the window frame. FIG. 8 is a diagram illustrating how a mask for generating a 3D point cloud of a window region is generated from the binary image representing the window frame and the non-window frame. FIG. 9 is a diagram illustrating how a 3D point cloud of a window region inside the window frame is generated. FIG. 10 is a diagram illustrating a 3D point cloud of the window frame and the result of superimposing the 3D point cloud of the window region on the 3D point cloud of the window frame.

[0011] Hereinafter, embodiments will be described with reference to the drawings. Note that the scale, aspect ratio, and the like of each part in each drawing used in the following description of the embodiments may be changed as appropriate for the sake of convenience. Also, in each drawing used in the following description of the embodiments, some of the configuration may be omitted to make the description easier to understand. Furthermore, each element shown in the drawing conceptually illustrates the configuration, and the ratios of each part, etc. are not necessarily strictly accurate, and do not limit the invention.

[0012] (Functional Configuration) First, the functional configuration of an image processing device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of an image processing device 10 according to an embodiment. The image processing device 10 has a window frame detection unit 20 and a window region processing unit 30.

[0013] The window frame detection unit 20 receives a real-world 3D point cloud as input. The real-world 3D point cloud is a collection of points in a 3D space in the real world, and is collected by a sensor such as a LiDAR or RGB-D camera. Hereinafter, the 3D point cloud may be simply referred to as a point cloud. The window frame detection unit 20 uses a deep learning model to detect window frames from the real-world 3D point cloud.

[0014] The window region processing unit 30 processes the window region inside the window frame based on the 3D point cloud of the window frame detected by the window frame detection unit 20. The window region processing includes, for example, a process of hiding the original 3D point cloud of the window region and a process of improving the appearance of the window region. In other words, the window region processing is a process of replacing the original data (3D point cloud) with other data (3D point cloud). An example of such a process is texturing.

[0015] More specifically, the window frame detection unit 20 includes a learning unit 21 and an inference unit 22 .

[0016] The learning unit 21 receives learning data as input and generates a deep learning model for detecting window frames from a 3D point cloud in the real world through deep learning. The learning unit 21 provides the generated deep learning model to the inference unit 22.

[0017] The inference unit 22 receives a real-world 3D point cloud as input, and detects a window frame from the real-world 3D point cloud using the deep learning model generated by the learning unit 21. The inference unit 22 provides the detected 3D point cloud of the window frame to the window region processing unit 30.

[0018] More specifically, the window area processing unit 30 includes a plane detection unit 31, a two-dimensional point cloud generation unit 32, a binary image generation unit 33, a mask generation unit 34, a three-dimensional point cloud generation unit 35, and a superposition unit 36.

[0019] The plane detection unit 31 receives the 3D point cloud of the window frame as input from the window frame detection unit 20, and detects the plane of the window frame from the 3D point cloud of the window frame. The plane of the window frame is the plane on which the reference plane (e.g., the center plane) of the window frame is located.

[0020] The two-dimensional point cloud generation unit 32 receives as input the three-dimensional point cloud of the window frame from the window frame detection unit 20 and the plane of the window frame from the plane detection unit 31, and generates a two-dimensional point cloud of the window frame from the three-dimensional point cloud of the window frame and the plane of the window frame.

[0021] The binary image generating unit 33 receives the two-dimensional point cloud of the window frame as an input from the two-dimensional point cloud generating unit 32, and generates a binary image representing the window frame and the parts other than the window frame from the two-dimensional point cloud of the window frame.

[0022] The mask generation unit 34 receives the binary image from the binary image generation unit 33 as an input, extracts a window region from the binary image, and generates a mask. The mask is used to generate a three-dimensional point cloud of the window region in the three-dimensional point cloud generation unit 35.

[0023] The three-dimensional point cloud generator 35 receives the window frame plane from the plane detector 31 and the mask from the mask generator 34 as inputs, and generates a three-dimensional point cloud of the window region from the window frame plane and the mask.

[0024] The superimposing unit 36 ​​receives as input the 3D point cloud of the window frame from the window frame detection unit 20 and the 3D point cloud of the window region from the 3D point cloud generation unit 35, and superimposes the 3D point cloud of the window region on the 3D point cloud of the window frame. The superimposing unit 36 ​​outputs the superimposed 3D point clouds of the window frame and window region.

[0025] (Hardware Configuration) Next, a hardware configuration of the image processing device 10 will be described. Here, an example will be described in which the image processing device 10 is configured as a computer. For example, the image processing device 10 may be configured as a personal computer, a server computer, or the like.

[0026] 2 is a block diagram showing an example of the hardware configuration of the image processing device 10 according to the embodiment. The image processing device 10 is configured, for example, by a computer. The image processing device 10 has a control device 50, an input device 70, and an output device 80.

[0027] The control device 50 controls the entire image processing device 10. The control device 50 includes a processor 51, a read only memory (ROM) 52, a random access memory (RAM) 53, and an auxiliary storage device .

[0028] The processor 51, ROM 52, RAM 53, auxiliary storage device 54, input device 70, and output device 80 are electrically connected to one another via a bus 60, and can transmit and receive data.

[0029] The processor 51 is configured by a general-purpose hardware processor including, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), etc. The processor 51 executes programs deployed in the RAM 53 to perform various functions of the image processing device 10.

[0030] The ROM 52 is a non-volatile memory that constitutes part of the main storage device. The ROM 52 non-temporarily stores a startup program required to start up the image processing device 10. The processor 51 loads the startup program in the ROM 52 into the RAM 53 and executes it to start up the image processing device 10. The ROM 52 is, for example, configured from an EPROM (Erasable Programmable Read Only Memory) and can store various startup settings in addition to the startup program.

[0031] The RAM 53 is a volatile memory that constitutes part of the main storage device. The RAM 53 temporarily stores programs required for processing by the processor 51 and data required for executing the programs. In other words, the RAM 53 functions as a work area for the processor 51.

[0032] The auxiliary storage device 54 is configured with a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary storage device 54 can non-temporarily store various programs executed by the processor 51 and data required for executing the programs. The processor 51 executes various functions of the image processing device 10 by loading the programs in the auxiliary storage device 54 into the RAM 53 and executing them.

[0033] The input device 70 is a device for a user to input information and instructions, and accepts input of information and instructions. The input device 70 includes a keyboard, a pointing device, etc. The pointing device includes a mouse, a trackpad, a touch screen, etc.

[0034] The output device 80 is a device that outputs information to provide information to a user. The output device 80 is, for example, a display device that displays characters, images, etc. on a screen. For example, the output device 80 is a liquid crystal display, an organic EL display, a plasma display, etc.

[0035] The output device 80 and the input device 70 may be configured as an input / output device having the functions of both. Such an input / output device may be configured as, for example, a touch panel.

[0036] The input device 70 may also include a device that reads data from a computer-readable recording medium 90 that non-temporarily records data such as a program. For example, the recording medium 90 includes disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. The input device 70 includes drives, readers, etc. for these.

[0037] The program stored in the auxiliary storage device 54 is provided to the image processing device 10, for example, via the recording medium 90. Alternatively, the program may be stored in a server on a network and provided to the image processing device 10 by downloading it.

[0038] For example, when the image processing device 10 is started up, the processor 51 executes a startup program in the ROM 52 to start up the operating system (OS). Under the control of the OS, the processor 51 monitors input instructions, connections to external devices, etc. Also, under the control of the OS, the processor 51 sets up a program area and a data area in the RAM 53.

[0039] In response to an instruction to start a program, processor 51 reads the program from auxiliary storage device 54 into the program area of ​​RAM 53, and also reads data necessary for executing the program from auxiliary storage device 54 into the data area of ​​RAM 53. Processor 51 calculates the data in the data area in accordance with the program and writes the calculation results into the data area.

[0040] Through these operations, the processor 51, RAM 53, and auxiliary storage device 54 work together to execute at least some of the functions of the control device 50. Furthermore, the control device 50, input device 70, and output device 80 work together to execute at least some of the functions of the image processing device 10.

[0041] The input device 70 functions as a functional unit that receives training data and a 3D point cloud of the real world, while the output device 80 functions as a functional unit that outputs a 3D point cloud of the window frame and window area.

[0042] The programs non-temporarily stored in the auxiliary storage device 54 include an image processing program that causes the processor 51 to execute at least some of the functions of the control device 50. In other words, the processor 51 executes this image processing program to execute at least some of the functions of the control device 50.

[0043] As a result, the control device 50 cooperates with the input device 70 and the output device 80 to execute at least some of the functions of the window frame detection unit 20 and the window area processing unit 30 .

[0044] (Example of Operation of Window Frame Detection Unit 20) Next, an example of operation of the window frame detection unit 20 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of operation of the window frame detection unit 20.

[0045] In step S11, training data is created as a preliminary step. As an example, the training data is prepared as follows. For 20 point cloud files, since the detection target is a window frame, objects that are advantageous for detecting the window frame are segmented. Specifically, window frame, house, building, and school objects are detected and segmented using CloudCompare. To avoid the problem of class imbalance, in which other point clouds are significantly more numerous than the window frame point cloud, point clouds other than window frames, houses, buildings, and schools are deleted. Furthermore, houses, buildings, and schools are classified as "other," and a binary point cloud of window frames and "other" is created, which is used as training data.

[0046] In step S12, the learning unit 21 uses the learning data created in step S11 to construct a learning model for detecting window frames. For example, the deep learning architecture is based on KPConv (Kernel Point Convolution) described in Literature 1 and is adapted to large-scale point clouds as described in Literature 2. Here, Literature 1 is "Hugues et al., "KPConv: Flexible and Deformable Convolution for Point Clouds," Proceedings of the IEEE / CVF International Conference on Computer Vision 2019," and Literature 2 is "Kurata, et al., "Aggregative Input Convolution for Large-Scale Point Cloud Semantic Segmentation," IIEEJ Transactions on Image Electronics and Visual Computing 10 (1), 11-18, 2022-06-15."

[0047] In step S13, the inference unit 22 receives the 3D point cloud of the real world as an input, and detects a window frame from the 3D point cloud of the real world using the learning model constructed in step S12.

[0048] In step S14 , the inference unit 22 outputs the three-dimensional point cloud of the detected window frame to the window region processing unit 30 .

[0049] (Example of Operation of Window Region Processing Unit 30) Next, an example of operation of the window region processing unit 30 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of operation of the window region processing unit 30.

[0050] In step S21, the plane detection unit 31 detects a plane of the window frame from the 3D point cloud of the window frame detected by the window frame detection unit 20. The plane of the window frame is a plane on which the reference plane (e.g., the center plane) of the window frame is located. Fig. 5 shows an example of the input 3D point cloud of the window frame and the detected plane of the window frame.

[0051] In step S22, the 2D point cloud generator 32 generates a 2D point cloud of the window frame from the 3D point cloud of the window frame detected by the window frame detector 20 and the plane of the window frame detected in step S21. For example, the 2D point cloud of the window frame is generated by projecting the 3D point cloud of the window frame onto the plane of the window frame. Figure 6 shows an example of the plane of the window frame and the generated 2D point cloud of the window frame (2D display and 3D display).

[0052] In step S23, the binary image generation unit 33 generates a binary image representing the window frame and parts other than the window frame from the two-dimensional point cloud of the window frame generated in step S22. For example, the generation of the binary image is performed according to the following procedure. Also, Fig. 7 shows how the binary image is generated.

[0053] First, the two-dimensional point cloud of the window frame is roughly rasterized. In other words, the two-dimensional point cloud of the window frame is converted into a raster image (a grid-like arrangement of pixels). In this case, the window frame is represented by a "2" and everything other than the window frame is represented by a "0." In Figure 7, the "0" is represented by a point (dot).

[0054] Next, a local maximum filter is applied to the raster image, which emphasizes the window frame. As a result, the data in the window frame are interpolated, and the data area of ​​the window frame is expanded.

[0055] Next, a local minimum filter is applied to the maximum filtered image. This emphasizes the area outside the window frame. As a result, the data area of ​​the window frame is narrowed. This generates a binary image of the window frame.

[0056] In step S24, the mask generation unit 34 extracts the window region inside the window frame from the binary image generated in step S23 to generate a mask. For example, the mask is generated in the following procedure. Fig. 8 shows how the mask is generated. In Fig. 8, the binary image generated in step S23 is shown on the left side, and in this binary image, "0" is represented by a point (dot).

[0057] First, the binary image generated in step S23 is scanned from above, below, left and right, and deleted until the first "2" is found. Next, the "2" is deleted. Next, the remaining "0" is converted to "1". As a result, a binary image mask is generated in which the window region inside the window frame is represented by "1".

[0058] In step S25, the 3D point cloud generator 35 generates a 3D point cloud of the window region inside the window frame from the plane of the window frame detected in step S21 and the mask generated in step S24. For example, the 3D point cloud of the window region is generated in the following procedure. Also, Fig. 9 shows how the 3D point cloud of the window region is generated.

[0059] First, in step S21, a large number of points with three-dimensional coordinates are generated randomly on the plane of the detected window frame, that is, a three-dimensional point group is generated on the plane of the window frame.

[0060] Next, the mask generated in step S24 is applied to the 3D point cloud on the plane of the generated window frame. That is, among the 3D point cloud on the plane of the window frame, the 3D point cloud that is not covered by the mask is deleted. As a result, only the 3D point cloud located in the window region inside the window frame remains.

[0061] In step S26, the superimposing unit 36 ​​superimposes the 3D point cloud of the window region inside the window frame generated in step S25 onto the 3D point cloud of the window frame detected by the window frame detection unit 20. Figure 10 shows the 3D point cloud of the window frame and the result of superimposing the 3D point cloud of the window region on the 3D point cloud of the window frame. The superimposing unit 36 ​​outputs the 3D point clouds of the superimposed window frame and window region.

[0062] (Effect) According to the embodiment, it is possible to provide an image processing device and an image processing program that process window areas inside window frames that are difficult to detect in images or point clouds in the real world. The processing of the window area includes, for example, processing to hide the original 3D point cloud of the window area and processing to improve the appearance of the window area. This makes it possible to use 3D point clouds collected by sensors in the real world in building the Metaverse world without causing privacy issues.

[0063] (Other) In the embodiment, an example has been described in which the image processing device 10 is configured as a computer having a processor and a storage device, the storage device stores an image processing program, and the processor executes the image processing program to perform the functions of the image processing device 10. However, the image processing program may cause the processor to perform part of the functions of the image processing device 10, that is, the image processing program may cause the processor to perform the functions of the image processing device 10 in combination with a program already stored in the computer. Furthermore, the image processing program may cause the processor to perform the functions of the image processing device 10 in combination with hardware such as a programmable logic device (PLD), a field programmable gate array (FPGA), or a graphic processing unit (GPU).

[0064] The embodiments of the present invention have been described above with reference to the drawings. However, the above embodiment is merely an example of a configuration that embodies the present invention. That is, it is clear that the present invention is not limited to the above embodiment. Therefore, addition, omission, substitution, and other modifications of components may be made within the scope of the technical concept of the present invention.

[0065] In short, the present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0066] DESCRIPTION OF SYMBOLS 10... Image processing device 20... Window frame detection unit 21... Learning unit 22... Inference unit 30... Window region processing unit 31... Plane detection unit 32... 2D point cloud generation unit 33... Binary image generation unit 34... Mask generation unit 35... 3D point cloud generation unit 36... Superposition unit 50... Control device 51... Processor 52... ROM 53... RAM 54... Auxiliary storage device 60... Bus 70... Input device 80... Output device 90... Recording medium

Claims

1. An image processing device comprising: a window frame detection unit that detects window frames from a real-world 3D point cloud using a deep learning model; and a window region processing unit that processes the window region inside the window frame based on the 3D point cloud of the window frame.

2. The image processing device of claim 1, wherein the window area processing unit comprises: a plane detection unit that detects the plane of the window frame from the three-dimensional point cloud of the window frame; a two-dimensional point cloud generation unit that generates a two-dimensional point cloud of the window frame from the three-dimensional point cloud of the window frame and the plane of the window frame; a binary image generation unit that generates a binary image representing the window frame and other components from the two-dimensional point cloud of the window frame; a mask generation unit that extracts the window area from the binary image and generates a mask; and a three-dimensional point cloud generation unit that generates a three-dimensional point cloud of the window area from the plane of the window frame and the mask.

3. The image processing device according to claim 2, wherein the window region processing unit further comprises a superimposing unit that superimposes the three-dimensional point cloud of the window region on the three-dimensional point cloud of the window frame.

4. An image processing program that causes a computer having a processor and a storage device to execute at least a part of the functions of at least the window frame detection unit and the window area processing unit of the image processing device according to any one of claims 1 to 3.

Images