Image processing apparatus, image processing method, and program

Abstract

To reduce memory usage in a case where learning of spatial information is performed.SOLUTION: An image processing apparatus 100 in accordance with the present disclosure acquires data of a plurality of captured images obtained by capturing an object from a plurality of directions and image capturing camera parameters, which are camera parameters corresponding to each of the plurality of captured images, generates a plurality of object images by extracting an image region corresponding to an image of the object from each of the plurality of captured images, generates object image camera parameters, which are camera parameters corresponding to each of the plurality of object images, based on the image capturing camera parameters and information indicating a position of an image region corresponding to the image of the object in the plurality of captured images, and acquires spatial information representing a space in which the object exists based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for estimating spatial information in a target area imaged by an imaging device. [Background technology]

[0002] There is a technique for estimating spatial information in a target area captured by an imaging device based on a plurality of captured image data obtained by capturing images from a plurality of viewpoints and the camera parameters of each imaging device used for the capture. According to this technique, it is possible to render an image (hereinafter referred to as a "virtual viewpoint image") corresponding to the appearance from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint"). Patent Document 1 discloses a NeRF (Neural Radiance Field) method as an algorithm for rendering a virtual viewpoint image. Specifically, the NeRF method disclosed in Patent Document 1 compares color information of each pixel in a virtual viewpoint image obtained from viewpoint information indicating the position of the captured viewpoint and the direction of the line of sight, etc., and estimated spatial information with color information of each pixel in a captured image. Next, the NeRF method generates spatial information corresponding to the captured image by feeding back the error between these color information to spatial information and performing learning. According to the NeRF method disclosed in Patent Document 1, it is possible to generate a highly realistic virtual viewpoint image using the generated spatial information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Pat. No. 1,130,8659 Summary of the Invention [Problem to be solved by the invention]

[0004] When learning spatial information, a sampling process and an error calculation process are performed on color information for each pixel in each of a plurality of captured images obtained by capturing images from multiple viewpoints. Therefore, in order to process the learning at high speed, a computer that learns the spatial information is required to have high parallelism in the above-mentioned sampling process and error calculation process, and generally, a GPU (Graphics Processing Unit) is used for these processes.

[0005] However, when the number of captured images, i.e., the number of viewpoints, is large and the resolution of each captured image is high, the amount of VRAM used becomes very large when multiple captured images are simultaneously deployed on a graphics memory (hereinafter referred to as "VRAM"). This causes a problem that the amount of VRAM available for other calculations is limited, or it may be difficult to deploy some of the multiple captured images on the VRAM. For example, when 4K (3840 x 2160 pixels) captured images each having a single-precision float type pixel value for each RGB channel are deployed on the VRAM in the number corresponding to 100 viewpoints, the VRAM requires a huge capacity of about 10 gigabytes.

[0006] The present disclosure has been made to solve the above-mentioned problems, and has an object to provide an image processing technique for reducing memory usage when learning spatial information. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the image processing device according to the present disclosure has a data acquisition means for acquiring data of a plurality of captured images obtained by capturing images of an object from a plurality of directions and imaging camera parameters which are camera parameters corresponding to each of the plurality of captured images, an object image generation means for generating a plurality of object images by extracting an image area corresponding to an image of the object from each of the plurality of captured images, a camera parameter generation means for generating object image camera parameters which are camera parameters corresponding to each of the plurality of object images based on the imaging camera parameters and information indicating the position of the image area corresponding to the image of the object in the plurality of captured images, and a space acquisition means for acquiring spatial information representing the space in which the object exists based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images. Effect of the Invention

[0008] According to the present disclosure, it is possible to reduce the amount of memory used when learning spatial information. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a logical configuration of an image processing device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 3] 5 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 4] 3A to 3C are diagrams illustrating examples of various data aspects in the spatial information acquisition process of the image processing device according to the first embodiment. [Diagram 5] 4A to 4C are diagrams for explaining an example of an aspect of a generation process of an object image camera parameter in a camera parameter generation unit according to the first embodiment. [Figure 6]4 is a schematic diagram showing an example of a change in memory usage for each processing step in the image processing apparatus according to the first embodiment. FIG. [Figure 7] FIG. 11 is a block diagram illustrating an example of a logical configuration of an image processing device according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of a processing flow of an image processing device according to a second embodiment. [Figure 9] 13A to 13C are diagrams illustrating examples of various data formats from captured image data acquisition processing to object image camera parameter generation processing in an image processing device according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a GUI screen displayed by the image processing device 100 according to the second embodiment. [Figure 11] 13 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to another embodiment. [Figure 12] 13A and 13B are diagrams illustrating an example of various data formats from the process of acquiring captured image data to the process of generating object image camera parameters in an image processing device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the configurations of the means required to solve the problems are not limited to the configurations shown in the embodiments.

[0011] [First embodiment] An image processing device 100 according to the first embodiment will be described with reference to Figs. 1 to 6. First, the configuration of the image processing device 100 according to the first embodiment will be described with reference to Figs. 1 and 2. Fig. 1 is a block diagram showing an example of a logical configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has a data acquisition unit 101, an object image generation unit 102, a camera parameter generation unit 103, and a space acquisition unit 104. Note that in the first embodiment, as an example, a form will be described in which the image processing device 100 separates a captured image into a foreground and a background, extracts the foreground as an object image, and acquires space information indicating a space including an object based on the extracted object image.

[0012] The data acquisition unit 101 acquires data of a plurality of captured images obtained by capturing images of at least one object from a plurality of directions, and camera parameters of the imaging device corresponding to each captured image. The camera parameters are a file in which the imaging conditions of each imaging device are described, or information indicating the imaging conditions. Hereinafter, the camera parameters will be described as including information on the position, optical axis direction (hereinafter referred to as "attitude"), focal length, and principal point of the imaging device, as well as the size of the captured image, and, if necessary, including information on coefficients related to lens distortion or shear in addition to the above information.

[0013] The object image generating unit 102 generates an object image. Specifically, the object image generating unit 102 first separates the captured image into a foreground and a background, and extracts a rectangular area in the captured image that includes pixels of the area separated as the foreground. Next, the object image generating unit 102 generates an image in the captured image that corresponds to the extracted rectangular area as an object image, and outputs data of the generated object image. The camera parameter generating unit 103 changes a part of information included in the camera parameters acquired by the data acquiring unit 101, and generates camera parameters (hereinafter referred to as "object image camera parameters") corresponding to the object image.

[0014] The space acquisition unit 104 acquires space information indicating a space including an object by using data of the object image and the object image camera parameters. The space information is information including at least one of density information indicating the density of the object for each position in the space, signed distance information indicating the distance from the object surface, color information, and color information depending on the direction.

[0015] The processing of each unit that the image processing device 100 has as a logical configuration is performed by hardware such as a CPU (Central Processor Unit) built into the image processing device 100. The processing of each unit provided in the image processing device 100 may be performed by software using a CPU or a GPU (Graphics Processor Unit) and a memory built into the image processing device 100.

[0016] With reference to Fig. 2, a hardware configuration of the image processing device 100 in the case where each unit that the image processing device 100 has as a logical configuration operates as software will be described. Fig. 2 is a block diagram showing an example of a hardware configuration in the image processing device 100 according to the first embodiment. The image processing device 100 is configured by a computer, and the computer has a CPU 201, a GPU 202, a ROM 203, a RAM 204, a VRAM 205, and an auxiliary storage device 206, as shown as an example in Fig. 2. Furthermore, the image processing device 100 has a display unit 207, an operation unit 208, a communication unit 209, and a bus 210 in addition to the above-mentioned configuration.

[0017] The CPU 201 controls the computer using a program or a program and data stored in the ROM 203 or the RAM 204, thereby making the computer function as each unit of the image processing device 100 shown in FIG. 1, which has a logical configuration. The spatial acquisition unit 104 functions not only by the CPU 201, but also by the CPU 201 and the GPU 202 performing processing in cooperation with each other. The GPU 202 controls the computer using a program or a program and data stored in the VRAM 205, thereby making the computer function as a main calculation unit of the spatial acquisition unit 104, which the image processing device 100 shown in FIG. 1 has as a logical configuration. The image processing device 100 may have one or more dedicated hardware different from the CPU 201 and the GPU 202, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0018] The ROM 203 is a storage device that stores information such as programs that do not require modification. The RAM 204 is a storage device that temporarily stores programs or data supplied from the auxiliary storage device 206, or data supplied from the outside via the communication unit 209, and operates as a work area for the CPU 201. The VRAM 205 is a storage device that temporarily stores information such as data supplied from the ROM 203, the RAM 204, or the auxiliary storage device 206, and operates as a work area for the GPU 202, and the information is used for processing by the GPU 202. The auxiliary storage device 206 is composed of a hard disk drive or the like, and stores various data such as image data or audio data.

[0019] The display unit 207 is configured with a liquid crystal display, an LED (light-emitting diode), or the like, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 100, or a GUI for browsing the processing contents, etc. of the image processing device 100. The operation unit 208 is configured with a keyboard, a mouse, a touch panel, or the like, and receives operations by the user and inputs various instructions corresponding to the operations to the CPU 201. The CPU 201 also operates as a display control unit that controls the display unit 207, and an operation control unit that controls the operation unit 208.

[0020] The communication unit 209 is used for communication between the image processing device 100 and an external device of the image processing device 100. For example, when the image processing device 100 is connected to an external device by wire, a communication cable is connected to the communication unit 209. When the image processing device 100 has a function of wireless communication with an external device, the communication unit 209 includes an antenna. The bus 210 connects each unit that the image processing device 100 has as a hardware configuration so that they can communicate with each other, thereby transmitting information. Hereinafter, the display unit 207 and the operation unit 208 will be described as existing inside the image processing device 100, but at least one of the display unit 207 and the operation unit 208 may be existing as a separate device outside the image processing device 100.

[0021] The operation of the image processing device 100 will be described with reference to Figs. 3 and 4. Fig. 3 is a flowchart showing an example of a processing flow of the image processing device 100 according to the first embodiment. Specifically, Fig. 3(a) shows an example of a series of processing flows in the image processing device 100. Fig. 3(b) will be described later. In the following description, the symbol "S" means a step (process). Fig. 4 is a diagram showing an example of various data modes in the spatial information acquisition process of the image processing device 100 according to the first embodiment. Specifically, Fig. 4(a) is a diagram for explaining the process of S301 shown in Fig. 3, Fig. 4(b) is a diagram for explaining the processes of S302 to S304 shown in Fig. 3, and Fig. 4(c) is a diagram for explaining the process of S305 shown in Fig. 3. First, in S301, the data acquisition unit 101 acquires data of multiple captured images 401 obtained by capturing images of at least one object from multiple directions and camera parameters 402 of the imaging device corresponding to each captured image 401.

[0022] Next, in S302, the object image generating unit 102 performs a separation process for separating the foreground region 403 and the background region for each captured image 401 acquired in S301. Here, for example, the foreground region 403 is an image region (hereinafter referred to as an "object region") corresponding to an image of an object in the captured image 401. The object image generating unit 102 separates the foreground region 403 and the background region in the captured image 401 by extracting the foreground region from the difference between the captured image 401 and a captured image (hereinafter referred to as a "background image") obtained by capturing an image in a state in which no object exists. The method of separating the foreground region 403 and the background region is not limited to the above-mentioned method. For example, the object image generating unit 102 may separate the foreground region 403 and the background region in the captured image 401 by extracting an image region other than a specific color in the captured image 401 as the foreground region, as in the case of capturing an image using a green screen or the like.

[0023] Next, in S303, the object image generating unit 102 performs the following process on each captured image 401 acquired in S301. Specifically, in S303, the object image generating unit 102 cuts out a rectangular area including the foreground area 403 from the captured image 401, generates the cut-out image as an object image 404, and outputs data of the generated object image 404. That is, the object image 404 is an image obtained by cutting out an image area determined as an area (object area) corresponding to an image of an object in the captured image 401 into a rectangle. The object image generating unit 102 may cut out a plurality of rectangular areas from one captured image 401 obtained by imaging from one viewpoint, and generate a plurality of object images 404 for one captured image 401. In addition, when the object area is specified in pixel units in S302, the object image generating unit 102 may be configured to exclude processing of the background area in the generated object image 404 in the acquisition process of spatial information. For example, in order to exclude processing of the background area in the object image 404, the object image generating unit 102 is configured as follows. Specifically, the object image generating unit 102 is configured to add an alpha channel that expresses transparency to the data of the object image 404, and change the transparency of pixels in the background region of the object image 404, i.e., the value of the added alpha channel.

[0024] Next, in S304, the camera parameter generating unit 103 performs the following process on each object image 404 generated in S303. Specifically, in S304, the camera parameter generating unit 103 generates camera parameters (object image camera parameters) 405 corresponding to the object image 404 from the camera parameters 402. Here, in order to generate the object image camera parameters 405, it is necessary to change the image size and correct the information of the principal point from the camera parameters 402. The camera parameter generating unit 103 changes the image size by replacing the image size of the camera parameters 402 with the image size of the object image 404. The principal point refers to the optical center point on the captured image, and is a parameter expressed using image coordinates. Since the origin of the image coordinate system in the object image 404 is different from the origin of the captured image 401, it is necessary to align the principal point expressed by the coordinates of the image coordinate system to the same position as the principal point of the captured image 401 by correcting the difference in the position of the origin of each image coordinate system.

[0025] 5 is a diagram for explaining an example of an aspect of the object image camera parameter generation process in the camera parameter generation unit 103 according to the first embodiment. An offset 502 indicates a coordinate difference in the image coordinate system between an origin 501 of a captured image 500 (hereinafter referred to as a "captured image origin") and an origin 511 of an object image 510 (hereinafter referred to as an "object image origin") 511. In the following description, the value of the offset 502 in the x direction (horizontal direction) in the image coordinate system is represented as "offset_x_sub" and the value of the offset 502 in the y direction (vertical direction) is represented as "offset_y_sub".

[0026] Difference 504 indicates the difference in coordinates in the image coordinate system between principal point 503 of captured image 500 (hereinafter referred to as "captured image principal point") and difference 504 of principal point 513 of corrected object image 510 (hereinafter referred to as "object image principal point") 513. Here, difference 504 is similar to offset 502. Therefore, the value (cx_sub) of difference 504 in the x direction (horizontal direction) and the value (cy_sub) of difference 504 in the y direction (vertical direction) of object image principal point 513 in the image coordinate system can be calculated based on the following formula (1).

[0027] (cx_sub,cy_sub)=(cx_img-offset_x_sub,cy_img-offset_y_sub) … Formula (1) Here, cx_img is the x-direction (horizontal direction) coordinate of the captured image principal point 503 in the image coordinate system, and cy_img is the y-direction (vertical direction) coordinate of the captured image principal point 503. By such correction, the position of the object image principal point 513 relative to the object image origin 511 in the camera coordinate system coincides with the position of the captured image principal point 503 relative to the captured image origin 501. Note that when generating the object image camera parameters, the same values ​​as those of the camera parameters may be used for information indicating the position, orientation, and focal length of the imaging device, which are camera parameters that do not change even if a part of the captured image is cut out.

[0028] After S304, in S305, the space acquisition unit 104 acquires spatial information of a space 406 including an object based on the object image 404 generated in S303 and the object image camera parameters 405 generated in S304. Details of the process of S305 will be described later. Note that, as an example, color information is handled as the spatial information here. After the process of S305, the image processing device 100 ends the process of the flowchart shown in FIG. 3.

[0029] FIG. 3B is a flowchart showing an example of the flow of the spatial information acquisition process in the space acquisition unit 104 according to the first embodiment, and is a flowchart showing an example of the processing flow in the process of S305 shown in FIG. 3A. First, in S311, the space acquisition unit 104 selects some or all of the object images 404 generated in S303. Next, in S312, the space acquisition unit 104 acquires color information in the space 406 (hereinafter referred to as "spatial color information") using the current spatial information and the object image camera parameters 405 corresponding to the object image 404 selected in S311. Specifically, first, the space acquisition unit 104 calculates three-dimensional light rays corresponding to each pixel in the object image 404 using the object image camera parameters 405 corresponding to the object image 404 selected in S311. Next, the space acquisition unit 104 acquires color information (spatial color information) in the space 406 by sampling the space 406 on each light ray or in the vicinity of the light ray including the light ray, using the current spatial information and each calculated light ray.

[0030] After S312, in S313, the space acquisition unit 104 calculates the color difference between the spatial color information acquired in S312 and the color information of the object image 404 selected in S311, and updates (learns) the spatial information so that the calculated color difference becomes smaller. Here, the spatial information may be expressed by weight parameters in a neural network as in Patent Document 1, or may be expressed in a data form having values ​​in a point cloud, volume, or multiple Gaussian distributions. Next, in S314, the space acquisition unit 104 determines whether or not all the object images 404 generated in S303 have been selected in S311. If it is determined in S314 that at least some of the object images 404 have not been selected, the space acquisition unit 104 returns to the process of S311. In this case, the space acquisition unit 104 selects some or all of the remaining object images 404 that have not yet been selected in S311. Thereafter, the space acquisition unit 104 repeatedly executes the processes from S311 to S314 until it is determined in S314 that all of the object images 404 have been selected.

[0031] If it is determined in S314 that all object images 404 have been selected, the space acquisition unit 104 ends the process of the flowchart shown in FIG. 3B, i.e., the process of S305. The processes of S312 and S313 by the space acquisition unit 104 may be performed in parallel by the GPU 202 or the like. That is, if multiple object images are selected in S301, the space acquisition unit 104 performs the processes of S312 and S313 in parallel for the multiple selected object images. For example, if the space acquisition unit 104 selects all object images 404 generated in S303 in S311, it may perform the processes of S312 and S313 in parallel for all the object images 404.

[0032] The amount of memory used in each process from S301 to S305 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of a change in memory usage in the image processing device 100 according to the first embodiment. Specifically, Fig. 6 is a schematic diagram showing an example of a change in the type and amount of data expanded on the RAM 204 and the VRAM 205 in each process from S301 to S305. In the following description, it is assumed that the VRAM 205 is used only in the process of the space acquisition unit 104 among the units that the image processing device 100 has as a logical configuration, and that the RAM 204 is used in the process of the other logical configurations.

[0033] The CPU 201 and the GPU 202 each have their own specialty processes. The CPU 201 uses the RAM 204, which can be easily enlarged, as a work area, so the amount of data handled by the CPU 201 is unlikely to be an issue. In contrast, the GPU 202 uses the VRAM 205, which has a smaller capacity than the RAM, as a work area, so it cannot handle a large amount of data in one processing, and the amount of data becomes an issue. Therefore, in order for the GPU 202 to perform processing efficiently, it is necessary to reduce the amount of data handled by the GPU 202 before the step in which the GPU 202 performs processing. Below, each process from S301 to S305 will be explained separately as the process by the CPU 201 and the process by the GPU 202, taking the amount of data into consideration.

[0034] Specifically, each process from S301 to S304 is a step for handling large volume data of the captured image 401, and a step for generating an object image 404 and a step for generating object image camera parameters 405, which do not require parallel processing. Therefore, each process from S301 to S304 is performed using the CPU 201 and the RAM 204.

[0035] The data amount 601 indicates the amount of data immediately after the processing of S301 is executed, and indicates the usage state of the RAM 204 when the data of the captured image 401 and the camera parameters 402 acquired in S301 are stored in the RAM 204. The data amounts 602 and 603 indicate the amount of data after the processing of S302 is executed and before the processing of S303 is executed. The data amounts 602 and 603 indicate the usage state of the RAM 204 when information indicating the foreground separated in S302 (hereinafter referred to as "foreground information") is stored in the RAM 204 in addition to the data of the captured image 401 and the camera parameters 402.

[0036] The data amount 604 indicates the amount of data after the process of S303 is executed and before the process of S304 is executed. Specifically, the data amount 604 indicates the usage state of the RAM 204 when the camera parameters 402 acquired in S301, the foreground information acquired in S302, and the data of the object image 404 generated in S303 are stored in the RAM 204. When the process of S303 is executed, the data of the captured image 401 is deleted, and the data of the object image 404 is developed in the RAM 20 in place of the captured image 401. Here, since the object image 404 is a part of the captured image 401 cut out, the amount of image data is reduced by the process of S303, and memory saving is realized.

[0037] For example, the total number of pixels of all captured images 401 acquired in S301 is img The total number of pixels of all the object images 404 generated in S303 is sub , and the amount of data per pixel is B. In this case, the total amount of data for all captured images 401 is B×pixel img The total data size of all the object images 404 is B×pixels. sub Therefore, the data amount 604 after the processing of S303 is compared with the data amount 603 before the processing of S303, and is calculated as B×(pixel img -pixel sub) is reduced. When the number of viewpoints, an increase in image resolution, or support for multiple frames due to changes in algorithms is taken into consideration, the reduction in the amount of VRAM 205 usage is expected to be even greater as the amount of captured image data increases.

[0038] The data amount 605 indicates the amount of data after the process of S304 is executed and before the process of S305 is executed. The data amount 605 indicates the usage state of the RAM 204 when the foreground information acquired in S302, the data of the object image 404 generated in S303, and the object image camera parameters 405 generated in S304 are stored in the RAM 204. When the process of S304 is executed, the camera parameters 402 are deleted, and the object image camera parameters 405 are deployed in the RAM 20 in place of the camera parameters 402. Note that the data amounts of the camera parameters 402 and the object image camera parameters 405 are sufficiently smaller than the data amount of the image data. Therefore, the difference in memory usage when the camera parameters 402 are replaced with the object image camera parameters 405 can be ignored.

[0039] The process of acquiring spatial information in S305 requires high parallelism. Therefore, the data of the object image 404 generated in S303 and the object image camera parameters 405 are transferred from the RAM 204 to the VRAM 205, and the process of S305 is performed using the GPU 202 and the VRAM 205. The data amount 606 indicates the data amount before the process of S305 is executed. Specifically, the data amount 606 indicates the usage state of the VRAM 205 when the data of the object image 404 transferred from the RAM 204 to the VRAM 205 and the object image camera parameters 405 are stored in the VRAM 205. In this way, the input data expanded on the VRAM 205 are only the object image 404 and the object image camera parameters 405. Therefore, the usage amount of the VRAM 205 is reduced compared to the case where the data of all the captured images 401 are expanded on the VRAM 205.

[0040] As described above, the image processing device 100 is configured to generate an object image and object image camera parameters corresponding to the object image from a plurality of captured images obtained by capturing images from a plurality of directions and camera parameters corresponding to each captured image. The image processing device 100 is also configured to update (learn) spatial information using data of each generated object image and object image camera parameters corresponding to each object image. According to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less usage of the VRAM 205 compared to the case where data of all captured images obtained is expanded in the VRAM 205. According to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less calculation amount of the GPU 202 compared to the case where data of captured images is used to update (learn) spatial information.

[0041] [Second embodiment] The image processing device 100 according to the first embodiment extracts object regions from a captured image to generate object images, so that the image sizes of the object images differ from one another. When the GPU 202 performs image processing in parallel, the processing time for an object image with the largest image size is the limiting factor, which may cause a decrease in the efficiency of the parallel processing. In the second embodiment, a form will be described in which the efficiency of parallel processing in the GPU 202 is improved by dividing all object images into image regions (hereinafter referred to as "unit regions") of a fixed image size.

[0042] An image processing device 100 according to the second embodiment will be described with reference to Figs. 7 to 10. Fig. 7 is a block diagram showing an example of a logical configuration of the image processing device 100 according to the second embodiment (hereinafter simply referred to as "image processing device 100"). The image processing device 100 has a data acquisition unit 101, a size setting unit 701, an object image generation unit 102, a camera parameter generation unit 103, and a space acquisition unit 104. The processing of the data acquisition unit 101 according to the second embodiment (hereinafter simply referred to as "data acquisition unit 101") is similar to the processing of the data acquisition unit 101 according to the first embodiment, so the description of the data acquisition unit 101 will be omitted. The size setting unit 701 sets the size of a unit area. The object image generation unit 102 according to the second embodiment will be described below.

[0043] The object image generating unit 102 according to the first embodiment generates an object image by cutting out a rectangular area including the foreground extracted by separating it from the background from the captured image. Therefore, the image sizes of the object images generated by the object image generating unit 102 according to the first embodiment are different for each object image.

[0044] In contrast, an object image generating unit 102 according to the second embodiment (hereinafter simply referred to as "object image generating unit 102") generates an object image of a fixed image size. Specifically, the object image generating unit 102 divides the extracted foreground region into unit regions of a size set by a size setting unit 701, and generates a plurality of object images by cutting out each of a plurality of image regions obtained by the division from a captured image. Since the object image generating unit 102 generates a plurality of object images each having the same image size as the unit region, the image size of the object images generated by the object image generating unit 102 is made common.

[0045] The processing of the camera parameter generation unit 103 according to the second embodiment (hereinafter simply referred to as the "camera parameter generation unit 103") is similar to the processing of the camera parameter generation unit 103 according to the first embodiment. Moreover, the processing of the space acquisition unit 104 according to the second embodiment (hereinafter simply referred to as the "space acquisition unit 104") is similar to the processing of the space acquisition unit 104 according to the first embodiment. Therefore, the description of the camera parameter generation unit 103 and the space acquisition unit 104 will be omitted.

[0046] FIG. 8 is a flowchart showing an example of a processing flow of the image processing device 100 according to the second embodiment. With reference to FIG. 8, the difference between the image processing device 100 and the image processing device 100 according to the first embodiment will be described. FIG. 9 is a diagram showing an example of the form of various data from the process of acquiring captured image data to the process of generating object image camera parameters in the image processing device 100 according to the second embodiment. First, in S301, the data acquisition unit 101 acquires data of a plurality of captured images 401 and camera parameters 402 corresponding to each captured image 401. FIG. 9(a) shows an example of data acquired by the process of S301, and is a diagram similar to FIG. 4(a).

[0047] Next, in S801, the size setting unit 701 sets the size of the unit area 901. FIG. 9B shows an example of the unit area 901 set by the process of S801. For example, the size setting unit 701 specifies the width of the unit area, that is, x_unit, which is the length of the unit area in the x direction (horizontal direction), and the height of the unit area, that is, y_unit, which is the length of the unit area in the y direction (vertical direction). Here, it is desirable that the size of the unit area 901 is set so that an integer multiple of the total number of pixels of one unit area is the batch size of the spatial information update (learning) process. By setting the size of the unit area 901 to such a size, the target to be processed by the spatial acquisition unit 104 in learning by one batch process can be specified in image units, not in pixel units. This makes it easier for the spatial acquisition unit 104 to process in batch units. Here, the batch size in the spatial information update (learning) process refers to the number of pixels that can be collectively subjected to error calculation in one learning. The ratio of x_unit to y_unit is not limited to 1, but may be set to any value.

[0048] After S801, the image processing device 100 executes the processes of S302 and S802. FIG. 9C shows an example of data acquired by the processes of S302 and S802. Specifically, after S801, in S302, the object image generating unit 102 performs a separation process for separating the foreground region 403 and the background region for each captured image 401 acquired in S301, as in the first embodiment. Next, in S802, the object image generating unit 102 generates an object image. More specifically, in S802, the object image generating unit 102 first performs a division process for dividing each foreground region 902 extracted in S302 into a plurality of unit regions 901 so that they do not overlap with each other. Note that "dividing so as not to overlap" as used here includes not only the meaning of dividing so as not to overlap strictly, but also the meaning of dividing so as to substantially not overlap, which allows some overlap. Subsequently, in S802, the object image generating unit 102 generates an object image 903 by cutting out each unit area 901 obtained by dividing the foreground area 902 from the captured image 401.

[0049] When dividing the foreground region 902 into a plurality of unit regions 901 without overlapping each other, there are cases where the foreground region 902 cannot be divided into an integer number of unit regions 901, and a part of the unit region 901 extends beyond the foreground region 902. In such a case, for example, the object image generating unit 102 may generate the object image 903 by cutting out the unit region 901 including the pixels 906 outside the foreground region 902 in the captured image 401 from the captured image 401. Furthermore, when the foreground region 403 of the captured image 401 is specified in pixel units, the number of pixels of the object image 903 can be reduced by adding a process of removing pixels corresponding to the pixels specified as the foreground region 403 from the object image 903. In other words, the addition of this process can reduce the usage amount of the VRAM 205.

[0050] After S802, in S304, the camera parameter generating unit 103 generates object image camera parameters 905 corresponding to each object image 903 generated in S802. The method of generating the object image camera parameters 905 is similar to the method of generating the object image camera parameters 405, and therefore the description will be omitted. After S304, in S305, the space acquiring unit 104 acquires space information of the space 406 including the object based on the object image 903 generated in S802 and the object image camera parameters 905 generated in S304. The process of S305 according to the second embodiment is similar to the process of S305 according to the first embodiment, and therefore the description will be omitted. After the process of S305, the image processing device 100 ends the process of the flowchart shown in FIG. 3.

[0051] Here, in the second embodiment, all object images 903 have a common image size. Therefore, when spatial information acquisition processing is performed in parallel for a plurality of object images 903, there is no significant difference in processing time for each object image 903. Therefore, the image processing device 100 according to the second embodiment can achieve more efficient parallel processing than the image processing device 100 according to the first embodiment.

[0052] As described above, the image processing device 100 is configured to generate an object image and object image camera parameters corresponding to the object image from a plurality of captured images obtained by capturing images from a plurality of directions and camera parameters corresponding to each captured image. Here, the image processing device 100 is configured so that the generated object image has an image size corresponding to a predetermined unit area or an image size equal to or smaller than the image size. Furthermore, the image processing device 100 is configured to update (learn) spatial information using data of each generated object image and object image camera parameters corresponding to each object image. According to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less usage of the VRAM 205 compared to the case where data of all captured images obtained is expanded in the VRAM 205. Moreover, according to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less calculation amount of the GPU 202 compared to the case where data of captured images is used to update (learn) spatial information. Furthermore, according to the image processing device 100 configured in this manner, in the spatial information acquisition process, it is possible to suppress a decrease in efficiency of parallel processing due to variations in image size for each object image, and to realize more efficient parallel processing for object images.

[0053] In the above description, the image processing device 100 has been described as acquiring spatial information as a result of learning, but the image processing device 100 may generate an image using the acquired spatial information, and display the generated image on the display unit 207 as a result of learning to present it to a user. Specifically, for example, first, the image processing device 100 determines a virtual viewpoint (hereinafter referred to as a "rendering viewpoint") for rendering an image based on the input camera parameters 402 or the object image camera parameters 405 or 905. Next, the image processing device 100 uses the acquired spatial information to generate an image corresponding to the appearance of the space 406 from the determined rendering viewpoint, and displays this on the display unit 207 as a result of learning.

[0054] In the above description, an object region in the captured image 401 is extracted as the foreground region 403 by foreground / background separation, but the method of extracting the foreground region 403 is not limited to the method by foreground / background separation. For example, the foreground region may be set by a user annotating an object region in the captured image 401 using a predetermined GUI. Hereinafter, a specific example of a user annotating an object region using a GUI will be described with reference to FIG. 10.

[0055] 10 is a diagram showing an example of GUI screens 1000, 1010 displayed by the image processing device 100 according to the second embodiment. A user sets an image area including a foreground area by adding an annotation to an object area in a captured image 401 using the GUI screen 1000. The GUI screen 1000 has a file name box 1002, an image display area 1003, an information box 1006, a decision button 1007, a done button 1008, a unit area division button 1009, and a slider bar 1001.

[0056] When a user inputs a file name in the file name box 1002, the image processing device 100 displays, in the image display area 1003, a captured image 401 corresponding to the file name among the multiple captured images 401 acquired by the data acquisition unit 101. When an object area 1005 is set in the image display area 1003 by the user operating a cursor 1004, the image processing device 100 displays area information indicating the set object area in an information box 1006. When the user presses a confirm button 1007 in a state in which the object area is set, the image processing device 100 generates an object image in the object image generation unit 102 based on the area information indicating the set object area. Furthermore, the image processing device 100 generates object image camera parameters in the camera parameter generation unit 103 based on the area information and the camera parameters 402 corresponding to the captured image 401 with the file name specified in the file name box 1002.

[0057] When the user presses the done button 1008, the image processing device 100 acquires spatial information based on all the generated object images and the object image camera parameters in the space acquisition unit 104. Specifically, the user presses the done button 1008 after finishing the generation of object images corresponding to all the captured images 401 acquired in the data acquisition unit 101. As a result, the space acquisition unit 104 of the image processing device 100 acquires spatial information based on the data of all the generated object images and the object image camera parameters corresponding to each object image.

[0058] The unit area division button 1009 is composed of a check box, and the user can switch the unit area division button 1009 between ON and OFF. When the unit area division button 1009 is set to ON, the image processing device 100 transitions the display from the GUI screen 1000 to the GUI screen 1010, and displays an image 1011 in which the set object area is divided by unit areas in the image display area 1003. The user can change the size of the unit area by performing an operation of sliding the slider of the slider bar 1001 using the cursor 1004, and the size of the unit area is set by the operation of the change. The user sets the object area by operating the cursor, various buttons, and the slider bar while referring to the captured image displayed in the image display area 1003. In addition, the image processing device 100 generates an object image and object image camera parameters based on the set object area, and acquires spatial information based on the generated object image and object image camera parameters.

[0059] With the above configuration, the user can set a desired image area in each captured image 401 as an object area.

[0060] [Other embodiments] The image processing device 100 according to the first and second embodiments extracts or sets an object region, i.e., a foreground region, by foreground background separation or by a user setting, and generates an object image based on the foreground region. The method of extracting or setting an object region in a captured image is not limited to the above, and the foreground region may be extracted or set by a method other than the above.

[0061] With reference to Figs. 11 and 12, another method for extracting a foreground region will be described. Fig. 11 is a flowchart showing an example of a processing flow of an image processing device 100 according to another embodiment. Fig. 12 is a diagram showing an example of the form of various data from the process of acquiring captured image data to the process of generating object image camera parameters in the image processing device 100 according to another embodiment. In another embodiment, as an example, a processing method in which the object image generating unit 102 in the image processing device 100 uses semantic region division to extract an object region in a captured image will be described. Note that semantic region division is a method of dividing a two-dimensional image data or the like into a plurality of image regions by assigning a semantic label to each region that is semantically different from each other.

[0062] First, in S1101, the data acquisition unit 101 acquires information on the semantic label 1201 corresponding to the object to be determined. In the following description, it is assumed that the object to be determined is set in advance, and the data acquisition unit 101 acquires the information by reading the information on the semantic label 1201 corresponding to the object to be determined from the auxiliary storage device 206. Note that the object to be determined may be set by a user using a GUI (not shown). After S1101, in S301, the data acquisition unit 101 acquires data on a plurality of captured images 401 and camera parameters 402 corresponding to each captured image 401. The process of S301 according to the other embodiments is similar to the process of S301 according to the first or second embodiment, and therefore detailed description thereof will be omitted.

[0063] After S301, in S1102, the object image generating unit 102 performs semantic region division on each captured image 401 acquired in S301, and assigns a semantic label 1202 to pixels in each captured image 401 corresponding to an image of an object to be determined. In the following description, the object to be determined is a natural person (hereinafter, simply referred to as a "person"). Next, in S1103, the object image generating unit 102 generates an object image 1203 by cutting out an image region including pixels to which the semantic label 1202 was assigned in S1102 as a foreground region in the captured image. At this time, as a result of cutting out the image region including the pixels to which the semantic label 1202 was assigned in a rectangular shape, a region 1204 of pixels to which a semantic label corresponding to an object other than the object to be determined is assigned may be included in the object image. In such a case, the object image generating unit 102 generates an object image so as not to include an area 1204 of pixels to which semantic labels corresponding to objects other than the object to be determined (hereinafter referred to as "semantic labels not to be determined").

[0064] In such a case, there are three methods for generating an object image, for example, methods (1) to (3) given below as examples.

[0065] (1) An object image 1203 is generated by assigning a non-learned label 1205 to a region 1204 of pixels to which a semantic label other than the target for judgment has been assigned.

[0066] (2) Object image 1203 is divided so as not to include region 1204 of pixels to which semantic labels other than those to be determined are assigned, and object images 1206 and 1207 corresponding to object image 1203 are generated.

[0067] (3) Divide the object image 1203 into unit areas of a preset size, and generate an object image 1209 corresponding to each unit area other than the unit area 1208, excluding the unit area 1208 that includes pixels assigned semantic labels that are not subject to judgment.

[0068] Here, the non-learning label 1205 is a label for designating pixels that are not used by the space acquisition unit 104 for updating (learning) the space information, from among the pixels included in the object image.

[0069] After S1103, the image processing device 100 executes the processes of S304 and S305, and after the process of S305, ends the process of the flowchart shown in Fig. 11. Note that the processes of S304 and S305 in other embodiments are similar to the processes of S304 or S305 in the first or second embodiment, and therefore detailed description thereof will be omitted.

[0070] As described above, the image processing device 100 is configured to generate an object image and object image camera parameters corresponding to the object image from a plurality of captured images obtained by capturing images from a plurality of directions and camera parameters corresponding to each captured image. The image processing device 100 is also configured to update (learn) spatial information using data of each generated object image and object image camera parameters corresponding to each object image. According to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less usage of the VRAM 205 compared to the case where data of all captured images obtained is expanded in the VRAM 205. According to the image processing device 100 configured in this way, it is possible to update (learn) spatial information with less calculation amount of the GPU 202 compared to the case where data of captured images is used to update (learn) spatial information.

[0071] Furthermore, the image processing device 100 is configured so that image regions of the object image corresponding to images of objects other than the object to be determined are not used for updating (learning) the spatial information. According to the image processing device 100 configured in this way, it is possible to update (learn) the spatial information with a smaller amount of usage of the VRAM 205. Also, according to the image processing device 100 configured in this way, it is possible to update (learn) the spatial information with a smaller amount of calculation of the GPU 202.

[0072] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0073] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0074] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0075] <Configuration 1> a data acquisition means for acquiring data of a plurality of captured images obtained by capturing images of an object from a plurality of directions and image capturing camera parameters which are camera parameters corresponding to each of the plurality of captured images; an object image generating means for generating a plurality of object images by extracting an image area corresponding to an image of the object from each of the plurality of captured images; a camera parameter generating means for generating object image camera parameters, which are camera parameters corresponding to each of the plurality of object images, based on the imaging camera parameters and information indicating positions of image areas corresponding to the images of the object in the plurality of captured images; a space acquisition means for acquiring space information representing a space in which the object exists, based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images; 13. An image processing device comprising:

[0076] <Configuration 2> the space acquisition means generates an image corresponding to a view from a virtual viewpoint based on the object image camera parameters based on virtual space information and the object image camera parameters, and acquires the space information by repeatedly updating the virtual space information so that a difference between the generated image and the object image corresponding to the object image camera parameters becomes small; 2. The image processing device according to claim 1,

[0077] <Configuration 3> the spatial information is information representing density at each position in the space; 3. The image processing device according to configuration 1 or 2,

[0078] <Configuration 4> the spatial information being information representing a signed distance from a surface of the object for each position in the space; 3. The image processing device according to configuration 1 or 2,

[0079] <Component 5> the spatial information is information representing a color for each position in the space; 3. The image processing device according to configuration 1 or 2,

[0080] <Component 6> the spatial information is information that represents a different color for each position and each direction in the space; 3. The image processing device according to configuration 1 or 2,

[0081] <Component 7> 7. The image processing device according to any one of configurations 1 to 6, further comprising: a generating means for generating an image corresponding to an appearance from an arbitrary viewpoint based on the spatial information.

[0082] <Component 8> The imaging camera parameters include information on the position, orientation, focal length, and principal point of an imaging device used for capturing the image, and information on the size of the captured image; 8. The image processing device according to any one of configurations 1 to 7,

[0083] <Component 9> the camera parameter generating means generates the object image camera parameters by changing information on a size of the captured image and information on a principal point of an imaging device used for capturing the image, the information being included in the imaging camera parameters corresponding to the captured image, based on information indicating a position of an image area corresponding to an image of the object in the captured image; 9. The image processing device according to any one of configurations 1 to 8,

[0084] <Component 10> an area setting means for specifying a resolution of the object image generated by the object image generating means; Further comprising: the object image generating means divides an image area corresponding to the extracted image of the object into a plurality of unit areas based on a specified resolution, and generates the object image by extracting each of the divided unit areas from the captured image; 10. The image processing device according to any one of configurations 1 to 9,

[0085] <Component 11> the object image generating means classifies the captured image into a foreground region and a background region, and generates the object image by extracting the foreground region from the captured image; 11. The image processing device according to any one of configurations 1 to 10,

[0086] <Component 12> The data acquisition means further acquires a semantic label that designates the object to be processed; the object image generating means generates the object image by assigning label information for semantic classification to a part of an image region or pixels in the captured image, and extracting an image region or pixels to which label information corresponding to the semantic label is assigned from the captured image; 12. The image processing device according to any one of configurations 1 to 11,

[0087] <Component 13> the object image generating means generates the object image based on a partial area or pixels of the captured image designated by a user; 13. The image processing device according to any one of configurations 1 to 12,

[0088] <Method> a data acquisition step of acquiring data of a plurality of captured images obtained by capturing images of an object from a plurality of directions and capturing camera parameters which are camera parameters corresponding to each of the plurality of captured images; an object image generating step of generating a plurality of object images by extracting an image region corresponding to an image of the object from each of the plurality of captured images; a camera parameter generating step of generating object image camera parameters, which are camera parameters corresponding to each of the plurality of object images, based on the imaging camera parameters and information indicating positions of image areas corresponding to the images of the object in the plurality of captured images; a space acquisition step of acquiring space information representing a space in which the object exists, based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images; 13. An image processing method comprising:

[0089] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0090] 100 Image processing device 101 Data Acquisition Section 102 Object Image Generation Unit 103 Camera parameter generator 104 Space acquisition part

Claims

1. a data acquisition means for acquiring data of a plurality of captured images obtained by capturing images of an object from a plurality of directions and image capturing camera parameters which are camera parameters corresponding to each of the plurality of captured images; an object image generating means for generating a plurality of object images by extracting an image area corresponding to an image of the object from each of the plurality of captured images; a camera parameter generating means for generating object image camera parameters, which are camera parameters corresponding to each of the plurality of object images, based on the imaging camera parameters and information indicating positions of image areas corresponding to the images of the object in the plurality of captured images; a space acquisition means for acquiring space information representing a space in which the object exists, based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images; 13. An image processing device comprising:

2. the space acquisition means generates an image corresponding to a view from a virtual viewpoint based on the object image camera parameters based on virtual space information and the object image camera parameters, and acquires the space information by repeatedly updating the virtual space information so that a difference between the generated image and the object image corresponding to the object image camera parameters becomes small; 2. The image processing device according to claim 1,

3. the spatial information is information representing density at each position in the space; 2. The image processing device according to claim 1,

4. the spatial information being information representing a signed distance from a surface of the object for each position in the space; 2. The image processing device according to claim 1,

5. the spatial information is information representing a color for each position in the space; 2. The image processing device according to claim 1,

6. the spatial information is information that represents a different color for each position and each direction in the space; 2. The image processing device according to claim 1,

7. The image processing apparatus according to claim 1 , further comprising: a generating unit configured to generate an image corresponding to an appearance from an arbitrary viewpoint based on the spatial information.

8. The imaging camera parameters include information on the position, orientation, focal length, and principal point of an imaging device used for capturing the image, and information on the size of the captured image; 2. The image processing device according to claim 1,

9. the camera parameter generating means generates the object image camera parameters by changing information on a size of the captured image and information on a principal point of an imaging device used for capturing the image, the information being included in the imaging camera parameters corresponding to the captured image, based on information indicating a position of an image area corresponding to an image of the object in the captured image; 2. The image processing device according to claim 1,

10. an area setting means for specifying a resolution of the object image generated by the object image generating means; Further comprising: the object image generating means divides an image area corresponding to the extracted image of the object into a plurality of unit areas based on a specified resolution, and generates the object image by extracting each of the divided unit areas from the captured image; 2. The image processing device according to claim 1,

11. the object image generating means classifies the captured image into a foreground region and a background region, and generates the object image by extracting the foreground region from the captured image; 2. The image processing device according to claim 1,

12. The data acquisition means further acquires a semantic label that designates the object to be processed; the object image generating means generates the object image by assigning label information for semantic classification to a part of an image region or pixels in the captured image, and extracting an image region or pixels to which label information corresponding to the semantic label is assigned from the captured image; 2. The image processing device according to claim 1,

13. the object image generating means generates the object image based on a partial area or pixels of the captured image designated by a user; 2. The image processing device according to claim 1,

14. a data acquisition step of acquiring data of a plurality of captured images obtained by capturing images of an object from a plurality of directions and capturing camera parameters which are camera parameters corresponding to each of the plurality of captured images; an object image generating step of generating a plurality of object images by extracting an image region corresponding to an image of the object from each of the plurality of captured images; a camera parameter generating step of generating object image camera parameters, which are camera parameters corresponding to each of the plurality of object images, based on the imaging camera parameters and information indicating positions of image areas corresponding to the images of the object in the plurality of captured images; a space acquisition step of acquiring space information representing a space in which the object exists, based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images; 13. An image processing method comprising:

15. A program for causing a computer to function as the image processing device according to any one of claims 1 to 13.

Images