Image processing device and image processing method

Abstract

An image processing device according to one embodiment of the present invention is provided with: an acquisition unit that acquires a plurality of images obtained by photographing a subject using a plurality of cameras and viewpoint information indicating the viewpoints of the plurality of cameras; a point group data generation unit that receives the plurality of images and viewpoint information as input, and generates point group data corresponding to the viewpoints of the plurality of cameras using an AI model that is machine-learned so as to output point group data for each viewpoint of the plurality of cameras, each point of the point group data including depth information and diffusion parameters indicating the distance from each camera to the subject; and an integration unit that integrates point group data corresponding to the viewpoints of the plurality of cameras to generate a three-dimensional model of the subject.

Description

Image processing device and image processing method

[0001] The present disclosure relates to an image processing device and an image processing method.

[0002] There are known techniques for generating an image from an arbitrary viewpoint based on a plurality of images taken from different viewpoints. For example, Non-Patent Document 1 describes a technique called 3D Gaussian splatting. 3D Gaussian splatting is a technique for generating point cloud data from a plurality of images taken from different viewpoints, applying a Gaussian function to parameters set for each point in the point cloud data, and smoothly blending the points with their surroundings to reconstruct a three-dimensional model of the subject.

[0003] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis, “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, SIGGRAPH 2023 (ACM Transactions on Graphics), Vol. 42, No. 4, (2023)

[0004] In the above-mentioned 3D Gaussian splatting, the reconstruction of a 3D model is performed by iteratively processing and gradually optimizing the parameters associated with each point in the point cloud data. This optimization process requires a large amount of computational processing, and the reconstruction of the 3D model requires a long processing time.

[0005] The present disclosure aims to reduce the time required to reconstruct a three-dimensional model.

[0006] An image processing device according to one aspect of the present invention comprises: an acquisition unit that acquires multiple images of a subject taken by multiple cameras and viewpoint information indicating the viewpoints of the multiple cameras; a point cloud data generation unit that inputs the multiple images and viewpoint information and generates point cloud data corresponding to the viewpoints of the multiple cameras using an AI model that has been machine-learned to output point cloud data for each viewpoint of the multiple cameras, wherein each point in the point cloud data includes depth information and diffusion parameters indicating the distance from each camera to the subject; and an integration unit that integrates the point cloud data corresponding to the viewpoints of the multiple cameras to generate a three-dimensional model of the subject.

[0007] According to the present invention, the time required to reconstruct a three-dimensional model can be reduced.

[0008] FIG. 1 is a block diagram showing the functional configuration of an image processing device according to an embodiment. FIG. 2 is a diagram schematically showing the arrangement of multiple cameras. FIG. 3 is a diagram schematically showing the functions of an AI model. FIG. 4 is a diagram schematically showing an example of a method for generating a three-dimensional model. FIG. 5 is a diagram schematically showing another example of a method for generating a three-dimensional model. FIG. 6 is a block diagram showing the functional configuration of an image processing device according to another embodiment. FIG. 7 is a flowchart showing the image generation phase of an image processing method according to an embodiment. FIG. 8 is a flowchart showing the learning phase of an image processing method according to an embodiment. FIG. 9 is a block diagram showing the hardware configuration of an image processing device.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicated description will be omitted.

[0010] An image processing device according to an embodiment will now be described. Fig. 1 is a block diagram showing the functional configuration of an image processing device 10 according to an embodiment. The image processing device 10 generates a three-dimensional model of a subject from a plurality of images captured by a plurality of cameras 2.

[0011] The multiple cameras 2 are imaging devices that capture images of the subject 3 from different viewpoints. In the following description, the term "image" includes both still images and moving images. In one embodiment, as shown in FIG. 2 , the multiple cameras 2 are arranged around the subject 3 and capture images of the subject 3 from different angles.

[0012] 1, the image processing device 10 includes, as functional components, an acquisition unit 11, a point cloud data generation unit 12, an integration unit 13, a rendering unit 14, a comparison unit 15, and a learning unit 16. In the embodiment shown in FIG. 1, the acquisition unit 11, the point cloud data generation unit 12, the integration unit 13, the rendering unit 14, the comparison unit 15, and the learning unit 16 are realized by a single image processing device 10, but these functional elements may be distributed across multiple devices.

[0013] Each functional element of the image processing device 10 can access the storage unit 20. The storage unit 20 is a storage device that stores a plurality of captured images Xj (j=1, 2, ..., L), shooting viewpoint information Cj (j=1, 2, ..., L) of a plurality of cameras 2, and a three-dimensional model Y. The storage unit 20 may be disposed in the image processing device 10 as shown in FIG. 1 , or may be disposed outside the image processing device 10 so as to be accessible from the image processing device 10.

[0014] The multiple captured images Xj are two-dimensional image data captured by multiple cameras 2. If the captured images Xj are color images, each pixel of the captured images Xj contains information related to RGB pixel values. The capturing viewpoint information Cj is information indicating the viewpoint of each camera 2. More specifically, it is a parameter representing the position information and attitude information of the camera 2. The position information of the camera 2 indicates, for example, the position of the camera 2 in three-dimensional space. The attitude information of the camera 2 indicates, for example, the roll angle, pitch angle, and yaw angle of the camera 2.

[0015] The acquisition unit 11 acquires, from the storage unit 20, a plurality of captured images Xj captured by a plurality of cameras 2 and imaging viewpoint information Cj corresponding to the plurality of cameras 2. The acquisition unit 11 outputs the acquired plurality of captured images Xj and imaging viewpoint information Cj to the point cloud data generation unit 12.

[0016] The point cloud data generator 12 generates point cloud data Yj (j=1, 2, ..., L) corresponding to the viewpoints of the multiple cameras 2, based on the multiple captured images Xj and the shooting viewpoint information Cj acquired by the acquirer 11. The point cloud data Yj is generated for each of the multiple captured images Xj.

[0017] The point cloud data Yj is a set of points corresponding to pixels in each captured image Xj. Each point in the point cloud data Yj includes two-dimensional position information as seen from the viewpoint of each camera 2, depth information indicating the distance from the viewpoint of the camera 2 to the subject 3, and a diffusion parameter. The diffusion parameter includes color information related to RGB, information indicating transparency, and information indicating covariance. The information indicating covariance is, for example, a 3x3 covariance matrix.

[0018] The point cloud data generator 12 receives a plurality of captured images Xj and shooting viewpoint information Cj as input, and generates point cloud data Yj using an AI model 12a that has been machine-learned to output point cloud data Yj for each viewpoint of a plurality of cameras 2. The AI ​​model 12a is constructed by machine learning training data that includes a plurality of training images captured in the past, viewpoint information of a plurality of cameras that captured the plurality of training images, and point cloud data corresponding to the plurality of training images.

[0019] The point cloud data generation unit 12 inputs a plurality of captured images Xj and shooting viewpoint information Cj into the AI ​​model 12a constructed as described above, and generates point cloud data Yj for each viewpoint of a plurality of cameras 2. For example, as shown in FIG. 3 , when three captured images X1, X2, and X3 captured by three cameras 2 and the shooting viewpoint information C1, C2, and C3 of the three cameras 2 are input to the AI ​​model 12a, three sets of point cloud data Y1, Y2, and Y3 corresponding to the viewpoints of the three cameras 2 are output. The point cloud data generation unit 12 outputs the generated point cloud data Yj to the integration unit 13.

[0020] The integration unit 13 integrates the point cloud data Yj corresponding to the viewpoints of the multiple cameras 2 to generate a three-dimensional model Y of the subject 3. The three-dimensional model Y is a three-dimensional digital model of the subject 3 created on a computer.

[0021] FIG. 4 is a diagram illustrating an example of a method for generating a three-dimensional model Y in the integrating unit 13. In this method, the union of multiple point cloud data Y1, Y2, and Y3 is mapped onto a three-dimensional space to generate the three-dimensional model Y. As shown in the example of FIG. 3, when three sets of point cloud data Y1, Y2, and Y3 are generated from three captured images X1, X2, and X3 and shooting viewpoint information C1, C2, and C3, the integrating unit 13 moves and rotates the three sets of point cloud data Y1, Y2, and Y3 based on the shooting viewpoint information C1, C2, and C3 so that they match the viewpoints of the three cameras 2. Next, the integrating unit 13 maps the union of the moved and rotated point cloud data Y1, Y2, and Y3 onto a three-dimensional space to generate the three-dimensional model Y. The generated three-dimensional model Y is stored in the storage unit 20.

[0022] FIG. 5 is a diagram schematically illustrating another example of a technique for generating a three-dimensional model Y in the integrating unit 13. In this technique, the integrating unit 13 generates the three-dimensional model Y by performing statistical processing on the point cloud data Yj. As shown in the example of FIG. 3 , when three sets of point cloud data Y1, Y2, and Y3 are generated from three captured images X1, X2, and X3 and shooting viewpoint information C1, C2, and C3, the integrating unit 13 moves and rotates the three sets of point cloud data Y1, Y2, and Y3 based on the shooting viewpoint information C1, C2, and C3 so as to match the viewpoints of the three cameras 2. Next, the integrating unit 13 calculates the average or median of the depth information and the diffusion parameter for each two-dimensional position of the point cloud data Y1, Y2, and Y3, and maps the calculated average or median into three-dimensional space to generate the three-dimensional model Y. The generated three-dimensional model Y is stored in the storage unit 20.

[0023] The rendering unit 14 generates a rendering image of the subject 3 viewed from an arbitrary viewpoint using the three-dimensional model Y. For example, the rendering unit 14 receives designated viewpoint information indicating a designated viewpoint from the line-of-sight setting unit 18, and generates a rendering image of the subject 3 corresponding to the designated viewpoint using the three-dimensional model Y. The designated viewpoint is, for example, a viewpoint designated by a user. In other words, the rendering unit 14 can generate an image of the subject 3 from a viewpoint different from the viewpoints of the multiple cameras 2. The rendering image output from the rendering unit 14 is displayed on a display device (not shown).

[0024] The comparison unit 15 compares the rendered image generated by the rendering unit 14 with one or more images captured by one or more cameras 2 out of the multiple cameras 2. For example, the comparison unit 15 acquires multiple rendered images Rj (j = 1, 2, ..., L) of the subject 3 corresponding to the shooting viewpoint information Cj of the multiple cameras 2, and calculates the differences between the multiple captured images Xj captured by the multiple cameras 2 and the multiple rendered images Rj. In one embodiment, the comparison unit 15 calculates the mean square error MSE between the pixel values ​​of the multiple captured images Xj and the pixel values ​​of the multiple rendered images Rj based on the following formula (1):

[0025]

[0026] The comparison unit 15 outputs information indicating the results of the comparison between the plurality of captured images Xj and the plurality of rendering images Rj to the learning unit 16 .

[0027] The learning unit 16 re-learns the AI ​​model 12a of the point cloud data generation unit 12 based on the comparison results between the multiple captured images Xj and the multiple rendered images Rj. For example, the learning unit 16 generates parameter update information based on the mean square error MSE calculated by the comparison unit 15, and updates the learning parameters of the AI ​​model 12a using the parameter update information. For example, the learning unit 16 updates the learning parameters of the AI ​​model 12a using an error backpropagation method that backpropagates the mean square error MSE from the output layer to the input layer of the AI ​​model 12a.

[0028] As described above, in the image processing device 10, a plurality of captured images Xj of the subject 3 captured by a plurality of cameras 2 and shooting viewpoint information Cj indicating the viewpoints of the plurality of cameras 2 are input to the AI ​​model 12a to generate point cloud data Yj including depth information and diffusion parameters, and the point cloud data Yj is integrated to generate a three-dimensional model Y of the subject 3. As described above, in this image processing device 10, the point cloud data Yj is generated directly by the AI ​​model 12a without performing optimization processing of the diffusion parameters, and therefore the three-dimensional model Y can be reconstructed in a short time.

[0029] In addition, the image processing device 10 re-learns the AI ​​model 12a based on the comparison results between the rendering image Rj generated by the rendering unit and the images captured by one or more of the multiple cameras 2, thereby making it possible to generate a three-dimensional model Y with high reproducibility.

[0030] Furthermore, since the integration unit 13 generates the three-dimensional model Y by performing statistical processing on the point cloud data Yj corresponding to the viewpoints of the multiple cameras 2, it is possible to generate a three-dimensional model Y that is robust to noise.

[0031] Next, an image processing device according to another embodiment will be described. Fig. 6 is a block diagram showing the functional configuration of an image processing device 10A according to another embodiment. The following mainly describes differences from the image processing device 10 shown in Fig. 1, and redundant explanations will be omitted.

[0032] The image processing device 10A differs from the image processing device 10 in that it further includes a pseudo viewpoint information generator 21. The pseudo viewpoint information generator 21 generates pseudo viewpoint information C'k (k = 1, 2, ..., K) that indicates a pseudo viewpoint different from the viewpoints of the multiple cameras 2, and outputs the pseudo viewpoint information C'k to the point cloud data generator 12. The pseudo viewpoint is set to a viewpoint that complements blind spots of the subject 3 in the multiple captured images Xj. For example, the pseudo viewpoint is one or more viewpoints that are virtually set at positions different from the shooting viewpoints of the multiple cameras 2.

[0033] The point cloud data generation unit 12 inputs the pseudo viewpoint information C'k in addition to the multiple captured images Xj and shooting viewpoint information Cj to the AI ​​model 12a to generate point cloud data Yj corresponding to the viewpoints of the multiple cameras 2 and point cloud data Y'k (k = 1, 2, ..., K) corresponding to the pseudo viewpoints. Then, the integration unit 13 integrates the point cloud data Yj and the point cloud data Y'k to generate a three-dimensional model Y.

[0034] In the image processing device 10A, point cloud data Y'k corresponding to a pseudo viewpoint that complements the blind spot of the subject 3 is generated, so that a three-dimensional model Y with higher reproducibility can be generated.

[0035] Next, an image processing method according to one embodiment will be described. This image processing method is executed by the image processing devices 10 and 10A described above. The image processing method according to one embodiment includes an image generation phase for generating a three-dimensional model Y and a learning phase for learning the AI ​​model 12a. Figure 7 is a flowchart showing the image generation phase of the image processing method.

[0036] As shown in FIG. 7, in the image generation phase, the acquisition unit 11 of the image processing device 10 acquires captured images Xj captured by the multiple cameras 2 and the imaging viewpoint information Cj of the multiple cameras 2 (step ST1).

[0037] Next, the point cloud data generator 12 inputs the captured image Xj and the shooting viewpoint information Cj into the AI ​​model 12a, which has been trained in advance by machine learning, and generates point cloud data Yj corresponding to the captured image Xj and the shooting viewpoint information Cj (step ST2). In one embodiment, the pseudo viewpoint information generator 21 may generate pseudo viewpoint information C'k indicating a pseudo viewpoint different from the viewpoints of the multiple cameras 2 (step ST3). In this case, the point cloud data generator 12 inputs the captured image Xj, the shooting viewpoint information Cj, and the pseudo viewpoint information Ck into the AI ​​model 12a, and generates point cloud data Y'k corresponding to the pseudo viewpoint in addition to the point cloud data Yj corresponding to the viewpoints of the multiple cameras 2 (step ST4).

[0038] Next, the integration unit 13 integrates the point cloud data Yj and the point cloud data Y'k to generate a three-dimensional model Y of the subject 3 (step ST5). Next, the rendering unit 14 acquires designated viewpoint information indicating a designated viewpoint from the user, and generates a rendering image of the subject 3 corresponding to the designated viewpoint using the three-dimensional model Y (step ST6). The generated rendering image is displayed on the display device.

[0039] 8 is a flowchart showing the learning phase of an image processing method according to an embodiment. As shown in Fig. 8, in the learning phase, the acquisition unit 11 first randomly selects a previously generated three-dimensional model Y, and acquires N sets of rendering images X'i (i = 1, 2, ..., N) corresponding to the selected three-dimensional model and shooting viewpoint information Ci (i = 1, 2, ..., N) from the storage unit 20 (step ST11).

[0040] Next, the acquisition unit 11 generates a random number M (1≦M≦N) and selects M sets of rendering images X'j (j = 1, 2, ..., M) and shooting viewpoint information Cj (j = 1, 2, ..., M) from the N sets of rendering images X'i and shooting viewpoint information Ci (step ST12).

[0041] Next, the point cloud data generation unit 12 inputs M sets of rendering images X'j and shooting viewpoint information Cj into the AI ​​model 12a and generates M sets of point cloud data Yj (j = 1, 2, ..., M) corresponding to the shooting viewpoint information Cj (step ST13).

[0042] Next, the integration unit 13 integrates the M sets of point cloud data Yj and outputs a three-dimensional model Y (step ST14).

[0043] Next, the rendering unit 14 generates M rendering images Rj (j=1, 2, . . . , M) corresponding to the shooting viewpoint information Cj using the three-dimensional model Y (step ST15).

[0044] Next, the comparison unit 15 compares the rendering image X'j with the rendering image Rj (step ST16). For example, the comparison unit 15 calculates the mean square error MSE between the rendering image X'j and the rendering image Rj based on the following equation (2).

[0045]

[0046] Next, the learning unit 16 re-learns the AI ​​model 12a based on the comparison result between the rendering image X'j and the rendering image Rj. For example, the learning unit 16 updates the learning parameters of the AI ​​model 12a using the MSE calculated by the backpropagation algorithm (step ST16). The above steps ST11 to ST17 are repeated a predetermined number of times or until the MSE converges. This optimizes the learning parameters of the AI ​​model 12a.

[0047] The image processing device 10, 10A and information processing method of the present disclosure may have the following configuration.

[0048] [1] An image processing device comprising: an acquisition unit that acquires a plurality of images of a subject taken by a plurality of cameras and viewpoint information indicating the viewpoints of the plurality of cameras; a point cloud data generation unit that receives the plurality of images and the viewpoint information and generates the point cloud data corresponding to the viewpoints of the plurality of cameras using an AI model that has been machine-learned to output point cloud data for each viewpoint of the plurality of cameras, wherein each point of the point cloud data includes depth information and a diffusion parameter indicating a distance from each camera to the subject; and an integration unit that integrates the point cloud data corresponding to the viewpoints of the plurality of cameras to generate a three-dimensional model of the subject.

[0049] [2] The image processing device according to [1], wherein the diffusion parameters include information indicating color, transparency, and covariance.

[0050] [3] The image processing device according to [1] or [2], further comprising a pseudo viewpoint information generation unit that generates pseudo viewpoint information indicating a pseudo viewpoint different from viewpoints of the plurality of cameras, wherein the point cloud data generation unit inputs the pseudo viewpoint information to the AI ​​model in addition to the plurality of images and the viewpoint information, and generates the point cloud data corresponding to the viewpoints of the plurality of cameras and the pseudo viewpoint, and the integration unit integrates the point cloud data corresponding to the viewpoints of the plurality of cameras and the pseudo viewpoint to generate the three-dimensional model.

[0051] [4] The image processing device according to any one of [1] to [3], further comprising a rendering unit that generates a rendering image of the subject viewed from a specified viewpoint using the three-dimensional model.

[0052] [5] The image processing device according to [4], further comprising a learning unit that re-learns the AI ​​model according to a comparison result between the rendering image generated by the rendering unit and one or more images among the plurality of images.

[0053] [6] The image processing device according to any one of [1] to [5], wherein the integration unit generates the three-dimensional model by performing statistical processing on the point cloud data corresponding to the viewpoints of the plurality of cameras.

[0054] [7] An image processing method comprising: a step of acquiring a plurality of images of a subject taken by a plurality of cameras and viewpoint information indicating the viewpoints of the plurality of cameras; a step of generating point cloud data corresponding to the viewpoints of the plurality of cameras using an AI model that has been machine-learned to input the plurality of images and the viewpoint information and output point cloud data for each viewpoint of the plurality of cameras, wherein each point of the point cloud data includes depth information and diffusion parameters indicating the distance from each camera to the subject; and a step of integrating the point cloud data corresponding to the viewpoints of the plurality of cameras to generate a three-dimensional model of the subject.

[0055] The block diagrams shown in FIGS. 1 and 6 show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0056] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0057] For example, the image processing device 10, 10A according to an embodiment of the present invention may function as a computer. Fig. 9 is a diagram showing an example of the hardware configuration of the image processing device 10, 10A. The image processing device 10, 10A may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0058] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the image processing apparatus 10 may be configured to include one or more of the apparatuses shown in Fig. 9, or may be configured to exclude some of the apparatuses.

[0059] Each function of the image processing device 10, 10A is realized by loading specified software (programs) onto hardware such as a processor 1001 and memory 1002, causing the processor 1001 to perform calculations and control communication via a communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.

[0060] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each of the functional elements shown in FIG. 1 may be realized by the processor 1001.

[0061] The processor 1001 also reads programs (program codes), software modules, and data from the storage 1003 and / or the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each functional element of the image processing device 10 may be implemented by a control program stored in the memory 1002 and running on the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0062] The memory 1002 is a computer-readable recording medium and may be composed of at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), and a random access memory (RAM). The memory 1002 may also be called a register, a cache, a main memory (primary storage device), or the like. The memory 1002 can store executable programs (program codes), software modules, and the like for implementing an information processing method according to one embodiment of the present invention.

[0063] Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including memory 1002 and / or storage 1003.

[0064] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0065] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0066] Furthermore, each device such as the processor 1001 and the memory 1002 is connected to a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

[0067] The image processing device 10, 10A may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

[0068] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0069] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (New Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and next-generation systems enhanced based on these. Furthermore, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.) may also be applied.

[0070] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0071] In the present disclosure, a specific operation described as being performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and another network node other than the base station (for example, an MME or an S-GW, etc., but are not limited to these). Although the above example illustrates a case where there is one other network node other than the base station, a combination of multiple other network nodes (for example, an MME and an S-GW) may also be used.

[0072] Information etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer), or may be input / output via multiple network nodes.

[0073] Input and output information may be stored in a specific location (for example, memory) or managed in a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0074] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0075] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0076] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0077] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0078] Software, instructions, etc. may also be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.

[0079] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0080] It should be noted that terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0081] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0082] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed as absolute values, as relative values ​​from a predetermined value, or as other corresponding information. For example, a radio resource may be indicated by an index.

[0083] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0084] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0085] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0086] When designations such as "first," "second," etc. are used in this disclosure, any reference to an element does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

[0087] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0088] To the extent that the terms "include," "including," and variations thereof are used herein or in the claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used herein or in the claims, is not intended to be an exclusive or.

[0089] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0090] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0091] 2...camera, 3...subject, 10, 10A...image processing device, 11...acquisition unit, 12...point cloud data generation unit, 12a...AI model, 13...integration unit, 14...rendering unit, 16...learning unit.

Claims

1. An image processing device comprising: an acquisition unit that acquires multiple images of a subject taken by multiple cameras and viewpoint information indicating the viewpoints of the multiple cameras; a point cloud data generation unit that receives the multiple images and the viewpoint information and generates point cloud data corresponding to the viewpoints of the multiple cameras using an AI model that has been machine-learned to output point cloud data for each viewpoint of the multiple cameras, wherein each point in the point cloud data includes depth information and diffusion parameters indicating the distance from each camera to the subject; and an integration unit that integrates the point cloud data corresponding to the viewpoints of the multiple cameras to generate a three-dimensional model of the subject.

2. The image processing apparatus according to claim 1, wherein the diffusion parameters include information indicating color, transparency, and covariance.

3. The image processing device according to claim 1, further comprising a pseudo viewpoint information generation unit that generates pseudo viewpoint information indicating a pseudo viewpoint different from the viewpoints of the multiple cameras, wherein the point cloud data generation unit inputs the pseudo viewpoint information to the AI ​​model in addition to the multiple images and the viewpoint information, and generates the point cloud data corresponding to the viewpoints of the multiple cameras and the pseudo viewpoint, and the integration unit integrates the point cloud data corresponding to the viewpoints of the multiple cameras and the pseudo viewpoint to generate the three-dimensional model.

4. The image processing device according to claim 1, further comprising a rendering unit that generates a rendering image of the subject viewed from a specified viewpoint using the three-dimensional model.

5. The image processing device according to claim 4, further comprising a learning unit that retrains the AI ​​model based on a comparison result between the rendering image generated by the rendering unit and one or more images from among the plurality of images.

6. The image processing device according to claim 1, wherein the integration unit generates the three-dimensional model by performing statistical processing on the point cloud data corresponding to the viewpoints of the plurality of cameras.

7. An image processing method comprising: a step of acquiring a plurality of images of a subject taken by a plurality of cameras and viewpoint information indicating the viewpoints of the plurality of cameras; a step of generating point cloud data corresponding to the viewpoints of the plurality of cameras using an AI model that has been machine-learned to input the plurality of images and the viewpoint information and output point cloud data for each viewpoint of the plurality of cameras, wherein each point in the point cloud data includes depth information and diffusion parameters indicating the distance from each camera to the subject; and a step of integrating the point cloud data corresponding to the viewpoints of the plurality of cameras to generate a three-dimensional model of the subject.

Images