Method and apparatus for processing 3D scenes

Abstract

A method and apparatus for processing a 3D scene are disclosed. (41) At least one virtual reference viewpoint in the 3D scene is determined. (42) A map of registered real cast shadows of objects in the 3D scene is captured from an input image captured by a camera located at a viewpoint different from the virtual reference viewpoint, the map of real cast shadows being registered relative to the virtual reference viewpoint. (44) Using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene projected by the at least one light source from the virtual reference viewpoint, parameters for at least one light source in the 3D scene are determined.

Description

1. Technical Field

[0001] The present disclosure relates to 3D scene lighting for mixed reality. More specifically, the present disclosure relates to virtual object lighting inserted into a 3D real scene. 2. Background technology

[0002] In mixed reality, i.e. when virtual objects are inserted into a 3D model of a real scene, the 3D lighting of the virtual objects is a key feature in order to provide realistic aspects of the 3D scene. Virtual objects should be correctly illuminated by mimicking the real lighting of the scene. However, the lighting environment can change dynamically. Therefore, the 3D lighting of the real scene must be continuously estimated.

[0003] However, lighting is a time-consuming task in real-time rendering and needs to be greatly simplified, especially when reproducing on mobile devices. Therefore, a compromise must be found between complex modeling and graphics rendering time.

[0004] Therefore, a new method is needed to model the light sources of complex indoor dynamic scenes. 3. Summary of the invention

[0005] According to one aspect of the present disclosure, a method for processing a 3D scene is disclosed. The method includes: determining at least one virtual reference viewpoint in the 3D scene; obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera at a viewpoint different from the virtual reference viewpoint, the real cast shadow map being registered relative to the virtual reference viewpoint; using the registered real cast shadow map and at least one map of virtual shadows of objects in the 3D scene projected by at least one light source from the virtual reference viewpoint, determining parameters for at least one light source in the 3D scene.

[0006] According to another aspect of the present disclosure, a device for processing a 3D scene is also disclosed. The device includes: a tool for determining at least one virtual reference viewpoint in the 3D scene; a device for obtaining a registered real projection shadow map of an object in the 3D scene from an input image captured by a camera at a viewpoint different from the virtual reference viewpoint, the real projection shadow map being registered relative to the virtual reference viewpoint; and a device for determining parameters for at least one light source in the 3D scene using the registered real projection shadow map and at least one map of virtual shadows of objects in the 3D scene projected by at least one light source from the virtual reference viewpoint.

[0007] 4. Brief description of the accompanying drawings

[0008] Figure 1 An exemplary method for determining a 3D position of a candidate point light source of a 3D scene according to an embodiment of the present disclosure is shown.

[0009] Figure 2A An image showing a 3D scene,

[0010] Figure 2B Shown from Figure 2A The image shown casts shadows of objects in the 3D scene.

[0011] Figure 2C shows an image of a 3D scene rendered with a point light source,

[0012] Figure 2D Another image of the 3D scene rendered with another point light source is shown,

[0013] Figure 3A shows different viewpoints for capturing a real scene from a camera embedded in a mobile device,

[0014] Figure 3B shows the virtual reference viewpoint introduced in the 3D scene,

[0015] Figure 4 An exemplary method for processing a 3D scene according to an embodiment of the present disclosure is shown.

[0016] Figure 5A An exemplary method for obtaining a registered true cast shadow map according to an embodiment of the present disclosure is shown.

[0017] Figure 5B An exemplary method for obtaining a registered true cast shadow map according to another embodiment of the present disclosure is shown.

[0018] Fig. 6A An exemplary method for determining a virtual reference viewpoint according to an embodiment of the present invention is shown.

[0019] Figure 6B An exemplary method for determining a virtual reference viewpoint according to another embodiment of the present invention is shown.

[0020] Figure 7 An exemplary method for determining lighting parameters according to an embodiment of the present disclosure is shown.

[0021] Figure 8 An exemplary apparatus for processing a 3D scene according to an embodiment of the present disclosure is shown.

[0022] 5. Description of the implementation plan

[0023] In order to correctly illuminate the virtual scene, it is necessary to correctly render the illuminated virtual objects and to faithfully render the shadows cast by the virtual objects onto the real scene. Two main features contribute to the realism of shadows and cast shadows: the 3D position / orientation of the shadow and the shadow intensity given by the falloff of the color intensity of the shadow surface.

[0024] In the presence of hard shadows, there is a set of virtual lighting models: point lights, spot lights, or directional lighting.

[0025] In the embodiments disclosed herein, illumination is modeled by a set of 3D point light sources. The 3D positions of such source light points and the shadow attenuation associated with each source light point can be estimated from a single video camera observing the real scene.

[0026] It should be noted that the lighting conditions in real scenes are unstable and continuously changing. Therefore, in order to improve the realism of mixed reality, the lighting environment must be estimated continuously.

[0027] In order to estimate the 3D position of the light source in the real scene, the real scene is analyzed by image processing.

[0028] Figure 1 An exemplary method for determining the position of a light source is shown.

[0029] It is assumed that a 3D geometric model of the real scene and an image of the scene captured from an input video camera with a known 3D pose in the 3D model coordinate system are available. The 3D pose of the input video camera includes not only the 3D position of the input video camera in the 3D model coordinate system, but also the intrinsic camera parameters.

[0030] For example, for a point light source from a candidate point light source set, the position of the point light source in the 3D scene is determined. Such a 3D point light source set may be a list of 3D poses or a structure tree of 3D positions.

[0031] In step 10, real cast shadows are detected in an input RGB (red, green, blue) image of a scene obtained for an input video camera. Real cast shadows correspond to shadows of real objects present in the scene, cast by real light sources and captured by the input video camera.

[0032] Figure 2A An example of an input RGB image is shown. Figure 2B The corresponding cast shadows detected in the input image are shown. Figure 2B , the detected cast shadow appears in light grey and is identified by an arrow pointing from reference 20 .

[0033] In step 11, for the candidate light source, the virtual shadow obtained by the candidate light source is matched with the detected projected shadow.

[0034] To do so, in step 11, a rendered image is obtained for the candidate point light sources. Such a rendered image may be obtained by any known method for 3D rendering of cast shadows from candidate point light sources. The rendered image includes shadows cast by associated candidate point light sources from the set of point light sources.

[0035] Figure 2C and 2D Examples of rendering images with different point light sources and using the viewpoint in the 3D scene as the viewpoint of the input video camera (similar to the viewpoint used to obtain the input RGB image) are shown.

[0036] exist Figure 2C and 2D As can be seen from the figure, the cast shadows 21 and 22 depend on the position of the 3D point light source used to render the image. Figure 2C In the figure, the 3D point light source is located on the left side of the projected scene, and Figure 2D In the figure, the 3D point light source is located on the right side of the projected scene.

[0037] Then, in step 11, the rendered image is matched with the mask of the real cast shadow detected in step 10. Matching the rendered image with the detected real cast shadow can be achieved by calculating the correlation between the binary mask of the detected real cast shadow and the binary mask of the virtual shadow cast by the candidate 3D point light source.

[0038] The 3D position of the point light source corresponds to the 3D position of the candidate point light source.

[0039] In step 11, once the position of the candidate point light source is determined, matching pixels that match both the detected mask of the real cast shadow and the virtual cast shadow obtained from the candidate point light source are marked so as to be discarded when estimating other point light source positions.

[0040] When estimating other point light sources, in step 11, pixels that have been marked in determining previous point light sources are discarded.

[0041] In step 12, it is verified whether the maximum number of point light sources has not been determined and whether the number of unmatched real shadow points is higher than a determined value.

[0042] For example, the maximum number of point lights may correspond to the maximum number of point lights that can be satisfactorily processed by the rendering engine.

[0043] If both conditions cited above are true, another point light source is selected from the set of candidate 3D point light sources and added to the 3D scene. The matching operation is repeated for the new 3D point light source (step 11). It should be noted that when the matching operation is repeated for the new 3D point light source, detected real shadow points that match virtual shadow points cast from the previously determined 3D point light source are discarded.

[0044] When at least one of the above two conditions is false, the step of determining the position of the 3D point light source ends, and in step 13 , the occlusion attenuation is calculated and assigned to each 3D point light source determined in step 11 .

[0045] As can be seen from the above exemplary method, a virtual shadow map (i.e., a shadow map cast by a candidate 3D light source point) must be calculated for each candidate source light. Therefore, creating a set of virtual shadow maps is expensive.

[0046] Furthermore, these maps are associated with a specific viewpoint that corresponds to the viewpoint of the input video camera that captures the image of the real scene. If the lighting environment is stationary and if the geometry of the scene is static, the above lighting determination can be performed offline. But on the other hand, if the lighting environment is dynamic, the lighting determination must be performed online and continuously.

[0047] Furthermore, the light source is determined based on the input video camera viewpoint. When the input video camera viewpoint changes, the cast shadows (eg, real and virtual) change.

[0048] For example, such a problem may arise when the above-mentioned 3D scene processing is used for mixed reality applications provided on mobile devices (such as mobile phones, tablet computers, etc.). In such applications, the real scene may be captured by a video camera of the mobile device. The user of the mobile device can move the device, and thus the input video camera viewpoint may change. For example, Figure 3A As shown, the input video camera can capture the real scene from the first viewpoint V1. Then, the user moves towards the viewpoint V2 with the mobile device on the right side, and then continuously captures the real scene from consecutive viewpoints along the input video camera trajectory.

[0049] In this case, since the light source must be continuously determined and the viewpoint of the input video camera changes, the set of virtual shadow maps must be continuously updated. However, calculating many shadow maps is time consuming. Therefore, it is an object of the present disclosure to limit the occurrence of shadow map calculations.

[0050] According to the principles proposed herein, a virtual reference viewpoint is introduced in a 3D scene. Therefore, a set of virtual shadow maps can be created relative to the virtual reference viewpoint. As long as the input video camera remains close to the virtual reference viewpoint, the virtual shadow map created with respect to the virtual reference viewpoint is used in determining the lighting parameters.

[0051] In the specific case where the position of a moving camera is roughly known, a set of virtual reference viewpoints can be predefined and the corresponding shadow maps can be precomputed. On the other hand, if the camera motion cannot be controlled, the virtual reference viewpoints have to be defined dynamically.

[0052] In the latter case, when the input video camera moves and reaches a position where the current virtual reference viewpoint is no longer valid, a new virtual reference viewpoint is defined and a new set of virtual shadow maps is created. The new virtual reference viewpoint may, for example, be selected as the new position of the input video camera.

[0053] Figure 3B Exemplary positions V1 , V3 , V4 , V2 for virtual reference viewpoints introduced in a 3D scene are shown.

[0054] According to an embodiment of the present principles, the virtual reference viewpoint may correspond to the viewpoint of the multi-view image capture. In this way, the camera parameters and positions are known.

[0055] According to the principles given here, the steps to determine the lighting parameters are ( Figure 1 Step 11) is modified as follows. Figure 4 An exemplary method for determining lighting parameters according to an embodiment of the present disclosure is illustrated.

[0056] The real scene is captured by a camera from a first viewpoint, which will be referred to as the input video camera viewpoint hereinafter. For example, a mobile device is located at a viewpoint V1' in a 3D scene, such as Figure 3B shown.

[0057] During the 3D scene rendering process, a virtual cast shadow map is determined relative to a current virtual reference viewpoint. At the beginning of the rendering process, the current virtual reference viewpoint corresponds to the input video camera viewpoint, since the input video camera may not have been moved by the user. Figure 1 The content is determined as discussed above.

[0058] In certain cases, the user can be asked to start from a rough position, from which the preprocessed reference viewpoint can be compared with its precomputed shadow Figure 1 Use together.

[0059] In step 40 , it is checked whether a new virtual reference viewpoint is needed for determining the lighting parameters.

[0060] According to an embodiment of the present disclosure, if the viewpoint of the input video camera and the current virtual reference viewpoint do not meet the determination criteria, a new virtual reference viewpoint is required. For example, if the number of overlapping pixels in the image captured from the current virtual reference viewpoint and the image captured from the current viewpoint of the input video camera is lower than a determined value, the new virtual reference viewpoint is determined. As an example, if the distance between two co-located pixels represented by the absolute difference in depth after one of the two images is registered on the other image is lower than a threshold, the pixels are declared to be "overlapping".

[0061] According to another example, if the number of pixels of the image captured from the current virtual reference viewpoint having no corresponding points in the image captured from the current viewpoint of the input video camera is higher than a determined value, a new virtual reference viewpoint is determined.

[0062] According to another embodiment of the present invention, if the distance between the input video camera viewpoint and the current virtual reference viewpoint is greater than a determined value, a new virtual reference viewpoint is required. Such a determined value may depend on the complexity of the real scene, taking into account, for example, the number of objects in the scene, the number and position of 3D light sources, etc.

[0063] If a new virtual reference viewpoint is needed, then in step 41 , the new virtual reference viewpoint is determined and an underlying candidate shadow map is obtained.

[0064] If the new virtual reference viewpoint has been previously used, or if the new virtual reference viewpoint has been previously determined and added to the set of available virtual reference viewpoints, its underlying candidate shadow map has been previously calculated and stored. In this case, the candidate shadow map associated with the new virtual reference viewpoint is retrieved from memory.

[0065] If a candidate shadow map associated with the new virtual reference viewpoint has not yet been calculated, then, for example, Figure 1 The process discussed computes them. Thus, a candidate shadow map is obtained for each candidate point light source using any known method for 3D rendering cast shadows.

[0066] For example, a new virtual reference viewpoint is selected from a set of candidate viewpoints whose positions are known. Figure 3B In the example of , the set may include viewpoints V1-V4.

[0067] According to an embodiment of the present invention, a virtual reference viewpoint in the candidate viewpoint set is selected as a viewpoint that is closest to the current input video camera viewpoint.

[0068] According to another embodiment of the present invention, the virtual reference viewpoint in the candidate viewpoint set is selected as a viewpoint for which the number of pixels overlapped in an image captured from the candidate viewpoint and an image captured from the current input video camera viewpoint is greater than a determined value. The selected viewpoint may be a candidate viewpoint having the most pixels overlapped with the input video camera viewpoint.

[0069] It has been assumed above that a set of virtual reference viewpoints is available, among which a satisfactory reference viewpoint can be identified regardless of the current input video camera viewpoint. This may be the case if the space of positions and orientations of the input video camera viewpoints is known a priori. Otherwise, if, for example, the camera motion cannot be controlled, then the reference viewpoints and the underlying candidate shadow maps are defined dynamically. The following discussion Figure 6B An exemplary method for dynamically determining a virtual reference viewpoint according to an embodiment of the present disclosure is disclosed.

[0070] In step 42, a new map of registered real cast shadows is obtained. Real cast shadows are shadows cast by real light sources from real objects in a real scene. From an input image captured by an input video camera at a current moment, a real projection map is obtained. Since the current viewpoint of the input video camera may be different from the new virtual reference viewpoint, the real projection map must be registered relative to the new virtual reference viewpoint.

[0071] The current map of registered real cast shadows is continuously updated due to the fact that the lighting may have changed or the real camera may have moved and thus the cast shadows detected in the current real viewpoint may have changed.

[0072] In step 44, lighting parameters are determined using the currently registered real projection map and the virtual shadow map projected by the 3D point light source and associated with the current virtual reference viewpoint. A virtual shadow map is generated for each candidate 3D point light source relative to the current virtual reference viewpoint.

[0073] According to the above Figure 1 The process in question determines the lighting parameters, matches each virtual shadow map with the map of registered real cast shadows obtained in step 42. And selects the virtual shadow map that best matches the map of registered real cast shadows. The 3D point light source associated with the selected virtual shadow map is selected as the 3D point light source. This step is iterated using other maps of virtual shadows to select other 3D point light sources. Pixels from the map of registered real cast shadows that have been marked as matching with a previous virtual shadow map are discarded in the matching process for determining subsequent 3D point light sources.

[0074] Figure 5AAn exemplary method for obtaining a registered true cast shadow map according to an embodiment of the present disclosure is shown.

[0075] According to this embodiment, in step 50, the Figure 1 In a similar manner as discussed above, real cast shadows are detected in the images captured from the input video camera. Thus, a map of detected real cast shadows is obtained. This map reflects the real cast shadows from the viewpoint of the input video camera.

[0076] In step 51, the detected map of real projected shadows is registered relative to the current virtual reference viewpoint. This registration is performed by determining, for each pixel of the detected map of real projected shadows, a corresponding pixel or pixels in the registered map of real projected shadows. This can be done by determining a 3D point in the 3D scene corresponding to a pixel projected in the detected map of real projected shadows. The determined 3D point in the 3D scene is then projected into the current virtual reference viewpoint. The pixels in the registered map of real projected shadows corresponding to the pixels in the detected map of real projected shadows are set to the same binary value, for example, to 0 if the pixel is a detected shadow pixel, and to 1 if the pixel is not a detected shadow pixel.

[0077] Figure 5B An exemplary method for obtaining a registered true cast shadow map according to another embodiment of the present disclosure is shown.

[0078] According to the present embodiment, in step 52, the image captured by the input video camera is registered with respect to the current virtual reference viewpoint. This can be performed in the same manner as described above, i.e. by projecting pixels in 3D space and then projecting 3D points with respect to the current virtual reference viewpoint. A registered input image is thereby obtained.

[0079] In step 53 , real cast shadows are then detected in the registered input images obtained in step 52 .

[0080] Fig. 6A An exemplary method for determining a virtual reference viewpoint according to an embodiment of the present disclosure is shown.

[0081] According to this embodiment, a new virtual reference viewpoint may be predetermined before it is needed for determining lighting parameters. According to this embodiment, the camera trajectory may be predicted. A new virtual reference viewpoint is predetermined based on the trajectory of the camera, and a virtual shadow map is predetermined based on the new virtual reference viewpoint. In this way, a predetermined new virtual reference viewpoint may be used when the current virtual reference viewpoint is no longer valid.

[0082] According to this embodiment, in step 60, the movement of a camera capturing a real scene is detected.

[0083] In step 61 , the camera trajectory is predicted by inferring the movement of the camera.

[0084] In step 62, a new virtual reference viewpoint is determined according to the predicted camera trajectory. A multi-view viewpoint on the camera trajectory or a viewpoint close to the camera trajectory may be selected.

[0085] A virtual shadow map can then be determined and stored for future use.

[0086] Figure 6B An exemplary method for determining a virtual reference viewpoint according to another embodiment of the present invention is shown. According to this embodiment, the virtual reference viewpoint is dynamically determined to anticipate Figure 4 The disclosed system may require the fact of a new virtual reference viewpoint.

[0087] The new reference viewpoint can be defined as one of the last real input video camera viewpoint, or the current real input video camera viewpoint, or a virtual viewpoint corresponding to the average position and direction of the last real input video camera viewpoint, for example. Since calculating the corresponding virtual shadow map is time-consuming, it needs to be anticipated.

[0088] In step 63, the camera pose is continuously tracked.

[0089] In step 64, it is checked whether the camera is moving away from the current reference viewpoint by a distance above a threshold. For example, the distance between the current virtual reference viewpoint and the camera viewpoint is determined and it is checked whether this distance is above a determined value. According to a variant, it can be checked whether the number of overlapping pixels in the image captured from the current virtual reference viewpoint and the image captured from the camera's viewpoint is below a determined value. In this case, it is considered that the camera has moved away from the current reference viewpoint by a distance above the threshold. Such overlap measurements on consecutive viewpoints can allow evaluating trends and predicting the evolution of overlap measurements in order to make better decisions with respect to the possible introduction of new reference viewpoints.

[0090] If this is the case, a new reference viewpoint is determined, for example at the current real camera pose, in step 65. A virtual shadow map associated with the new reference viewpoint is calculated.

[0091] While defining this new reference viewpoint, the current reference virtual viewpoint is still used to determine the lighting parameters. Therefore, the threshold used in step 64 is selected so that outside this threshold, the current reference viewpoint can still be used while calculating the virtual shadow map of the new reference viewpoint.

[0092] When the new reference viewpoint is ready, in step 66, the poses of the two reference viewpoints (the current reference viewpoint and the new reference viewpoint) are compared with the current real camera pose, and the reference viewpoint closest to the current real camera pose is selected. Figure 4 The selected reference viewpoint is used in the disclosed process for illumination parameter determination.

[0093] The reference virtual viewpoints and their corresponding shadow maps may also be stored for later use, so that they enrich the set of available reference virtual viewpoints, at step 67. In this case, the selection is hybrid: if a satisfactory virtual viewpoint exists, a reference virtual viewpoint is selected among the available virtual viewpoints, or a new virtual viewpoint is calculated and stored.

[0094] The virtual reference viewpoint may be selected to be close to the area where the video camera moves in order to minimize the effect of errors on the 3D position of the virtual reference viewpoint and, therefore, on the registered real shadow map. For example, it may be a real viewpoint that is selected once it is captured. The selected virtual viewpoint may then be maintained as long as, for example, the overlap between the real viewpoint image and the virtual viewpoint image is greater than a percentage of pixels.

[0095] The condition for changing the virtual reference viewpoint is that the system has time to generate a new virtual shadow map corresponding to the next virtual reference viewpoint.

[0096] The creation of the virtual map for a new virtual reference viewpoint can also be performed in a coarse-to-fine mode by first ensuring that the 3D space is covered by distant light positions and then gradually introducing new light positions. In this mode, once a new virtual viewpoint is selected, the determination of the lighting parameters is first performed using a subset of the virtual shadow map, which may result in a coarse estimate of the light position. Then, as new virtual shadow maps are created, the estimate can be made more accurate.

[0097] In the above description, it has been assumed that the geometry of the observed 3D scene has not changed significantly. On the other hand, if a part of the scene is modified (e.g. an object is moved, inserted or removed), the scene geometry changes should be detected and taken into account for calculating the shadow map associated with the virtual reference viewpoint and determining the lighting parameters.

[0098] Figure 7 An exemplary method for determining lighting parameters when the geometry of a scene changes according to an embodiment of the present disclosure is shown.

[0099] In step 70, geometry changes in the scene are detected. For example, the current geometry of the scene is captured and compared to a currently stored 3D model.

[0100] If there is no significant change in geometry, then at step 75, use Figure 4 The discussed method performs lighting parameter determination and it is important to understand here that geometry changes have an impact on the virtual cast shadows associated with the current virtual reference viewpoint.

[0101] If there is a significant change in the geometry, the new 3D model and the currently stored 3D model are projected onto the current viewpoint in step 71. 2D regions affected by the modification in the 3D scene are identified. These 2D regions are discarded in the input image of the viewpoint, for example by defining a mask identifying the discarded regions.

[0102] In step 72, using a mask, using Figure 4 The discussed method performs the lighting parameter determination such that no discarded 2D regions are analyzed in the input image.

[0103] At the same time, in step 73, the geometric model is updated.

[0104] Once the geometric model is updated, at step 74 a new virtual shadow map is calculated in the virtual reference viewpoint.

[0105] Once the new virtual shadow map is ready, the 3D model is updated for all tasks (including the processing of the input frame) using the updated virtual reference viewpoint and removing the old stored virtual reference viewpoint. Thus, the management of the virtual reference viewpoint is reinitialized.

[0106] Then, in step 75, use Figure 4 The method in question performs the lighting parameter determination without the mask defined in step 71 , since the management of the virtual reference viewpoint has been reinitialized.

[0107] Figure 8 An exemplary apparatus for processing a 3D scene according to an embodiment of the present disclosure is shown. Figure 8 A block diagram of an exemplary system 800 in which various aspects of the exemplary embodiments may be implemented is shown. The system 800 may be implemented as a device including the various components described below and configured to perform the above process. Examples of such devices include, but are not limited to, mobile devices, personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The system 800 may be communicatively coupled to other similar systems and coupled to a display via a communication channel.

[0108] Various embodiments of system 800 include at least one processor 810, which is configured to execute instructions loaded therein, for implementing various processes as described above. Processor 810 may include embedded memory, input and output interfaces, and various other circuits known in the art. System 800 may also include at least one memory 820 (e.g., volatile memory device, non-volatile memory device). System 800 may additionally include storage device 840, which may include non-volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive and / or optical drive. As a non-limiting example, storage device 840 may include internal storage device, attached storage device and / or network accessible storage device.

[0109] Program code to be loaded onto the one or more processors 810 to perform the various processes described above may be stored in the storage device 840 and subsequently loaded onto the memory 820 for execution by the processor 810. According to an exemplary embodiment, one or more of the processor(s) 810, the memory 820, and the storage device 840 may store one or more of the various items during execution of the processes discussed above, including, but not limited to, real and virtual cast shadow maps, input images, 3D scene geometry, 3D poses of virtual reference viewpoints, variables, operations, and operation logic.

[0110] The system 800 may also include a communication interface 850 that enables communication with other devices via a communication channel. The communication interface 850 may include, but is not limited to, a transceiver configured to send and receive data from the communication channel. The communication interface 850 may include, but is not limited to, a modem or a network card, and the communication channel 850 may be implemented in a wired and / or wireless medium. The various components of the system 800 may be connected or communicatively coupled together (not shown) using various suitable connections (including, but not limited to, internal buses, wires, and printed circuit boards).

[0111] The system 800 further includes a user interaction device 830 coupled to the processor for receiving user input.

[0112] The system 800 also includes a video capture tool 860 (eg, a camera) coupled to the processor for capturing video images.

[0113] The exemplary embodiments may be performed by computer software implemented by a processor 810, or hardware, or a combination of hardware and software. As a non-limiting example, the exemplary embodiments may be implemented by one or more integrated circuits. The memory 820 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 810 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0114] The implementations described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). For example, the device may be implemented with appropriate hardware, software, and firmware. The method may be implemented in, for example, a device (such as a processor), which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, cellular phones, portable / personal digital assistants ("PDAs"), tablet computers, and other devices that facilitate information communication between end users.

[0115] A method for processing a 3D scene is disclosed. The method includes: determining at least one virtual reference viewpoint in the 3D scene; obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera at a viewpoint different from the virtual reference viewpoint, the map of real cast shadows being registered relative to the virtual reference viewpoint; determining parameters for at least one light source in the 3D scene using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by at least one light source from the virtual reference viewpoint.

[0116] According to the present principles, a virtual reference viewpoint is determined for determining lighting parameters. The virtual reference viewpoint may be selected to be close to the area where the video camera moves so as to minimize the impact of errors on the 3D position of the virtual reference viewpoint and, therefore, minimize the impact of errors on the registered real shadow map. For example, it may be a real viewpoint that is selected once it is captured.

[0117] Then, as long as the overlap between, for example, the camera's viewpoint and the virtual viewpoint image is greater than a percentage of pixels, the selected virtual viewpoint can be maintained. Thus, the complexity of determining online lighting parameters is reduced and online 3D rendering processing is possible on mobile devices.

[0118] Also disclosed is an apparatus for processing a 3D scene. The apparatus includes: means for determining at least one virtual reference viewpoint in the 3D scene; means for obtaining a map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera at a viewpoint different from the virtual reference viewpoint, the map of real cast shadows being registered relative to the virtual reference viewpoint; means for determining parameters for at least one light source in the 3D scene using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene projected by at least one light source from the virtual reference viewpoint.

[0119] According to an embodiment of the present disclosure, obtaining the registered real projected shadow map includes: detecting the real projected shadow in the input image to obtain the detected real projected shadow map; and registering the detected real projected shadow map relative to the virtual reference viewpoint.

[0120] According to an embodiment of the present disclosure, obtaining the registered real projected shadow map includes: registering the input image relative to the virtual reference viewpoint to obtain a registered image; and detecting the real projected shadow in the registered image.

[0121] According to an embodiment of the present invention, using a registered map of real projected shadows and at least one map of virtual shadows of objects in the 3D scene projected by the at least one light source from the virtual reference viewpoint includes: matching the registered map of real projected shadows and at least one map of virtual shadows; and selecting the map of virtual shadows that best matches the registered map of real projected shadows.

[0122] According to an embodiment of the present disclosure, the virtual reference viewpoint corresponds to a viewpoint from which a camera has previously captured a real scene represented by a 3D scene.

[0123] According to an embodiment of the present disclosure, a virtual reference viewpoint in a 3D scene is determined as a viewpoint for which the number of pixels overlapping in an image captured from the virtual reference viewpoint and an image captured from a viewpoint of a camera is greater than a determined value.

[0124] According to an embodiment of the present disclosure, when a virtual reference viewpoint and a viewpoint of a camera do not satisfy a determination criterion, a new virtual reference viewpoint is determined.

[0125] According to an embodiment of the present disclosure, if the number of pixels overlapping in an image captured from a virtual reference viewpoint and an image captured from a viewpoint of a camera is lower than a determined value, the virtual reference viewpoint and the viewpoint of the camera do not satisfy a determination criterion.

[0126] According to an embodiment of the present disclosure, when the distance between the virtual reference viewpoint and the viewpoint of the camera is greater than a determined value, the virtual reference viewpoint and the viewpoint of the camera do not satisfy a determination criterion.

[0127] According to an embodiment of the present invention, the method for processing a 3D scene also includes: comparing the postures of the virtual reference viewpoint and the new virtual reference viewpoint with the camera posture; selecting a virtual reference viewpoint closest to the camera viewpoint among the virtual reference viewpoint and the new virtual reference viewpoint, and the light source parameters in the 3D scene are determined relative to the selected virtual reference viewpoint.

[0128] According to this embodiment, new virtual reference viewpoints are pre-calculated, stored and selected when needed.

[0129] According to an embodiment of the present disclosure, determining the at least one virtual reference viewpoint in the 3D scene includes: detecting the movement of a camera that captures a real scene; predicting the camera trajectory in the 3D scene based on the detected movement, and the at least one virtual reference viewpoint is determined based on the predicted camera trajectory.

[0130] According to this embodiment, by anticipating the movement of the user and the movement of the camera, a new virtual reference viewpoint can be predetermined. Thus, the calculation of the virtual shadow map relative to the new virtual reference viewpoint can be performed in advance before the new virtual reference viewpoint is needed.

[0131] According to an embodiment of the present disclosure, determining the at least one virtual reference viewpoint in the 3D scene includes: determining and storing at least one shadow map associated with a light source of the 3D scene. According to this embodiment, before a virtual reference viewpoint associated with the virtual shadow map is needed, the virtual shadow map can be dynamically calculated and stored for future use. In this way, virtual shadow map calculations can be anticipated.

[0132] According to an embodiment of the present disclosure, the method for processing a 3D scene also includes: detecting geometric structure changes in the 3D scene; determining an area in an input image captured by a camera that is affected by the geometric structure changes, and discarding the determined area when using a virtual reference viewpoint to determine parameters for at least one light source in the 3D scene until a virtual shadow map associated with the virtual reference viewpoint is updated based on the geometric structure changes.

[0133] According to this embodiment, geometry changes are taken into account when determining the lighting parameters.

[0134] One or more of the embodiments of the present invention further provides a computer-readable storage medium having stored thereon instructions for processing a 3D scene according to any of the embodiments of the method described above. The embodiments of the present invention further provide a computer program product including instructions for executing any of the embodiments of the described method.

[0135] The implementation described herein can be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). For example, the device can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, an apparatus such as a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as smart phones, tablet computers, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0136] The embodiments of the various processes and features described herein may be embodied in a variety of different devices or applications, in particular, for example, devices or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such devices include encoders, decoders, post-processors for processing outputs from decoders, pre-processors for providing inputs to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptop computers, personal computers, cellular phones, PDAs, and other communication devices. It should be clear that the device can be mobile and can even be installed in a mobile vehicle.

[0137] In addition, the method may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by the implementation) may be stored on a processor-readable medium, such as an integrated circuit, a software carrier, or other storage device, such as a hard disk, a compact disk ("CD"), an optical disk (such as a DVD, commonly known as a digital versatile disk or a digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application program tangibly embodied on a processor-readable medium. The instructions may be, for example, hardware, firmware, software, or a combination thereof. The instructions may be found, for example, in an operating system, a separate application program, or a combination of both. Thus, a processor may be characterized as, for example, a device configured to perform a process and a device including a processor-readable medium (such as a storage device) having instructions for performing the process. In addition, the processor-readable medium may store data values ​​produced by the implementation in addition to or in lieu of the instructions.

[0138] As will be appreciated by those skilled in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the grammar of the described embodiment, or to carry as data the actual grammar values ​​written by the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0139] Many implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. In addition, it will be appreciated by those skilled in the art that other structures and processes may replace the disclosed content, and the resulting implementations will perform at least substantially the same function (one or more) in at least substantially the same manner (one or more) to achieve at least substantially the same result (one or more) as the disclosed implementation. Therefore, the present application contemplates these and other implementations.

Claims

1. A method comprising: Tracking a current viewpoint of a camera capturing at least one object in a 3D scene; determining a reference viewpoint relative to the current viewpoint of the camera; as well as Determine the location of the light source, including: obtaining a map of registered real cast shadows of said at least one object from an input image captured by said camera, wherein said map is registered with respect to said reference viewpoint, For a light source candidate, obtaining a map of a corresponding virtual shadow of the at least one object created relative to a reference viewpoint, and determining the position of the light source based on the light source candidates having corresponding maps of virtual shadows that match the registered map of real cast shadows, wherein determining the reference viewpoint comprises selecting the reference viewpoint closest to the current viewpoint of the camera from a plurality of reference viewpoints, and Wherein obtaining the corresponding virtual shadow map comprises retrieving the corresponding virtual shadow map pre-calculated for the selected reference viewpoint from a memory.

2. The method according to claim 1, further comprising: The obtained map of corresponding virtual shadows of the reference viewpoints is stored.

3. The method of claim 1 , wherein if a previous reference viewpoint previously determined for the tracked viewpoint of the camera satisfies a determination criterion: The determined reference viewpoint is the previous reference viewpoint; and The obtained corresponding map of virtual shadows is a map of virtual shadows previously obtained for the previous reference viewpoint. 4 . The method of claim 3 , wherein the determination criterion is satisfied if the distance between the current viewpoint of the camera and the previous reference viewpoint is below a determined value. 5 . The method of claim 3 , wherein the determination criterion is satisfied if the number of corresponding pixels projected from the current viewpoint of the camera and from the previous reference viewpoint is above a determined value.

6. The method according to claim 1, further comprising: Predicting the trajectory of the tracked camera; determining another reference viewpoint relative to the predicted trajectory of the camera; as well as For the light source candidate, a map of a corresponding virtual shadow of the at least one object created relative to the another reference viewpoint is obtained.

7. The method of claim 1, wherein obtaining the registered true cast shadow map comprises: Detecting the real cast shadow from the input image to obtain a map of the real cast shadow; as well as A map of the real cast shadows is projected relative to the reference viewpoint.

8. The method of claim 1, wherein obtaining the registered true cast shadow map comprises: projecting the input image relative to the reference viewpoint to obtain a registered image; as well as The real cast shadow is detected from the registered images.

9. The method according to claim 1, further comprising: detecting changes in the geometry of the 3D scene, and Regions of the input image affected by the detected changes are determined, wherein the determined regions are discarded from the map of virtual shadows during determination of the light source position.

10. A device comprising: at least one processor; as well as A memory storing instructions which, when executed by the at least one processor, cause the apparatus to perform a method according to any one of claims 1-9.

11. A non-transitory computer readable medium comprising instructions executable by at least one processor to perform the method according to any one of claims 1-9.

Images