Cloud-based camera calibration

Abstract

A processor system (200) processes image data from a camera (20) to render a virtual object in a 3D virtual environment, and communicates with a server system (300) via a network. A processor (220) obtains image data of a physical space from the camera and obtains second data of a second device (30). The processor system sends the image data, the second data, and a calibration command to the server system. In the server, a processing resource (320) processes the data according to the calibration command and sends calibration data indicating the camera posture and the second posture of the second device. The processor generates 3D object data of the object by processing the image data based on the calibration data. The 3D object data is provided for rendering the virtual object in the 3D virtual environment. Therefore, the processor is in control, and the calibration is performed by the server.

Description

Technical Field

[0001] The invention relates to a processor system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment, the virtual object representing an object in the physical space, the processor system being arranged to communicate with a server system comprising processing resources via a network. The invention further relates to a server system, a processing method and a computer program comprising instructions for causing the processor system to perform these methods.

[0002] The present invention relates to the field of providing a virtual reality (VR) environment in which one or more users or other objects present in a physical space (such as a room) can be rendered as virtual objects. Such an environment may be referred to as a social VR environment, wherein these objects are captured by one or more cameras. For various image processing, mainly for creating a 3D model of the user to be used for a so-called self-view (the self-view is a virtual 3D image of the user himself in a 3D virtual environment), camera calibration is required with respect to the camera pose (i.e., the position and orientation of the camera in physical space or relative to another object, camera or device). Various cameras or devices in the physical space may have their own position and / or axial references for position and orientation, which may be mapped on the physical space and / or relative to each other during the calibration process. Background Art

[0003] Calibration of camera(s) used to capture objects such as people in a social VR environment can be quite complex, especially in an automated process that needs to handle multiple cameras. Some challenges are:

[0004] -Using a single image and depth camera: to detect a person (e.g., a head or VR headset) to correctly place the person's virtual self-view in the VR environment and to correctly position the virtual user in the VR environment. Also, the room's landmarks (e.g., floor or chair) need to be detected to correctly place the user in the remote user's VR environment. For example, when recording a person in a chair, the captured chair needs to be properly placed on the floor in the virtual environment;

[0005] - Using multiple image and / or depth cameras: Different cameras need to be aligned for capturing together, e.g. capturing a user from different angles to create a better 3D capture of the user. This alignment can be done by point matching of at least 3 points using depth-based calibration;

[0006] -Using multiple image-only (non-depth) cameras: Creating a depth image from multiple regular RGB cameras requires complex stereo image point matching on each frame;

[0007] - In order to accurately display all camera images in a geometric VR space, camera characteristics including camera focal length and lens distortion are required, which can be detected with the help of vision patterns, for example.

[0008] Thus, camera calibration is known in the art. However, one problem is that such calibration requires considerable processing and may even require dedicated hardware, such as a graphics processing unit GPU (see Section IV.B.2 of reference [1]) or an FPGA (see reference [2]).

[0009] From a processing perspective, it may make sense to perform the processing remotely (e.g., in the "cloud") somewhere where specific hardware is available. Examples of performing such processing on a system different from the capture system (e.g., in a multi-camera sensor system) are provided in references [3] and [4]. Summary of the invention

[0010] Prior art methods can achieve the transfer of image processing tasks to remote processing resources. However, this cannot solve the situation where the processing results are needed quickly locally. However, this situation may be hindered by limited local processing capabilities, limited network connection capabilities, delays caused by image data transmission, or may cause additional battery consumption when using wireless systems. For example, a local image processing system (i.e., a PC) may include one or more (depth) cameras and may need to use captured images locally. Other examples of local processing using a local camera include a smartphone, a smartphone with 1 or more external cameras, or two smartphones that may be directly connected (one of which serves as the main capture system). The external camera or smartphone can be connected, for example, via a Bluetooth connection.

[0011] The need to use local processing may arise due to end-to-end latency. For example, see Section 4.1.6 of [4], where all video is sent to the network, processed there and then sent on. This transmission may incur additional latency, for example, due to additional encoding / decoding steps in the end-to-end video transmission chain. If the local system constructs the required output (e.g., 3D point cloud or mesh), this will avoid or reduce end-to-end latency.

[0012] An exemplary purpose of using images locally is to create a self-view, which is a 3D representation of the user's own body, which is usually visualized through a VR head-mounted device (also known as an HMD, head-mounted display). In this document, the user (e.g., the user's body or head) captured by the camera or any other element in the room or location can be named as: an object in the physical space. Data corresponding to such an object that can be visually represented as a virtual object in a VR environment can be referred to as 3D object data. The self-view is an example of a virtual object that corresponds to an actual user in a room in which a camera is placed. The self-view in the virtual environment should preferably be well aligned with the user's body to ensure that the user's proprioceptive experience matches the visual experience. Moreover, the self-view may include views of other local physical objects (such as chairs, coffee cups, etc.). Another purpose of the 3D object data is for local use in augmented reality, that is, for creating a local 3D model of the room and rendering virtual objects in the virtual room, for example, in a VR / AR hybrid scene where some users wear AR head-mounted devices and see other users projected in their own physical environment. In this document, this AR environment including virtual objects may also be referred to as a 3D virtual environment.

[0013] Therefore, there may be a need for a system that locally provides 3D object data for rendering virtual objects in a 3D virtual environment while reducing the need for local processing power.

[0014] According to a first aspect of the present invention, there may be provided a processor system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment,

[0015] The virtual object represents an object in the physical space.

[0016] The processor system is arranged to communicate with a server system comprising processing resources via a network, wherein the processor system may include:

[0017] - a network interface for exchanging data via the network;

[0018] - a capture interface to the camera;

[0019] - a second device interface to a second device in the physical space;

[0020] The camera has a camera pose in the physical space, and the second device has a second pose in the physical space, the second pose being different from the camera pose; and

[0021] - a processor that can be configured to

[0022] - obtaining image data of the physical space from the camera via the capture interface;

[0023] - obtaining second data of the second device via the second device interface;

[0024] - sending the image data, the second data and the calibration command to the server system;

[0025] - receiving calibration data from the server system according to the calibration command, the calibration data indicating the camera pose and the second pose;

[0026] - generating 3D object data of the object by processing the image data based on the calibration data, the 3D object data being provided for rendering the virtual object in the 3D virtual environment.

[0027] According to another aspect of the present invention, a server system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment may be provided.

[0028] The virtual object represents an object in the physical space.

[0029] The server system is arranged to communicate with the processor system via a network,

[0030] The server system includes

[0031] - a network interface for exchanging data via the network, and

[0032] - Processing resources, which can be configured as

[0033] - receiving, via the network interface, image data of the physical space obtained by the camera, second data of a second device in the physical space, and a calibration command from the processor system;

[0034] - processing the image data and the second data according to the calibration command to generate calibration data indicating the camera pose and the second pose; and

[0035] - sending the calibration data to the processor system via the network interface.

[0036] According to another aspect of the present invention, there is provided a processing method for a processor system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment.

[0037] The virtual object represents an object in the physical space.

[0038] The processor system may be arranged to communicate with a server system via a network, the camera having a camera pose in the physical space, and the second device having a second pose in the physical space, the second pose being different from the camera pose.

[0039] The method may include:

[0040] - obtaining image data of the physical space from the camera;

[0041] - obtaining second data of the second device;

[0042] - sending the image data, the second data and the calibration command to the server system;

[0043] - receiving calibration data from the server system according to the calibration command, the calibration data indicating the camera pose and the second pose; and

[0044] - generating 3D object data of the object by processing the image data based on the calibration data, the 3D object data being provided for rendering the virtual object in the 3D virtual environment.

[0045] According to another aspect of the present invention, there is provided a processing method for a server system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment.

[0046] The virtual object represents an object in the physical space.

[0047] The server system may be arranged to communicate with the processor system via a network, wherein the method comprises:

[0048] - receiving image data of the physical space obtained by the camera, second data of a second device in the physical space, and a calibration command from the processor system;

[0049] - processing the image data and the second data according to the calibration command to generate calibration data indicating the camera pose and the second pose; and

[0050] - sending the calibration data to the processor system.

[0051] In addition, a transient or non-transient computer-readable medium including a computer program is provided, the computer program including instructions for causing a processor to perform one or both of the above two methods. Furthermore, a signal data to be transmitted between the above processor system and the above server system is provided, the signal data being configured to carry the calibration command or the calibration data.

[0052] The measures in the various systems and methods described above can have the following effects. The process can be constructed as follows. First, a local processor system (also referred to as a client) sends image data and calibration instructions to a server. In the present disclosure, the term 'server' may be used, and the server may include any one or more server systems in a network that have processing resources capable of performing calibration, including systems that include dedicated hardware for image processing. This may include, but is not limited to, cloud computing, edge computing, fog computing, and mist computing, and may also include another local computer with sufficient processing power.

[0053] On the server, the image data is analyzed according to the calibration instructions to determine the calibration data. The server then sends the calibration data to the client. Finally, the client uses the calibration data to process the image data to generate 3D object data. Note that the image data processed on the client can be different data from the image data sent to the server (e.g., future captured frames). Therefore, the processor system locally controls the capture and processing of image data from the camera to obtain 3D object data, which enables the use of 3D object data in real time. Calibration is a complex operation that typically uses complex algorithms to extract the required calibration data from visual images, and is performed remotely on the server under the control of the client according to the calibration commands. Therefore, advantageously, performing calibration and optional related processing (such as, for example, monitoring) on ​​the server offloads a lot of work from the local system.

[0054] In practice, the local client delegates the calibration to a server system in the network. Depending on latency requirements, this could be to a system at the edge of the network (e.g. 5G edge computing) or to any system in the network (e.g. typically 'in the cloud').

[0055] Advantageously, the total processing load is divided into a relatively complex remote part and a time-critical local part. The local part is to generate 3D object data for the object by processing the image data based on the calibration data. In the remote part at the server system, the calibration data is generated by instructing the server system. To this end, a calibration command and the corresponding image data and second data are sent to the server system. Subsequently, the server system performs the complex calibration task and sends the calibration data back to the local processing system. Advantageously, the processing of the video data to obtain the 3D object data is performed locally while using the remotely generated calibration data.

[0056] Additionally, there may be some delay in obtaining the calibration data, for example, in the case of recalibration due to a change in camera configuration. This may temporarily cause some misalignment until updated calibration data has been generated and transmitted to the client. However, advantageously, the real-time behavior of the 3D object data still closely follows the actual physical object, because the local processor is not slowed down by performing the calibration.

[0057] The second device may be a head mounted display (HMD) or a user interaction device for sensing the movement of a user in a physical space or any other reference device. The HMD may render a virtual 3D environment. In an embodiment of the processor system, the processor is configured to obtain data about the position or orientation of the second device in the physical space as the second data. The data may include at least one of the following:

[0058] - the displacement distance of the second device;

[0059] - the displacement direction of the second device;

[0060] - the rotation angle of the second device;

[0061] - A rotation direction of the second device. The displacement distance or displacement direction may be defined with reference to a reference point or a reference plane. The rotation angle or rotation direction may be defined with reference to a rotation axis. Advantageously, the spatially specific second data about the position, orientation or movement of the second device may be processed and matched with the image data of the camera in order to enhance the calibration data about the pose of the camera and the pose of the second device.

[0062] In a practical case, the second device may be an HMD for rendering a virtual 3D environment, and the object may be a person wearing the HMD in a physical space. The processor may be configured to generate the position data and / or orientation data of the person as the 3D object data by processing the image data to determine the posture of the head mounted display, and the 3D object data is provided for rendering a self-view as a virtual object in the 3D virtual environment. The HMD may report its own axis and orientation (i.e., it has its own (0,0,0) point and orientation angle (usually horizontal direction + north direction), i.e., data from the HMD about its relative position and orientation in the physical space, as the second data. At the same time, the image data of the camera is also about the physical space. By determining the posture of the camera and the HMD relative to each other, the calibration associates these two data types together. Advantageously, a person will experience himself in a realistic position and orientation in a 3D virtual environment, because the self-view is positioned and oriented according to the position of the HMD (i.e., the person's head), and the image data is processed to determine the posture of the HMD.

[0063] The second device may include a second camera and / or a depth camera. In an embodiment of the processor system, the processor is configured to obtain at least one of second image data of the physical space and depth data of the physical space as the second data. Advantageously, the other image and / or depth second data may be processed and matched with the image data of the camera to enhance the calibration data about the pose of the camera and the second device.

[0064] In an embodiment of the processor system, the processor may be configured to obtain metadata indicating at least one parameter of the camera or the second device; and include the metadata in the calibration command. Optionally, the metadata includes static device parameters, including at least one of device type, lens parameters, lens distortion data, image sensor resolution, depth sensor resolution, and device data transmission format. Optionally, the metadata may include dynamic device parameters, including at least one of focal length parameters, tilt parameters, pan parameters, camera focus parameters, camera aperture parameters, and synchronization data. Advantageously, the metadata can be applied during calibration to map image data to physical space. Static parameters can be applied to all calibration types, for example, during initial calibration or reference calibration. Dynamic parameters can be used to update calibration when parameters change.

[0065] In fact, the process of calibration can cause problems when starting capture. When setting up one or more capture devices for a session, most calibration needs to be done once at the beginning of the session.

[0066] In an embodiment of the processor system, the processor may be configured to determine a need for reference calibration data, the reference calibration data comprising at least one of the following:

[0067] - a reference view of the physical space without the object;

[0068] - fixed elements in that physical space;

[0069] - Visual markers in that physical space;

[0070] - predefined objects in the physical space;

[0071] And the processor is configured to include in the calibration command a request regarding said need for reference calibration data.Advantageously, the reference calibration data may be used to process the image data, for example, to track elements, markers or objects, or to isolate such elements from physical space in the image data.

[0072] In an embodiment of the processor system, the processor may be configured to determine a need for recalibration after receiving calibration data from an earlier calibration. The need may include one of the following:

[0073] - detecting movement of the camera relative to the camera pose based on the early calibration;

[0074] - detecting movement of the second device;

[0075] - adding another device in the physical space, the other device being able to provide another image data, depth data or data about the position or orientation of the other device in the physical space. Optionally, the processor may be configured to include a request about the need for recalibration in the calibration command. Advantageously, by knowing the specific need, the server may perform efficient processing of the image data and / or the second data and / or the other data while taking into account earlier calibration data.

[0076] In practice, things can change during a session. Cameras can be moved during a session, for example, the user can move them if they find that their placement is not optimal. Movement is also a factor with handheld cameras, for example, using a smartphone as a capture system. Cameras can zoom in or out or pan during a session (i.e., for PTZ (pan-tilt-zoom) cameras). Cameras can be added during a session, for example, the user connects additional cameras to improve the 3D reconstruction. The HMD can drift in its tracking process and therefore needs to be recalibrated over time if the drift becomes too large.

[0077] In an embodiment of the processor system, the processor may be configured to recalibrate the data from the receiving server system after receiving the calibration data of the earlier calibration, and use the recalibrated data to process the image data to obtain the image data of the object. Advantageously, upon receiving the recalibrated data, the processor may immediately replace the earlier calibration data. Moreover, the processor may efficiently process the image data and / or the second data while taking into account the earlier calibration data and the recalibrated data.

[0078] In an embodiment of the server system, the processing resources may be configured to determine the need for recalibration after an earlier calibration. The need may be one of the following:

[0079] - receiving a recalibration command indicating the need for recalibration;

[0080] - detecting movement of the camera relative to the camera pose based on the early calibration;

[0081] - detecting movement of the second device relative to the second posture based on the earlier calibration;

[0082] -Adding another device in the physical space, which is capable of providing another data about the physical space. The processing resource can be configured to: process the image data and the second data to generate recalibration data indicating the camera pose and the second pose; and send the recalibration data to the processor system. In particular, detecting movement may include determining whether the change in pose exceeds a threshold value of a position change, an orientation change, a speed change, or an acceleration change. Advantageously, by understanding specific needs, the server system can efficiently process the image data and / or the second data and / or the other data while taking into account earlier calibration data.

[0083] In an embodiment of the server system, the processing resource may be configured to, upon receiving a monitoring command,

[0084] - monitoring a data stream from the camera or the second device about the physical space;

[0085] - determining a need for recalibration based on the data stream and the calibration data;

[0086] - recalibrating to generate recalibration data for sending the recalibration data to the processor system. For example, the data stream may be generated and transmitted by the processor system for said monitoring, for example, image data streamed at a lower rate (such as once per second), or it may be a data stream transmitted from the processor system to another destination, the data stream being routed via the server system. Advantageously, the monitoring and recalibration may be performed by the server while substantially occupying no processing power on the client side.

[0087] In fact, detecting the need for calibration may be the first step of the calibration process. When additional cameras are added, this is an obvious sign of calibration. But when, for example, the cameras are only moved slightly, this is not so obvious, i.e., it may only lead to small errors in the captured results. Therefore, monitoring the need for calibration can be seen as part of the calibration. Detecting the need for the actual calibration and detecting the actual calibration itself can be a heavy processing load on the local system, which may not be able to do this in real time, which may cause system overload or affect overall performance, and it may drain the battery, or may even cause the local system to (over)heat.

[0088] In an embodiment of the server system, the image data may include a first image at a first moment and a second image at a second moment, and the second data may indicate an actual movement of the second device from a second posture at the first moment to a subsequent second posture at the second moment. In an embodiment, the processing resources may be configured to: detect the actual movement of the second device as represented in the image data, obtain at least one camera parameter from the actual movement as represented, and generate the calibration data using the camera parameter as obtained. Advantageously, the camera parameters can be obtained on the server side while substantially not occupying processing power on the client side.

[0089] In an embodiment of the server system, the processing resource may be configured to determine reference calibration data comprising at least one of the following:

[0090] - a reference view of the physical space without the object;

[0091] - fixed elements in that physical space;

[0092] - Visual markers in that physical space;

[0093] - predefined objects in that physical space,

[0094] The reference calibration data is sent to the processor system. Advantageously, the reference calibration data may be generated upon request or automatically generated on the server side. The processor system may receive the reference calibration data while substantially not occupying processing power on the client side.

[0095] It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and / or aspects of the invention may be combined in any way deemed useful.

[0096] Those skilled in the art can perform modifications and variations of the processor system, device, server system and / or computer program on the basis of this description, which modifications and variations correspond to the modifications and variations of the described method, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. In the drawings:

[0098] Figure 1 An example of a system for rendering a virtual object in a 3D virtual environment is shown.

[0099] Figure 2 An example of a system for rendering a self-view in a 3D virtual environment is shown,

[0100] Figure 3 An example of a system for 3D object creation is shown,

[0101] Figure 4 shows the axial system,

[0102] Figure 5a shows a camera pointed at a user,

[0103] Figure 5b Shows Figure 5a The scene shown in the figure is captured by a camera.

[0104] Figure 5c and Figure 5d The calibration of the camera and HMD is schematically shown.

[0105] Figure 6a The user's self-view and plane AB to be positioned are shown,

[0106] Figure 6b shows the front view of the self image,

[0107] Figure 6c The angular coordinates of A and B are determined as shown,

[0108] Figure 7a A method for a processor system is shown,

[0109] Figure 7b A server method for a server system is shown,

[0110] Figure 8 A transitory or non-transitory computer readable medium is shown; and

[0111] Fig. 9 An exemplary data processing system is shown.

[0112] It should be noted that similar items in different drawings may have the same reference numerals, may have similar structural features, functions or logos. Where the function and / or structure of such an item has been explained, it will not be repeatedly explained in the detailed description. DETAILED DESCRIPTION

[0113] Several embodiments of a processor system and a server system that embody 3D object data generation are described below. First, some further general descriptions of the technical concept of calibration are provided. In this context, calibration is to make image data available in a 3D virtual environment. Examples of image data that need to be calibrated and enable retrieval of 3D objects therefrom are:

[0114] - Single RGB image + depth frame, calibrated relative to the physical space of the image. A HMD as a second device may be present in the room. This may be sufficient to detect the HMD position in the image, and therefore allow the self-view to be properly placed in the 3D virtual environment.

[0115] - Multiple RGB+depth images at a single instant, i.e. from different cameras. Such data allows to align the cameras using at least 3 points (i.e. using depth-based multi-camera calibration).

[0116] -Multiple RGB images at a single moment in time, using vision-based methods to align the camera images.

[0117] To detect the need for (re)calibration, new frames are needed over time to detect changes in the image, e.g. for detecting camera movement. In practice, the client may also send a stream containing images via the network, e.g. to other clients, and such a stream may also be used to detect the need for calibration. Alternatively, e.g. in case of local use (such as for self-view), the client may send regular image data updates to the server for this purpose.

[0118] Multi-camera calibration may require time synchronization between camera images, for example, when multiple mobile phones are used as cameras. In a distributed camera network, this problem can be solved using synchronization techniques known per se, for example, using clock synchronization between systems and timestamping of images. When multiple cameras are connected to a single local system, time synchronization can be arranged at a local level. For example, a local capture system can capture images (RGB and / or depth) from multiple cameras simultaneously.

[0119] In this document, the concept of a network includes various types of networks, both local, regional, and global. A network may include one or more network portions (such as a home network, a corporate network, a network domain under the control of a specific service provider such as an Internet Service Provider (ISP)), and may include both fixed (i.e., wired) and wireless (e.g., Wi-Fi or cellular) connections. Such a network may include a large number of network resources and optionally network controllers, which include servers, nodes, and links connecting these nodes, and which have network controller interfaces for exchanging network control data. A network may be configured to transmit data via a chain of network resources between a first end node and a second end node, while routing data streams via a specific server. It is well known that each end node may have a network interface and another control logic for exchanging data via a network. A server system may be coupled to a network near a user's location to provide so-called edge computing that further limits transmission delays, or may be coupled at a remote location or node in the network.

[0120] Figure 1 An example of a system for rendering a virtual object in a 3D virtual environment is shown. The system includes a processor system 200, which may be referred to as a client, and a server system 300, which may be referred to as a server. The processor system 200 is arranged to process image data 22 from a camera 20 in a physical space to render a virtual object in a 3D virtual environment. The virtual object represents an object in the physical space. The processor system is arranged to communicate with the server system 300 via a network 100. The processor system has: a network interface 240 for exchanging data 24 via the network 100; a capture interface 210 to the camera 20; and a second device interface 230 to a second device 30 (e.g., a head mounted display (HMD)) in the physical space. The camera has a camera pose in the physical space, and the second device has a second pose in the physical space. The second pose is different from the camera pose and may be any pose that defines the absolute position and orientation of the device in the physical space or the position and orientation relative to another object, camera, or device.

[0121] The processor system has a processor 220, which is configured to obtain image data 22 of a physical space via a camera 20, and obtain second data 23 via a second device 230. The processor is also arranged to send the image data, the second data and a calibration command to a server system 300 via a network 100. The processor is also arranged to receive calibration data from the server system 300 according to the calibration command. The calibration data indicates a camera pose and a second pose. The processor is also arranged to generate 3D object data 25 of an object by processing the image data based on the calibration data. The 3D object data is provided for rendering a virtual object in a 3D virtual environment. For example, the 3D object data can be used to render a virtual object in a virtual environment in the HMD 30, which rendering can also be performed by the processor 220. Alternatively, the 3D object data 25 can be transmitted to another device, for example, a client at a different location, via the network 100. At the different location, a 3D virtual environment including a 3D virtual object can be rendered.

[0122] Figure 1A server system 300 is also shown. The server system is arranged to process image data from a camera in a physical space to render virtual objects in a 3D virtual environment. The server system is arranged to communicate with a processor system 200 as described above via a network 100. The server system has a network interface 310 and a processing resource 320, the network interface being used to exchange data 31 via a network. The processing resource is configured to receive image data, second data, and calibration commands from the processor system 200. The processing resource is also configured to process the image data and the second data according to the calibration command to generate calibration data indicating the camera posture and the second posture, and send the calibration data to the processor system. The processing resource may have an image processor 330, which includes dedicated hardware for high-speed processing of image data.

[0123] Advantages of the above system include offloading the processing of the client to the server, thereby reducing the burden on the client. The server can have specific hardware 330 (e.g., GPU or FPGA hardware) to accelerate processing, so that the processing can be done more efficiently than the client can do. Moreover, since the server can be calibrated faster than the client, and the network latency is smaller (i.e., a particular advantage when using 5G edge computing), this arrangement can save time. Moreover, this arrangement can save battery power for mobile / battery-powered clients.

[0124] In an embodiment, the second device is a head mounted display (HMD). The HMD may be used to render a 3D virtual environment. Alternatively, the second device may be a user interaction device for sensing the movement of a user in a physical space, for example, a game movement sensor to be manipulated by a user. The processor is further configured to obtain data about the position or orientation of the second device in the physical space as second data. In an embodiment, the second data may be a displacement distance of the second device or a displacement direction of the second device. Moreover, the second data may be a rotation angle of the second device or a rotation axis of the second device. Moreover, the second data may be a combination of these. Note that the second data is data reported by or about the second device and is therefore independent: it may not be data about an absolute position and orientation in a physical space, but may be data about the position and orientation of the second device relative to an (internally determined) axis system. For example, when the HMD is first turned on, its first position may be determined as its (0,0,0) point and its first orientation may be determined as its "north orientation" or "0 orientation". Note that the HMD typically has an internal inertial sensor that can determine whether the HMD is in a horizontal orientation.

[0125] Figure 2An example of a system for rendering a self-view in a 3D virtual environment is shown. The structure of the client system 201 and the server system 301 may correspond to the above Figure 1 , and the figure schematically shows the data to be transmitted and the functions to be performed. In this example, the user wears an HMD connected to a client system (PC), to which an RGB+depth camera (e.g., Microsoft's Kinect or Intel's RealSense D415) is also connected. This example shows how a server can be used to calibrate the position of the HMD relative to the camera so that the created self-view can be correctly positioned in the virtual environment. Calibration can include spatially matching depth information with RGB information.

[0126] The client system 201 has a capture interface coupled to one or more cameras. A head mounted display is shown as a second device. In this example, the subject is a person wearing a head mounted display and sitting on a chair in a physical space. The client transmits image data (e.g., RGB and depth data) and calibration commands (CMD) to the server 301.

[0127] The server 301 receives the calibration command, as well as the image data and HMD data, such as the position and orientation of the HMD. The server then performs calibration according to the calibration command.

[0128] At the client 201, the processor is configured to generate the position data and / or orientation data of the person as 3D object data. To this end, the client receives calibration data from the server, for example, as detected by the server and subsequently transmitted to the HMD posture of the client, as shown in the figure. Now, the processor in the client is enabled to process the captured image data to determine the actual posture (i.e., 3D object data) of the head mounted display by using the calibration data. Based on this, the self-view is created using the 3D object data rendering as a self-view of a virtual object in a 3D virtual environment. Moreover, the client can track the movement of the HMD to adjust the posture of the virtual object (i.e., the self-view) in the 3D virtual environment. The client can store the position data and / or orientation data of the HMD when capturing an image or an image sequence, and the HMD data can be used for calibration.

[0129] The captured image containing the RGB image and the depth image and / or HMD data is sent to the server. The server can use a vision-based method to detect the HMD in the image. This can be facilitated by using additional markers on the HMD (e.g., a QR code-like marker) to make detection easier.

[0130] The server may need to detect both the position and orientation of the HMD in the image. The position may consist of a placement (X, Y coordinates) in the image frame and a depth value. The orientation may consist of a vector or of rotation values ​​on various axes. The axis system may be such that the direction pointing directly at the camera is the north direction. The detected HMD pose (position and orientation) is sent back to the client to correctly place the self-view. In order to correctly position the self-view based on the alignment of the reported HMD pose with the detected pose, the size and shape of the HMD may be required. For brevity, the size and shape of the HMD are omitted here, and a single position / orientation is used.

[0131] Further, for the self-view, the captured image and the correct metric size of the person or object in the image may be required. The captured 2D RGB image can be converted into a 3D point cloud together with the captured depth image. In virtual reality, this point cloud can be positioned at the exact user position based on the detected HMD position / orientation. In order to correctly determine the size of the virtual self-view, more information can be used, for example, as explained in application EP16201955.8. For example, this information can be obtained as follows:

[0132] - By 'measuring' objects in the image based on camera characteristics (ie focal length, sensor size) and the measured depth of the objects in the captured image:

[0133]

[0134] - By knowing the exact dimensions of the objects in the image. For example, if the client tells the server the brand and model of the HMD, and the exact dimensions and shape of the HMD are known at the server, the server can use this information to determine the camera characteristics and therefore also measure other objects in the image;

[0135] - By combining the tracking information across the various video frames with the detected position and orientation. If the captured user moves 50cm to the right, the HMD tracking on the client will track this. Knowing this movement will allow the server to measure this movement between the captured images and therefore also allow the camera characteristics to be determined. This requires the client to report this movement to the server based on the HMD tracking data.

[0136] The above calibration information can be used to correctly determine the size of the self-view, but can be further used to keep the placement correct when the user moves. If the user moves 50cm to the right, the self-view created using the captured RGB+depth image should remain correctly placed. The movement information can be obtained locally, for example, the client can obtain this information from the camera. Or it can be detected during calibration, and then this information must be sent to the client together with the detected position and orientation of the HMD.

[0137] Figure 3 An example of a system for creating a 3D object is shown. The structure of the client system 203 and the server system 303 may correspond to the above Figure 1 , and the corresponding systems and elements in the figure schematically illustrate the data to be transmitted and the functions to be performed. The figure shows the use of two cameras to capture a user. Then, when the server performs calibration between the cameras and returns the relative camera pose, 3D user capture can be performed to generate 3D object data for the user.

[0138] The client system 203 has a capture interface coupled to two RGB cameras, so the second camera constitutes a second device. In this example, the object is a person wearing a head mounted display, sitting on a chair in a physical space. In this example, the HMD is not detected in this way, that is, the HMD is just an object worn by the user. The client transmits image data (e.g., two RGB images) and calibration commands (CMD) to the server 303.

[0139] The server 303 receives the calibration command and the two images, for example, via a network (such as the Internet). Then, as schematically shown, the server performs calibration according to the calibration command by matching elements in the two images. The server generates calibration data, for example, a relative camera pose of the second camera relative to the first camera. Moreover, the server can use the difference of the two RGB images and the relative pose of the two cameras to generate a depth map as part of the calibration data.

[0140] In this embodiment, the second device is a second camera. Alternatively or additionally, the second device may also be a depth camera, or a combined image and depth camera. The processor is configured to obtain second image data of the physical space and / or depth data of the physical space as the second data. In the calibration command, the client may command the server to provide calibration data about the pose of the second camera relative to the first camera, and / or provide calibration data that maps the coordinate system and / or matches the parameters or other characteristics of the two cameras.

[0141] In such Figure 3In the example shown, the user is recorded using 2 RGB cameras. This allows the creation of a 3D version of the user, which can be streamed to other users in a 3D virtual environment (such as a VR conference) as a point cloud or a time-varying mesh (TVM) or as an RGB+depth stream. This may require calibrating the 2 cameras in physical space, i.e., in order to determine the pose of each camera relative to the other. To this end, the client sends two RGB images to the server. The server can use vision-based methods (i.e., perform complex stereo image point matching, see e.g. [6]) to determine the relative position and relative orientation of the two cameras. Next, the server sends the calibration data back to the client, so the client can create 3D object data, e.g., a 3D version of the user for streaming.

[0142] In this example, the calibration data defines the position and orientation of the camera. For this description, an axis system can be defined and the orientation can be defined, for example, as pitch / roll / yaw or as a pointing vector. Alternatively, a new axis system is selected and the calibration data contains data for the poses of the two cameras to be matched to the new axis system. Alternatively, one of the cameras is selected for the axis system, for example, calling that camera the (0,0,0) point and its orientation the (0,0,0) orientation, and then the calibration data contains the relative pose for the second camera.

[0143] Figure 4 shows an axial system. Figure 4a An axial system without roll is shown. Figure 4b An axial system with roll is shown. If the camera is horizontal, then roll does not need to be defined.

[0144] Using the axial system shown in FIG4 , the position of the first camera can be defined as (x, y, z) = (0, 0, 0) and its orientation as (yaw, pitch, roll) = (0, 0, 0) or as a vector (x, y, z) = (1, 0, 0) with the position as origin. These coordinates can be measured in the metric system. If the second camera is placed about 30 cm to the left at the same height and also pointed at the object, the position of the second camera can be (x, y, z) = (0, 0.3, 0) and its orientation can be (yaw, pitch, roll) = (-30°, 0°, 0°) or represented by the vector (x, y, z) = (1, -0.3, 0). Here, all position data can describe the lens position, i.e. the so-called pinhole. Furthermore, as metadata, camera characteristics can be used, for example, as explained in [5].

[0145] An initial calibration may be sufficient to provide reference calibration data, for example, when the camera position is static. However, if the camera is moved during a session, the calibration needs to be repeated, which can be called recalibration. If the user moves the camera, recalibration may be needed because the user may move or the user may detect that he is not captured properly. This change in position / orientation of the camera must first be detected by the system, or indicated to the system, and then new calibration data must be generated, i.e., the new relative position and orientation must be determined. Detection of the need for calibration can be independent (i.e., detecting camera movement or zoom, etc.), or by performing calibration continuously and seeing if the results change.

[0146] In an embodiment, the processor is configured to determine a need for reference calibration data and include a request regarding said need in the calibration command, thereby indicating the calibration data required at the client. The reference calibration data may include one or more of the following:

[0147] - a reference view of the physical space without the object;

[0148] - fixed elements in that physical space;

[0149] - Visual markers in that physical space;

[0150] - Predefined objects in that physical space.

[0151] In fact, in order to track certain objects such as HMDs, physical markers can be used to facilitate object detection. If the exact size and shape of these markers are known (e.g., as provided by reference calibration data), these markers can also be used for camera calibration, i.e., for detection of camera characteristics.

[0152] In an embodiment, the processor may be configured to determine a need for recalibration after receiving calibration data from an earlier calibration, and include a request regarding the need in the calibration command. For example, the request may indicate a reason for the calibration. The need may be one of the following:

[0153] -Detect camera movement relative to the camera pose based on early calibration;

[0154] - detecting movement of the second device;

[0155] - detecting drift in the posture updates of the second device;

[0156] - Adding another device in the physical space that is able to provide another image data, depth data, or data about the position or orientation of the other device in the physical space. For example, drift can occur in actual HMDs, especially mobile-based HMDs such as Samsung's gear VR or Google's Daydream.

[0157] In an embodiment or in any of the above embodiments, the processor may be configured to receive recalibration data from the server system after receiving the calibration data of the earlier calibration. Subsequently, the processor may use the recalibration data to process the image data to obtain image data of the object.

[0158] In practice, camera movement can be a reason for recalibration. The movement can be manual, but it can also be a change in a pan-tilt-zoom (PTZ) camera, for example, a camera that is able to track an object to be captured. Typically, zooming a camera changes the camera's lens properties, for example, the focal length. Panning or tilting will change the camera orientation, which can be detected or measured by tracking elements in the image. In extreme cases, the camera may be mounted on a drone or other moving object and continuous calibration is required.

[0159] In an embodiment, the processor is configured to obtain metadata indicating one or parameters of the camera, or one or more parameters of the second device; and include the metadata in the calibration command. For example, the metadata may include static device parameters. Static parameters may include device type, lens parameters, lens distortion data, image sensor resolution, depth sensor resolution, device data transmission format, etc. Moreover, the metadata may include dynamic device parameters. Dynamic parameters may include one or more of focal length parameters, tilt parameters, pan parameters, camera focus parameters, camera aperture parameters, synchronization data.

[0160] In order to correctly place the self-view, a calibration is required between the user pose (e.g., wearing an HMD) and the camera pose. The camera can be an RGB+depth camera whose output is sufficient to create a 3D image of the user. The camera is likely to capture the user from the front, i.e., the user is sitting directly in front of the camera. By using the camera characteristics (i.e., focal length), the user can be correctly measured based on the depth value.

[0161] Placing a 3D model of the user in a virtual environment requires coordinates and orientation. The user sees the virtual environment through a virtual camera (or multiple virtual cameras, one for each eye). As the user moves his head, i.e., by rotating or moving the head in space, the movement is tracked by the HMD system (e.g., using sensors inside the HMD or external tracking sensors, or a combination thereof). The virtual camera pose is updated based on the tracking data, thereby ensuring the correct visual response of the virtual environment to the user's head movements.

[0162] When creating a self-view, it needs to be placed correctly in the virtual environment in terms of position and orientation, as well as in terms of size. Therefore, the camera output needs to be associated with the axial system of the HMD, that is, it needs to be calibrated. Since the user (including his HMD) is within the field of view of the camera, this can be used for calibration purposes. But it is not enough to just detect the HMD in the image (RGB and depth image). The pose in the image needs to be associated with the pose of the HMD when the image was captured. Once the axial systems of both the camera and the HMD are calibrated, any image captured by the camera can be correctly placed in the virtual environment.

[0163] Figure 5a A camera is shown pointing towards the user. The camera may be an RGB+depth camera. The user is positioned a little to the left of the camera and is looking straight ahead while wearing the HMD.

[0164] Figure 5b Shows Figure 5a The camera captures the scene shown in FIG. The camera also provides a depth image, which represents the distance between the lens and the captured object. In this case, the front of the HMD is 1 meter in front of the camera, and the HMD is at the same height as the camera.

[0165] Figure 5c and Figure 5d The calibration of the camera and HMD is schematically shown. Figure 4a The axial system shown in is used as the axial system of the camera. The center of the camera lens following the pinhole model is located at the origin of the system, that is, the (0,0,0) point, and the X axis is the North axis, that is, the X axis is the direction the camera points to. Moreover, it is assumed that the camera is placed horizontally.

[0166] Then, the detected HMD will be located at, for example, (1, 0.2, 0), and its orientation will be (described as a vector from the object origin) (-1, 0, 0), described using the camera's axial system. Here, it is assumed that the center point of the HMD is indicated for this purpose. Therefore, the calibration command may include:

[0167] -RGB image and associated depth image.

[0168] -The focal length of the camera.

[0169] - The current HMD pose (ie when capturing RGB+D images) and the axial system used for this (see below).

[0170] A default calibration request may mean that the camera is described in terms of the HMD axial system. For example, assume that the HMD internally uses the same axial system as shown in Figure 4. Next, assume that the HMD pose when capturing the RGB+depth image consists of a position (-0.2, 0.6, 0) and an orientation described as a vector (0.5, 0.5, 0). For localization purposes, the center of the HMD is considered, and the orientation is straight ahead.

[0171] The result of the calibration can be the camera posture described by this axis system. The position is:

[0172] -x axis (in cm): -20+sqrt(100^2 / 2)-sqrt(20^2 / 2)=37 (rounded off)

[0173] -y axis (in cm): 60 + sqrt(100^2 / 2) + sqrt(20^2 / 2) = 145 (rounded off)

[0174] Therefore, the position of the camera is (0.37,1.45,0). The orientation of the camera is directly opposite to the direction of the user's gaze and is therefore (-0.5,-0.5,0). This is described in the HMD axis system. Therefore, the result of the calibration will be this pose. This will allow the client system to process the camera images and place the self-view at the user's position in the virtual environment. Even if the user moves later, this is not a problem because the HMD tracker is constantly monitoring the pose of the HMD.

[0175] The accurate placement of the self-view is an additional step. The RGB+depth image has to be converted into a 3D image, for example using a point cloud representation: each pixel becomes a pixel with a 3-dimensional placement value. This can be a color image that includes the depth. The image can be rendered on a plane in the virtual environment, where the pixels will of course be placed in front of or behind the plane based on the depth values. Note that the points can also be curved behind or in front of the plane based on the focal length of the camera and the specific camera image mapping technique used. However, the plane must be placed exactly in the right place in the virtual environment to ensure proper alignment of the self-view. Typically, the plane is placed by indicating its four corners or by indicating two opposite corners.

[0176] Figure 6a The user's self-view to be positioned is shown, as well as the plane AB. In this figure, a top view is shown, where the depth of the user's image is completely behind the plane, i.e., the front of the HMD has a depth value of 0 and the back of the head has a depth value of, for example, 0.2 meters. The left side of the frame is A and the right side is B as seen from the front.

[0177] Figure 6bA front view of the self-image is shown, which shows the entire plane. Here, A is shown as the upper left corner of the frame, and is therefore Figure 6a The left side is seen from above, and B is shown as the lower right corner of the frame, and is therefore Figure 6a In order to place the self-view correctly, it is necessary to determine the correct coordinates of angles A and B so that the self-view is correctly placed and correctly sized.

[0178] As in Figure 6b As shown in , as can be determined from the focal length, assume that the frame size at a distance of 1 meter is 1 meter high and 1.6 meters wide.

[0179] Figure 6c The angular coordinates for determining A and B are shown. In this figure, the coordinates are determined as:

[0180] - For angle A

[0181] ox axis (in cm): -20+sqrt(40^2 / 2)=8.3cm

[0182] Oy axis (in cm): 60-sqrt(40^2 / 2)=31.7cm

[0183] - For angle B

[0184] ox axis (in cm): -20-sqrt(120^2 / 2)=-104.9cm

[0185] Oy axis (in cm): 60 + sqrt (120^2 / 2) = 144.9 cm

[0186] Assume that the height of the center of the HMD is at the level of z=0. Therefore, the coordinates of corner A will be (0.083, 0.317, 0.5), and the coordinates of corner B will be (-104.9, 144.9, -0.5). By placing the video plane at this position, the self-view will be correctly placed and sized. Note that this needs to be corrected for the placement of the eyes relative to the front of the HMD, i.e., the left eye is placed about 5 cm back and about 3.5 cm to the left, and the right eye is placed about 5 cm back and about 3.5 cm to the right, assuming a 7 cm interocular distance.

[0187] Other ways of projecting the self-view in the 3D virtual environment are also possible. For example, it is also possible to project all the points of the point cloud from a single point, which is a bit of an inversion of the capture process. The position and orientation of the projected point must be determined based on the alignment of the HMD, which maps to the pinhole point of the camera. Next, RGB+depth information is projected outward from this point using the focal length of the camera used to record. Other ways of projecting this information are also possible, for example, indicating the position of a single point from the point cloud (e.g., the center of the HMD), and calculating the relative positions of all other points and placing them directly as points in the 3D scene.

[0188] There are various ways in which this calibration information can be exchanged. One of these ways may be a default way, and any calibration request is a request to apply this default to the calibration. Alternatively, the calibration command may contain instructions on how the calibration should be described. Alternatives are:

[0189] -Describe the camera output in terms of the HMD's axis system;

[0190] - describing the HMD in terms of the camera axial system (i.e. the axial system described from the camera's point of view);

[0191] - Choose some reference axis system, and describe both the camera output and the HMD in terms of that reference system.

[0192] Furthermore, the reference system can be based on static parts of the captured images, e.g., on a table or a wall. Note that if one of the devices (such as an HMD) cannot look outward, it cannot conform to external reference points by itself: in this case, the camera images need to be used both to conform to the external reference system and for calibration of the other devices.

[0193] Furthermore, the axis system to be used may be defined in advance or may be agreed upon in preliminary communication. Typically, the X-axis is horizontal, the Y-axis is also horizontal but orthogonal to the X-axis, and the Z-axis is vertical. However, other systems use the Y-axis as the vertical and the Z-axis as the second horizontal axis. In addition, rotation about each axis may be right-handed (i.e., the right thumb is in the direction of the axis and the other fingers show the true orientation direction) or left-handed (i.e., the left thumb is in the direction and the other fingers show the orientation). Figure 4b A right-hand system is shown. Orientation can be indicated in yaw-pitch-roll using the orientation system, or as a vector defined pointing from the origin. Any calibration requires selection of a system, e.g. either defining a system as default or indicating the system used or to be used.

[0194] Calibration between two cameras, or calibration between an independent RGB camera or stereo camera and a depth camera can be implemented similarly. Note that the above examples assume a perfect camera, i.e., there is no deformation in the image. Since the camera is not perfect by definition, i.e., the lens may cause distortion (e.g., barrel distortion, pincushion distortion), the placement of the lens and the sensor may not be perfectly aligned. In order to obtain the correct geometric information, the output of the camera needs to be adjusted for this. If the camera is calibrated using a known geometric shape (e.g., a chessboard), this allows the creation of a filter that can be applied to the raw camera output to adjust for this, and obtain the correct geometric information. This information can be sent to the server as part of the calibration request. Alternatively, the filter may have been applied on the capture side before encoding and transmission, and therefore no further correction is required.

[0195] Another part that may need to be part of the calibration request is the principal point offset. Some cameras may not have their sensor directly behind the lens, i.e. the principal point may not be the center of the image. If this is the case, this principal point offset needs to be part of the calibration request.

[0196] Figure 7a A method 500 for a processor system is shown. The method is used to process image data from a camera in a physical space to render a virtual object in a 3D virtual environment. The virtual object represents an object in the physical space, for example, a human head. The processor system (also referred to as a client) is arranged to communicate with a server system (also referred to as a server) via a network, as schematically indicated by arrow 560, which represents signal data to be transmitted between the client and the server as described above. The signal data is constructed to carry calibration commands or calibration data. The processor system has: a network interface for exchanging data via a network; a capture interface to a camera; and a second device interface to a second device in the physical space. The camera has a camera pose in the physical space, and the second device has a second pose in the physical space, which is different from the camera pose.

[0197] The method may start with a capture process 510 to obtain image data of a physical space from a camera via a capture interface. Moreover, for example, in parallel, the method performs a second process 520 to obtain second data of a second device (e.g., an HMD, a second camera, or a depth camera) via a second device interface. Subsequently, in a transmission process 540, the method sends image data, second data, and a calibration command to a server in one or more transmission steps. Then, in response, in another communication process 550, the method receives calibration data from a server according to the calibration command. The calibration data indicates a camera pose and a second pose. The communication process with the server is schematically illustrated by a dashed box 530 and a signal 560 to be transmitted via a network. Finally, in a generation process 570, the method generates 3D object data of an object by processing the image data based on the calibration data. The 3D object data is provided to a head-mounted display, for example, for rendering virtual objects in a 3D virtual environment.

[0198] Figure 7b A server method 600 for a server system is shown. The server method is for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment. The server system is arranged to communicate with a processor system via a network, as schematically indicated by arrow 560, which represents signal data to be transmitted between the processor system and the server system as described above.

[0199] The server method starts with a first communication process 640 to receive image data of the physical space obtained by the camera, second data of the second device in the physical space, and a calibration command from the processor system via a network interface. Then, the server method proceeds to a processing resource step 610 to process the image data and the second data according to the calibration command to generate calibration data indicating the camera posture and the second posture. Next, in a second communication process 650, the method sends the calibration data to the processor system via the network interface. The communication process with the client is schematically shown by a dotted box 630 and a signal 560 to be transmitted via the network.

[0200] In an embodiment of the server system, the processing resources are configured to determine the need for recalibration after an earlier calibration. This need can be determined based on receiving a recalibration command indicating the need for recalibration. Moreover, the need can be determined by the server based on image data and / or data of a second device, for example, by detecting movement of the camera relative to a camera pose according to an earlier calibration, or by detecting movement of the second device relative to a second pose according to an earlier calibration. Moreover, the server can detect that another device has been added to the physical space, which other device is capable of providing another data about the physical space. For example, another person with an active mobile phone can enter the physical space and provide image data.

[0201] Subsequently, the processing resource is configured to: process the image data and the second data to generate recalibration data indicating the camera pose and the second pose; and send the recalibration data to the processor system.

[0202] In an embodiment of the server system, the processing resource may be configured to monitor data such as provided by the client upon receiving a monitoring command. Subsequently, the processing resource is configured to monitor a data stream about the physical space from a camera or a second device. Upon receiving such a data stream, the processing resource determines the need for recalibration based on the data stream and the calibration data, and performs recalibration to generate recalibration data for sending the recalibration data to the processor system.

[0203] In an embodiment of the server system, the image data includes a first image at a first moment and a second image at a second moment, and the second data indicates an actual movement of the second device from a second posture at the first moment to a subsequent second posture at the second moment. The processing resource is configured to detect the actual movement of the second device as represented in the image data. Subsequently, the processing resource obtains at least one camera parameter from the actual movement as represented; and generates calibration data using the obtained camera parameters.

[0204] Furthermore, in the server system, the processing resource may be configured to determine reference calibration data and send the reference calibration data to the processor system. The reference calibration data may include one or more of the following:

[0205] - a reference view of the physical space without the object;

[0206] - fixed elements in that physical space;

[0207] - Visual markers in that physical space;

[0208] - Predefined objects in that physical space.

[0209] Performing this processing on the server may cause some latency, since the image data is first sent to the server, and then the calibration data is received from the server. This is acceptable, as the server processing can be completed faster. Also, zero latency is not required in many cases. If the camera is moved, the user is likely to accept that it will take some time for the 3D environment to be 'stable' again.

[0210] In the above configuration of the client and server, the server system can be located anywhere. Moreover, the server system can be one server, but it can also be distributed or the functions can be divided among multiple servers. For example, any location in the Internet can be referred to as being in the 'cloud', or it can be a conventional server connected to the Internet. However, the server can also be strategically involved, for example, at a special location near the client, i.e., a capture point, which can be referred to as being at the 'edge' or 'edge computing' of the network. Processing near the capture point is more efficient for the network because the image data does not need to be transmitted through the network, which can also minimize latency. Moreover, streams can be sent to the user, and these streams can pass through the server, or can be forked to the server for the calibration. This routing brings additional efficiency because the client only needs to send image data once for both purposes. The network 100 is only schematically shown, but can have a large number of network resources (such as nodes coupled via links), and can have at least one network controller arranged to perform the above routing or forking.

[0211] In practice, the processor system 200 for rendering virtual objects in a virtual environment can, but need not be, integrated in a single device, such as a smartphone that is also a camera, a smartphone in a VR shell, a personal computer, a laptop, a tablet device, a set-top box, a smart watch, smart glasses, a television, a monitor, a projector, a media player, a media recorder, an audio system, a game console, etc. Alternatively, the system 200 can be distributed across multiple devices or subsystems, such as two or more smartphones that are locally connected (e.g., via Wi-Fi or via a network such as 4G or 5G). The system 200 is shown as being connected to a camera 20, from which image data 22 of a physical space can be received. Alternatively, the system 200 can include the camera 20. Alternatively, the system 200 can be integrated into the camera 20.

[0212] The server system 300 may receive the image data obtained from the camera 20. The server 300 may then further transmit the image data 22 to other devices participating in the virtual environment. In addition to providing the image data 22 to the server, the system 200 may further provide the server 300 with 3D object data so that the server and / or other rendering devices can render a 3D virtual object (e.g., a virtual representation of a user in the virtual environment). For this purpose, the client may provide different types of 3D object data depending on which type of second device is participating and / or which type of object is to be rendered.

[0213] Optionally, the processor system 200 and / or the server system 300 may be arranged to communicate with a database. The database may include metadata for one or more different camera types or other second devices, which metadata may be stored and retrieved based on type identifiers of these cameras or second devices. Accordingly, the system 200 may obtain camera metadata or additional device metadata, such as, for example, physical size and shape, by obtaining a type identifier of the camera or second device and looking up the type identifier in a database. It will be understood that the database may be an internal database, but may also be an external database, for example, a network-hosted database. As an alternative to using a database, metadata may also be queried from another entity or service, such as a search engine or an auxiliary service based on 'artificial intelligence'. For this purpose, an appropriate API may be used.

[0214] In addition, the processor system 200 and the server system 300 are typically located in different locations, such as different rooms, buildings or places. Thus, the communication between the devices can be remote communication, for example, involving data communication via a network (such as or including one or more access networks and / or the Internet).

[0215] In addition, in addition to the above-mentioned network interface and camera interface, the processor system 200 may also have a memory including instruction data representing a set of instructions, and the processor 220 is configured to communicate with the memory and execute the set of instructions, wherein the set of instructions, when executed by the processor, enables the processor 220 to perform various functions as described above.

[0216] The network interface 240 may take any suitable form, including but not limited to a wireless network interface based on Wi-Fi, Bluetooth, ZigBee, 4G mobile communication or 5G mobile communication, or a wired network interface based on Ethernet or optical fiber. The network interface 240 may be a network interface to a local area network (LAN) or a network interface to a wide area network (WAN), such as the Internet.

[0217] The server system 300 to which the camera metadata is provided may be a server configured to host a virtual environment or a rendering device configured to render a virtual environment. This may be similar to current video conferencing setups, where either a video multipoint control unit is used to mix the videos of all participants in a particular manner (i.e., server-based), or point-to-point communication is used between all users, and where each user's device renders all inputs locally (e.g., based on a rendering device). The network entity 300 may further include a memory including instruction data representing a set of instructions. The server has a processing resource 320 configured to communicate with the network interface 310 and the memory and execute the set of instructions, where the set of instructions, when executed by the processor 320, may cause the processor 320 to generate calibration data as described above.

[0218] Typically, the processing system 200 and the server system 300 can each be embodied as a device or apparatus, or in a device or apparatus. The device or apparatus may include one or more (micro) processors that execute appropriate software. The processors of the system and communication equipment may be embodied by one or more of these (micro) processors. The software implementing the functions of the system or network entity may have been downloaded and / or stored in one or more corresponding memories, for example, volatile memories such as RAM or non-volatile memories such as flash memory. Alternatively, the processor of the system or network entity can be implemented in a device or apparatus in the form of programmable logic, for example, as a field programmable gate array (FPGA). Any input and / or output interface can be implemented by a corresponding interface (such as a network interface) of the device or apparatus. Typically, each unit of the system or network entity can be implemented in the form of a circuit. Note that the processor system or server can also be implemented in a distributed manner, for example, involving different devices or apparatuses.

[0219] Typically, the rendered 3D virtual environment can be displayed using a display. The display can be, for example, a head-mounted VR device of the same or similar type as 'Oculus Rift', 'HTC Vive' or 'PlayStation VR' or simply a VR head-mounted device. Other examples of VR devices are so-called augmented reality (AR) devices, such as Microsoft's HoloLens or Google's Glass goggles, or mobile VR devices, such as Samsung's Gear VR or Google's Cardboard. It will be understood that the display may not necessarily be head-mounted, but for example a stand-alone holographic display or a CAVE-like system.

[0220] Figure 8 A transitory or non-transitory computer readable medium, such as an optical disk 900, is shown. Figure 5a and Figure 5bInstructions for a computer (e.g., executable code) for one or more of the methods presented may be stored on a computer-readable medium 900, for example, in the form of a series of machine-readable physical marks 910 and / or as a series of elements with different electrical (e.g., magnetic or optical) properties or values. The executable code may be stored in a transient or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc.

[0221] Fig. 9 A block diagram is shown, showing an exemplary data processing system that can be used in an embodiment of a processor system or server system as described above. Such a data processing system includes a data processing entity described in the present disclosure, including but not limited to a processor system for generating 3D object data or a server system for generating calibration data. The data processing system 1000 may include at least one processor 1002 coupled to a memory element 1004 via a system bus 1006. In this way, the data processing system can store program code in the memory element 1004. Further, the processor 1002 can execute program code accessed from the memory element 1004 via the system bus 1006. In one aspect, the data processing system can be implemented as a computer suitable for storing and / or executing program code. However, it will be appreciated that the data processing system 1000 can be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0222] Memory element 1004 may include one or more physical memory devices, such as, for example, local memory 1008 and one or more mass storage devices 1010. Local memory may refer to a random access memory or other (multiple) non-persistent memory devices commonly used during the actual execution of program code. Mass storage devices may be implemented as hard disk drives, solid state drives or other persistent data storage devices. Processing system 1000 may also include one or more cache memories (not shown), which provide temporary storage of at least some program codes to reduce the number of times program codes must be retrieved from mass storage devices 1010 during execution.

[0223] Input / output (I / O) devices, depicted as input devices 1012 and output devices 1014, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, microphones, keyboards, pointing devices such as mice, touch screens, and the like. Examples of output devices may include, but are not limited to, for example, monitors or displays, speakers, and the like. Input devices and / or output devices may be coupled to the data processing system directly or through an intermediate I / O controller. Network interface 1016 may also be coupled to the data processing system or as part of the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intermediate private or public network. The network interface may include a data receiver for receiving data transmitted by the system, device, and / or network to the data and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network interfaces that may be used with data processing system 1000.

[0224] As shown in FIG7 , memory element 1004 may store application 1018. It should be understood that data processing system 1000 may further execute an operating system (not shown) that can facilitate the execution of application programs. Application programs implemented in the form of executable program code may be executed by data processing system 1000 (e.g., by processor 1002). In response to executing the application program, the data processing system may be configured to perform one or more operations as described in further detail above.

[0225] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by hardware comprising several distinct elements and by a suitably programmed computer. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The fact that certain measures are stated in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0226] References

[0227] [1]Alexiadis, D.S., Zarpalas, D., and Daras, P. (2013). Real-time, full 3D reconstruction of moving foreground objects from multiple consumer depth cameras. IEEE Transactions on Multimedia, 15(2), 339-358.

[0228] [2] Mielczarek, A., Makowski, D., Perek, P., Napieralski, A., and Sztoch, P. (June 2016). FPGA-based image analyzer for calibration of stereo vision rigs. In Real Time Conference (RT), 2016 IEEE-NPSS (pp. 1-2). IEEE.

[0229] [3] Doubek, P., Svoboda, T. and Van Gool, L. (April 2003). Monkeys—a software architecture for ViRoom—low-cost multicamera system. In International Conference on Computer Vision Systems (pp. 386-395). Springer, Berlin, Heidelberg.

[0230] [4] MPEG, Use cases and requirements for NBMP (v4), April 2018. https: / / mpeg.chiariglione.org / standards / mpeg-i / network-based-media-processing / use-cases-and-requirements-nbmp-v4

[0231] [5] Remondino, F. and Fraser, C. (2006). Digital camera calibration methods: considerations and comparisons. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 36(5), 266-272.

[0232] [6] Remondino, F., Spera, MG, Nocerino, E., Menna, F. and Nex, F. (2014). State of the art in high density image matching. The Photogrammetric Record, 29(146), 144-166.

Claims

1. A processor system for processing image data from a camera in a physical space to render virtual objects in a 3D virtual environment, The virtual object represents an object in the physical space. The processor system is arranged to communicate via a network with a server system comprising processing resources, wherein The processor system includes a network interface for exchanging data via the network; a capture interface to the camera; a second device interface to a second device in the physical space, wherein the second device comprises at least one of: a head mounted display, a user interaction device for sensing movement of a user in the physical space; The camera has a camera pose in the physical space, and the second device has a second pose in the physical space, the second pose being different from the camera pose; and a processor configured to obtain image data of the physical space from the camera via the capture interface; Obtaining second data of the second device via the second device interface, wherein the second data is data about the position or orientation of the second device in a physical space, including at least one of the following: the displacement distance of the second device; a displacement direction of the second device; the rotation angle of the second device; a rotation direction of the second device; sending the image data, the second data and a calibration command to the server system; receiving calibration data from the server system according to the calibration command, the calibration data indicating the camera pose and the second pose; 3D object data of the object is generated by processing the image data based on the calibration data, and the 3D object data is provided for rendering the virtual object in the 3D virtual environment.

2. The processor system of claim 1, wherein: The second device includes a head-mounted display, and the object includes at least a portion of a person wearing the head-mounted display in the physical space, and the processor is configured to generate position data and / or orientation data of the person as the 3D object data by processing the image data, and the 3D object data is provided for rendering as a self-view of a virtual object in the 3D virtual environment.

3. The processor system of claim 1, wherein: The second device includes at least one of the following Second camera; a depth camera; and The processor is configured to obtain at least one of the following as the second data: second image data of the physical space; Depth data of this physical space.

4. The processor system of claim 1, wherein: The processor is configured as obtaining metadata indicating at least one parameter of the camera or the second device; and Include the metadata in the calibration command.

5. The processor system of claim 4, wherein: The metadata includes static device parameters, the static device parameters including at least one of device type, lens parameters, lens distortion data, image sensor resolution, depth sensor resolution, and device data transmission format; or The metadata includes dynamic device parameters including at least one of focal length parameters, tilt parameters, pan parameters, camera focus parameters, camera aperture parameters, and synchronization data.

6. The processor system of claim 1, wherein: The processor is configured to determine a need for reference calibration data, The reference calibration data includes at least one of the following: a reference view of the physical space without the object; fixed elements in that physical space; visual markers in that physical space; predefined objects in the physical space; and The processor is configured to include a request regarding said need for reference calibration data in the calibration command.

7. The processor system of claim 1, wherein: The processor is configured to determine a need for recalibration after receiving calibration data of an earlier calibration, the need comprising one of: detecting movement of the camera relative to the camera pose based on the early calibration; detecting movement of the second device; adding another device in the physical space, the other device being capable of providing another image data, depth data, or data regarding the position or orientation of the other device in the physical space; and The processor is configured to include a request regarding the need for recalibration in the calibration command.

8. The processor system of claim 1, wherein: The processor is configured to, after receiving calibration data of an earlier calibration, receiving recalibration data from the server system; and The image data is processed using the recalibration data to obtain image data of the object.

9. A server system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment, the virtual object representing an object in the physical space, The camera has a camera pose in the physical space, and the second device has a second pose in the physical space, the second pose being different from the camera pose, wherein, The second device includes at least one of the following: a head mounted display, a user interaction device for sensing movement of a user in the physical space; The server system is arranged to communicate with the processor system via a network, wherein the server system comprises a network interface for exchanging data via the network, and A processing resource, the processing resource being configured as: Receiving, via the network interface, image data of the physical space obtained by the camera, second data of a second device in the physical space, and a calibration command from the processor system, wherein the second data is data about a position or orientation of the second device in the physical space, including at least one of the following: the displacement distance of the second device; a displacement direction of the second device; the rotation angle of the second device; a rotation direction of the second device; Processing the image data and the second data according to the calibration command to generate calibration data indicating the camera pose and the second pose; and The calibration data is sent to the processor system via the network interface.

10. The server system according to claim 9, wherein: The processing resource is configured to determine a need for recalibration after an earlier calibration, the need comprising one of: receiving a recalibration command indicating the need for recalibration; detecting movement of the camera relative to the camera pose based on the early calibration; detecting movement of the second device relative to the second posture based on the early calibration; Adding another device in the physical space, the other device being capable of providing another data about the physical space; And the processing resources are configured as processing the image data and the second data to generate recalibration data indicating the camera pose and the second pose; and The recalibration data is sent to the processor system.

11. The server system according to claim 9, wherein: The processing resource is configured to monitor a data stream about the physical space from the camera or the second device upon receiving a monitoring command; determining a need for recalibration based on the data stream and the calibration data; Recalibration is performed to generate recalibration data for sending the recalibration data to the processor system.

12. The server system according to claim 9, wherein: The image data includes a first image at a first moment and a second image at a second moment, and the second data indicates an actual movement of the second device from a second posture at the first moment to a subsequent second posture at the second moment, and the processing resource is configured to detecting actual movement of the second device as represented in the image data; deriving at least one camera parameter from the actual movement as represented; and The calibration data is generated using the obtained camera parameters.

13. The server system according to claim 9, wherein: The processing resource is configured to determine reference calibration data, the reference calibration data comprising at least one of: a reference view of the physical space without the object; fixed elements in that physical space; visual markers in that physical space; predefined objects in that physical space, and The reference calibration data is sent to the processor system.

14. A processing method for a processor system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment, the virtual object representing an object in the physical space, the processor system being arranged to communicate with a server system via a network, the camera having a camera pose in the physical space, and the second device having a second pose in the physical space, the second pose being different from the camera pose; in, The device includes at least one of: a head mounted display, a user interaction device for sensing movement of a user in the physical space; Wherein, the method comprises: Obtaining image data of the physical space from the camera; Obtaining second data of the second device, wherein the second data is data about the position or orientation of the second device in a physical space, comprising at least one of the following: the displacement distance of the second device; a displacement direction of the second device; the rotation angle of the second device; a rotation direction of the second device; sending the image data, the second data and a calibration command to the server system; receiving calibration data from the server system according to the calibration command, the calibration data indicating the camera pose and the second pose; and 3D object data of the object is generated by processing the image data based on the calibration data, and the 3D object data is provided for rendering the virtual object in the 3D virtual environment.

15. A processing method for a server system for processing image data from a camera in a physical space to render a virtual object in a 3D virtual environment, the virtual object representing an object in the physical space, wherein: The camera has a camera pose in the physical space, and the second device has a second pose in the physical space, the second pose being different from the camera pose, The server system is arranged to communicate with the processor system via a network, wherein the method comprises: receiving image data of the physical space obtained by the camera, second data of a second device in the physical space, and a calibration command from the processor system, The second device includes at least one of the following: a head mounted display, a user interaction device for sensing the movement of a user in the physical space, The second data is data about the position or orientation of the second device in the physical space, including at least one of the following: the displacement distance of the second device; a displacement direction of the second device; the rotation angle of the second device; a rotation direction of the second device; Processing the image data and the second data according to the calibration command to generate calibration data indicating the camera pose and the second pose; and The calibration data is sent to the processor system.

16. A transitory or non-transitory computer readable medium comprising a computer program comprising instructions for causing a processor to perform the method according to any one of claims 14 to 15.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method of any one of claims 14 to 15 are performed.

Images