Data compression for medical images

Abstract

The application describes an embodiment in which data indicative of a camera transformation matrix of a display unit is receive and an image rendered based at least in part on the camera transformatio

Description

BACKGROUND The present invention relates to data compression. The present invention is, in particular (but not exclusively), concerned with data compression to reduce latency when transmitting medical images over a network such as the internet. A sequence of consecutive medical images may be sent over a network, for example to facilitate surgery, or to help perform diagnoses, or to facilitate collaborative work. Virtual reality medical images may be streamed from a therapist to a client, or from an operating room (OR) to a remote expert. Similarly, virtual reality medical images of complex data may be displayed to an expert such as a surgeon. Examples of such complex data are data resulting from a 4D ultrasound scan, or data from a computed tomography (CT) scan. Such data may be in the form of 3D voxels and may also be time varying (for a 4D ultrasound). As those skilled in the art will appreciate, the size of voxel datasets can be many megabytes or gigabytes and can be difficult to render in real time. In either case, latency can be highly objectionable. For time-varying data (such as 4D ultrasound data), latency that varies may also be objectionable as latency that varies may result in “stuttering”, giving an incorrect representation of the time-varying data. Medical virtual reality preferably uses higher frame rates than conventional video. For example, whereas films and games may use frame rates of 30 Hz or 60 Hz, medical virtual reality can benefit from frames rates such as 90 Hz or higher (for example 120 Hz). To sustain a frame rate of 120 Hz, each frame of video must be processed within a time period of only 8.33 milliseconds. Some known video compression methods can use what are sometimes called inter frames (I-frames), predicted frames (P-frames) and bi-directionally predicted frames (B-frames). l-frames represent complete images. P-frames are compressed relative to a previous l-frame. B-frames are compressed relative to a past l-frame and relative to a future l-frame. Thus, the use of B-frames can result in excessive latency as the decoder cannot decode a B-frame until the future l-frame has been received by the decoder. Examples of known approaches are described in US 10,477,149, KR 20190113760 and US 2014176661. The contents of each of these documents is hereby included by reference. Figure 11 of the present disclosure shows a portion of a system known from US 10,477,149; the system of Figure 11 of the present disclosure appears to involve point cloud data but does not reduce latency in the sense of the present application. The inventors of the present application have determined that the latency associated with encoding high resolution images is a problem with prior art techniques. The inventors of the present application have also determined that synchronization (between video data and point cloud data) is a problem with prior art techniques. Different compression techniques (for video data and for point cloud data) may result in different latencies for the different compression algorithms. Also, point cloud data can have a relatively constant compression latency whereas video data may have a more variable compression latency. BRIEF DESCRIPTION OF INVENTION According to an aspect of the present invention, there is provided an apparatus and a method according to Claims 1 to 19. Disclosed embodiments provide a system comprising: a matrix module operable to determine a camera transformation matrix; a render module operable to render an image based at least in part on the camera transformation matrix; a compression module operable to compress the rendered image to form compressed image data; a decompression module operable to decompress the compressed image data to form a decompressed image; an overlay module operable to overlay the decompressed image onto a projection plane to form a projected image, wherein the projection plane is based on the camera transformation matrix. FIGURES Figure 1 shows a system according to an embodiment of the present invention. A camera device is used to generate data indicative of a compressed texture and a 3-dimensional mesh. The compressed texture and the 3-dimensional mesh are rendered by a remote virtual reality display that is remote relative to the camera device. Figure 2 shows an example of an image that is converted to a compressed texture. Figure 2 also shows an example of a 3-dimensional mesh that is generated from point-cloud data. Figure 3 shows an example of UV mapping. Figure 4 shows an example of a frame of data that may be sent, by the system of Figure 1, over a network. Figure 5 shows a system according to an embodiment of the present invention. The system of Figure 5 is analogous to the system of Figure 1. Figure 6 shows a system-level diagram of another embodiment of the present invention. Unlike Figures 1 &5, Figure 6 renders an image on a remote device. The remotely rendered image is then compressed and sent over a network to a display device. Figure 7 shows an example of two frames of data that may be used by the system of Figure 6. Figure 8 shows in more detail the embodiment of Figure 6. Figure 9 shows an embodiment of a method that generally corresponds to Figures 1 to 5. Figure 10 shows an embodiment of a method that generally corresponds to Figures 6 to 8. Figure 11 shows a portion of a system known from US 10,477,149. DESCRIPTION Figure 1 shows a system 100 comprising: 110 a video camera, 113 the field of view of the video camera 110, 115 a data output of the video camera 110, 120 a depth camera, 123 the field of view of the depth camera 120, 125 a data output of the depth camera 120, 130 a person, 140 compression hardware, 143 a texture compression unit, 144 compressed texture data, 146 a point-cloud to 3-dimensional mesh conversion unit, 147 mesh data, 150 a network (such as the internet), 160 a display unit for rendering images, 164 a pair of displays (one for each eye of a user), and 168 an inertial measurement unit (IMU). The system 100 uses the video camera 110 to form an image of a scene. The system 100 uses the depth camera 120 to form a depth image (for example, a 3-dimensional point cloud) of the scene. For Figure 1, the scene include the person 130. The texture compression unit 143 (which, in this embodiment, forms part of the compression hardware 140) converts the data output 115 from the video camera 110 to compressed texture data 144. As those skilled in the art will appreciate, the video camera 110 outputs data 115 indicative of image frames. The image frames may have a rate of, for example, 60 frames per second. The texture compression unit 143 may compress the data output 115 using, for example, the MPEG-4 Part 10 video compression standard (also known as H.264) or the High Efficiency Video Coding video compression standard (also known as HEVC or H.265). Alternatively, the texture compression unit 143 may encode each of the image frames using, for example, the Joint Photographic Experts Group image standard (also known as JPEG or JPG). The texture compression unit 143 may use lossy or lossless compression. The point-cloud to 3-dimensional mesh conversion unit 146 (which, in this embodiment, forms part of the compression hardware 140) converts the data output 125 from the depth camera 120 to 3-dimensional mesh data 147. The depth camera 120 may output image data 125 in which each pixel represents the distance (that is, depth) of the corresponding part of the scene from the depth camera 120. In this embodiment, the point-cloud to 3-dimensional mesh conversion unit 146 converts the point cloud data 125 to a mesh 147 of triangle data 225 (see Figure 2). A brief diversion will be made to Figure 2 and to Figure 3. Figure 2 shows: 130 the person from Figure 1, 215 an image from the video camera 110, 216 an open door, 217 a whiteboard, 225 a mesh from the mesh conversion unit 146 (showing the person 130). Figure 2 shows an example of an image 215 from the video camera 110. The image 215 shows the person 130, an open door 216, and a whiteboard 217. Figure 2 shows an example of a mesh 225 (in this example, of triangle data) from the mesh conversion unit 146. The mesh 225 comprises data indicative of a plurality of points, each point having an x-axis, a y-axis and a z-axis position. As those skilled in the art will appreciate, the mesh 225 may be in the “OBJ” file format that was developed by Wavefront Technologies, Inc. The OBJ file format includes a list of x-y-z vertices, and specifies the position of each vertex. The OBJ file format also includes a list of faces; for a triangle mesh, each face (in the list of faces) species 3 of the vertices from the list of vertices. It will be noted that the image 215 and the mesh 225 both show a woman 130 but the image 215 and the mesh 225 do not exactly correspond. For example, the mesh 225 omits the open door 216 and omits the whiteboard 217. (Also, Figure 2 shows that the posture of the woman 130 has changed slightly in the mesh 225 compared to the image 215.) It will be appreciated that the open door 216 and the whiteboard 217 are static, in the sense that they do not change in consecutive image frames. Accordingly, the compression hardware 140 may extract and remove static image features such as the open door 216 and the whiteboard 217 from the compressed texture 144 and I or from the mesh 147. In other words, the compression hardware 140 may compare two preceding frames from the video camera 115 and I or may compare two preceding frames from the texture compression unit 143. Based on the comparison, the compression hardware 140 may determine that one or more regions (such as the open door 216 and I or the whiteboard 217) are substantially invariant. For a subsequent frame, the compression hardware 140 may omit the one or more invariant regions from the compressed texture data 144 as the data for the one or more invariant regions will already have been sent for a preceding frame. Similarly, the compression hardware 140 may compare two preceding frames from the depth camera 120 and I or may compare two preceding frames form the point-cloud to 3-dimensional mesh conversion unit 146. Based on the comparison, the compression hardware 140 may determine that one or more regions (such as the open door 216 and I or the whiteboard 217) are substantially invariant. For a subsequent frame, the compression hardware 140 may omit the one or more invariant regions from the mesh data 147 as the data for the one or more invariant regions will already have been sent for a preceding frame. Data indicative of static features (such as the open door 216 and the whiteboard 217) may be sent once at the start of a videoconferencing session. For subsequent image frames, only the changes (relative to a previous frame) may be sent. For Figure 2, the mesh 225 may correspond to a different image frame than the image 215. Thus the compression hardware 140 can further improve the compression by avoiding sending data for frame features that are substantially unchanged compared to the corresponding features in a previous frame. The compression hardware 140 may use a threshold to determine when to re-send a frame. For 6 example, if the woman 130 closes the door 216 then the threshold may be exceeded, with the result that a complete frame from the video camera 110 may be sent, together with a complete frame from the depth camera 120. Alternatively, the compression hardware 140 may periodically, for example at a fixed or an adaptive time interval, resend a complete compressed texture 147 and / or a complete mesh 147. Figure 3 shows an example of UV mapping. As those skilled in the art will appreciate, some or all points of the mesh data 147 may be associated with respective UV values in which U has the range of 0.0 to 1.0 and in which V has the range of 0.0 to 1.0. Each triangle of the mesh 225 may have a respective UV mapping, thereby allowing the 3-dimensional display unit to “paint” (that is, overlay) the compressed texture 144 onto the mesh 147. Returning to Figure 1, the depth camera 120 generates the point cloud data 125 based on the time-of-flight of an array of light beams (one beam per pixel) to the scene and back to the depth camera 120. (For clarity, Figure 1 does not show a generator of an array of light beams.) The depth camera 120 may have the same image resolution (width, height) as the video camera 110 or may have a different resolution. In alternative embodiments, instead of using the depth camera 120, the system 100 may use a different sensor. For example, the system 100 may obtain depth information based on the time-of-flight of an ultrasonic pulse, or a microwave pulse. In another embodiment, the system 100 may obtain depth information using 2 or more video cameras (not shown) and by processing the outputs of the 2 or more video cameras to determine depth information based on stereoscopy. For convenience, these other types of distance measurement sensors will also be called “depth cameras” even though they may use, for example, non-optical signals (such as ultrasonic signals). The “depth camera” may instead be renamed as a depth sensor. The depth camera 120 forms a depth image of substantially the same scene as the video camera 110. The width and height of the field of view 113 of the video camera 110 and the width and height of the field of view 123 of the depth camera 120 are preferably similar or identical. The centre of the field of view 113 of the video camera 110 and the centre of the field of view 123 of the depth camera 120 are preferably similar or identical. The video camera 110 and the depth camera 120 may be coaxial or they may be spatially separated by a relatively small distance (for example, about 2 centimetres) compared to the distance of the person 130 from the cameras 110, 120. The compression hardware 140 may transfer data (Indicative of the compressed texture 144 and the mesh 147) via the network 150. The network 150 may include the internet. Alternatively, the network 150 may comprise a universal serial bus (USB) connection from the compression hardware 140 to the display unit 160, and / or a Wi-Fi wireless connection based on the IEEE 802.11 family of standards. A brief diversion will be made to Figure 4. Figure 4 shows an example of a data format that may be used by the compression hardware 140, and shows: 400 a frame of data, 405 an optional synchronisation field, 415 a data field containing a compressed texture 144, 425 a data field containing a mesh 147. Figure 4 shows that the frame 400 may be divided into an optional synchronisation field 405, a field 415 for a compressed texture 144, and field 425 for a mesh 147. As will be appreciated by those skilled in the art, an alternative term for the frame 400 is a network packet 400. The compression hardware 140 may transmit network packets 400 at a frequency of 60 Hz. The video camera 110 and the depth camera 120 preferably operate at the same frame rate as each other and preferably operate at the same rate as the rate at which the compression hardware 140 sends network packets 400. The optional synchronization field 405 may increase by one for each consecutive network packet 400 and may then roll over to 0, for example as 65533, 65534, 65535, 0, 1, 2, 3 etc. The use of synchronization field 405 allows the display unit 160 to display frames in the correct sequence in the event that a network packet 400 with a larger value (e.g. 65535) in the synchronization field 405 arrives at the display unit 160 before a network packet 400 with a smaller value (e.g. 65530) in the synchronization field 405. As those skilled in the art will appreciate, some networks 150 (such as the internet) may have a transmission latency that varies for different network packets 400. The compression hardware 140 may transmit data using a format different to that of the network packet 400. For example, data bits of the data fields 415, 425 may be interleaved. In another embodiment (not shown), the data associated with the compressed texture 415 and I or the data associated with the mesh 425 may be divided into 2 or more network packets. For example, a first network packet (not shown) may contain information associated with a relatively coarse texture 415 and with a relatively coarse mesh 425. A second network packet (not shown) may contain information associated with a relatively detailed texture 415 and with a relatively detailed mesh 425. The first network packet may be sent via a first route (over the network 150) that has a relatively low data rate but has a relatively high probability of being correctly received by the display unit 160. The second network packet may be sent via a second route (for example, using a network that is different to the network 150) that has a relatively high data rate but has a lower probability of being correctly received by the display unit 160. In some embodiments, some network packets 400 may omit either the texture data field 415 and I or the mesh data field 425. For example, if the person 130 moves by a relatively small amount (less than a threshold), the compression hardware 140 may decide to transmit a network packet (not shown) that contains a mesh 425 but omits a texture 415. On receipt of the mesh 425, the display unit 160 may use a compressed texture 415 that was previously received. For example, the display unit 160 may use the most recent compressed texture 415 that was received by the display unit 160. As another alternative, a network packet (not shown) may omit a texture 415 but may specify the identity of a previous network packet, the texture data 415 of which previous network packet is to be used. For example, a network packet (not shown) may specify that the texture data 415 of a network packet that was transmitted 5 packets earlier is to be used. As an example of when a network packet (not shown) may omit a mesh 425, suppose that the field of view 113 of the video camera 110 includes a television (not shown by Figure 1). An image displayed by the television may change but the data output of the depth camera 120 would not change. Accordingly, the compression hardware 140 may decide to transmit a network packet that contains a texture 415 but omits a mesh 425. Returning to Figure 1, the display unit 160 may receive network packets 400 and may display an image to a user (the user is not shown by Figure 1). An example of a suitable display unit 160 is the “HoloLens 2” augmented reality headset that is sold by Microsoft. Another example of a suitable display unit 160 is the “Quest 2” virtual reality headset that is sold by Meta (and was formerly sold by Oculus). As those skilled in the art will appreciate, the display unit 160 may comprise a pair of displays 164 (one display for each eye of a user). Each display 164 of the pair may comprise a liquid crystal display (LCD), optionally with a backlight, or an organic light emitting diode (OLED) display, or a moving beam display (such as a laser beam that is deflected in 2-axes by a pair of mirrors that are actuated by respective galvanometers). The display unit 160 may also comprise an inertial measurement unit (IMU) 168 for determining the position and I or orientation of the display unit 160 (and thus the position and I or orientation of a user’s head) relative to a datum (for clarity, Figure 1 does not show a datum). As those skilled in the art will appreciate, the IMU 168 may comprise one or more of an accelerometer, a gyroscope, and a magnetometer. The IMU 168 may comprise a positional tracking device, for example with 6 degrees of freedom, to determine the position and orientation of a user’s head (it will be appreciated that, strictly speaking, a positional tracking device is not an IMU but, for convenience, the IMU terminology will be retained). In some embodiments, the IMU 168 need not determine the position of a user’s head but only the view direction of a user; accordingly, other terms for the function that is provided by the IMU 168 are: view angle sensor, and orientation sensor. A positional tracking device may use optical tracking, for example using one or more cameras in conjunction with infrared light emitting diodes (LEDs), and I or acoustic time-of-flight sensing. The display unit 160 receives the compressed texture 144, 415 and the 3-dimensional mesh 147, 425. The display unit 160 may comprise hardware (not shown) that forms an image (not shown), on the pair of displays 164. The image (not shown) combines the compressed texture 144 and the 3-dimensional mesh 147 together. In another embodiment (not shown), the display unit does not comprise hardware that forms an image; instead the hardware that forms an image is separate from the display unit and the display unit functions merely as a “dumb” display. The display unit 160 decompresses the compressed texture 144 to form a decompressed image. The decompressed image may be substantially the same as the image data 115 (of the corresponding frame of video) from the video camera 110. Alternatively, the decompressed image may, for example, be a higher resolution image (in pixel wide and I or pixels high) or a lower resolution image than the image data 115. In another embodiment (not shown), the display unit does not decompress the compressed texture 144; instead, the decompression is performed remotely, by a separate decompressor, and the decompressed texture is provided to the display unit. The display unit 160 overlays the decompressed texture 144 (either directly or as a decompressed image) onto the 3-dimensional mesh 147. By “overlay”, it is meant that the display unit 160 may generate a pair of stereoscopic images (one image for a user’s left eye and another image for the user’s right eye) in which, for example, the image 215 is UV-mapped onto the mesh 225. (It will be recalled that the image 215 and the mesh 225 are similar but that the mesh 225 does not exactly correspond to the image 215 as the posture of the woman 130 is slightly different between the image 215 and the mesh 225.) In another embodiment (not shown), the display unit is not operable to perform the overlay function; instead, the overlay function is performed remotely, by a separate overlay unit which overlays the decompressed texture onto the mesh 147 and sends an overlayed image to the display unit for display. In some embodiments of the compression hardware 140, if the 3-dimensional mesh 147 changes between two frames of the depth camera 120, then instead of sending a new mesh 147, the compression hardware 140 may determine an affine transformation. Suppose a subsequent mesh 147 is only slightly different (for example, slightly wider, slightly taller, and I or slightly rotated) compared to a preceding mesh 147. The compression hardware 140 may use a threshold to determine if a subsequent mesh 147 is only slightly different to a preceding mesh 147. If the subsequent mesh 147 is determined to be only slightly different then the compression hardware 140 may determine a matrix and a vector that together specify a affine transformation that, when applied to the preceding mesh 147, substantially generates the subsequent mesh 147. The threshold that is used by the compression hardware 140 to determine whether two depth images from the depth camera 120 are sufficiently similar to allow the use of an affine transformation may be applied to two depth frames of the raw data output 125 from the depth camera 120, or to two 3-dimensional meshes 147 after encoding by the point-cloud to 3-dimensional mesh conversion unit 146. In some embodiments, for an affine transformation, the compression hardware 140 may determine only a matrix or only a vector. The compression hardware 140 may send a modified network packet (not shown) in which, instead of a data field 425 for a mesh 147, the modified network packet specifies an affine transformation that is to be applied to a mesh 147 from a preceding network packet. On receipt of data indicative of an affine transformation, the display unit 160 may transform a previous mesh 147. As will be appreciated, the use of an affine transformation may reduce the size of the modified frame compared to a network packet 400 which includes the full data field 425. Figure 5 shows an embodiment 500 that has some similarities to the embodiment of Figure 1. Figure 5 shows: 510 a Microsoft Kinect device, 520 a desktop computer (having a graphics card, not shown), 522 a module to generate a mesh and a texture, 524 a module to encode the texture using H.264 or HEVC, 530 a network, in this embodiment the “Accord” network, 540 a Microsoft HoloLens device, 542 a module to decompress a texture from H.264 to RGB, 544 a module to UV-map a decompressed image onto a mesh, 546 a module to display an image to a user. The Kinect device 510 captures data indicative of a coloured 3D point cloud of a subject (one or more persons and / or objects) using a 3D camera, in this case a Microsoft Azure Kinect connected to the personal computer (PC) such as the desktop 520. Whereas Figure 1 has two separate streams of data (the stream 115 from the video camera 110 and the stream 125 from the depth camera 120), for Figure 5 the Kinect device 510 internally provides both streams. Also, whereas Figure 1 had compression hardware 140 that was distinct from the video camera 110 and the depth camera 120, in Figure 5 the functions of the texture compression unit 143 and the point-cloud to 3-dimensional mesh conversion unit 146 are integrated into the Kinect device 510. The desktop computer 520 may be an IBM compatible personal computer (PC). Preferably, the desktop computer 520 comprises a graphics card (not shown) as manufactured by Nvidia or AMD, for example. Alternatively, the computer 520 may be a laptop. The module 522 (within the computer 520) generates a spatial mesh with RGB texture data, from the point cloud. This is a 3D representation of the subject (one or more persons and / or objects). The module 524 (within the computer 520) performs realtime texture compression on a frame-by-frame basis to an H.264 or HEVC stream, for example using a graphics processor unit (GPU) that forms part of the graphics card. The computer 520 sends a network message, using the network 530, to the HoloLens device 540. In this embodiment, the network 530 is the HoloCare network stack (Accord). Further information regarding the Accord network may be found at: http: / / accord-framework. net https: / / github.com / accord-net The contents of both of these websites are hereby included by reference. Each network message represents one single temporal frame (of video and depth) and comprises: (a) the 3D mesh as vertices, triangles and texture coordinates (the complete mesh is sent for every frame), and (b) the compressed texture bitmap. The module 542 (within the HoloLens device 540) decompresses the encoded H.2641 HEVC frame on the HoloLens hardware in realtime using the DirectX application programming interface (API) from Microsoft. The frame is decoded on a graphics processor unit (GPU) of the device 540 directly as a single texture to a hardware texture without going via a central processing unit (CPU) of the device 540. The GPU texture is created only once. For every following frame, the compressed bitmap is uploaded to the GPU and decoded directly to the GPU texture. Note that while the present examples describe the use of DirectX, other graphics APIs may be used depending on hardware. For example, while DirectX may be used on the decoding side for a HoloLens implementation, other APIs with direct graphics hardware access (such as OpenGL) could also be used (e.g. when using other VR / mixed reality devices such as Apple Vision Pro or Meta Quest 3). Expressed using different words, the module 542 receives compressed H.2641 HEVC image frames. The module 542 decompresses the compressed image frames, to form decompressed image frames. Each decompressed image frame may define a two-dimensional array of pixels, and each pixel may have respective red, green, blue (RGB) data values. The module 544 (within the HoloLens device 540) creates a mesh (for the initial frame) or updates the mesh with a new mesh. To minimize the overhead of creating new CPU resources, the mesh may be reused between frames. The mesh may be updated using the Unity application programming interface (API) from Unity Software Inc. Internally, the same DirectX resources are used. The module 544 performs UV-mapping of the texture onto the mesh, to form a textured mesh. Expressed using different words, the module 544 receives a series of images and a series of meshes. Each image may be a 2-dimensional array of RGB pixels. For each image-mesh pair, the module 544 performs UV-mapping to map the image onto the mesh to form a textured mesh. The module 546 (within the HoloLens device 540) causes the textured mesh to be visible in the HoloLens as a “hologram”. Note that a visible image displayed to a user by a HoloLens device 540 is sometimes colloquially called a “hologram” although strictly speaking, a hologram is a recording of an interference pattern which can reproduce a 3D light field using diffraction. The various modules 522, 524, 542, 544, 546 may be implemented using software or hardware. In an alternative embodiment (that is not shown) of Figure 5, some other display device may be used instead of a HoloLens device. The modules 542, 544, 546 may be implemented using separate devices. For example (analogous to the discussion above for Figure 1), a “dumb” display device may be connected to a separate device that implements the decompression of module 542 and to a separate device that implements the UV-mapping of module 544. Figure 6, Figure 7 and Figure 8 relate to another embodiment of the present invention. Whereas Figure 1 related to an embodiment which used a video camera 110 and a depth camera 120, the embodiment of Figures 6, 7 &8 remotely renders an image, then compresses the remotely rendered image, then transfers the compressed image for display on a display unit such as a virtual reality headset. Figure 6 shows an embodiment 600, and shows: 610 a display unit for rendering images, 620 a network message to a remote renderer, 630 a remote renderer, 635 data (for example voxel data of a human heart), and 640 a network message from the remote renderer to the display unit. For the embodiment 600, the display unit 610 is a Microsoft HoloLens device (as was used for the display unit 510 of Figure 5). However, variations of the embodiment 600 may use other types of display unit 610. The network message 620 includes data that describes the viewpoint of a user that is using the display unit 610. (As will be explained in more detail for Figure 8, the viewpoint may include a camera transformation matrix.) Based at least in part on the data of the viewpoint network message 620, the remote renderer 630 generates a rendering of, for example, data 635. The data 635 may be medical data indicative of 3-dimensional voxel data of a human heart. The medical data 635 may be stored in RAM (not shown), or in flash memory (not shown) or on a hard disk (not shown). For clarity, Figure 6 does not show the rendered image. The renderer 630 compresses the rendered image using, for example, H.264 or HEVC encoding as was discussed for the embodiment of Figure 1. The renderer 630 sends a network message 640 to the display unit 610; the network message 640 includes data indicative of the compressed rendered image. The remote renderer 630 may comprise a desktop computer. The remote renderer 630 may comprise a graphics processor unit (GPU), for example mounted on a graphics card. Although the renderer 630 is termed a “remote” renderer, the renderer 630 may be physically near (for example in the same building or the same room) or adjacent to the display unit 610. However, the renderer 630 may be remote in the sense that the functions of the renderer 630 are implemented in a unit that is distinct from the display unit 610. The renderer 630 may be remote in the sense that communications 620, 640 between the display unit 610 and the renderer 630 are conveyed over a network (not shown) such as the network 150 of Figure 1. The embodiment 600 uses the capabilities of the display unit 610 and remote renderer 630 to achieve a result that neither the display unit 610 alone, nor the remote renderer 630 alone, could provide. That is, the display unit 610 can provide a convenient display capable of displaying a stereoscopic image to a user, and capable of measuring an orientation of a user’s head (and thereby determining the appropriate orientation of a projection plane). However, such a display unit 610 is preferably lightweight and thus may lack the computational ability to render complex images such as images based on voxel data 635. Conversely, the remote renderer 630 is preferably capable of rendering complex data, such as medical voxel data 635, to form, for example, a pair of stereoscopic images at a rate of 120 Hz. However, the renderer 630 may be too heavy and I or may use too much electrical power to be mounted onto a user’s head. The display unit 610 may send messages 620 at a rate of 120 Hz. Correspondingly, the renderer 630 may send messages 640 at a rate of 120 Hz. Figure 7 shows the network messages of Figure 6 in more detail, and shows: 620 the network message to a remote renderer, 640 the network message from the remote renderer, 722 an optional synchronization data field, 725 a field indicative of a camera transformation matrix, 742 an optional synchronization data field, and 745 a field indicative of a compressed image. The optional synchronization data field 722 can be used to facilitate synchronization between different network packets. The data field 722 may be a frame identifier, for example an integer number that is incremented by 1 for each successive frame (that is, for each successive network packet). The data field 725 contains data indicative of a camera transformation matrix of the display unit 610. The data field 725 may contain position and I or direction information for the display unit 610 (that is, the position of the display unit 610 and I or the gaze direction of the eyes of a user of the display unit 610). The data field 725 may provide a camera transformation matrix for each eye of a user of the display unit 610. Alternatively, the field 725 may specify the camera transformation matrix for a single eye (for example the left eye) and the remote renderer 630 may infer the camera transformation matrix for the right eye based on the interpupillary distance between the eyes of the user. The optional synchronization data field 742 can be used to facilitate synchronization between different network packets. The data field 742 may be a frame identifier, for example an integer number that is incremented by 1 for each successive frame (that is, for each successive network packet). The field 745 specifies data of a compressed image (using, for example, the H.264 or HEVC format) of an image that was rendered by the remote renderer 630. Figure 8 shows functional blocks of the embodiment of Figure 6 in more detail, and shows: 610 the display unit for rendering images, 620 the network message to a remote renderer, 630 the remote renderer, 640 the network message from the remote renderer to the display unit, 812 a module that determines a camera transformation matrix, 813 a communication path from module 812 to module 814, 814 a module that decompresses a texture, 816 a module that updates a projection plane that is to be used for display, 818 a module that causes a pair of stereoscopic images to be displayed, 832 a module that renders a pair of stereoscopic images, one for each eye, 834 a module that compresses the pair of images. The modules 812, 814, 816, 818, 832, 834 may be implemented using hardware or software. For Figure 8, the modules 812, 814, 816, 818 form part of the display unit 610. For Figure 8, the modules 832, 834 form part of the remote renderer 630. The matrix module 812 determines at least one camera transformation matrix for the display unit 610. The matrix module 812 may determine a pair of camera transformation matrixes, one for each eye of a user. The matrix module 812 may determine a skewed projection view matrix for each eye which describes a camera location and a field of view. The pair of matrices may have a parallel view direction, with an offset corresponding to the distance between a user’s eyes. The matrix module 812 may comprise an inertial measurement unit (IMU) 168 as was described for Figure 1. For Figure 8, the network message 620 comprises the transformation matrix (or matrices) from module 812. The synchronization field 722 is a sequential index. For Figure 8, the network message 620 is conveyed using the Accord network that was discussed in more detail for the network 530 of Figure 5. The render module 832 renders an image (not shown), for example based on the data 635 (shown for Figure 6 but, for clarity, omitted from Figure 8). The rendered image may be a volumetric rendering of a stack of computed tomography (CT) images. The rendered image may be rendered as a stereo pair. The rendered image may be a bitmap for each eye. The compression module 834 compresses (that is, encodes) the image to an H.264 or HEVC frame. For increased speed, the compression module 834 may use a graphics processing unit (GPU) such as an Nvidia GPU. The compression module 834 preferably operates in realtime. For Figure 8, the message 640 may be sent using the Accord network that was discussed in more detail for the network 530 of Figure 5. The network message 640 may comprise a compressed (that is, encoded) pair of stereo images. The synchronization field 742 (see Figure 7) may be the same as the synchronization field 722. The synchronization fields 722, 742 may be a frame identifier (ID). The frame ID may be used by the display unit 610 to match the rendered image with the appropriate camera transformation matrix. It will be appreciated that there may be a stream, for example at 120 Hz, of network messages 120 and a stream, for example at 120 Hz, of network messages 640. The user may change the position and I or orientation of their gaze between network messages; the frame ID allows the display unit 610 to correlate a rendered image (within the network message 640) with the appropriate camera transformation matrix. The use of the frame ID to associate a rendered image with the appropriate camera transformation matrix is discussed in more detail for module 816. The decompression module 814 decompresses (that is, decodes) the compressed H.264 or HEVC frame (that was rendered by the remote renderer 630). The frame may be decompressed by hardware of the display unit 610; the DirectX application programming interface (API) may be used to interface to the hardware. The decompressed image (or pair of stereoscopic images) may be in red, green, blue (RGB) format. Preferably, the frame is decompressed by a graphics processor unit (GPU) of the display unit 610 to a texture that is then used by display hardware of the display unit 610. Preferably, the decompression does not involve decompression by a central processing unit (CPU) of the display unit 610. The overlay module 816 updates a projection plane (not shown) using a camera transformation matrix from module 812. (The projection plane is used by module 818.) The overlay module 816 selects the appropriate camera transformation matrix by looking up the frame ID; it will be noted that Figure 8 shows that the module 812 has a communication path 813. The communication path 813 is used to transfer messages (not shown) that have a similar data content to that of the network messages 620. The overlay module 816 renders the texture from the decompression module 814 onto the projection plane. As will be appreciated by those skilled in the art, if the user (of the display device 610) moves their head around, the current projection plane may no longer be perpendicular to the view direction that was used by module 812; the camera transformation matrix compensate for this. The display module 818 displays, for example as a stereoscopic pair of images, the textured projection plane to the user. (As such, the display module 818 may include a pair of displays 164 as was discussed for Figure 1.) Such a display on a HoloLens device is sometimes colloquially called a “hologram” although it will be appreciated that, strictly speaking, the image that is displayed to the user is not a hologram as, strictly speaking, a hologram is a recording of an interference pattern which can reproduce a 3D light field using diffraction. The embodiments 600, 800 shown by Figure 6 and by Figure 8 obtained a camera transformation matrix from a display unit 610 and displayed an image on the same display unit 610. In an alternative embodiment (not shown), two or more display units 610 are used. One of the display units 610 is used to determine the camera transformation matrix; this display unit 610 may be regarded as a master display unit. The remotely rendered image is displayed on at least one slave display unit (the remotely rendered image may also be displayed on the master display unit). This alternative embodiment may be useful for teaching. A teacher may user the master display unit. One or more students, each using a slave display unit, may view the same rendered image as the teacher. The modules of Figure 5 or of Figure 8 may be implemented using one or more application specific integrated circuits (ASICs). Alternatively, one or more of the modules may be implemented using a configurable integrated circuit, for example a field programmable gate array (FPGA). Alternatively, one or more of the modules may be implemented using software that defines instructions that specify the operation of a processor. The processor may be a reduced instruction set computer (RISC) architecture, a complex instruction set computer (CISC) architecture, a very long instruction word (VLIW) architecture, or a digital signal processing (DSP) architecture. Similarly, it will be appreciated that any of the hardware functions or software functions described above may be implemented using ASICs, FPGAs or software running on a processor (whether a RISC, CISC, VLIW or DSP architecture). Figure 9 shows an embodiment 900 of a method that generally corresponds to Figures 1 to 5, and shows: 910 mesh generation, 920 texture generation, 930 compression (encoding) of the texture, 940 data transmission, 950 decoding (decompression) of the texture, 960 texture mapping the texture onto the mesh. Steps 910 to 940 may be regarded as relating to clause 13 appended to this disclosure as originally filed. Steps 940 to 960 may be regarded as relating to clause 14 appended to this disclosure as originally filed. At step 910, a mesh is generated. The mesh may comprise data indicative of triangles. The mesh may be generated based on point cloud data from, for example, a camera such as the depth camera 120. At step 920, a texture is generated. The texture may comprise data indicative of a frame of video data. The texture may be generated based on data from, for example, a camera such as the video camera 110. At step 930, the texture is compressed (that is, encoded). The texture may be compressed using, for example, the H.264 or HEVC video encoding standards. At step 940, data is transmitted. The data may be transmitted as a single data frame (that is, a single network packet) comprising both mesh data and texture data. Alternatively, the data may be transmitted using two separate data frames (that is, two separate network packets), one data frame for the mesh data and another data frame for the texture data; both data frames may comprise synchronization data. At step 950, the compressed texture is decoded (that is, decompressed) to form data indicative of a decompressed texture. The compressed texture may be decompressed using, for example, the H.264 or HEVC video standards. At step 960, the decompressed texture is texture-mapped on to the mesh to form data indicative of an image. Figure 10 shows an embodiment of a method 1000 that generally corresponds to Figures 6 to 8, and shows: 1010 generation of a camera transformation matrix, 1020 transmission of the matrix, 1030 rendering of an image (using the matrix), 1040 compression (encoding) of the image, 1050 transmission of the compressed image, 1060 decompression (decoding) the compressed image, 1070 overlaying of the decompressed image onto a projection plane. Steps 1010 to 1070 may be regarded as relating to claim 12 of this disclosure as originally filed. At step 1010, a camera transformation matrix is generated. The camera transformation matrix may be generated using the display unit 610. At step 1020, data indicative of the camera transformation matrix is transmitted to a renderer. The renderer may be the remote renderer 630. At step 1030, the renderer renders an image, taking into consideration the camera transformation matrix. At step 1040, the renderer compresses (that is, encodes) the image. The image may be compressed using, for example, the H.264 or HEVC video standards. At step 1050, the renderer transmits the compressed image. At step 1060, the compressed image is decompressed (that is, decoded). The image may be decompressed using, for example, the H.264 or HEVC video standards. At step 1070, the decompressed image is overlayed onto a projection plane. As those skilled in the art will appreciate, the methods shown at Figure 9 and at Figure 10 may be modified. Some steps (for example data transmission) may be removed. Other steps may be added. The abstract as filed is hereby included by reference. Further aspects and features of the invention are set out in the following numbered clauses. Clause 1. A system (100, 500) comprising: a video camera (110) operable to generate video data (115), a compression unit (143) operable to compress the video data (115) to form compressed video data (144) using a first compression algorithm, a depth camera (120) operable to generate point cloud data (125), a point-cloud to 3-dimensional mesh conversion unit (146) operable to generate mesh data (147) based on the point cloud data (125), using a second compression algorithm, wherein the mesh data (147) defines a 3-dimensional mesh; a data packetiser for producing packets comprising compressed texture data and packets comprising mesh data, the packets including sequence information (405); a transmission system (150) for transmitting the data packets; a decompressor (542) operable to decompress the compressed video data (144) to form decompressed image data; an overlay unit (544) operable to overlay the decompressed image data onto a 3-dimensional mesh defined by the mesh data (147) to form a textured mesh; and a display unit (160, 540) operable to display the textured mesh. Clause 2. A system (100, 500) according to clause 1, wherein the compression unit (143) and the decompressor (542) are operable to use the H.264 or HEVC video compression formats. Clause 3. A system (100, 500) according to clause 1 or 2, wherein the point-cloud to 3-dimensional mesh conversion unit (146) is operable to form mesh data (147) comprising a mesh of triangles. Clause 4. A system (100, 500) according to any preceding clause, wherein the point-cloud to 3-dimensional mesh conversion unit (146) is operable to remove substantially static features from the mesh data (147). Clause 5. A system (100, 500) according to any preceding clause, wherein the point-cloud to 3-dimensional mesh conversion unit (146) is operable to determine an affine transformation. Clause 6. A system (100, 500) according to any preceding clause, wherein the overlay unit (544) is operable to perform UV-mapping. Clause 7. A system (100, 500) according to any preceding clause, wherein the display unit (160, 540) comprises a pair of displays (164), one of the displays for a user's left eye, the other display for a user's right eye. Clause 8. A system (100, 500) according to any preceding clause, wherein the display unit (160, 540) comprises an inertial measurement unit (168). Clause 9. A system (100, 500) according to any preceding clause, comprising a unitary compression hardware device (140), wherein the compression hardware (140) comprises: the video camera (110), the compression unit (143), the depth camera (120), and the point-cloud to 3-dimensional mesh conversion unit (146). Clause 10. A system (100, 500) according to any preceding clause, wherein the system is operable to transfer the compressed video data (144) and the mesh data (147) over a network (150). Clause 11. A system (100, 500) according to any preceding clause, wherein the system is operable to transfer the compressed video data (144) and the mesh data (147) using network packets (400). Clause 12. A system (100, 500) according to clause 11, wherein the network packets (400) comprise synchronization data (405). Clause 13. A method of compressing video data, comprising the steps of: receiving video data (115); compressing (143) the video data (115) to form compressed video data (144), using a first algorithm; receiving point cloud data (125); compressing (146), using a second algorithm, the point cloud data (125) to generate mesh data (147) indicative of a 3-dimensional mesh; and transmitting the compressed video data (144) and the mesh data (147). Clause 14. A method of decompressing video data, comprising the steps of: receiving data (415) indicative of compressed video; decompressing (542) the data (415) to form decompressed image data; receiving data (425) indicative of a 3-dimensional mesh; and overlaying (544) the decompressed image data onto the 3-dimensional mesh, to form an overlaid image. Clause 15. A method according to clause 14, wherein the step of overlaying (544) comprises UV-mapping the decompressed image data onto the 3-dimensional mesh. Clause 16. A method according to clause 14 or 15, further comprising the step of displaying (546) the overlayed image. Clause 17. A method of transporting video data over a network (150), comprising the steps of: compressing and transmitting data according to the method of clause 13; and decompressing data according to the method of any of clauses 14 to 16. 08 05 25

Claims

1. A system comprising:a stereoscopic display unit and a remote renderer;the stereoscopic display unit comprising a matrix module operable to determine at least one camera transformation matrix, the display unit operable to send the at least one camera transformation matrix over a network to the remote renderer;the remote renderer comprising:a render module operable to render image data based at least in part on the at least one camera transformation matrix, the image data comprising a stereoscopic image pair;a compression module operable to compress the rendered image data to form compressed image data;wherein the remote renderer is operable to send the compressed image data to the display unit via the network;wherein the display unit further comprises:a decompression module operable to decompress the compressed imagedata to form decompressed image data; andan overlay module operable to overlay the decompressed image data onto a projection plane to form projected image data, wherein the projection plane is based on the at least one camera transformation matrix;wherein the display module is operable to display the projected image data as a stereoscopic image pair.

2. A system according to claim 1, wherein the display unit is operable to measure an orientation of a user’s head and thereby determine the orientation of the projection plane.

3. A system according to claim 1 or 2, wherein the matrix module comprises an inertial measurement unit (IMU).

4. A system according to any of the preceding claims, wherein the render module is operable to render the image data based on voxel data.

5. A system according to any of the preceding claims, wherein the compression module and the decompression module use the H.264 or HEVC video compression standard.

6. A system according to any of the preceding claims, wherein the overlay module is08 05 25operable to overlay an RGB texture onto the projection plane.

7. A system according to any of the preceding claims, wherein the at least one cameratransformation matrix is sent over the network as part of a matrix network frame.

8. A system according to any of the preceding claims, wherein the compressed image data is sent over a network as part of a compressed image data network frame.

9. A system according to claim 7 and / or claim 8, wherein the matrix network frame and / or the compressed image data network frame, comprises synchronization data.

10. A system according to any of the preceding claims, comprising a master display unit and a slave display unit.

11. A method of compressing a rendered image, the method comprising the steps of: receiving data indicative of a camera transformation matrix of a display unit from a remote stereoscopic display unit via a network;rendering an image based at least in part on the camera transformation matrix, to form a rendered image;compressing the rendered image to form a compressed image;transmitting the compressed image to the stereoscopic display unit for display.

12. A method according to claim 11, wherein the step of compressing comprises using H.264 or HEVC compression.

13. A method according to claim 11 or 12, wherein the display unit performs the step of decompressing the compressed image to form a decompressed image.

14. A method according to claim 13, wherein the display unit performs the step of overlaying the compressed image onto a projection plane.

15. A method according to claim 14, wherein the step of overlaying comprises UV-mapping.

16. A method according to any of claims 11 to 15, wherein the display unit performs the step of displaying an image to a user.

17. A method according to any of claims 11 to 16,wherein the received data is associated with synchronization data, andwherein the step of transmitting the compressed image to the display unit comprises including synchronization data that is based on the synchronization data.

18. A method according to any of claims 11 to 17,wherein a first display unit is operable as a master display unit to determine thecamera transformation matrix, andwherein a second display unit is operable as a slave display unit to render the rendered image.08 05 25

Images