Apparatus and method for obfuscating information

Abstract

There is provided herein an apparatus for processing an imaging feed from an imaging device, comprising: a device for converting the imaging feed; and a communication interface for transmitting the imaging feed to another device; wherein only the converted imaging feed is transmitted out of the communication interface.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for blurring sensitive information. In particular, the present invention relates to an apparatus and method for processing an imaging feed from an imaging device to blur certain areas as required. Background Art

[0002] In various situations, it is desirable to obscure sensitive information from an imaging feed, such as a video stream. For example, in the field of telemedicine or telesurgery, a patient's data, face, or tattoos may be visible. A surgeon or other user may wish to obscure this data in order to protect the patient's privacy. The present invention relates to an improved scheme for achieving this purpose. Summary of the invention

[0003] According to a first aspect of the invention, there is provided an apparatus for processing an imaging feed from an imaging device, comprising: a device for converting the imaging feed; and a communication interface for transmitting the imaging feed to another device; wherein only the converted imaging feed is transmitted out of the communication interface.

[0004] By transmitting only the converted imaging feed out of the communication interface, the security of the device may be increased. Thus, the original or "unconverted" imaging feed is not distributed and cannot be intercepted. The conversion may be used to obfuscate areas of the imaging feed, preferably areas that include sensitive information. This means that these areas are not transmitted out of the communication interface. This helps to ensure that the converted imaging feed is immutable (i.e., the original, unconverted imaging feed cannot be restored). Preferably, only the immutable converted imaging feed is transmitted out of the communication interface.

[0005] Preferably, the other device is located outside the communication interface. As used herein, the term "external" preferably means that the other device is logically located outside the communication interface. The communication interface preferably provides an interface between a logical network and an external entity. The communication device is preferably located inside or locally of the imaging device. "Local or internal" preferably means that it is provided as part of the same logical network. This may be the case even if they are provided on different hardware, such as in adjacent rooms. In addition, the conversion is preferably performed locally. The apparatus may also include a processor for converting the imaging feed. The processor may be internal to the communication interface because it is part of the same logical network. Therefore, the original (unconverted) version of the imaging feed may remain within the same logical network. Preferably, the device for converting the imaging feed and the communication interface may be provided on the same (independent) device. For example, the device for converting the imaging feed and the communication interface may be provided on the same computer device. In this case, the device for converting the imaging feed may be a processor of the computer device. In this case, the communication interface enables the computer device to communicate with another (external) device. The communication may be via an Internet connection (e.g., to facilitate communication with another external device located in a physically remote location) or via a wired connection (e.g., to facilitate communication with another external device located in the same location (e.g., a building or room). The logical network preferably includes the imaging device. For example, the computer device may be connected to the imaging device or may include the imaging device.

[0006] The other device may be a server, wherein the server is preferably configured to distribute the converted imaging feed. As used herein, the term "server" preferably refers to a computer device or computer program configured to manage a centralized resource or service in a network, and / or manage access to the centralized resource or service. As used herein, the term "distribution" preferably encompasses broadcasting, transmitting, displaying, recording, saving, and / or assembling (although the term is not limited to these actions). For example, a server may receive a converted imaging feed and then broadcast it to one or more other entities and / or devices. A server may distribute to a network of other entities (e.g., devices).

[0007] Preferably, the other device is remote and / or geographically distant from the imaging device and the communication interface. As used herein, the terms "geographically distant" and "remote" preferably encompass being located in a different country, city, building or even room. The term "remote" preferably means being part of a separate logical network. This may include being geographically close (e.g., in the same room) but logically remote (i.e., logically distinct and / or separate).

[0008] In a preferred embodiment, the imaging feed is not recorded before conversion by the conversion device. This can improve the security of any information that the user wishes to obscure from the imaging feed, as it cannot be recovered from the stored file.

[0009] Preferably, the imaging feed is transmitted from the imaging device to the conversion device without being recorded. As used herein, the term "transmit" preferably means any way of sending the imaging feed to another entity, including electronic transmission or broadcast. Preferably, the imaging feed is transmitted from the imaging device to the conversion device without being recorded by any recording device. More preferably, the imaging feed is transmitted from the imaging device to the conversion device without being broadcast. Thus, preferably only the converted imaging feed may be transmitted or recorded.

[0010] The apparatus preferably also includes means (preferably in the form of an authorizing device or module, and / or preferably embodied in a processor and associated memory) for authorizing an entity to control the imaging feed before transmitting the imaging feed to another device. This functionality may also be provided independently. In this case, the entity preferably only receives the converted imaging feed.

[0011] According to another aspect of the present invention, there is provided an apparatus for processing an imaging feed from an imaging device, comprising: a communication interface for transmitting the imaging feed to another device; and a device (preferably in the form of an authorization device or module, and / or preferably embodied in a processor and associated memory) for authorizing an entity to control the imaging feed before transmitting the imaging feed to the other device.

[0012] In other words, the apparatus may be configured to authorize an entity to control the imaging feed before transmitting the imaging feed to another device.

[0013] The entity may be a (human) user or a (computer) device. An "authorizing device" may be or be embodied in a module of a computer device or server or in a separate and / or different device. As used herein, the term "control" preferably includes passive actions such as (but not limited to) access and viewing, and active actions such as (but not limited to) editing, manipulating and indicating. The other device may be a server. This may enhance the security of the imaging feed by requiring authorization to control the imaging feed. This may prevent external entities from controlling (e.g. viewing and / or editing and / or manipulating) the imaging feed.

[0014] As used herein, the term "before" preferably refers to an earlier position in a communication path and / or an earlier point in time.

[0015] Preferably, the authorizing device is configured to authorize an entity to view and / or edit the imaging feed before the imaging feed is transmitted to another device. This may occur before a transformation is applied. This may authorize an entity to view the original or untransformed imaging feed. This may authorize an entity to edit the original or untransformed imaging feed, for example by defining a transformation. These may be separate and / or different authorizations.

[0016] Preferably, the authorizing device is configured to authorize an entity to define transformations to be applied to the imaging feed before the imaging feed is transmitted to another device. This may authorize an entity (internal or external) to define transformations, preferably to apply the transformations before the imaging feed is transmitted outside the internal device. This may be outside the communication interface and / or logical network. The entity authorized to define transformations may be the conversion device or another device (e.g., a device external to the conversion device).

[0017] Preferably, the authorizing device is configured to prohibit the entity from controlling the imaging feed before the imaging feed is transmitted to another device. This may include prohibiting the entity from viewing and / or editing and / or manipulating the imaging feed. These may be separate and / or different authorizations. This may authorize the entity to define and / or edit transformations applied to the imaging feed.

[0018] In a preferred embodiment, the authorizing device is configured to prohibit an entity from viewing the imaging feed before the imaging feed is transmitted to another device, and to authorize the same entity to define transformations applied to the imaging feed, preferably before the imaging feed is transmitted to another device. This can authorize external entities to define transformations while still ensuring that only the transformed imaging feed is transmitted outside the internal device (e.g., a logical network, which preferably contains the imaging device and may also contain a communication interface for transmitting the imaging feed). For example, an entity may be authorized to define (new) transformations and / or edit transformations, but still only view the transformed imaging feed. Preferably, only the transformed imaging feed is transmitted (and / or transferred) to the entity even if the entity is defining another (new) transformation and / or editing a transformation. Preferably, the entity views a preview of another (new) transformation and / or edited transformation applied to and / or transformed imaging feed (i.e., views the current transformation and a preview of another (new) transformation and / or edited transformation at the same time).

[0019] In a preferred embodiment, there may be different levels of authorization. The authorization device may define different authorizations and / or authorization levels for different entities. The authorization device may define different authorization combinations for different entities. Different (or differentiated) authorization levels may be defined by different authorization level combinations.

[0020] Preferably, the authorization device is configured to determine the authorization level of the entity based on the state of the entity. As used herein, the term "state" may refer to the relationship of the entity to the imaging feed and / or the attributes assigned to the entity. For example, an authorization level may be assigned to a user. As another example, an authorization level may be assigned to a device, preferably depending on the state of whether the device is located inside or outside the imaging feed (e.g., inside or outside the logical network of the imaging device and / or communication interface). As another example, an authorization level may be assigned to an entity based on its state as a controller, initiator, or participant in a "session." A "session" may refer to the implementation of instructions according to the present invention.

[0021] In a preferred embodiment, the authorization device is configured to authorize the entity controlling the communication interface to view the imaging feed before transmitting the imaging feed to another device. This can authorize the entity to view the original or unconverted imaging feed, but only allow the converted imaging feed to be transmitted out of the communication interface.

[0022] In some embodiments, the apparatus may further include another communication interface for transmitting another imaging feed to the server. This may allow two or more imaging feeds to be transmitted to the server. The authorization device may be configured to define different authorizations for the another communication interface. The authorization device may be configured to define different authorizations for the entity for each imaging feed and / or communication interface.

[0023] The transformed imaging feed may include an imaging feed from which at least one region has been extracted. In this way, at least one region of the imaging feed may be completely removed from the imaging feed such that it cannot be restored. Such an implementation helps ensure that the transformation is immutable.

[0024] The apparatus may further comprise a processor for extracting at least one region of the imaging feed and replacing it with at least one region of the substitute image to generate a transformed imaging feed. Thus, the processor may be configured to transform the imaging feed. This functionality may be provided independently.

[0025] According to another aspect of the present invention, there is provided an apparatus for processing an imaging feed from an imaging device, comprising: a processor for extracting at least one region of the imaging feed and replacing it with at least one region of a substitute image to generate a transformed imaging feed.

[0026] The method may advantageously convert the imaging feed in an immutable manner. At least one region of the imaging feed may include sensitive information. In this way, at least one region of the imaging feed may be completely removed from the imaging feed such that it cannot be recovered. Thus, such conversion may be immutable. Thus, the processor may be configured to convert the imaging feed.

[0027] A "substitute image" may include a blank image, an image having uniform pixel values ​​(e.g., uniform color), or a substitute image (e.g., displaying a non-uniform display, including an image of varying pixel values). The substitute image may include a processed copy of the imaging feed, for example, the substitute image may include a blurred copy (e.g., a pixelated or blurred copy or version) of the imaging feed.

[0028] The processing step may be applied to an image of the imaging feed.The processing step may also include the step of generating an image of the imaging feed.

[0029] The converted imaging feed may be composed of a series of converted images. The processing step may also include performing these steps on images of the imaging feed. The processing step may further include: generating the converted imaging feed using the converted images. The processor may generate the converted imaging feed using the timestamp.

[0030] Preferably, at least one region of the substitute image spatially corresponds to at least one region of the imaging feed. As used herein, "spatially corresponding" preferably means that at least one region of the substitute image has the same configuration (e.g., size, position, and orientation) as at least one region of the imaging feed. If the substitute image and the imaging feed have the same size, then at least one region of the substitute image may have the same coordinates as at least one region of the imaging feed.

[0031] The substitute image may be an image of the imaging feed to which the processor is configured to apply a filter. The image of the imaging feed may be viewed as another imaging feed. Thus, the substitute image may be a "copy" of the imaging feed. The image may be temporarily recorded. The image may have an image file format, such as a bitmap image file format.

[0032] Preferably, the processor is configured to apply the filter to a larger portion of the image than the at least one region of the replacement image. More preferably, the processor is configured to apply the filter to the entire replacement image. This may prevent degradation of the filtering quality at the edges of the at least one region. This may also improve computational efficiency.

[0033] Preferably, the filter is an occlusion filter. As used herein, "occlusion filter" preferably means any filter that reduces image detail. Preferably, the filter is a blur filter. As used herein, "blur filter" preferably means a filter that blurs an image, thereby reducing clarity and / or detail. Even more preferably, the filter is a Gaussian blur filter. In some embodiments, the filter can be a pixelation filter. As used herein, "pixelation filter" preferably means a filter that reduces pixel density by replacing a number of pixels greater than one with a color value that is the average of the color values ​​of these pixels. The same device can be configured to apply different filters.

[0034] Preferably, the filter can be selected based on a performance indicator of the device. In a preferred embodiment, the performance indicator is related to processing power. As used herein, "indicator" preferably means a measured and / or calculated numerical or non-numerical value and / or parameter and / or set of values ​​and / or parameters. The performance indicator can be determined continuously or intermittently over time (e.g., by measurement or calculation). The performance indicator may be related to processing speed and / or connection speed. Different filters can be selected based on changes in the performance indicator over time. This helps prevent interruptions in imaging feed conversion and / or transmission.

[0035] In a preferred embodiment, the imaging feed is real-time, and / or the conversion of the imaging feed is performed in real-time to generate a real-time converted imaging feed. Preferably, the conversion is performed within 500 milliseconds of receiving the imaging feed, preferably within 300 milliseconds, and more preferably within 200 milliseconds. As used herein, the term "real-time" preferably includes transmitting the imaging feed outside the processor (e.g., outside the communication interface) within 500 milliseconds, preferably within 300 milliseconds, and more preferably within 200 milliseconds after the imaging feed is generated and / or transmitted to the processor.

[0036] The apparatus preferably also includes a user interface configured to enable a user to define and / or edit transformations applied to the imaging feed. As used herein, "user interface" preferably refers to any device or means that allows a user to interact with the apparatus of the present invention. Preferably, it allows a user to provide instructions to the processor. The ability to define and / or edit transformations preferably depends on the authorization and / or status of the entity and / or user.

[0037] Preferably, the user interface is configured to enable a user to define at least one area. The user interface and / or the processor are preferably configured to convert input to and / or interaction with the user interface into computer executable instructions, thereby defining at least one area.

[0038] In a preferred embodiment, the user interface is configured to enable a user to define the position and / or shape and / or size and / or orientation of at least one area. Preferably, the user interface is configured to enable a user to select a shape. More preferably, the shape can be selected from a list including at least a rectangle and an ellipse. These shapes preferably include specific examples of squares and circles. The shape can be used to define the imaging feed area to be blocked. This allows the user to quickly and efficiently define these areas.

[0039] Preferably, the user interface is configured to enable a user to define and / or edit transformations when transmitting an imaging feed. This may enable a user to edit the transformations of an imaging feed when transmitting and / or transferring the imaging feed. As described above, this may depend on the authorization of the user and / or device. The user interface and / or processor may be configured to convert the user's interaction into computer executable instructions. The instructions may define the edited transformations. The instructions may be sent and / or transmitted to a server. The instructions may be sent and / or transmitted to the imaging device and / or associated processor (which may be done by the server). Preferably, the editing of the transformations is completed only when the user enters confirmation (preferably in the user interface).

[0040] In a preferred embodiment, the apparatus is further configured to output an alert upon detecting that the imaging device is capable of changing the configuration of the imaging feed. Preferably, this occurs upon detecting that the imaging device is capable of changing the pan, tilt and / or zoom settings of the imaging feed. This may alert a user or device that the imaging feed may be reconfigured so that a region contains different information.

[0041] The device preferably also includes an imaging device. The imaging device may be located in a medical facility. Preferably, the imaging feed is from a medical facility. Any features of the device may be provided in a medical facility. The device may be provided in a medical facility. For example, the medical facility may be a surgical facility. The medical facility may be an operating room. The user interface may be provided in a medical facility. The device may include multiple imaging feeds from different medical facilities. For example, they may come from different departments within a hospital, including different operating rooms, laboratories, wards, or offices. Alternatively, they may come from different hospitals, clinics, or universities. Users who may interact with the user interface may be medical professionals, such as doctors, dentists, nurses, medical students, medical assistants, administrators, or the like.

[0042] According to another aspect of the present invention, there is provided herein a method for processing an imaging feed from an imaging device, the method comprising: converting the imaging feed; and transmitting only the converted imaging feed to another device.

[0043] According to another aspect of the present invention, there is provided herein a method for processing an imaging feed from an imaging device, the method comprising: transmitting the imaging feed to another device; and authorizing an entity to control the imaging feed before transmitting the imaging feed to the other device.

[0044] According to another aspect of the invention, there is provided herein a method for processing an imaging feed from an imaging device, the method comprising: extracting at least one region of the imaging feed; and replacing at least one region of a substitute image to generate a transformed imaging feed.

[0045] The method may further comprise the device features described above.The described method and device features may be provided in any combination.

[0046] The term "device" as used herein preferably encompasses a device or a network of devices. As described herein, each "device" may be provided as a device comprising a processor and associated memory. Alternatively, the device may be provided as a module of a larger device, wherein the larger device comprises a processor and associated memory that communicates with the module.

[0047] The invention extends to methods and / or apparatus substantially as herein described with reference to the accompanying drawings.

[0048] Any apparatus features described herein may also be provided as method features, and vice versa.

[0049] Any feature in any aspect of the present invention may be applied to other aspects of the present invention in any appropriate combination. Specifically, method aspects may be applied to device aspects, and vice versa. In addition, any, some and / or all features in one aspect may be applied to any, some and / or all features in any other aspect in any appropriate combination.

[0050] Furthermore, features implemented in hardware may also be implemented in software and vice versa. Any references herein to software and hardware features should be interpreted accordingly.

[0051] Any device features described herein may also be provided as method features, and vice versa.As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0052] It should also be understood that specific combinations of the various features described and defined in any aspect of the present invention may be implemented and / or provided and / or used independently.

[0053] These and other aspects of the present invention will become apparent from the exemplary embodiments described below with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 An exemplary computer device that can implement the methods described herein is shown;

[0055] Figure 2 An overview of an exemplary device network is shown;

[0056] Figure 3 An exemplary user interface display of a multi-feed session is shown, including a set of feeds transmitted over a network of devices;

[0057] Figure 4ashows an example user interface display during session initialization;

[0058] Figure 4b shows an exemplary user interface display of a multi-feed session during the process of adding an additional feed;

[0059] Figure 5 showing a user interface display of a preview display when applying an occlusion shape;

[0060] Figure 6 shows that when further occlusion shapes are applied Figure 5 A preview display of the imaging feed;

[0061] Figure 7 An exemplary workflow of a user interacting with a user interface is shown;

[0062] Figure 8 an exemplary user interface display showing a multi-feed display of a multi-feed session to be edited;

[0063] Fig. 9 shows a preview display of the imaging feed while editing the occlusion shape;

[0064] Fig.10 The logical flow of constructing the imaging feed stream is shown;

[0065] Fig.11 shows the conversion logic flow of the imaging feed; and

[0066] Fig.12 The flow of imaging feed conversion is intuitively illustrated. DETAILED DESCRIPTION

[0067] System Overview

[0068] refer to Figure 1 The methods, apparatuses, and systems disclosed herein are typically implemented using at least one computer device 1000, 1100, 1200, 1300 connected or connectable to a server 2000, and are typically computer-implemented methods.

[0069] The computer device 1000 includes a processor in the form of a computer processing unit (CPU) 1002, a communication interface 1004, a memory 1006, a storage 1008, a user interface 1010, and an authorization module 1016, and is coupled to each other via a bus 1012. The CPU 1002 is also connected to an imaging device 1014, which is typically a camera, but can be any form of device for generating an imaging feed (real or computer-generated). For example, it can provide an imaging feed including visual test results, X-rays, computer-generated visualizations, etc. Each computer device 1000, 1100, 1200, 1300 can include or be connected to more than one imaging device 1014.

[0070] CPU 1002 executes instructions, including instructions stored in memory 1006 and / or storage 1008 .

[0071] The communication interface 1004 enables the computer device 1000 to communicate with the server 2000 and other computer devices. The communication interface 1004 may include an Ethernet network adapter that couples the bus 1012 to an Ethernet socket. Such an Ethernet socket may be coupled to a network, such as the Internet. It should be understood that the communication interface may be arranged to communicate using a variety of media, such as a local area network (e.g., the Internet), infrared, and

[0072] The memory 1006 stores instructions and other information for use by the CPU 1002. The memory is the main memory of the computer device 1000. It typically includes random access memory (RAM) and read-only memory (ROM).

[0073] The memory 1008 provides mass storage for the computer device 1000. In various implementations, the memory is an integrated storage device in the form of a hard disk device, a flash memory, or some other similar solid-state memory device, or an array of such devices.

[0074] The user interface 1010 enables a user to interact with the computer device 1000 and includes an input device and / or an output device. For example, the user interface may include a display and an input / output device such as a keyboard, a mouse, or a touch screen interface.

[0075] The authorization module 1016 defines and implements the authorization of the device 1000, 1100, 1200, 1300. This can define which entities (e.g., other devices and / or users) are authorized to provide instructions executed by the computer device 1000, 1100, 1200, 1300. The "authorization module" of each device preferably simply represents an entity (processor and / or device) configured to perform the authorization function of the device and / or the authorization device for the device. The authorization module can be a module provided on each device or located at a server. It can be a module of the processor, a separate module, or a module embodied in the processor. It may refer to any entity configured to execute authorization instructions. These instructions can be recorded in the memory of the device and / or stored in the server and transmitted to the device.

[0076] The present invention can preferably be implemented on general-purpose computer equipment. This allows participants to participate in remote surgery without the need for specialized hardware.

[0077] A computer program product is disclosed herein, which includes instructions for performing various aspects of the methods described below. The computer program product can be stored in any of the memory 1006, the storage 1008 and / or the removable storage (e.g., a universal serial bus storage device) at different stages. The storage of the computer program product is non-temporary, unless the instructions contained in the computer program product are being executed by the CPU 1002, in which case the instructions are sometimes temporarily stored in the CPU or the memory. The instructions of the present invention are usually recorded on the server 2000. The server 2000 can preferably send the required instructions to the CPU 1002 via the communication interface 1016 to execute the methods described herein. The server 2000 can send updated or modified instructions.

[0078] It should also be noted that the removable memory may be removed from the computer device 1000 so that the computer program product may be stored separately from the computer device from time to time. Different computer program products, or different aspects of a single overall computer program product, may be present on any computer device used by a user.

[0079] This paper also discloses a system (e.g., network) comprising multiple computer devices, wherein each computer device can perform different parts of the method disclosed herein. Each computer device can be configured for a specific purpose. Typically, multiple computer devices are arranged to communicate with each other and / or communicate with server 2000 so that data can be transmitted between computer devices or between equipment and server 2000. Each computer device can be defined by different authorizations.

[0080] Figure 2A schematic overview of devices forming a network that can be used to implement the present invention is shown. Server 2000 transmits instructions to other devices of the network for execution. The system according to the present invention allows a first computer device (referred to as session owner device 1000) to initiate a telesurgery session in which multiple imaging feeds can be collated and displayed to form an aggregate display. The imaging feeds can originate from imaging devices of session owner device 1000, "first participant" device 1100, "second participant" device 1200, and / or "third participant" device 1300. Each of these devices typically includes a computer device associated with the session owner device 1000. Figure 1 . Specifically, each of the session owner device 1000, the "first participant" 1100, the "second participant" 1200, and the "third participant" 1300 devices includes an imaging device 1014, and the communication interface 1004 is configured to transmit an imaging feed from the imaging device 1014 to the server 2000. The imaging feed may originate from a physical imaging device, such as a computer camera, a capture card, or other imaging feed (e.g., video) generating device. The imaging feed may also be generated by a user, such as through screen sharing or a virtual camera. Typically, up to four imaging feeds can be active at any particular time.

[0081] The session owner device 1000, the "first participant" device 1100, the "second participant" device 1200, and the "third participant" device 1300 may be located at the same location or at different locations (including some devices at the same location and some devices at other locations), and may be located in any combination. For example, the session owner device 1000 may be located in a surgeon's office in a hospital, the first participant device 1100 and the second participant device 1200 may be located in an operating room in the hospital, and the third participant device 1300 may be located at a university in a different city.

[0082] The imaging feed is converted on its source device to generate a converted imaging feed with some areas blocked (this is performed by methods described later). The conversion is performed by the CPU 1002 on the device. The converted feed is then sent to the server 2000 via the communication interface 1004. The imaging feed is not transmitted to the server 2000 in an unconverted state, nor is it recorded in an unconverted state. This enhances privacy control of the imaging feed.

[0083] The server 2000 collates the converted feeds into a "masked" multi-stream feed 232. The multi-stream feed 232 is distributed to any viewing device of the session, including the session owner 1000 and participants 1100, 1200, 1300, and any other remote viewing devices 2400. These devices can view the converted or "masked" multi-stream feed through their user interfaces 1010. Continuing with the above example, the user interfaces of the devices located in the surgeon's office, the operating room, and the university will all display the multi-stream feed transmitted by the server 2000. The remote viewing devices 2400 can be located anywhere in the world and can include multiple remote viewing devices located in different locations around the world. Typically, remote viewing devices need to obtain authorization to access the server's information. This can be an authorization associated with the user and / or the device.

[0084] The multi-stream feed 232 may also be recorded to the remote memory 2500 and / or the internal memory 1008 of one or more devices. This means that the feed is only transmitted or saved if the occlusion regions are applied (i.e., the imaging feed is only transmitted or saved in a "converted" state). Therefore, any viewing device, whether viewing in real time or via playback, will only see the imaging feed with the occlusion applied. As will be explained in more detail later, the imaging feed is transmitted and saved with the occlusion information embedded in the file so that it cannot be easily removed. This provides security to ensure that sensitive information cannot be compromised.

[0085] The user interface of the privacy control tool is displayed

[0086] The server 2000 receives imaging feeds from devices 1000, 1100, 1200, 1300 and then collates them into a collective display or "multi-feed" display. The server 2000 distributes the multi-feed display to any viewing device of the session, including the session owner 1000, participant devices 1100, 1200, 1300, and any other remote viewing devices 2400. Figure 3 An exemplary embodiment of a user interface display 200 of software according to the present invention is shown. Figure 3 The user interface display 200 of is in a multi-feed display 232 configuration, which displays four imaging feeds 202, 204, 206, and 208. The multi-feed display 232 is divided into four sections, each section displays an imaging feed. In this example, four different imaging feeds display a video feed of the operating room from four different angles: the main video feed 208 displays the surgeon, the first video feed 202 displays the anesthesiologist, the second video feed 204 displays a close-up view of the operating room tool table, and the third video feed 206 displays a close-up view of the surgical site. Alternative imaging feeds may also be used, including a combination of imaging feeds from different geographic locations.

[0087] The session owner 1000 initiates the session, and the session owner 1000 and each participant device 1100 , 1200 , 1300 can add imaging feeds. Figure 4a 2 shows what the user interface display 200 may look like before any imaging feeds are added. The user may select the "Add Video Device" button 212 in the center of the user interface display 200, or may select the "Add Video Device" button 210 in the toolbar 216. This may prompt the user interface to display a list of available imaging devices, such as cameras, connected to the device, from which the user may select to add an imaging feed.

[0088] Figure 4b The user interface display 200 is shown as it would appear after a number of feeds have been posted to the session. In this case, the user interface display 200 shows a primary imaging feed 208, a secondary imaging feed 204, and a tertiary imaging feed 206. These have been added by, for example, the second participant 1200, the third participant 1300, and the session owner 1000. Three of the four sections of the multi-feed display 232 are populated with these feeds. The fourth section displays an "Add Video Device" button 212, which the user can select to initiate the process of adding a new feed. Alternatively, the user can select the "Add Video Device" button 210 in the toolbar 216. Selecting either of the "Add Video Device" buttons 210, 212 prompts the user interface to display a selectable list 214 of available imaging devices. These are not limited to video devices, but could include any imaging device.

[0089] After selecting one of the cameras from the selectable list 214, the user interface 200 displays a preview display 230. The preview display 230 is as shown in FIG. Figure 5 and Figure 6 As shown. The preview display 230 shows the imaging feed from the selected imaging device (in this case, a camera). The CPU 1002 at the relevant device executes instructions locally to display the preview display 230 on the local user interface 1010. These instructions may have been sent from the server 2000 and stored in the local memory 1006 of the relevant device (such storage may be temporary). At this time, the preview display 230 and the imaging feed have not yet been transmitted to the server 2000. Until the user at the relevant device 1000, 1100, 1200, 1300 confirms that the imaging feed should be shared to the session, the imaging feed is only visible to the device 1000, 1100, 1200, 1300 that issued the imaging feed. This allows sensitive information to be hidden in the imaging feed before it is transmitted or shared. This may be referred to as "privacy control." The user interface display 200 includes a privacy control toolbox 240, an enlarged view of which is shown in FIG. Figure 5 shown.

[0090] In the exemplary embodiment shown, the privacy control toolbox 240 includes an optional tool for adding an occluding shape or "masking" shape. The user can select the shape to use and define its position, size and / or orientation. This means that the user can add and edit shapes to place over the area of ​​the imaging feed they want to block. The participant device implements the occluding shape by applying a "transformation" to the imaging feed to generate a transformed imaging feed. The details of how the transformation is implemented will be explained in detail in a later section.

[0091] Specifically, the privacy control toolbox 240 includes a button for applying a rectangular mask 242a to the preview display 230 and a button for applying an elliptical mask 242b to the preview display 230. The privacy control toolbox 240 also includes an "undo" button 244a, a "redo" button 244b, a "delete" button 246, an "add video" button 248, and a "cancel" button 250 for performing operations in the preview display 230. The buttons generally display representative icons and / or text to indicate their purpose. For example, the rectangular mask button 242a may display a rectangular icon, the elliptical mask button 242b may display an elliptical icon, the "undo" button 244a may display a curved arrow icon pointing to the left, the "redo" button 244b may display a curved arrow icon pointing to the right, and the "delete" button 246 may display a trash can icon.

[0092] exist Figure 5, the rectangular mask is selected so that the associated button 242a is displayed in "active" mode. The buttons of the privacy control toolbox 240 generally provide one of four states: "disabled", "inactive", "hovered", and "active". In the "disabled" state, the button is not selectable and is generally displayed with low color contrast. For example, the button body may be white and the icon is displayed in light gray. The "undo" button 244a is generally in a disabled state until a change is made. Once a change is made, the "undo" button 244a moves to the "inactive" state. In the "inactive" state, the button is selectable but has not yet been selected. In this state, the button is generally displayed with greater color contrast. For example, the button body may remain white and the icon is displayed in dark gray. If the user hovers the cursor over the button, it moves from the "inactive" state to the "hovered" state. In this state, the button may appear "highlighted" with the intensity of both the background and the icon increasing. For example, the background may be light gray and the icon may be dark gray or black. If the user subsequently selects the button (e.g., rectangular mask button 242a and elliptical mask button 242b), it moves to "active" mode. For example, when rectangular mask button 242a is in "active" mode, the user can draw a rectangular mask over the video feed. The icon shape being filled can indicate "active" mode. The buttons can also display colors, for example, the shape buttons (rectangular 242a and elliptical 242b) can display a low-intensity blue background, while the icons can display a higher-intensity blue. The "Undo" 244a and "Redo" 244b buttons can remain grayscale, while the "Delete" button 246 can be displayed in red (which can serve as a warning that the shape is about to be discarded). The rectangular mask button 242a and the elliptical mask button 242b cannot be selected at the same time, but the user can switch between them. The "Undo" button 244a, the "Redo" button 244b, and the "Delete" button 246 are not activated for a long time, but are activated instantly when selected to perform a discrete operation. When rectangular mask button 242a or elliptical mask button 242b is activated, the cursor display changes from a selection arrow to a cross, indicating that the drawing mode of the relevant shape is activated. In some embodiments, other methods that interact with user interface display 200 may be used, such as also or alternatively enabling keyboard shortcuts to allow the user to perform relevant operations.

[0093] Figure 5Another illustration in FIG. 2 shows a rectangular mask 220 applied to an area of ​​an imaging feed 202. In this example, the upper left corner of the imaging feed 202 displays personal details of a patient that the user wishes to block out. Therefore, drawing a rectangular mask 220 at this location can cause the area defined by the shape to become obscured, thereby blocking the information. When the rectangular mask button 242a is selected, the user can click and drag to draw a shape. In this state, the cursor 224 is displayed as a cross to indicate that the drawing mode is activated. In the preview display 230, an outline is provided while the shape is selected to clarify the location and shape on the imaging feed 202. In the illustrated embodiment, the outline is implemented as a white dashed line. Once the user creates a shape and releases the mouse click, the shape will remain active and selected, displaying a border, rotation handle (not displayed when the shape was drawn), and resize handle 222. However, at this time, the rectangular mask button 242a on the privacy control toolbox 240 is converted to an "inactive" state. This allows the user to click outside the shape to deselect it. The cursor display changes back to an arrow display. When a shape is deselected, it will appear as it would in the live feed, allowing the user to preview how the shape will look in the session (e.g., without a border). The shape can be selected again by clicking on it or by clicking and dragging a selection rectangle that encompasses the shape. Multiple shapes can be selected by either method. In this mode, the cursor appears as an arrow to indicate selection mode.

[0094] The rectangular mask 220 can be resized by selecting one of a plurality of resize handles 222, which are typically located at the corners of the shape and at the center of each edge. The user can resize the shape by dragging the resize handles 222 to a new location to enlarge or reduce the rectangle. The rectangular mask 220 can also be rotated by selecting one end of a rotation handle extending from the center of one edge of the rectangle 220 (not in the center). Figure 5 , because the rotation handle is not displayed when the user draws the shape). The user can change the rotation by selecting the rotation handle and dragging it until the shape is rotated to the desired direction. In addition, when the shape is selected, the user can move it by dragging the entire shape to a new location. During such operations, the cursor is displayed as an arrow cross.

[0095] Various areas of the imaging feed 202 may be obscured at any time. Figure 6An imaging feed 202 display of the user interface 200 in a preview display mode 230 is shown as further shapes are applied. In this example, the user has selected an elliptical mask 260. This is accomplished by selecting the elliptical mask button 242b. The user can adjust the size and shape of the elliptical mask 260 by selecting a corner 262 and dragging the corner until the corner is moved to a position where the desired shape and size are achieved. The shape can be rotated by selecting a rotation handle 266 via a cursor 264. The rotation handle 266 includes a line extending perpendicularly from one side of the ellipse with a selectable handle at its distal end. The user can move the rotation handle 266 until the desired orientation of the shape 260 is achieved. In Figure 6 In the example shown, the elliptical mask 260 has been resized, rotated, and positioned so that the patient's face is obscured. In this way, the surgeon can enhance the privacy of the patient.

[0096] The privacy control toolbox 240 also includes an "Undo" button 244a and a "Redo" button 244b. The "Undo" button 244a is disabled until the user makes a change (e.g., edits the size or positioning of a shape). After the user makes a change, the "Undo" button 244a becomes enabled and the user can select it to undo the change. At this point, the "Redo" button 244b becomes enabled. If selected by the user, it can cause the operation to be re-performed.

[0097] The privacy control toolbox 240 also includes a "Delete" button 246 that can be used to delete a selected shape from the preview session. During a preview session, any shape can be reconfigured by selecting it with a cursor click. When a shape is selected, the "Delete" button moves from a "disabled" state to an "inactive" state (or a "hover" state or an "active" state, depending on where the user places and clicks the cursor). When no shape is selected, the "Delete" button returns to a "disabled" state.

[0098] The shapes defined by the user in the user interface display 200 define the areas that should be occluded. Thus, these shapes define the transformations that will be applied to the imaging feed to generate the transformed imaging feed.

[0099] Once the user is satisfied with the shape applied to the imaging feed, they can select the "Add Video" button 248 to publish the transformed version of the imaging feed to the session. This causes the transformed imaging feed to be transmitted to the server 2000. The server 2000 then broadcasts the transformed imaging feed as part of the session so that all viewing devices of the session (i.e., the session owner 1000, participant devices 1100, 1200, 1300, and remote viewing devices 2400) can view it.

[0100] The conversion of each imaging feed is performed locally on the participant device from which the imaging feed originates. The converted imaging feed is then sent to the server 2000, such as Figure 2 The server 2000 collates the converted feeds into a converted multi-feed broadcast and then distributes them. The collated imaging feeds of the session are transmitted from the server 2000 to each of the session owner 1000, the participant devices 1100, 1200, 1300, any remote viewing device 2400, and any remote storage 2500 as a converted multi-feed display 232.

[0101] The untransformed imaging feed can only be viewed on the relevant participant device from which it originated and is not transmitted beyond that device nor recorded. Therefore, the image feed can only be transmitted in a transformed state and any sensitive information may be obscured by the blurred shapes. Neither the server 2000 nor the session owner 1000 receives the untransformed version of the imaging feed. Therefore, the unobstructed imaging feed cannot be recovered and therefore the obstruction is immutable. This increases the security of the privacy control tool.

[0102] In some embodiments, when the user selects the "Add Video" button 248 after applying the blur shape, an alert or warning may be output suggesting that the participant be instructed not to pan, tilt, or zoom the imaging device. For example, it may read: "PTZ Camera - We recommend that participants be instructed not to pan, tilt, or zoom a camera that has a privacy mask applied." This is because when any of these functions are applied to the imaging feed, the areas that the user wishes to block out may move to other locations on the display screen, causing them to no longer be blocked by the shape. This warning may be output based on the system detecting that the imaging device is a PTZ camera or other imaging device with PTZ functionality, or it may be output as a default. This alert may also be output when the system detects that the user is about to control a PTZ camera (or imaging device) and the feed has a defined transform.

[0103] The privacy control toolbox 240 also includes a “Cancel” button 250 for canceling edits in the preview session and causing the user interface display 200 to exit the preview display 230 .

[0104] Figure 7An example workflow is shown of a user interacting with a user interface display 200 of one of the devices 1000, 1100, 1200, 1300. At step 100, the user may add an imaging feed by selecting an "add video device" button 212 in the center of the screen, or by selecting an "add video device" button 210 in a toolbar 216. The system may then output a selectable list 214 of available imaging devices at step 102, from which the user may then select a device. This prompts the system to display a preview display 230 at step 104. In this preview display 230, the user may manually overwrite any sensitive information through the process described above at step 106. At step 114, any changes may be "undone", and occlusion shapes may be added and further manipulated in an iterative process. Once the user is satisfied with the configuration of the applied occlusion shapes, they may then choose to add the imaging feed to the session at step 108. Alternatively, the user may choose to add the imaging feed directly to the session at step 110. If the user chooses to add the imaging feed without adding any occluding shapes, the system will output an alert or prompt to the user to check for any sensitive information at step 112. For example, sensitive information may include personally identifiable information (PII). Typically, the user is required to confirm that they wish to proceed before the imaging feed is posted to the session.

[0105] Editing privacy control masks and authorizations

[0106] After the imaging feeds have been transmitted to the session, the server 2000 transmits the multi-feed display 232 to the viewing devices, which typically include the session owner 1000, each participant 1100, 1200, 1300, and any other remote viewing devices 2400. The four converted imaging feeds 202, 204, 206, 208 are arranged side by side, such as in a quadrant arrangement, such as Figure 3 These converted imaging feeds will include the occlusion shapes that were applied when the user added the feed, as described above. However, in some cases it may be desirable or necessary to edit the shapes during a session. This may be because the patient has moved, or the imaging device has been moved or reconfigured (e.g., by changing the pan, tilt, or zoom settings). These changes may result in areas believed to contain sensitive information no longer being occluded.

[0107] The user may select the privacy control icon 282 from the side navigation bar 280 of the user interface display 200. This will cause the privacy control toolbox 240 to be displayed, such as Figure 8. The user can then select one of the imaging feeds to edit, typically by hovering over the imaging feed and clicking to select it. Only the participant device that added the feed (where the feed originated) or the session owner 1000 can edit the feed. For example, if the first video feed 202 is transmitted from the first participant 1100 device, only the first participant 1100 or the session owner 1000 can edit the privacy control settings (blocking shape). The second participant 1200 and the third participant 1300 cannot edit the first feed 202. If the second feed 204 is added by (and originated from) the second participant device 1200, only the second participant 1200 or the session owner 1000 can edit the transformations applied to it (i.e., by editing the blocking shape). The first participant 1100 and the third participant 1300 cannot edit the second feed 204. This allows the session owner 1000 to edit the transformations of each feed. This may be advantageous, for example, when one or more users on participant devices 1100, 1200, 1300 are busy (e.g., a surgeon may be scrubbing) when a shape needs to be moved. However, it does not allow other participants to edit the transformation, as these edits may reveal areas (which may contain sensitive information) that the original participant wished to hide.

[0108] The authorization module 1016 of each computer device (session owner 1000 and participant devices 1100, 1200, 1300) defines which other devices are authorized to control the imaging feeds originating from that device. For example, the authorization module 1016 of the session owner device 1000 defines that only the session owner 1000 is authorized to control the primary imaging feed 208 (originating from the imaging device 1014 of the session owner 1000). The authorization module of the first participant device 1100 defines that the first participant device 1100 is authorized to control the first imaging feed 202 (including viewing it in an untransformed state) and that the session owner 1000 is authorized to edit the transformation. The authorization modules 1016 of the second participant device 1200 and the third participant device 1300 define similar authorizations, i.e., the session owner 1000 is authorized to make changes to the transformation of the second feed 204 or the third feed 206 (respectively), but only the same device (the second participant device 1200 and the third participant device 1300, respectively) is authorized to view the untransformed imaging feed.

[0109] If a participant device is not authorized to edit a particular feed, the privacy control toolbox 240 will not become active on that participant device's user interface display 200. This may occur, for example, if the second participant 1200 attempts to edit an occluding shape on the first video feed 202 (originated from the first participant 1100). The system may output a message "You do not have permission to modify this feed" (or similar message).

[0110] If the participant device is authorized to edit the feed, the user interface display 200 enters a preview mode 230 and the privacy control toolbox 240 becomes active. Fig. 9 As far as the user is adding imaging feeds, the editing controls and methods are the same as previously described for preview mode 230. The user can use the same commands to resize, rotate, add, delete, and move occlusion shapes.

[0111] The imaging feed is transmitted from the participant device 1100 to the server 2000 only in the transformed state, and is broadcast from the server 2000 to the session owner 1000 only in the transformed state. This means that if the session owner 1000 enters the preview mode 230 of the first feed 202 and performs an edit to move an occluding shape, the unoccluded imaging feed in that area is not available. Fig. 9 As shown, the "old" position 290 of the occluding shape still remains occluded (in this case, blurred). The user interface display 200 can indicate this old position 290 by displaying an indicative marker in this area. For example, Fig. 9 As shown, a border and shading may be applied to the old position 290. The new editable position 292 of the shape will typically display a preview of the occlusion effect (in this case, blur). A different border may be applied to the new position to indicate that it is currently selected and editable (e.g., a white dashed line, also used in preview mode when a user adds an imaging feed). The old position 290 and the new position 292 may also be labeled accordingly.

[0112] Edits in preview mode 230 are not posted to the session until the user selects the "Apply Changes" button 252 on the privacy control toolbox 240. Therefore, typically, no other device can see the changes made until the changes are complete. Typically, the preview display 230 may never be transmitted beyond the device displaying it. For example, if the session owner 1000 enters the preview display 230 and makes changes to the occluding shape, those modifications are made locally. The associated participant devices 1110 do not see the changes being made. Only when the modifications are finalized is the data transmitted to the server 2000 and then broadcast to the associated participant devices 1100. In some embodiments, a notification that a change is being made may be transmitted to the associated participant device 1100, optionally dependent on the participant device 1100 attempting to enter preview mode. When the user attempts to enter the preview display 230 to modify the shape, the system may output a warning that another device is making changes. In some embodiments, the system may prevent the user from entering the preview display 230 and / or making changes at the same time.

[0113] The imaging video feed is transformed on the relevant participant device and then sent to the server 2000 along with the applied mask. Once the user clicks "Apply Changes" 252, the transformation is modified at the participant device 1100, even if the user is at the session owner device 1000. If the shape has been defined and / or edited at the session owner 1000, the new, updated transformation (defined by the new configuration of the occlusion shape) is transmitted from the session owner 1000 to the relevant participant device 1100 via the server 2000. The updated transformation is typically transmitted as a set of modified computer executable instructions. These instructions typically include a list (array) of shapes (i.e., occlusion shapes), each of which has a coordinate location (e.g., xy coordinates), a rotation angle, a shape type, and shape dimensions (width and height for a rectangle; or radii Rx and Ry for an ellipse). These instructions can be received by the participant device via the communication interface 1004 and stored in the memory 1006 and / or storage 1008 of the computer device. The CPU 1002 of the participant device 1100 then executes the new, modified instructions to apply the new, modified transformation. The transformed imaging feed 202 (now including the modified occluded region) is transmitted from the participant device 1100 to the server 2000 where it is collated into the multi-feed display 232. Thus, only the transformed imaging feed 202 is transmitted out of the participant device 1100.

[0114] Video feed logic flow

[0115] Fig.10 An exemplary flow chart of the logic flow associated with the imaging feed stream constructor 700 is shown. In this exemplary embodiment, the imaging feed consists of a series of individually imaged frames. Each resulting image is then transformed and the series of transformed images are combined to generate a transformed imaging feed stream. In step 702, the imaging feed (e.g., a video stream) is input from the imaging device 1014 to the CPU 1002, and the CPU 1002 performs the method processing steps according to instructions stored in the memory 1006 or storage 1008 and / or instructions transmitted through the communication interface 1004. In step 704, the CPU processes the imaging feed and inputs it into the transformation stream 705. Generally, the media stream track processor can be used to transmit the imaging feed frame by frame through the transformation function so that the occlusion shape can be applied as needed.

[0116] During the conversion stream 705, each frame of the imaging feed stream is input at 706, and an image of the frame is created at 708 (e.g., as a bitmap file). At step 710, the imaging is converted so that the area indicated by the user is blocked. A new converted frame is queued at 712 in which the relevant area is blocked. At step 714, the frames are assembled into an imaging feed stream or "track". Typically, this is performed by a media stream track generator, which can queue a series of converted frames according to the original timestamps. At step 716, the new, converted imaging feed stream is output. The converted imaging feed is output from the CPU 1002 of the participant device to the server 2000 via the communication interface 1004 to be added to the session, such as Figure 3 shown.

[0117] Fig.11 A more detailed flow chart of the logic flow of transformation 800 is shown. Transformation 800 may be performed at transformation step 710 of the constructor logic flow, such as Fig.10 shown. Fig.12 A visual diagram shows the feed image created during the entire conversion process. Fig.12 As shown, an untransformed imaging feed 900 is first input. At step 802, the system first determines whether transformation is enabled (i.e., whether any occlusion shapes should be applied). If transformation is not enabled, the processor 1002 simply outputs a new frame from the image (step 812) and exits the logic flow. If transformation is enabled, the processor 1002 proceeds to the next step in the logic flow. The processor 1002 can check whether the frame has a defined width and height, and if not, create a new video frame using the last known frame.

[0118] In step 804, the CPU 1002 generates a mask wizard 902. The mask wizard 902 is a set of instructions that contains the configuration of all occlusion shapes, such as Fig.12As shown. The mask wizard 902 contains a blank transparent image or canvas of the same size as the original video frame, and all shapes are drawn in their correct positions and filled with a solid color (can be any color, but usually white). When further instructions are applied, these areas of defined shapes will become the "active areas" that affect the output. The mask wizard 902 is constructed and defined according to the arrangement of the occlusion shapes added and configured by the user in the preview mode 230 of the user interface 200. Therefore, it can be considered to use the mask wizard to define the transformation. The mask wizard 902 has the same size as the image of the frame. By using a single mask wizard 902 containing all occlusion shapes (rather than multiple smaller masks), processing efficiency can be optimized. This is because the mask wizard 902 is only calculated once when the occlusion shape changes (such as moving, scaling, deleting or adding), and then the occlusion instructions can be applied in each frame with only one traversal, without the need to recalculate each independent mask wizard multiple times. Compared with looping through each individual occlusion shape on each frame and applying the mask instruction multiple times, this can greatly improve the computational efficiency.

[0119] If the configuration of the occluding shape has not changed, the system may not need to generate a new mask sprite 902 for each frame. Instead, the mask sprite 902 can be saved in the device's memory 1006 and the CPU 1002 calls it when needed. The CPU 1002 can advantageously first check whether the size of the image 900 is still the same as the size of the mask sprite 902. The CPU 1002 can resize the mask sprite 902 to the size of the image 900 based on determining the size difference.

[0120] At step 806, the system draws the image as the basis for the output canvas frame (this will also overwrite any old image that may exist, such as a previous frame image from a previous run of the conversion logic flow 800).

[0121] In step 808, the mask wizard 902 is applied to the output canvas, thereby cutting out (i.e., completely removing) the area defined by the occluding shape from the image. The resulting image 904 includes the image after the area is removed, such as Fig.12 shown.

[0122] In step 814, in parallel with the above process steps, a temporary canvas is created and the image is applied to the canvas. CPU 1002 applies a filter effect to the temporary canvas to achieve the effect of blurring the image. This is usually a blur filter, so that the video frame image 906 appears completely blurred on the temporary canvas (i.e., the entire image is blurred, such as Fig.12). The filter effect used is typically a Gaussian blur filter. The standard deviation value of the Gaussian blur function is selected to maintain a compromise between achieving sufficient blurring of information and minimizing computational cost. For a 720p video feed (i.e., a high-definition (HD) display with a resolution of 1280x720 pixels), this is typically selected to be 25 pixels. However, the standard deviation may also be any value greater than 20 pixels, and is preferably between 20 pixels and 45 pixels, more preferably between 20 pixels and 35 pixels, and even more preferably between 20 pixels and 30 pixels. For imaging feeds of different resolutions, the standard deviation value used will be different. However, typically, the same ratio of standard deviation to resolution will be used (i.e., typically 25:720, preferably between 20:720 and 45:720, more preferably between 20:720 and 35:720, and even more preferably between 20:720 and 30:720). For example, a 360p video feed uses a standard deviation of 12.5 pixels. This helps maintain similar blur effects across imaging feeds of different resolutions.

[0123] At step 810, the corresponding areas of the blurred temporary canvas 906 are applied to the output canvas to replace those areas that were clipped by applying the mask wizard in step 808. This means that for those areas defined by the mask wizard, the blurred temporary canvas (e.g. Fig.12 The corresponding pixels (i.e., pixels at the same coordinates) of the original image (as shown in the image 908 in FIG. 1 ) completely replace the removed areas in the original image. Therefore, the resulting image 910 is a "patchwork" of the original frame image - where the areas 904 defined by the blurred shapes have been removed and replaced with corresponding areas 908 of the blurred version of the original image. The location, shape, and orientation of these areas are determined by the sprite mask 902, which is defined by the shapes applied and edited by the user through the privacy control toolbox 240 (note that Fig.12 Borders around occluded areas of converted image 910 are shown for emphasis, but these areas may not appear in the output image.)

[0124] In an exemplary embodiment, a Gaussian blur filter is applied to a copy of the image on a temporary canvas. Gaussian blur achieves the blurring effect by convolving the image with a Gaussian function. The Gaussian function is defined by a standard deviation value (e.g., 25 pixels). Therefore, the blur depends on the weighted average of neighboring pixels. Typically, if the Gaussian blur is applied only to the cropped area of ​​the image, this will result in a decrease in the blur quality at the outer edges of the area because the radius of the function exceeds the boundaries of the area. In contrast, the method of the present invention can advantageously produce a more uniform blurring effect in the occluded area inserted into the imaging feed. This is because the blur function is applied uniformly to the entire image, and then only the desired portion ("area") is extracted and output to the resulting image 910. This is conducive to the efficient use of smaller areas, which may otherwise be severely impaired by unclear edges.

[0125] The method of the present invention also means that the "blocked" areas consist only of the blurred image, rather than applying a blur mask over the sensitive information. Therefore, the imaging feed is not "masked", but transformed. This transformation involves completely replacing the area with the corresponding area of ​​the substitute image. This means that the occlusion is an inherent part of the generated image and cannot be removed afterwards. Sensitive parts of the image that the user wishes to occlude will be completely deleted and replaced with the corresponding area of ​​the substitute image (e.g., the occluded image). This occlusion itself is unchangeable.

[0126] Finally, in step 816, the resulting image 910 is output to form a new converted frame. In step 714, the media stream track generator assembles the frame to output the converted imaging feed. The device 1000, 1100, 1200, 1300 then sends the converted imaging feed 208, 202, 204, 206 to the server 2000.

[0127] The conversion occurs simultaneously with the imaging feed, so the video frames are converted in real time. Typically, the processing delay time does not exceed 500 milliseconds (ms), preferably does not exceed 300ms, and most preferably does not exceed 200ms. This delay time includes the time between the imaging device 1014 generating the feed and the communication device 1004 transmitting the converted imaging feed to the server 2000.

[0128] If the user edits the occlusion shape by the method outlined above, this will have the effect of changing the sprite mask 902 (i.e., changing the transform) immediately after the user "publishes" their changes to the session (by selecting the "Apply Changes" button 252 once, e.g. Fig. 9). This means that all subsequent frames are transformed using the updated sprite mask 902 so that the occluding shapes are in the new configuration. This transformation is performed on the relevant participant devices. For example, if the session owner 1000 changes the position of the mask applied to the first video feed 202, those changes to the sprite mask will be implemented at the first participant device 1100. In this case, the session owner device 1000 sends instructions related to the modified sprite mask (i.e., the modified transformation) to the first participant device 1100 via the server 2000. The first participant device 1100 stores the new instructions defining the modified sprite mask 902 in its memory 1006. The CPU 1002 of the first participant device 1100 executes the new instructions from then on to perform the processing steps as described above using the new sprite mask 902. The first participant device 1100 outputs the transformed imaging feed 202 to the server 2000, which can then broadcast it to the session owner 1000 (and other devices). Thus, even if the session owner 1000 implements changes to the sprite mask 902, it still cannot view or receive the untransformed version of the imaging feed (unless, of course, the user removes all occluding shapes, in which case all participants and viewing devices will also see the untransformed imaging feed).

[0129] Alternative occlusion techniques

[0130] In some embodiments, occlusion can be achieved by applying a different effect to the replacement image. This may include applying a different filter to the copy of the image. For example, the processor may apply a pixelation filter to the image on the temporary canvas instead of applying a Gaussian blur filter.

[0131] In some simplified embodiments, the imaging feed may not be imaged at all. Instead, the processor may apply a solid color to a temporary canvas. The cutout areas of the image (corresponding to the shapes) may then simply be replaced with corresponding areas of solid color. The color may be pre-defined (e.g. gray), or may be based on the average color of surrounding pixels or the entire image. Nonetheless, these areas are completely replaced in the image (via the "patchwork" technique described above), rather than applying the mask directly to the image (onto). This means that the masking of the feed is immutable, thereby increasing the security of the privacy control tool.

[0132] The occlusion method may be selected based on the performance of the relevant participant devices. The performance may be related to or defined by processing and / or connection speeds (which may include upload and / or download speeds). Performance may be further defined using metrics consisting of measured parameters, and these parameters may be measured continuously or intermittently. This may be done using standard metrics and methods known in the art. For example, a metric may include the time it takes for a page to process 1 video frame (which may be extracted from WebRTC statistics). Other metrics may include the number of frames per second (fps) that the page renders, which defines how many times per second the page is rendered. On a standard screen, this may ideally be 60fps, with lower values ​​indicating poor performance. The time per page frame (TPF) may also be used as a metric, which defines how long it takes for the page to fully render during each rendering cycle. For a standard 60fps screen, this may ideally be 17 milliseconds per frame, but higher values ​​may indicate worse performance. Additionally, the system may use the ratio of processed (encoded / decoded) video stream frames to video frames received from the source (camera or received over the network) as a metric. Ideally, the processing ratio may be 100% (or very close), with lower values ​​indicating poor performance. These are only example metrics and may be used in any combination. Any other metric deemed relevant to the overall performance of the system may also be used.

[0133] The occlusion method may be determined based on whether the measured or calculated values ​​of certain parameters are above or below a preset threshold and / or within a specific range. For example, if the metric is below a first threshold, the CPU 1002 executes a set of instructions to apply a pixelation filter to the replacement image instead of a blurring filter. If the metric drops even below a second threshold, the CPU 1002 may execute a set of instructions to replace the area with a replacement image formed of a solid color, and the process does not include applying a filter to the image at all. The occlusion method may change in real time as the determined and / or measured performance indicators and parameters change. This may prevent the stability of the imaging feed from being disrupted due to the computational cost of applying the transformation. Thus, different occlusion methods may be implemented on different participant devices of a session.

[0134] Other alternatives and modifications

[0135] Various other modifications will be apparent to those skilled in the art. For example, while the detailed description provides an exemplary use in a telesurgery system, it should be understood that the invention may be implemented in any situation where sensitive information may be transmitted via an imaging feed and a user may wish to securely hide that information locally.

[0136] The imaging feeds described in the present invention may encompass any type of imaging. These may be static imaging feeds or real-time imaging feeds, such as video feeds. Imaging feeds are not limited to video photography, but may also include other imaging techniques such as ultrasound. Imaging feeds may also include computer-generated 2D or 3D modeling visualizations.

[0137] An exemplary embodiment of the imaging feed stream construction method includes imaging each frame of the feed and then applying a transformation to each of these images. However, this transformation can be applied to imaging feeds with different structures or formats. This transformation can also be applied directly to the feed.

[0138] A mask sprite can be provided as a set of instructions indicating that the value of a set of pixels should be set to zero or to a specified value (e.g., indicating a specific color). The instructions can include a matrix for transforming the pixel values ​​of an image.

[0139] In the multi-feed display mode, the user interface 200 is described as being divided into four equal areas to carry four video streams. It should be understood that different numbers of imaging feeds may be used and that the imaging feeds may be arranged in alternative configurations. These feeds may all be displayed on the user interface 200 at the same size, or different sizes may be assigned to them. For example, this may be based on the importance of the content or feed, its magnification, and / or the level of detail it contains. The imaging feeds do not necessarily originate from different devices, but rather each device may send multiple imaging feeds or no imaging feeds.

[0140] Each computer device may take a variety of forms, for example, each computer device may be provided as a desktop, laptop, tablet, mobile phone, or other computer device.

[0141] This description refers to an action performed by a user, such as making a selection on a user interface by clicking a computer mouse. It should be understood that alternative methods of implementing actions such as selections on a user interface may be used. For example, an interactive touch screen display, keyboard shortcuts, and / or alternate controllers may be used. In these embodiments, a "click" may be implemented by an equivalent action.

[0142] The computer devices of the present invention have been described as each comprising an authorization module. However, one or more "central" authorization modules or devices may be provided that define the authorizations of different network computer devices. A central authorization module may be provided on a server or on one of the computer devices of the network. An "authorization module" may be provided as any device, machine, entity and / or authorization device.

[0143] It will be understood that the present invention has been described above by way of example only and that modifications of detail may be made within the scope of the invention.

[0144] Reference signs appearing in the claims are for illustrative purposes only and shall have no limiting effect on the scope of the claims.

[0145] The term "comprising" as used in the present description and claims preferably means "consisting at least in part of..." When interpreting statements in the present description and claims containing the term "comprising", other features may be present in addition to the features beginning with this term in the respective statement. Related terms such as "comprising" and "including" should be interpreted in a similar manner.

Claims

1. An apparatus for processing an imaging feed from an imaging device, wherein include: means for converting said imaging feed; as well as a communication interface for transmitting the imaging feed to another device; Therein only the converted imaging feed is transmitted out of the communication interface. The apparatus of claim 1 , wherein the other device is external to the communication interface.

3. An apparatus according to any preceding claim, wherein the other device is a server, preferably the server being configured to distribute the converted imaging feed.

4. An apparatus according to any preceding claim, wherein the further device is remote and / or geographically distant from the imaging device and communications interface.

5. Apparatus according to any preceding claim, wherein the imaging feed is not recorded prior to conversion by a conversion device.

6. Apparatus according to any preceding claim, wherein the imaging feed is transmitted from the imaging device to a conversion device without being recorded.

7. Apparatus according to any preceding claim, further comprising means for authorising an entity to control the imaging feed prior to transmitting the imaging feed to the further device, preferably in the form of an authorising device or module, and / or preferably embodied in a processor and associated memory.

8. An apparatus for processing an imaging feed from an imaging device, wherein include: a communications interface for transmitting the imaging feed to another device; as well as Means for authorising an entity to control said imaging feed prior to transmitting said imaging feed to said other device, preferably in the form of an authorising device or module, and / or preferably embodied in a processor and associated memory.

9. An apparatus according to claim 7 or 8, wherein the authorising device is configured to authorise an entity to view and / or edit the imaging feed before transmitting the imaging feed to the further device.

10. An apparatus according to any of claims 7 to 9, wherein the authorising device is configured to authorise an entity to define transformations applied to the imaging feed prior to transmitting the imaging feed to the further device.

11. An apparatus according to any one of claims 7 to 10, wherein the authorising device is configured to inhibit an entity from controlling the imaging feed prior to transmitting the imaging feed to the other device.

12. An apparatus according to any one of claims 7 to 11, wherein the authorizing device is configured to prohibit an entity from viewing the imaging feed before transmitting the imaging feed to the other device, and to authorize the same entity to define transformations applied to the imaging feed, preferably before transmitting the imaging feed to the other device.

13. The apparatus according to any one of claims 7 to 12, wherein the authorization device is configured to determine the authorization level of the entity according to the status of the entity.

14. An apparatus according to any one of claims 7 to 13, wherein the authorising device is configured to authorise an entity controlling the communications interface to view the imaging feed before transmitting the imaging feed to the further device.

15. An apparatus as claimed in any preceding claim, further comprising a further communication interface for transmitting a further imaging feed to a server.

16. An apparatus according to any preceding claim, further comprising a processor for extracting at least one region of the imaging feed and replacing it with at least one region of a substitute image to generate a transformed imaging feed.

17. An apparatus for processing an imaging feed from an imaging device, wherein include: A processor is provided for extracting at least one region of the imaging feed and replacing it with at least one region of a substitute image to generate a transformed imaging feed.

18. An apparatus as claimed in claim 16 or 17, wherein at least one region of the substitute image corresponds spatially to at least one region of the imaging feed.

19. An apparatus according to any one of claims 16 to 18, wherein the substitute image is an image of the imaging feed, the processor being configured to apply a filter to the substitute image.

20. An apparatus according to claim 19, wherein the processor is configured to apply the filter to a portion of the image that is larger than at least one area of ​​the replacement image, preferably wherein the processor is configured to apply the filter to the entire replacement image.

21. The apparatus according to claim 19 or 20, wherein the filter is an occlusion filter, preferably the filter is a blur filter, even more preferably the filter is a Gaussian blur filter.

22. The device according to claim 21, wherein the filter is selected according to a performance indicator of the device, preferably the performance indicator is related to processing power.

23. An apparatus according to any preceding claim, wherein the imaging feed is real-time and / or conversion of the imaging feed is performed in real-time to generate a real-time converted imaging feed, preferably within 500 milliseconds of receiving the imaging feed, more preferably within 300 milliseconds, even more preferably within 200 milliseconds.

24. An apparatus according to any preceding claim, further comprising a user interface configured to enable a user to define and / or edit transformations applied to the imaging feed.

25. Apparatus according to claim 24 as appended to any one of claims 16 to 23, wherein the user interface is arranged to enable a user to define at least one region.

26. An apparatus according to claim 25, wherein the user interface is configured to enable a user to define a position and / or shape and / or size and / or orientation of at least one area, preferably wherein the user interface is configured to enable a user to select a shape, more preferably wherein the shape can be selected from a list including at least a rectangle and an ellipse.

27. An apparatus as claimed in any preceding claim, wherein the user interface is configured to allow a user to define and / or edit transformations when transmitting an imaging feed.

28. An apparatus according to any preceding claim, further configured to output an alert upon detecting that the imaging device is capable of changing the configuration of the imaging feed, preferably upon detecting that the imaging device is capable of changing the pan, tilt and / or zoom settings of the imaging feed.

29. Apparatus according to any preceding claim, further comprising an imaging device, preferably located in a medical facility.

30. An apparatus according to any preceding claim, wherein the imaging feed is from a medical facility.

31. A method for processing an imaging feed from an imaging device, the method include: converting the imaging feed; as well as Only the converted imaging feed is transmitted to the other device.

32. A method for processing an imaging feed from an imaging device, the method include: transmitting the imaging feed to another device; as well as An authorized entity controls the imaging feed before transmitting the imaging feed to another device.

33. A method for processing an imaging feed from an imaging device, the method include: extracting at least one region of the imaging feed; as well as At least one region of the substitute image is replaced to generate a transformed imaging feed.

Images