Facial anonymization in the surgical environment

Abstract

The system can anonymize faces in a surgical environment. The system can generate a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras. Each camera may provide a different view of the surgical environment. The 3D model may indicate the person's position in global coordinate space. The system can determine a first location containing the person's face in the 3D model based on the person's position in global coordinate space. The system can match the first location to a second location in each video stream. The system can then anonymize faces in each video stream based on the visibility of faces at the second location. Other embodiments are also described and claimed.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 468,546, filed on May 24, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] (Field of the Invention) The present invention generally relates to the surgical environment, and more specifically, to systems and methods for anonymizing faces in a surgical environment. Other aspects are also described.

Background Art

[0003] Many operating rooms (ORs) have cameras installed to monitor the OR workflow. The OR videos captured by the cameras can provide visual feedback from events occurring during surgery. Thus, analyzing and mining the recorded OR videos can lead to improvements in OR efficiency, which can later reduce costs for both patients and hospitals. However, OR videos need to be anonymized first by removing Personally Identifiable Information (PII). As a result, the anonymized OR videos can be stored and passed on to post - processing services without leaking the PII of patients and OR staff.

[0004] The main sources of PII in OR videos are the faces of patients and OR staff. To anonymize the captured faces in the video, the faces must first be detected. Conventional face - detection techniques are generally constructed to utilize a bottom - up approach that relies on detecting features of the face such as the nose and eyes to construct and infer the location of the face in the video.

Summary of the Invention

Means for Solving the Problems

[0005] Implementations of the present disclosure include using a three-dimensional (3D) model generated from video streams corresponding to different views of a surgical environment to identify the location of a person's face in the video stream and anonymize them if visible. In some implementations, the system can generate a 3D model of a person in a surgical environment based on multiple video streams received from multiple cameras placed in the surgical environment. Each camera may provide a different view of the surgical environment. The 3D model may indicate the person's position in global coordinate space. The system can determine a first location, including the person's face, in the 3D model, based on the person's position in global coordinate space. The system can then match the first location to a second location in each video stream. The system can then anonymize faces in the video stream based on identifying the second location. Other embodiments are also described and claimed.

[0006] The above summary does not constitute an exhaustive list of all aspects of this disclosure. This disclosure is intended to include all systems and methods that can be practiced from all preferred combinations of the various aspects summarized above, as well as those disclosed in the following “Modes for Carrying Out the Invention,” particularly those indicated in the “Claims” section. Such combinations may have specific advantages not specifically described in the above summary. [Brief explanation of the drawing]

[0007] Some aspects of the disclosure described herein are illustrated in the accompanying drawings as examples, not as limitations, and in the drawings, similar reference symbols indicate similar elements. It should be noted that references to “an” or “one” aspects in this disclosure do not necessarily refer to the same aspect, but rather to at least one. Furthermore, for the sake of brevity and to reduce the total number of figures, a given figure may be used to illustrate features of two or more aspects of this disclosure, and not all elements in the figure may be required for a given aspect. [Figure 1] This is an example of a system for anonymizing faces in a surgical environment. [Figure 2] This is an example of a 3D model generated from video streams from different cameras placed in a surgical environment. [Figure 3] This is an example of the location of face matching in an unprocessed video stream. [Figure 4] This is an example of an anonymized face in an anonymized video stream. [Figure 5] This is a block diagram of an exemplary internal configuration of a computer system for anonymizing faces in a surgical environment. [Figure 6] This is a flowchart illustrating an example of a process for anonymizing faces in a surgical setting. [Modes for carrying out the invention]

[0008] With reference to the accompanying drawings, several aspects of this disclosure are described here. Wherever the shape, relative position, and other aspects of the described parts are not expressly defined, the scope of the invention is not limited to the illustrated parts, which are intended solely for illustrative purposes. Furthermore, although numerous details are described, it should be understood that some aspects of this disclosure can be carried out without these details. In other cases, well-known circuits, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0009] It is often desirable to anonymize OR (Operational Surgical) video from the surgical environment to obscure or remove the faces of patients and OR staff from the video. However, the facial features of people in the OR are often heavily obscured by personal protective equipment (PPE) such as face masks, face shields, goggles, and glasses, and are also occluded by other OR staff and OR equipment, which may render existing face detection techniques ineffective. These challenges can be exacerbated by poses that obscure faces, faces turned away, small faces with low resolution, low light, and sometimes, excessively bright light. Conventional automated two-dimensional (2D) anonymization systems can struggle when such occlusion / obstruction is present, sometimes requiring manual intervention. Further complicating matters is the large volume of video data containing images to be anonymized, making manual anonymization cumbersome. What is needed is a robust and effective OR video anonymization technique that does not suffer from the shortcomings of existing techniques.

[0010] Implementations of this disclosure address this problem by utilizing 3D models generated from video streams corresponding to different views of a surgical environment, identifying the visible locations of a person's face in the video streams, and anonymizing them when visible. In some implementations, the system can generate a 3D model of a person in a surgical environment based on multiple video streams (raw video streams) received from multiple cameras placed in the surgical environment. Each camera may provide a different view of the surgical environment. The 3D model may indicate the person's position in a global coordinate space, for example, by using a Cartesian coordinate system or a spherical coordinate system. The system can then determine a first location containing the person's face in the 3D model based on the person's position in the global coordinate space. The system can then match the first location to a second location in each video stream, for example, by projecting a 3D face mesh model to the second location in the raw video stream. The system can then anonymize faces in each video stream based on the visibility of faces at the second location, generating an anonymized video stream instead of the raw video stream.

[0011] In some implementations, the system can determine a global 3D face mesh model that overlaps with each person in the scene from multiple raw video streams corresponding to different camera views. The system can perform 2D anonymization of faces in the raw video stream by reprojecting the mesh from the 3D model onto the raw video stream from the camera. In some cases, faces may be occluded in a particular view and therefore invisible, for example, due to OR lighting. The system can determine whether a 3D face is visible in a 2D image by searching for the difference between the depth map generated by the camera and the generated 3D face mesh. The system can then perform anonymization of visible faces, such as 2D rendering adjustments to obtain a more realistic face replacement. For example, the system can merge the 3D face mesh model with a background image. In some cases, the system can utilize face replacement templates that can be individualized and modified for each person, for example, to influence factors such as age, gender, and / or ethnicity.

[0012] In some implementations, generating a 3D model of a person may involve combining 2D poses of the person from multiple video streams to create a 3D pose. In some implementations, determining the person's position in global coordinate space may involve regressing the human model to detected human keypoints within the 3D model. In some implementations, the 3D model may include a 3D point cloud, and each video stream may include a sub-point cloud registered in the global coordinate space of the 3D point cloud. In some implementations, the 3D model may be generated based on a combination of red, green, blue (RGB) images and depth maps from multiple video streams (collectively, red, green, blue, depth (RGB-D) data). In some implementations, matching a first location to a second location may involve projecting a 3D face model onto a video stream to determine the location of the face. In some implementations, anonymizing the face at the second location may involve determining that the face in the 3D model is visible within the video stream. In some implementations, face anonymization may involve replacing the face with an artificially generated face. In some implementations, facial anonymization may involve using a personalized face replacement template tailored to the individual.

[0013] Referring to the diagram, Figure 1 shows an example of a system 100 for anonymizing faces in a surgical environment. For example, it may be desirable to anonymize OR video from a surgical environment to obscure or remove the faces of patients and OR staff from the video. System 100 can receive multiple video streams (e.g., raw video streams) from multiple cameras placed in the surgical environment. There may be n cameras in the surgical environment, each capable of providing n video streams, where n is an integer greater than 1. For example, system 100 can receive video stream 1 (VS1) from camera 1, video stream 2 (VS2) from camera 2, and so on.

[0014] Each camera may provide a video stream containing multiview RGB-D images with color and depth. Furthermore, each video stream may correspond to different views of the surgical environment, such as different angles or viewpoints of the scene. For example, a first set of one or more cameras may be surgical cameras that capture the scene from a narrower viewpoint above the surgical procedure, and a second set of one or more cameras may be workflow cameras that capture the scene from a wider viewpoint within the OR. In some cases, system 100 may receive video streams from the cameras in real time, and in other cases, system 100 may receive video streams from a storage device that holds recordings from the cameras.

[0015] System 100 can generate 3D models of people in a surgical environment using the 3D model generation device 102. For example, the 3D model generation device 102 can generate 3D mesh models of people in the environment. The 3D model generation device 102 can generate 3D models based on video streams from cameras (e.g., VS1, VS2). The 3D model generation device 102 can detect 2D person keypoints in each video stream and regress the human model in global coordinate space. The 3D model can indicate the position of the person in a global coordinate space such as a Cartesian coordinate space or a spherical coordinate space with XYZ axes. For example, referring further to Figure 2, the 3D model generation device 102 can generate 3D models 120 of people such as people 122 and 124 corresponding to OR staff in global coordinate space.

[0016] In some implementations, the 3D model 120 may represent a point cloud containing specific data points within the point cloud corresponding to people in the environment (e.g., people 122 and 124). Each video stream may contain subpoint clouds registered in the global coordinate space of the 3D point cloud. For example, two or more subpoint clouds from VS1, VS2 may be registered in one global coordinate space of the 3D model 120 by minimizing photometric reprojection errors across keypoints on large visual markers.

[0017] In some implementations, determining the position of a person within the 3D model 120 may involve the generator 102 regressing a human model (e.g., a statistical parametric human mesh model) onto detected human keypoints. The generator 102 can then align the face of the human model onto the keypoints in the 3D point cloud of the 3D model 120. Thus, the 3D model 120 may be generated based on utilizing raw video streams from cameras with different angles, viewpoints, and / or views of the scene, such as VS1 and VS2. The 3D model 120 may be generated based on a combination of RGB-D images from each video stream. For example, RGB images and depth maps / images, or multi-view RGB-D data from these cameras may be fused into the 3D point cloud of the 3D model 120 to represent a scene within a surgical environment.

[0018] In some implementations, generating a 3D model of a person for 3D model 120 may involve combining 2D poses of a person from a video stream to form a 3D pose of the person in 3D model 120. For example, referring further to Figure 3, generating a 3D model 120 of person 122 may involve incorporating 2D poses of person 122 from video streams VS1 and VS2 (shown in the focus regions of VS1 and VS2 in Figure 3, highlighting different 2D poses of person 122) into a single 3D pose of person 122 (shown in 3D model 120 in Figure 2). In some implementations, the 3D poses from 3D model 120 may be reprojected onto each 2D image in the video stream as input to guide the 3D model regression (reprojection). Thus, system 100 can incorporate 2D human poses from multiple views into a single integrated 3D human pose, per person in the scene. System 100 can perform temporal smoothing on each sequence of 3D human poses to interpolate missing poses and / or reduce noise.

[0019] Referring again to Figure 1, the system 100 can then use the face locator 104 to determine a first location containing the face of a person in the 3D model 120, based on the person's position in global coordinate space. The face locator 104 can determine the first location based on the determined position of the person constructed in global coordinate space. For example, referring again to Figure 2, the face locator 104 can determine a first location 132 registered in the global coordinate space of the 3D model 120 as corresponding to the face of person 122. The face locator 104 can also determine another first location 134 registered in the global coordinate space of the 3D model 120 as corresponding to the face of person 124, and so on.

[0020] System 100 can then use the matching system 106 to match a first location to a second location in each video stream (e.g., an unprocessed video stream). The matching system 106 can project a 3D face model from the first location onto the video stream to determine the location of the face in the second location. For example, referring further to Figure 3, the matching system 106 can match a first location 132 corresponding to the face of person 122 to second locations 142 and 144 in each video stream. To perform the matching, the matching system 106 can project a 3D face model of person 122 from the first location 132 onto video streams VS1 and VS2 to determine the location of person 122's face in second locations 142 and 144, respectively. The matching system 106 can similarly match a first location 134 corresponding to the face of person 124 in the 3D model 120 to video streams VS1 and other second locations within VS1. In some implementations, matching a first location to a second location may involve comparing a face in the 3D model 120 with a face in a depth map or color image from a video stream.

[0021] System 100 can then utilize anonymizer 108 to anonymize the face in each video stream based on the visibility of the face at a second location, generating an anonymized video stream. For example, System 100 can generate anonymized video stream 1 (AS1, an anonymized version of VS1 corresponding to camera 1), anonymized video stream 2 (AS2, an anonymized version of VS2 corresponding to camera 2), and so on. To perform anonymization, anonymizer 108 may determine that the face within 3D model 120 is visible at a second location within the video stream (e.g., not obscured by other OR staff or equipment within the view). When the face is visible, anonymizer 108 can render the face of a human model (e.g., a statistical parametric human mesh model) in the video stream and replace that face with an artificially generated face (e.g., changing facial features such as adding / removing glasses, changing eyes, ears, nose, hair).

[0022] For example, referring again to FIG. 3, anonymizer 108 can anonymize the face of person 122 based on the visibility of the face in video streams VS1 and VS2 at second locations 142 and 144. Referring further to FIG. 4, anonymizer 108 can anonymize the face of person 122 by replacing it with artificially generated faces 152 and 154 (e.g., adding glasses) at the respective second locations of anonymized video streams AS1 and AS2.

[0023] In some implementations, system 100 can determine a global 3D face mesh model that overlaps each person in the video stream (e.g., VS1 and VS2). Then, system 100 (e.g., anonymizer 108) can perform 2D anonymization of the face by re-projecting the mesh onto the video stream to generate anonymized video streams AS1 and AS2 (re-projection). In some cases, the 3D face can be occluded in a particular view, e.g., by OR light, and thus may be invisible. System 100 can determine whether the 3D face is visible in the 2D image by searching for the difference between the generated depth map of the camera and the 3D face mesh. System 100 can also perform 2D rendering adjustments to obtain a more realistic face replacement, such as by merging the 3D face mesh model with the background image. In some implementations, anonymizer 108 can utilize a face replacement template that can be individualized and changed for each person (e.g., to generate artificially generated faces 152 and 154). This may enable adjustment of factors such as age, gender, and / or ethnicity. In some implementations, anonymizing the face by anonymizer 108 may include pixelating, blurring, and / or hiding the face that is visible in a second location.

[0024] In some implementations, system 100 can first perform multi-view 2D face localization by utilizing 3D information. This may enable consistent anonymization across multiple camera views. In some cases, system 100 can control 3D face replacement by utilizing mesh-based anonymization. The images anonymized by system 100 can then be made available for use by other systems further downstream.

[0025] Figure 5 is a block diagram of an exemplary internal configuration of a computer system 500 for anonymizing faces in a surgical environment. For example, the computer system 500 may be a client, server, computer, smartphone, PDA, laptop, or tablet computer with one or more processors embedded or combined, or another computing device. The computer system 500 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The computer system 500 includes a bus 502, one or more processors 504, system memory 506, read-only memory (ROM) 508, storage device 510, input device 512, output device 514, and network interface 516. In some embodiments, the computer system 500 may be part of a robotic surgical system.

[0026] Bus 502 collectively represents all system buses, peripheral buses, and chipset buses that communicate with numerous internal devices of the computer system 500. For example, bus 502 can communicate with one or more processors 504, ROM 508, system memory 506, and storage devices 510.

[0027] One or more processors 504 retrieve instructions to execute and data to process from various storage / memory units to perform various processes described herein, such as the process described above for anonymizing faces in a surgical environment. One or more processors 504 may include, but are not limited to, microprocessors, graphics processing units (GPUs), tensor processing units (TPUs), intelligent processor units (IPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). One or more processors 504 may be single-processor or multi-core processors in different implementations, such as distributed computing.

[0028] ROM 508 stores static data and instructions that can be used by one or more processors 504 and other modules of the computer system 500. Meanwhile, storage device 510 is a read / write memory device. Storage device 510 may be a persistent non-volatile memory unit that stores instructions and data even when the computer system 500 is turned off. Some implementations of the subject disclosure use a mass storage device (such as a magnetic disk or optical disk, and its corresponding disk drive) as storage device 510. Other implementations use a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) as storage device 510.

[0029] Like the storage device 510, the system memory 506 is a read / write memory device. However, unlike the storage device 510, the system memory 506 is a volatile read / write memory, such as random access memory (RAM). The system memory 506 can store some of the instructions and data that one or more processors 504 will use at runtime. In some implementations, various processes described herein, such as the process described above for anonymizing faces in a surgical environment, may be stored in the system memory 506, ROM 508, and / or storage device 510. From these various storage / memory units, one or more processors 504 can retrieve the instructions to execute and the data to process in order to execute processes in some implementations.

[0030] Bus 502 can also be connected to input device 512 via an input device interface, or to output device 514 via an output device interface. Input device 512 allows the user to communicate information to computer system 500 and select commands on computer system 500. Examples of input device 512 include an alphanumeric keyboard and a pointing device (also called a "cursor control device"). Output device 514 can, for example, enable the display of images generated by computer system 500 to the user. Examples of output device 514 include a printer and a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD). Some implementations include devices such as touchscreens that function as both input and output devices.

[0031] Bus 502 may also connect the computer system 500 to a network via the network interface 516. Thus, the computer system 500 may be part of a network of computers (such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet), or a network of networks (such as the Internet). Any or all components of the computer system 500 may be used in conjunction with the disclosures herein.

[0032] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To demonstrate this hardware and software compatibility, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms with respect to their function. Whether such functions are implemented as hardware or as software may depend on the design constraints imposed on the application and / or the system as a whole. Those skilled in the art may implement the described functions in various ways depending on the application. Such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0033] The hardware used to implement the various exemplary logics, logic blocks, modules, and / or circuits described in relation to the embodiments disclosed herein may be implemented or run by general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate logic, discrete transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of receiver devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuits specific to a given function.

[0034] In one or more exemplary embodiments, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-temporary computer-readable storage medium. Steps of the methods or algorithms disclosed herein may be embodied in processor-executable instructions that may reside on a non-temporary computer-readable medium. The non-temporary computer-readable storage medium or non-temporary processor-readable storage medium may be any storage medium accessible by a computer or processor. Examples of such non-temporary computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that is accessible by a computer. As used herein, “disk” and “disc” include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, and a disc can reproduce data optically using a laser. The above combinations also fall within the scope of non-temporary computer-readable media and non-temporary processor-readable media. Furthermore, the operation of a method or algorithm may reside as one or any combination of code and / or instructions, or a set thereof, on a non-temporary computer-readable storage medium that can be incorporated into a computer program product.

[0035] Figure 6 is a flowchart of an example of process 600 for anonymizing faces in a surgical environment (e.g., OR video anonymization). Process 600 can be performed using computing devices such as the systems, hardware, and software described with respect to Figures 1 to 5. Process 600 can be performed, for example, by executing machine-readable programs or other computer-executable instructions such as routines, instructions, programs, or other code. The operation of process 600, or any other techniques, methods, processes, or algorithms described with respect to the implementations disclosed herein, may be implemented directly in hardware, firmware, software executed by hardware, circuits, or a combination thereof.

[0036] For the sake of simplicity, Process 600 is shown and described herein as a series of actions. However, the actions provided herein may occur in various orders and / or simultaneously. Furthermore, other actions not presented or described herein may be used. Moreover, not all exemplified actions may be required to carry out the technique in accordance with the disclosed subject matter.

[0037] In operation 602, the system may generate a 3D model of a person in the surgical environment based on video streams received from cameras in the surgical environment. For example, system 100 may generate a 3D model 120 of a person in the OR based on video streams VS1 and VS2 received from cameras 1 and 2, respectively. Each camera may provide different views of the surgical environment, such as different angles, viewpoints, and / or views of the scene, based on their different positions and / or orientations in the environment. The 3D model may indicate the position of the person in the global coordinate space, for example, using a Cartesian coordinate system or a spherical coordinate system. In some cases, the 3D model may include a 3D point cloud, and each video stream may include a sub-point cloud registered in the global coordinate space of the 3D point cloud.

[0038] In operation 604, the system may determine a first location containing a person's face in global coordinate space based on its position. For example, system 100 may determine first locations 132 and 134 containing the faces of people 122 and 124 in 3D model 120, respectively. System 100 may determine the first location based on regressing the human model to detected human keypoints in the 3D model and aligning the faces of the human model to detected human keypoints of faces in the 3D model.

[0039] In operation 606, the system can match a first location to a second location within each of multiple video streams. For example, system 100 can match first locations 132 and 134 in 3D model 120 to second locations 142 and 144 in video streams VS1 and VS2. System 100 can perform matching based on projecting a 3D face model onto the video stream to determine the location of the face, and / or by comparing the face in the 3D model with the face in a depth map / image from the video stream. In some cases, the system can perform matching using bounding boxes.

[0040] In operation 608, the system may anonymize faces in each video stream based on the visibility of faces in a second location (e.g., a second location where faces are not obscured by other OR staff or equipment in the view). For example, system 100 may anonymize the faces of persons 122 and 124 in video streams VS1 and VS2 based on the visibility of faces in second locations 142 and 144. System 100 can then generate anonymized video streams AS1 and AS2 based on performing anonymization of visible faces. In various implementations, face anonymization may include replacing faces with artificially generated faces (e.g., adding / removing glasses, changing facial features such as eyes, ears, nose, and hair), pixelating faces, blurring, and / or hiding faces, or a combination thereof. In some implementations, face anonymization may include utilizing face replacement templates that can be individualized for each person.

[0041] While specific embodiments have been described and illustrated in the accompanying drawings, these embodiments are merely illustrative of the broader invention and do not limit it. Various other modifications can be conceived by those skilled in the art; therefore, the present invention is not limited to the specific configurations and arrangements shown and described. Accordingly, this description should be considered illustrative rather than restrictive.

[0042] [Implementation Method] (1) A method for anonymizing a face in a surgical environment, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. A first location including the face of a person in the 3D model is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, A method comprising anonymizing faces in the plurality of video streams based on the visibility of the faces in the second location. (2) The method according to Embodiment 1, wherein generating the 3D model of the person includes combining the two-dimensional (2D) poses of the person from the plurality of video streams to form a 3D pose. (3) The method according to Embodiment 1, wherein determining the position includes causing the human model to regress to the detected human keypoints in the 3D model. (4) The method according to Embodiment 1, wherein the 3D model includes a 3D point cloud, and each video stream includes a subpoint cloud of the 3D point cloud registered in the global coordinate space. (5) The method according to Embodiment 1, wherein the 3D model is generated based on a combination of red, green, and blue (RGB) images and depth maps from the plurality of video streams.

[0043] (6) The method according to Embodiment 1, wherein matching the first location to the second location includes projecting a 3D face model onto a video stream to determine the location of the face. (7) The method according to Embodiment 1, wherein anonymizing the face includes replacing the face with an artificially generated face. (8) The method according to Embodiment 1, wherein the anonymization of the face includes using a face replacement template that has been customized to the person. (9) A system for anonymizing faces in a surgical environment, Memory and It is a processor, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. A first location including the face of a person in the 3D model is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, Based on the visibility of the face at the second location, the faces are anonymized in the multiple video streams. A system comprising a processor configured to execute instructions stored in the memory in order to perform a certain action. (10) The system according to embodiment 9, wherein generating the 3D model includes generating a 3D mesh model for each person in the surgical environment.

[0044] (11) The system according to Embodiment 9, wherein determining the position includes aligning the face of a human model with a detected human keypoint of the face in the 3D model. (12) The system according to Embodiment 9, wherein the partial point cloud from each video stream is registered in the global coordinate space. (13) The system according to Embodiment 9, wherein the 3D model is generated based on red, green, blue, and depth (RGB-D) data from the images of the plurality of video streams. (14) The system according to Embodiment 9, wherein matching the first location to the second location includes comparing a face in the 3D model with a face in a depth map from a video stream. (15) Non-temporary computer-readable media, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. A first location including the face of a person in the 3D model is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, A non-temporary computer-readable medium storing instructions that can be operated to cause one or more processors to perform an operation including anonymizing faces in the plurality of video streams based on the visibility of the faces in the second location.

[0045] (16) A non-temporary computer-readable medium according to Embodiment 15, wherein generating the 3D model includes utilizing red, green, blue, and depth (RGB-D) data from each camera. (17) A non-temporary computer-readable medium according to Embodiment 15, wherein determining the position involves causing a statistical parametric human mesh model to regress to detected human keypoints in the 3D model. (18) The non-temporary computer-readable medium according to Embodiment 15, wherein matching the first location to the second location includes projecting a 3D face mesh model onto a video stream to determine the location of the face. (19) The non-temporary computer-readable medium according to Embodiment 15, wherein the anonymization of the face includes merging a 3D face mesh model with a background image. (20) A non-temporary computer-readable medium according to Embodiment 15, wherein a partial point cloud of a video stream is registered in the global coordinate space by minimizing the photometric reprojection error.

Claims

1. A system for anonymizing faces in a surgical environment, Memory and It is a processor, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. The first location, including the face of the person in the 3D model, is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, Based on the visibility of the face at the second location, the faces are anonymized in the multiple video streams. A system comprising a processor configured to execute instructions stored in the memory in order to perform a certain action.

2. The system according to claim 1, wherein generating the 3D model includes generating a 3D mesh model for each person in the surgical environment.

3. The system according to claim 1, wherein determining the position includes aligning the face of a human model with a detected human keypoint of the face in the 3D model.

4. The system according to claim 1, wherein the partial point clouds from each video stream are registered in the global coordinate space.

5. The system according to claim 1, wherein the 3D model is generated based on red, green, blue, and depth (RGB-D) data from the images of the plurality of video streams.

6. The system according to claim 1, wherein matching the first location to the second location includes comparing a face in the 3D model with a face in a depth map from a video stream.

7. Non-temporary computer-readable media, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. The first location, including the face of the person in the 3D model, is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, A non-temporary computer-readable medium storing instructions that can be operated to cause one or more processors to perform an operation including anonymizing faces in the plurality of video streams based on the visibility of the faces in the second location.

8. The non-temporary computer-readable medium according to claim 7, wherein generating the 3D model includes utilizing red, green, blue, and depth (RGB-D) data from each camera.

9. The non-temporary computer-readable medium according to claim 7, wherein determining the position involves causing a statistical parametric human mesh model to regress to detected human keypoints within the 3D model.

10. Matching the first location to the second location comprises projecting a 3D face mesh model onto a video stream to determine the location of the face, according to claim 7, for a non-temporary computer-readable medium.

11. The non-temporary computer-readable medium according to claim 7, wherein anonymizing the face includes merging a 3D face mesh model with a background image.

12. The non-temporary computer-readable medium according to claim 7, wherein a partial point cloud of the video stream is registered in the global coordinate space by minimizing the photometric reprojection error.

13. A method for anonymizing faces in a surgical environment, The method involves generating a three-dimensional (3D) model of a person in a surgical environment based on multiple video streams received from multiple cameras, wherein each camera provides a different view of the surgical environment, and the 3D model indicates the position of the person in a global coordinate space. The first location, including the face of the person in the 3D model, is determined based on the position of the person in the global coordinate space. Matching the first location to a second location within the plurality of video streams, A method comprising anonymizing faces in the plurality of video streams based on the visibility of the faces in the second location.

14. The method according to claim 13, wherein generating the 3D model of a person includes combining two-dimensional (2D) poses of the person from the plurality of video streams to form a 3D pose.

15. The method according to claim 13, wherein determining the position includes causing the human model to regress to the detected human keypoints in the 3D model.

16. The method according to claim 13, wherein the 3D model includes a 3D point cloud, and each video stream includes a sub-point cloud of the 3D point cloud registered in the global coordinate space.

17. The method according to claim 13, wherein the 3D model is generated based on a combination of red, green, and blue (RGB) images and depth maps from the plurality of video streams.

18. The method according to claim 13, wherein matching the first location to the second location includes projecting a 3D face model onto a video stream to determine the location of the face.

19. The method according to claim 13, wherein anonymizing the face includes replacing the face with an artificially generated face.

20. The method according to claim 13, wherein the anonymization of the face includes using a face replacement template that has been customized to the person.

Images