Blockchain-based face anonymization system

Abstract

A method performed by one or more network devices to execute one or more smart contracts stored in a blockchain for anonymizing a face appearing in digital media content. The method includes, for each of a plurality of users, obtaining a face model associated with the user, obtaining digital media content provided by a media platform, detecting one or more faces appearing in the digital media content, determining whether the detected face matches a face of any of the plurality of users based on applying one or more of the face models associated with the plurality of users to the detected face, anonymizing the detected face in response to a determination that the detected face matches a face of one of the plurality of users to generate an anonymized face, and providing the anonymized face to the media platform.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data anonymization, and more specifically, to a blockchain-based facial anonymization system. Background Technology

[0002] A blockchain consists of a continuously expanding set of blocks linked and secured using cryptography. Specifically, each block in a blockchain can include a cryptographic hash of the preceding block, a timestamp of the current block, and transaction data. The blockchain can be shared and managed through a peer-to-peer network of computers where new blocks to be added are peer-to-peer verified / confirmed, ensuring that blocks in the blockchain cannot be altered without changing all subsequent blocks—a requirement of network consensus. This architecture allows for the security of information stored within blocks through: the use of cryptography; information sharing / distribution through a peer-to-peer network; trust through consensus achieved via block addition; and the immutability of information stored within blocks through the use of cryptography, the chaining / linking of blocks, and peer-to-peer distribution (e.g., each peer in the network can maintain a ledger of all verified / confirmed transactions).

[0003] A smart contract is a computer protocol designed to digitally facilitate, verify, and / or enforce the negotiation or execution of a contract. Using smart contracts, a third party is not required to execute a trusted transaction between the parties, as the transaction is traceable and irreversible. Smart contracts can be used for general computation on the blockchain, and in this sense, they can be any kind of computer program, and do not necessarily have to be related to the classical meaning of a legal contract. Parties can use smart contracts not only to define the contractual conditions upon which they are obliged to maintain the defined arrangement, but also to define what the smart contract will automatically execute when those conditions are met.

[0004] Data anonymization is the process of encrypting or removing personally identifiable information from a dataset to keep the person / object described / portrayed by the data anonymous. Data anonymization is commonly used for privacy protection purposes. Image anonymization is a type of data anonymization that removes identifiable information from an image, which typically involves making faces in the image unrecognizable. This can be done by blurring faces, replacing faces with synthetic faces, and / or blending faces with the surrounding background scene.

[0005] Numerous tools exist for anonymizing images. For example, mobile apps exist that automatically anonymize images captured by a user's mobile device camera and / or images selected from the user's image library. These apps can automatically detect and hide faces in images by overlaying them with interesting illustratives (e.g., synthetic faces), allowing these images to be shared on social media without revealing the actual face. As another example, anonymization software exists that can detect and blur various objects in images. For instance, this software can detect faces, license plates, and / or other types of objects in an image and apply blur filters to these objects to make them unrecognizable / unreadable. Such software is commonly used to anonymize street view images, webcam images, and other images involving privacy.

[0006] However, with the anonymization methods mentioned above, the owner / uploader of the images / videos controls which images / videos are being anonymized and what / who is being anonymized in those images / videos. Other people appearing in the images / videos (who are not the owners / uploaders) have little editorial control over them. Therefore, a person's face can appear in images / videos shared / uploaded by others in public spaces (e.g., on social media or video-sharing platforms), sometimes without the person's knowledge, thus compromising their identity / privacy. This can happen accidentally (e.g., in the case of travel photos) or intentionally (e.g., when intended to reveal / compromise the person's identity). More generally, this problem arises because the person only has control over the images / videos they share / upload, and no editorial control over those shared / uploaded by others. Summary of the Invention

[0007] The embodiment overcomes some of the drawbacks of existing anonymization methods by allowing users to anonymize their faces in digital media content provided by one or more media platforms (e.g., even if the user is not the owner / uploader of the digital media content). Facial anonymization can be provided using blockchain technology and one or more smart comparisons to offer autonomous, transparent, and verifiable facial anonymization. The embodiment is a method by which one or more network devices execute one or more smart contracts stored in a blockchain to anonymize faces appearing in digital media content. The method includes: for each of a plurality of users, obtaining a facial model associated with that user and a profile of that user, wherein the facial model associated with the user is a computational model that can be used to detect the user's face; obtaining media content provided by a digital media platform; and detecting one or more faces appearing in the digital media content. The method further includes performing the following operations for each of one or more detected faces: generating a profile of the detected face; identifying one or more users from a plurality of users who have profiles that match the profile of the detected face; determining whether the detected face matches the face of any of the identified one or more users based on applying one or more facial models associated with the identified one or more users to the detected face; anonymizing the detected face to generate an anonymized face in response to the determination that the detected face matches the face of one of the identified one or more users; and providing the anonymized face to a media platform to allow the media platform to publish media content with the anonymized face.

[0008] Another embodiment is a network device configured to execute one or more smart contracts stored in a blockchain to anonymize faces appearing in digital media content. The network device includes: a collection of one or more processors; and a non-transitory machine-readable storage medium storing one or more smart contracts, which, when executed by the collection of one or more processors, cause the network device to: obtain, for each of a plurality of users, a facial model associated with the user and a user profile, wherein the facial model associated with the user is a computational model that can be used to detect the user's face; obtain digital media content provided by a media platform; detect one or more faces appearing in the digital media content; and for each of the one or more detected faces: generate a profile of the detected face; determine one or more users from the plurality of users who have profiles matching the profile of the detected face; determine whether the detected face matches the face of any of the determined one or more users based on applying one or more facial models associated with the determined one or more users to the detected face; anonymize the detected face to generate an anonymized face in response to the determination that the detected face matches the face of one of the determined one or more users; and provide the anonymized face to the media platform.

[0009] Another embodiment is a non-transitory machine-readable storage medium storing one or more smart contracts that, when executed by a set of one or more processors of one or more network devices implementing a blockchain, cause the one or more network devices to perform operations for anonymizing faces appearing in digital media content. The operations include: for each of a plurality of users, obtaining a facial model associated with that user and a profile of that user, wherein the facial model associated with the user is a computational model that can be used to detect the user's face; obtaining media content provided by a digital media platform; and detecting one or more faces appearing in the digital media content. The operations also include performing the following operations for each of the one or more detected faces: generating a profile of the detected face; identifying one or more users from the plurality of users who have profiles matching the profile of the detected face; determining whether the detected face matches the face of any of the identified one or more users based on applying one or more facial models associated with the identified one or more users to the detected face; anonymizing the detected face to generate an anonymized face in response to the determination that the detected face matches the face of the identified one or more users; and providing the anonymized face to the media platform. Attached Figure Description

[0010] The present invention can be best understood by referring to the following description and accompanying drawings, which illustrate embodiments of the invention. In the drawings:

[0011] Figure 1 This is a block diagram of a blockchain-based facial anonymization system according to some embodiments.

[0012] Figure 2 This is a diagram illustrating the interactions between components of a blockchain-based facial anonymization system according to some embodiments.

[0013] Figure 3 This is a flowchart of a process for anonymizing faces appearing in digital media content, according to some embodiments.

[0014] Figure 4A The connectivity between network devices (NDs) within an exemplary network according to some embodiments is illustrated, as well as three exemplary implementations of NDs.

[0015] Figure 4B Exemplary ways of implementing a dedicated network device according to some embodiments are shown. Detailed Implementation

[0016] The following description describes methods, apparatus, and systems for anonymizing faces appearing in digital media content. Numerous specific details, such as logic implementations, opcodes, means of specifying operands, resource partitioning / sharing / copying implementations, types and interrelationships of system components, and logical partitioning / integration choices, are set forth in this description to provide a more comprehensive understanding of the invention. However, those skilled in the art will recognize that the invention can be practiced without these specific details. In other instances, control structures, gate-level circuits, and full software instruction sequences have not been shown in detail so as not to obscure the invention. Using the included description, those skilled in the art will be able to implement appropriate functionality without excessive experimentation.

[0017] References to "an embodiment," "embodiment," "example embodiment," etc., in the specification indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include that particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it should be assumed that implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) is within the knowledge of those skilled in the art.

[0018] In this document, text enclosed in parentheses and boxes with dashed borders (e.g., long dashed dotted lines, short dashed lines, dotted lines, and dots) may be used to indicate optional operations for adding additional features to embodiments of the invention. However, such annotations should not be construed as implying that, in some embodiments of the invention, they are the only options or optional operations, and / or that boxes with solid borders are not optional.

[0019] In the following description and claims, the terms “coupled” and “connected” and their derivatives may be used. It should be understood that these terms are not intended to be synonyms with each other. “Coupled” is used to indicate that two or more elements may be in direct or non-direct physical or electrical contact with each other, cooperate with each other, or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled to each other.

[0020] Electronic devices use machine-readable media (also known as computer-readable media) to store and (internal and / or with other electronic devices on a network) transmit code (which consists of software instructions and is sometimes referred to as computer program code or computer program) and / or data. Machine-readable media are, for example, machine-readable storage media (e.g., disks, optical discs, solid-state drives, read-only memory (ROM), flash memory devices, phase-change memory) and machine-readable transmission media (also known as carriers) (e.g., electrical, optical, radio, acoustic, or other forms of propagation signals—e.g., carrier waves, infrared signals). Therefore, electronic devices (e.g., computers) include hardware and software, such as a collection of one or more processors (e.g., where the processors are microprocessors, controllers, microcontrollers, central processing units, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, other electronic circuits, or combinations thereof), coupled to one or more machine-readable storage media to store code for execution on the collection of processors and / or to store data. For example, an electronic device may include non-volatile memory containing code, because the non-volatile memory can retain code / data even when the electronic device is off (when power is lost), and when the electronic device is turned on, the portion of code to be executed by the processor of the electronic device is typically copied from the slower non-volatile memory to the volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of the electronic device. A typical electronic device also includes a set of one or more physical network interfaces (PNIs) for establishing network connections with other electronic devices (to transmit and / or receive code and / or data using propagated signals). For example, the set of physical NIs (or a combination of a set of physical NIs and a set of processors executing code) can perform any formatting, encoding, or conversion to allow the electronic device (via wired and / or wireless connections) to send and receive data. In some embodiments, the physical NIs may include radio circuitry capable of receiving data from other electronic devices via a wireless connection and / or sending data to other devices via a wireless connection. The radio circuitry may include transmitters, receivers, and / or transceivers suitable for radio frequency communications. Radio circuits can convert digital data into radio signals with appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). These radio signals can then be transmitted via an antenna to a suitable receiver. In some embodiments, the collection of physical NIs may include a network interface controller (NIC), also known as a network interface card, network adapter, or local area network (LAN) adapter. By plugging a cable into a physical port connected to the NIC, the NIC facilitates the connection of electronic devices to other electronic devices, allowing them to communicate via wired connections. One or more portions of the embodiments may be implemented using different combinations of software, firmware, and / or hardware.

[0021] A network device (ND) is an electronic device that enables communication interconnection between other electronic devices (e.g., other network devices, end-user devices) on a network. Some network devices are "multi-service network devices" that support multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session boundary control, quality of service, and / or subscriber management) and / or multiple application services (e.g., data, voice, and video).

[0022] As stated above, using existing anonymization methods, the owner / uploader of an image / video controls which images / videos are being anonymized and what / who is being anonymized in those images / videos. Other people appearing in the images / videos (who are not the owners / uploaders) have little to no edit control over them. Therefore, a person's face can appear in images / videos shared / uploaded by others in public spaces (e.g., on social media or video-sharing platforms), sometimes without the person's knowledge, thus compromising their identity / privacy. This can happen accidentally (e.g., in the case of travel photos) or intentionally (e.g., when intended to reveal / compromise the person's identity). More generally, this problem arises because the person only has control over the images / videos they share / upload, and no edit control over those shared / uploaded by others.

[0023] This disclosure describes a blockchain-based facial anonymization system that allows users and media platforms to register facial anonymization. The blockchain-based facial anonymization system can automatically anonymize the faces of registered users detected in media content provided by the registered media platform. The blockchain-based facial anonymization system can be implemented using one or more smart contracts stored in the blockchain to provide autonomous, transparent, and verifiable facial anonymization. An embodiment is a method by which one or more network devices execute one or more smart contracts stored in the blockchain to anonymize faces appearing in digital media content. The method includes: for each of a plurality of users whose faces are registered for anonymization, obtaining a facial model associated with that user and a profile of that user, wherein the facial model associated with the user is a computational model that can be used to detect the user's face; obtaining media content provided by the digital media platform whose faces are registered for anonymization; and detecting one or more faces appearing in the digital media content. The method may further include performing the following operations on each of one or more detected faces: generating a profile of the detected face; identifying one or more users from a plurality of users who have profiles matching the profile of the detected face; determining whether the detected face matches the face of any one of the identified one or more users based on applying one or more facial models associated with the identified one or more users to the detected face; anonymizing the detected face to generate an anonymized face in response to the determination that the detected face matches the face of one of the identified one or more users; and providing the anonymized face to a media platform to allow the media platform to publish media content with the anonymized face. Various embodiments thereof are further described below with reference to the accompanying drawings.

[0024] Figure 1 This is a block diagram of a blockchain-based facial anonymization system according to some embodiments. As shown, the system includes users 110A to 110N, media platforms 120A to 120C, and a blockchain 130. User 110 may correspond to a person who wishes to anonymize their face in public digital media content (e.g., digital images and / or digital videos). As will be further described herein, user 110 can register for facial anonymization to request anonymization of their face in digital media content provided by registered media platform 120. Media platform 120 is a platform that allows the public sharing of digital media content. As shown, media platform 120 includes social media platform 120A (e.g., (etc.), video sharing platform 120B (e.g., (etc.), and blog platform 120C (for example, (etc.). However, it should be understood that media platform 120 may include other types of platforms besides those shown in the figure (e.g., public camera platforms). As will be further described herein, media platform 120 may register facial anonymization to consent to the provision of digital media content that will be anonymized.

[0025] A face model generator 140 can generate a face model associated with a user interested in facial anonymization. As used herein, a face model associated with a user is a computational model that can be used to detect the user's face in digital media content. As used herein, digital media content refers to any media encoded in a machine-readable format, such as digital images and digital videos. The face model generator 140 can generate a face model associated with the user based on receiving digital images / videos containing a user's face as training input and applying machine learning techniques to the training input. Examples of machine learning techniques that can be used to train / generate a face model associated with the user include, but are not limited to, convolutional neural network algorithms and support vector machine algorithms. In one embodiment, the face model generator 140 can use a system such as FaceNet, DeepFace, OpenFace, or similar systems to generate a face model associated with the user. In one embodiment, the face model generator 140 is implemented as a mobile application that can be installed on a user's mobile device (e.g., a smartphone, tablet, or laptop). The mobile application can provide a user interface that allows the user to input / select digital images / videos containing the user's face (e.g., from various different angles) as training input to the mobile application. Mobile applications can apply machine learning techniques (e.g., the aforementioned techniques) to training inputs to generate facial models associated with users, which can be used to detect user faces in digital media content. Therefore, user 110 can download the mobile application to their respective mobile devices and use it to generate facial models. Additionally or alternatively, in one embodiment, facial model generator 140 is implemented as a server to which users can upload digital images / videos containing their faces. The server can receive digital images / videos containing user faces as training inputs and apply machine learning techniques to the training inputs to generate facial models associated with users, which can be used to detect user faces in digital media content. Therefore, user 110 can upload digital images / videos containing their faces to the server (e.g., via a secure website or application programming interface (API)), and the server can generate facial models associated with each user.

[0026] Blockchain 130 is a growing list of records linked using cryptography, referred to as “blocks.” Blockchain 130 can store smart contracts that allow the execution of trusted transactions / operations without third-party intermediaries, where the transactions / operations are traceable and irreversible. As shown in the diagram, Blockchain 130 stores user anonymization smart contracts 150, media platform smart contracts 160, face detection smart contracts 170, localization smart contracts 180, and anonymization smart contracts 190. These smart contracts can all include computer-executable code that can be executed by one or more network devices implementing Blockchain 130. As will be described in further detail herein, a smart contract can be executed to anonymize the face of a registered user appearing in digital media content provided by a registered media platform. In one embodiment, the smart contract is an Ethereum smart contract written in the Solidity programming language (which is then compiled into lower-level bytecode that runs in the Ethereum Virtual Machine (EVM)). However, it should be understood that other types of smart contracts can be used, and smart contracts can be written in other types of programming languages. Furthermore, while the diagram illustrates a face anonymization feature implemented using five separate smart contracts (e.g., to provide modularity), it should be understood that the same / similar functionality can be implemented using fewer or more smart contracts than shown (e.g., two or more smart contracts can be combined into the functionality of a single smart contract, or a single smart contract can be divided into the functionality of multiple smart contracts).

[0027] User anonymization smart contract 150 may include computer-executable code that allows user 110 to register facial anonymization. User anonymization smart contract 150 may also include computer-executable code that receives a facial model associated with user 110 (e.g., generated by facial model generator 140) and a user profile. The user profile may include attribute values ​​corresponding to one or more of the user's attributes, such as the user's age group (e.g., teen, twenties, thirties, forties, etc.), the user's gender (e.g., male or female), the user's skin color, and / or the user's location.

[0028] The media platform smart contract 160 may include computer-executable code that allows the media platform 120 to register facial anonymization. The media platform smart contract 160 may also include computer-executable code that receives digital media content to be anonymized provided by the media platform 120 and provides the received digital media content to the facial detection smart contract 170.

[0029] The face detection smart contract 170 may include computer executable code that receives digital media content provided by media platform 120 (e.g., via media platform smart contract 160 or directly from media platform 120 itself) and detects faces appearing in the digital media content. Faces can be detected in the digital media content using any type of face detection algorithm. Examples of face detection algorithms that can be used include, but are not limited to, the Haar-cascade algorithm, the Eigenfaces algorithm, the Fisherfaces algorithm, the Local Binary Pattern algorithm, and the Convolutional Neural Network algorithm. The face detection smart contract 170 may also include computer executable code that sends the detected faces to a localization smart contract 180. In one embodiment, faces detected in the digital media content are cropped from the digital media content, and the cropped faces are sent to the localization smart contract 180 for further processing (cropping faces reduces the amount of data compared to sending / processing the entire image / video). In another embodiment, faces are sent to the localization smart contract 180 by sending data about the faces without cropping the faces from the digital media content (e.g., by specifying the location of the faces detected in the digital media content, for example, using bounding boxes).

[0030] The localized smart contract 180 may include computer-executable code that receives faces detected in digital media content (e.g., from face detection smart contract 170) and generates a profile of the detected face. The profile of the detected face may include one or more attribute values ​​corresponding to one or more attributes of the detected face. The detected face attributes may be the same as or overlap with attributes included in a user profile. For example, the detected face attributes may include the age group of the detected face (e.g., teenager, twenties, thirties, forties, etc.), the gender of the detected face (e.g., male or female), the skin color of the detected face, and / or the location of the detected face. The attribute values ​​of the detected face may be determined based on applying one or more computational models to the detected face, these computational models being trained using machine learning techniques to specifically determine these attribute values ​​(e.g., computational models trained to detect the age group of faces, detect the gender of faces, etc.). In one embodiment, each of the attribute values ​​is assigned a confidence level that indicates the confidence level of the classification / determination performed using the computational model (e.g., the confidence level for the detected face being in the twenties age group is 80%). The localized smart contract 180 may also include computer-executable code that sends the detected face and its profile to the anonymous smart contract 150. As will be further described herein, the profile of the detected face can be used to reduce the search space of the facial model applied to the detected face.

[0031] User-anonymous smart contract 150 may include computer-executable code that receives a detected face and a profile of the detected face from a localized smart contract 180. User-anonymous smart contract 150 may also include computer-executable code that determines which registered users have profiles that match the profile of the detected face and determines whether the detected face matches any of those users based on applying facial models associated with those users to the detected face. For example, if the profile of the detected face includes attribute values ​​indicating that the detected face is a woman in her 20s with a darker complexion, the detected face will be matched against the faces of users with the same / similar profiles. This helps reduce the search space of the facial models applied to the detected face, thus allowing for lower computational costs and faster matching speeds. In one embodiment, only attribute values ​​of the detected face with a confidence level above a threshold level are used to determine which registered users have profiles that match the profile of the detected face. If the detected face does not match the face of any user with a profile that matches the detected face, then it is determined whether the detected face matches the face of any other remaining registered user (based on applying the facial models associated with these other users to the detected face). If it is determined that the detected face matches the face of a registered user, the user anonymization smart contract 150 may also include computer-executable code that sends the detected face to the anonymization smart contract 190.

[0032] Anonymous smart contract 190 may include computer executable code that receives a face to be anonymized (e.g., from user anonymized smart contract 150) and anonymizes the face. Anonymization can be achieved, for example, by pixelating / blurring the face, replacing the face with a synthetic face, blending the face into a background scene, or otherwise hiding the face. Anonymized smart contract 190 may also include computer executable code that provides the anonymized face to media platform 120 (which provides digital media content including the face) to allow the media platform to publish digital media content with the anonymized face. In one embodiment, the anonymized face is provided to media platform 120 in a cropped form, and the media platform is responsible for “stitching” the anonymized face to its original position / time within the digital media content. Alternatively or additionally, in one embodiment, anonymized smart contract 190 includes computer executable code that stitches the anonymized face to its original position / time within the digital media content and provides digital media content including the stitched-in anonymized face to media platform 120.

[0033] There are certain use cases that may require special handling. One use case that may require special handling is when a user wants to appear in certain digital media content but not in others. This could be the case, for example, if the user appears in an advertisement. In this situation, the user may not want to be anonymized in the advertisement but in other (non-advertising) digital media content. To allow for this use case, the system could provide the user with options (e.g., during or after registration) to specify certain conditions under which the user's face should be anonymized in digital media content. For example, the system could allow the user to specify that face anonymization should be disabled for the user in digital media content identified as an advertisement. If the user specifies that face anonymization should be disabled for advertisements, the system can avoid anonymizing the user's face if it appears in digital media content identified as an advertisement. Another use case that may require special handling is when a user's face is very similar to another user's face. This could lead to two (or more) facial models identifying the same face in digital media content, which could result in the wrong face being anonymized without the user's consent. To address this situation, the system could provide options for resolving conflicts. For example, the system could allow the user to specify the action to be taken in this situation. This action could be to inform the user to ignore the event or to notify them of it. If the user is notified of such an event, the system can allow the user to provide notes related to the digital media content (e.g., location, known people (if they appear in the digital media content)), which can be sent to the registered media platform, which will be able to verify and subsequently decide whether to continue anonymization. For example, the registered media platform can act as a mediator by deciding whether to continue anonymization based on matching the metadata of the digital media content with the verification notes. In the rare cases where the match is uncertain, the registered media platform can decide to maintain anonymization.

[0034] The embodiments disclosed herein offer several benefits. One benefit is that the embodiments can automatically and autonomously (e.g., without involving third-party intermediaries) anonymize the faces of registered users appearing in digital media content provided by a registered media platform. Therefore, even if a user does not own / share / upload the digital media content, they can still have their face anonymized within it. Another benefit of the embodiments disclosed herein is that even if a user is not a user of the media platform, they can still anonymize the faces of users appearing in digital media content provided by the media platform (provided both the user and the media platform have registered using a blockchain-based face anonymization system). Another benefit of the embodiments disclosed herein is that they provide transparency through the use of smart contracts stored in a blockchain. Both the user and the media platform can access the smart contracts stored in the blockchain (therefore the specifications / behavior of these smart contracts are known to both the user and the media platform), which establishes trust between the parties. Another benefit of the embodiments disclosed herein is that they provide verifiable results. The use of smart contracts and blockchain ensures that the face anonymization process is always verified through proof-of-work or other types of consensus mechanisms.

[0035] Furthermore, as mentioned above, localization helps reduce the search space of facial models applied to the detected face, thus allowing for lower computational costs and faster matching speeds. However, it should be noted that blockchain-based facial anonymization systems can be implemented without localization, although such implementations typically involve higher computational costs and longer matching times compared to implementations with localization.

[0036] Figure 2This diagram illustrates the interactions between components of a blockchain-based facial anonymization system according to some embodiments. At circle "1", user 110 provides training images (e.g., digital images of the user's face from different angles) to facial model generator 140. User 110 may also provide a user profile to facial model generator 140. At circle "2", facial model generator 140 generates a facial model associated with user 110 and sends this facial model along with user 110's profile to user anonymization smart contract 150. At circle "3", media platform 120 provides digital media content to be anonymized to facial detection smart contract 170. At circle "4", facial detection smart contract 170 detects faces appearing in the media content and sends the detected faces to localization smart contract 180. At circle "5", the localization smart contract generates a profile of the detected face and sends the detected face and its profile to user anonymization smart contract 150. At circle "6", the anonymous smart contract 150 applies the facial model associated with the registered user to the detected face to determine which (if any) of the detected faces match the registered user's face. To reduce the search space of facial models that need to be applied to the detected face, the anonymous smart contract 150 can determine which registered users have profiles that match the profile of the detected face and apply the facial models associated with these users to the detected face. If the detected face does not match any of these users' faces, the anonymous smart contract 150 can expand the search space (e.g., to include facial models of registered users who do not have profiles that match the profile of the detected face or otherwise have profiles that are less compatible). The anonymous smart contract 150 can then send the detected faces that match the face of user 110 (or the face of any other registered user) to the anonymous smart contract 190. At circle "7", the anonymous smart contract 190 anonymizes the detected face and sends the anonymized face to the media platform 120.

[0037] Figure 3 This is a flowchart illustrating a process for anonymizing faces appearing in digital media content, according to some embodiments. In one embodiment, the process is implemented by one or more network devices executing one or more smart contracts stored in a blockchain. The process can be implemented using hardware, software, firmware, or any combination thereof. The operations in the flowchart will be described with reference to exemplary embodiments in the other accompanying drawings. However, it should be understood that the operations in the flowchart can be performed by embodiments other than those described with reference to the other accompanying drawings, and the embodiments discussed with reference to those other accompanying drawings can perform operations different from those discussed with reference to the flowchart.

[0038] In one embodiment, in response to a request to register facial anonymization received from a user, one or more network devices register the user's facial anonymization, wherein an indication that the user has been registered for facial anonymization is stored in a blockchain. In response to a request to register facial anonymization received from a media platform, one or more network devices register the media platform's facial anonymization, wherein an indication that the media platform has been registered for facial anonymization is stored in a blockchain; and

[0039] At box 310, for each of a plurality of users (e.g., those registered for facial anonymization), one or more network devices obtain a facial model associated with that user and a profile of that user (e.g., including the user's age group, gender, and / or skin color), wherein the facial model associated with that user is a computational model that can be used to detect the user's face in digital media content. In one embodiment, each of the plurality of users is registered for facial anonymization.

[0040] At box 320, one or more network devices acquire digital media content provided by a media platform (e.g., one with registered facial anonymization). In one embodiment, the media platform is a social media platform, video sharing platform, blogging platform, or public camera platform (or any combination thereof). In one embodiment, the media platform is registered with facial anonymization. In one embodiment, the media platform is one of multiple media platforms with registered facial anonymization. In one embodiment, the digital media content includes digital images or digital videos uploaded to the media platform by users of the media platform. In one embodiment, at least one of the multiple users is not a user of the media platform (e.g., a user without a media platform account).

[0041] At box 330, one or more network devices detect one or more faces appearing in digital media content. For each of the detected faces, the one or more network devices execute boxes 340 through 395. At box 340, the one or more network devices generate a profile of the detected face (e.g., which includes the age group of the detected face, the gender of the detected face, and / or the skin color of the detected face). At box 350, the one or more network devices determine one or more users from a plurality of users who have profiles that match the profile of the detected face. In one embodiment, the profile of a given user among the plurality of users and the profile of a given detected face among the one or more detected faces each include one or more attribute values ​​corresponding to one or more attributes, wherein the profile of a given user is determined to match the profile of a given detected face if one or more attribute values ​​included in the profile of a given user match the corresponding attribute values ​​included in the profile of a given detected face. In one embodiment, the one or more attributes include one or more of the following: age group, gender, skin color, and location. At decision box 360, one or more network devices determine whether the detected face matches the face of any of the identified one or more users (based on applying a facial model associated with the identified one or more users to the detected face). If the detected face matches the face of any of the identified one or more users, the process proceeds to box 390, where one or more network devices anonymize the detected face to generate an anonymized face. In one embodiment, anonymizing a given detected face is done by pixelating the given detected face, replacing the given detected face with a synthetic face, or blending the given detected face with a background scene included in digital media content. At box 395, one or more network devices provide the anonymized face to a media platform (e.g., to allow the media platform to publish digital media content with the anonymized face). Returning to decision box 360, if one or more network devices determine that the detected face does not match the face of any of the identified one or more users, the process proceeds to decision box 370. At decision box 370, one or more network devices determine whether the detected face matches the face of any of the other users. If the detected face matches the face of any other user, the process proceeds to box 390, where one or more network devices anonymize the detected face to generate an anonymized face. At 395, one or more network devices provide the anonymized face to the media platform. Returning to decision box 370, if one or more network devices determine that the detected face does not match the face of any other user, the process proceeds to box 380, where it is determined that the detected face was not anonymized.

[0042] If one or more network devices complete processing of the detected face group, the process proceeds to block 320, where one or more network devices obtain additional digital media content to be anonymized from the media platform. One or more network devices may repeat blocks 330 through 395 to process the additional digital media content in a manner similar to that described above.

[0043] Figure 4A The connectivity between network devices (NDs) within an exemplary network according to some embodiments is illustrated, as well as three exemplary implementations of NDs. Figure 4A The diagram illustrates ND400A through ND400H and the connections between them, shown via lines between 400A and 400B, 400B and 400C, 400C and 400D, 400D and 400E, 400E and 400F, 400F and 400G, and between 400A and 400G, as well as between 400H and each of 400A, 400C, 400D, and 400G. These NDs are physical devices, and the connectivity between them can be wireless or wired (often referred to as links). Additional lines extending from NDs 400A, 400E, and 400F illustrate that these NDs act as entry and exit points for the network (hence, these NDs are sometimes referred to as edge NDs; while other NDs may be referred to as core NDs).

[0044] Figure 4A Two exemplary ND implementations are: 1) a dedicated network device 402 using a custom application-specific integrated circuit (ASIC) and a dedicated operating system (OS); and 2) a general-purpose network device 404 using a common off-the-shelf processor (COTS) and a standard operating system (OS).

[0045] Dedicated network device 402 includes networking hardware 410, which includes a collection of one or more processors 412, forwarding resources 414 (which typically include one or more ASICs and / or network processors), and physical network interfaces (NI) 416 (through which network connections are made, such as those shown by the connectivity between ND400A and ND400H), and a non-transitory machine-readable storage medium 418 in which networking software 420 is stored. During operation, networking software 420 can be executed by networking hardware 410 to instantiate a collection of one or more networking software instances 422. Each networking software instance 422 and the portion of networking hardware 410 that executes that networking software instance (if it is hardware dedicated to that networking software instance and / or a time slice of hardware shared by that networking software instance with other networking software instances 422) form separate virtual network elements 430A to 430R. Each virtual network element (VNE) 430A to 430R includes control communication and configuration modules 432A to 432R (sometimes referred to as local control modules or control communication modules) and forwarding tables 434A to 434R, such that a given virtual network element (e.g., 430A) includes a control communication and configuration module (e.g., 432A), a set of one or more forwarding tables (e.g., 434A), and a portion of the networking hardware 410 that executes the virtual network element (e.g., 430A).

[0046] Software 420 may include code such as a face anonymization smart contract 425 (which may include one or more of a user anonymization smart contract 150, a media platform smart contract 160, a face detection smart contract 170, a localization smart contract 180, and anonymization smart contract 190), which, when executed by networked hardware 510, causes a dedicated network device 402 to perform the operations of one or more embodiments described above as part of networked software instance 422 (e.g., to provide the blockchain-based face anonymization functionality described herein).

[0047] The dedicated network device 402 is often considered, physically and / or logically, to include: 1) an ND control plane 424 (sometimes referred to as the control plane), including processor 412 that performs control communication and configuration modules 432A to 432R; and 2) an ND forwarding plane 426 (sometimes referred to as the forwarding plane, data plane, or media plane), including forwarding resources 414 utilizing forwarding tables 434A to 434R and physical NI 416. As an example of an ND being a router (or implementing routing functions), the ND control plane 424 (processor 412 that performs control communication and configuration modules 432A to 432R) is typically responsible for participating in controlling how data (e.g., packets) is routed (e.g., the next hop of data and the output physical NI of that data) and for storing that routing information in forwarding tables 434A to 434R, while the ND forwarding plane 426 is responsible for receiving the data on physical NI 416 and forwarding the data to the appropriate physical NI in physical NI 416 based on forwarding tables 434A to 434R.

[0048] Figure 4B Exemplary ways of implementing a dedicated network device 402 according to some embodiments are shown. Figure 4B A dedicated network device including card 438 (typically hot-swappable) is shown. Although in some embodiments, card 438 has two types (one or more that operate as ND forwarding plane 426 (sometimes referred to as line cards) and one or more that operate to implement ND control plane 424 (sometimes referred to as control cards)), alternative embodiments may combine functionality onto a single card and / or include additional card types (e.g., an additional type of card is referred to as a service card, resource card, or multi-application card). Service cards can provide specialized processing (e.g., Layer 4 to Layer 7 services such as firewalls, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection Systems (IDS), Peer-to-Peer (P2P), Voice over IP (VoIP) Session Border Controllers, Mobile Radio Gateways (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateways)). As an example, service cards can be used to terminate IPsec tunnels and perform accompanying authentication and encryption algorithms. These cards are coupled together via one or more interconnect mechanisms shown as backplane 436 (e.g., a first full-mesh coupled line card and a second full-mesh coupled all cards).

[0049] Return to Figure 4AThe general network equipment 404 includes hardware 440, which includes a collection of one or more processors 442 (typically COTS processors), a physical NI 446, and a non-transitory machine-readable storage medium 448 in which software 450 is stored. During operation, the processors 442 execute the software 450 to instantiate one or more collections of applications 464A to 464R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 454 represents the kernel of an operating system (or a shim executing on a base operating system) that allows the creation of multiple instances 462A to 462R called software containers, each instance of which can be used to execute one (or more) of the application collections 464A to 464R. In this embodiment, multiple software containers (also referred to as virtualization engines, virtual private servers, or jails) are user-space instances (typically virtual memory spaces) that are isolated from each other and from the kernel space of the running operating system; and in which, unless explicitly permitted, the set of applications running in a given user space cannot access the memory of other processes. In another such alternative embodiment, virtualization layer 454 represents a supervisor (sometimes called a virtual machine monitor (VMM)) or a supervisor that executes on top of the host operating system, and each of the set of applications 464A to 464R runs on top of a guest operating system within instances 462A to 462R, referred to as a virtual machine running on top of the supervisor (in some cases, this can be considered as a form of tightly isolated software container)—the guest operating system and applications may not be aware that they are running on a virtual machine rather than on “bare metal” host electronic devices, or through paravirtualization, the operating system and / or applications may be aware of the existence of virtualization for optimization purposes. In other alternative embodiments, one, some, or all of the applications are implemented as a single kernel, which can be generated by directly compiling only a limited set of libraries (e.g., from the Library Operating System (LibOS), including drivers / libraries for the OS services) that provide the specific OS services required by the application. Since a single kernel can be implemented directly on hardware 440, directly on the supervisor (in which case, the single kernel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented entirely by a single kernel that: runs directly on the supervisor represented by virtualization layer 454, runs within a software container represented by instances 462A to 462R, or is a combination of a single kernel and the aforementioned techniques (e.g., both the single kernel and the virtual machine run directly on the supervisor, or a single kernel and a collection of applications running within different software containers).

[0050] The instantiation and virtualization (if implemented) of one or more sets of applications 464A to 464R are collectively referred to as software instance 452. Each set of applications 464A to 464R, the corresponding virtualization construct (e.g., instances 462A to 462R) (if implemented), and a portion of the hardware 440 that executes them (a time slice of hardware dedicated to that execution and / or temporarily shared hardware) form a separate virtual network element 460A to 460R.

[0051] Virtual network elements 460A to 460R perform similar functions to virtual network elements 430A to 430R—for example, similar to control communication and configuration modules 432A and forwarding tables 434A (this virtualization of hardware 440 is sometimes referred to as Network Functions Virtualization (NFV)). Therefore, NFV can be used to unify many network device types into industry-standard high-capacity server hardware, physical switches, and physical storage devices, which can reside in data centers, NDs, and customer premises equipment (CPEs). Although embodiments of the invention are shown to correspond to one VNE 460A to 460R for each instance 462A to 462R, alternative embodiments may implement this correspondence at a finer granular level (e.g., line card virtualization of line cards, control card virtualization of control cards, etc.); it should be understood that the techniques described herein with reference to the correspondence between instances 462A to 462R and VNEs are equally applicable to embodiments using this finer granular level and / or a single core.

[0052] In some embodiments, virtualization layer 454 includes a virtual switch that provides forwarding services similar to those of a physical Ethernet switch. Specifically, the virtual switch forwards traffic between instances 462A to 462R and the physical NI 446, and optionally between instances 462A to 462R; furthermore, the virtual switch can enforce network isolation (e.g., by implementing Virtual Local Area Networks (VLANs)) between VNEs 460A to 460R that are not allowed to communicate with each other.

[0053] Software 450 may include code such as a face anonymization smart contract 463 (which may include one or more of a user anonymization smart contract 150, a media platform smart contract 160, a face detection smart contract 170, a localization smart contract 180, and anonymization smart contract 190), which, when executed by processor 442, causes general-purpose network device 404 to perform the operations of one or more embodiments described above as partial software instances 462A to 462R (e.g., to provide the blockchain-based face anonymization functionality described herein).

[0054] Figure 4AA third exemplary ND implementation is a hybrid network device 406, which includes a custom ASIC / proprietary OS and a COTS processor / standard OS in a single ND or a single card within an ND. In some embodiments of such a hybrid network device, a platform VM (i.e., a VM that implements the functionality of the dedicated network device 402) can provide quasi-virtualization to the networking hardware present in the hybrid network device 406.

[0055] Regardless of the example implementation of the ND described above, when considering a single VNE among multiple VNEs implemented by the ND, or where the NV currently implements only a single VNE, the abbreviated term Network Element (NE) is sometimes used to refer to that VNE. Similarly, in all the example implementations described above, each VNE (e.g., VNE430A-R, VNE460A-R, and those in Hybrid Network Device 406) receives data on a physical NI (e.g., 416, 446) and forwards that data to the appropriate physical NI (e.g., 416, 446). For example, a VNE implementing IP router functionality forwards IP packets based on some IP header information in the IP packets; where the IP header information includes the source IP address, destination IP address, source port, destination port (wherein this document, "source port" and "destination port" refer to protocol ports, as opposed to the physical ports of the ND), transport protocol (e.g., User Datagram Protocol (UDP), Transmission Control Protocol (TCP), and Differentiated Code Point (DSCP) value).

[0056] A Network Interface (NI) can be physical or virtual; and in the context of IP, an interface address is the IP address assigned to an NI, whether it is a physical NI or a virtual NI. A virtual NI can be associated with a physical NI, associated with another virtual interface, or standalone (e.g., a loopback interface, a point-to-point protocol interface). NIs (physical or virtual) can be numbered (NIs with IP addresses) or unnumbered (NIs without IP addresses). A loopback interface (and its loopback address) is a specific type of virtual NI (and IP address) often used for management purposes on a NE / VNE (physical or virtual); where such an IP address is called a node loopback address. The IP address assigned to an NI on an ND is called the IP address of that ND; at a more granular level, the IP address assigned to an NI on an NE / VNE implemented on an ND can be called the IP address of that NE / VNE.

[0057] Some parts of the invention have been described in detail regarding the algorithms and symbolic representations of transactions of data bits already stored in computer memory. These algorithms and representations are methods commonly used by those skilled in the art of data processing to communicate their work to others skilled in the art. Algorithms are generally conceived herein as a self-consistent sequence of transactions that leads to a desired result. A transaction is a step requiring physical manipulation of a physical quantity. Typically (though not always), these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that, for convenience and primarily for general use, these signals can be represented using bits, values, elements, symbols, features, items, numbers, etc.

[0058] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient notations applied to those quantities. Unless otherwise explicitly stated as is evident in the above discussion, it should be understood that throughout the specification, discussions using terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data representing physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0059] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used with programs according to the teachings of this document, or it can be demonstrated that more specialized devices can be readily constructed to perform the desired methodological transactions. The required structures for various such systems are apparent from the description above. Furthermore, no particular programming language is referenced in the description of the embodiments. It should be understood that the teachings of the embodiments described herein can be implemented using various programming languages.

[0060] One embodiment may be an article of manufacture in which instructions (e.g., computer code) for programming one or more data processing components (collectively referred to herein as "processors") to perform the operations described above are stored on a non-transitory machine-readable storage medium (e.g., microelectronic memory). In other embodiments, some of these operations may be performed by specific hardware components (e.g., dedicated digital filter blocks and state machines) containing hard-wired logic. Alternatively, these operations may be performed by any combination of programmable data processing components and fixed hard-wired circuit components.

[0061] Throughout this specification, embodiments have been presented by means of flowcharts. It should be understood that the transactions and their order described in these flowcharts are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art will recognize that variations can be made to the flowcharts without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0062] In the foregoing specification, embodiments have been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made to the invention without departing from the broader spirit and scope of the invention as set forth in the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for anonymizing faces appearing in digital media content by executing one or more smart contracts stored in a blockchain using one or more network devices, the method comprising: For each of a plurality of users, obtain (310) a facial model associated with the user and a profile of the user, wherein the facial model associated with the user is a computational model capable of detecting the user’s face, and wherein the profile of the user includes multiple attribute values ​​of the user. Obtain (320) digital media content provided by a media platform; Detecting (330) one or more faces appearing in the digital media content; and For each of one or more of the detected faces: Generate (340) a profile of the detected face, the profile of the detected face including multiple attribute values ​​of the detected face. Based on comparing the multiple attribute values ​​of the detected face with the corresponding attribute values ​​of the multiple users included in the profiles of the multiple users, one or more users are identified from the multiple users (350) who have profiles that match the profile of the detected face. Based on applying one or more facial models from a set of facial models associated with one or more identified users to the detected face, it is determined (360) whether the detected face matches the face of any one of the one or more identified users. In response to the determination that the detected face matches the face of one or more of the identified users, the detected face is anonymized (390) to generate an anonymized face, and Provide the anonymized face (395) to the media platform.

2. The method according to claim 1, wherein, The detected faces in one or more detected faces are cropped from the digital media content before being anonymized.

3. The method according to claim 1, wherein, The multiple attribute values ​​of the detected face correspond to multiple attributes, wherein the multiple attributes include one or more of the following: age group, gender, skin color, and location.

4. The method according to claim 1, wherein, At least one of the multiple users is not a user of the media platform.

5. The method according to claim 1, wherein, The digital media content includes digital images or digital videos uploaded to the media platform by users of the media platform.

6. The method according to claim 1, wherein, Each of the plurality of users is registered with facial anonymization, and the media platform is also registered with facial anonymization.

7. The method according to claim 6, further comprising: In response to a request to register facial anonymization received from the user, the user is registered for facial anonymization, wherein an indication that the user has been registered for facial anonymization is stored in the blockchain; and In response to a request for facial anonymization received from the media platform, the media platform is registered for facial anonymization, wherein an indication of the media platform being registered for facial anonymization is stored in the blockchain.

8. The method according to claim 6, wherein, The media platform in question is one of several media platforms that have registered facial anonymization.

9. The method according to claim 1, wherein, Anonymize a given detected face by: pixelating the given detected face, replacing the given detected face with a synthetic face, or mixing the given detected face with a background scene included in the digital media content.

10. The method according to claim 1, wherein, The media platform mentioned is a social media platform, video sharing platform, blog platform, or public camera platform.

11. A network device (404) configured to execute one or more smart contracts stored in a blockchain to anonymize faces appearing in digital media content, said network device comprising: A collection of one or more processors (442); as well as A non-transitory machine-readable storage medium (448) storing the one or more smart contracts (463), which, when executed by a set of one or more processors, cause the network device to: For each of a plurality of users, a facial model associated with that user and a user profile are obtained, wherein the facial model associated with the user is a computational model capable of detecting the user's face, and the user profile includes multiple attribute values ​​of the user. Obtain digital media content provided by media platforms. Detect one or more faces appearing in the digital media content, and For each of one or more of the detected faces: Generate a profile of the detected face, the profile of the detected face including multiple attribute values ​​of the detected face. Based on comparing the multiple attribute values ​​of the detected face with the corresponding attribute values ​​of the multiple users included in the profiles of the multiple users, one or more users are identified from the multiple users who have profiles that match the profile of the detected face. Based on applying one or more facial models from a pool of identified facial models associated with one or more users to the detected face, it is determined whether the detected face matches the face of any one of the identified one or more users. In response to the determination that the detected face matches the face of one or more determined users, the detected face is anonymized to generate an anonymized face, and The anonymized face is provided to the media platform.

12. The network device according to claim 11, wherein, The detected faces in one or more detected faces are cropped from the digital media content before being anonymized.

13. The network device according to claim 11, wherein, The multiple attribute values ​​of the detected face correspond to multiple attributes, wherein the multiple attributes include one or more of the following: age group, gender, skin color, and location.

14. The network device according to claim 11, wherein, At least one of the multiple users is not a user of the media platform.

15. The network device according to claim 11, wherein, The digital media content includes digital images or digital videos uploaded to the media platform by users of the media platform.

16. A non-transitory machine-readable storage medium storing one or more smart contracts, said one or more smart contracts, when executed by a collection of one or more processors of one or more network devices implementing a blockchain, causing said one or more network devices to perform operations for anonymizing faces appearing in digital media content, said operations including: For each of a plurality of users, obtain (310) a facial model associated with the user and a profile of the user, wherein the facial model associated with the user is a computational model capable of detecting the user’s face, and wherein the profile of the user includes multiple attribute values ​​of the user. Obtain (320) digital media content provided by a media platform; Detecting (330) one or more faces appearing in the digital media content; and For each of one or more of the detected faces: Generate (340) a profile of the detected face, the profile of the detected face including multiple attribute values ​​of the detected face. Based on comparing the multiple attribute values ​​of the detected face with the corresponding attribute values ​​of the multiple users included in the profiles of the multiple users, one or more users are identified from the multiple users (350) who have profiles that match the profile of the detected face. Based on applying one or more facial models from a set of facial models associated with one or more identified users to the detected face, it is determined (360) whether the detected face matches the face of any one of the one or more identified users. In response to the determination that the detected face matches the face of one or more of the identified users, the detected face is anonymized (390) to generate an anonymized face, and Provide the anonymized face (395) to the media platform.

17. The non-transitory machine-readable storage medium according to claim 16, wherein, At least one of the multiple users is not a user of the media platform.

18. The non-transitory machine-readable storage medium according to claim 16, wherein, Each of the plurality of users is registered with facial anonymization, and the media platform is also registered with facial anonymization.

19. The non-transitory machine-readable storage medium according to claim 16, wherein, Anonymize a given detected face by: pixelating the given detected face, replacing the given detected face with a synthetic face, or mixing the given detected face with a background scene included in the digital media content.

20. The non-transitory machine-readable storage medium according to claim 16, wherein, The media platform in question is one of several media platforms that have registered facial anonymization.

Images