Detecting motion to prevent recognition

Abstract

A method for determining whether to identify an image or video. The method includes detecting a person in an image or video by determining a person image area corresponding to a boundary of the person or a facial image area corresponding to a position of the person's face. The method further includes analyzing pixel values ​​corresponding to the person or the position of the person's face based on the boundary of the person or the facial image area to generate an identification avoidance signal. The method further includes determining whether the identification avoidance signal indicates that an action was taken to prevent identification of the person. The method further includes refusing to identify the person in response to the identification avoidance signal indicating that the action was taken to prevent identification of the person.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 15 / 943,961, filed on April 3, 2018, entitled “Detecting Action to Prevent Recognition,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] The present specification relates to a recognition application for determining whether to recognize a person in an image or video.

[0004] When a person does not want to be identified in an image or video, it was previously sufficient to occlude the person's face, as occluding the person's face would prevent the facial recognition algorithm from identifying enough facial features to perform facial recognition. However, in modern computer vision systems, it is possible to identify a person in an image or from a sequence of images or in a video by extracting features from the entire body, even when the face is not visible or is highly occluded. For example, a person wearing a unique shirt in a first photo can be identified by first finding that shirt worn in another photo where the face is visible. Similarly, a person can be identified by a unique hairstyle, jewelry, tattoos, an object held in the hand, posture, or any other indicator. As a result, modern computer vision systems can identify people against their will.

[0005] The background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors described in this background section, as well as aspects of the description that may not otherwise be qualified as prior art at the time of filing, should neither be explicitly nor implicitly identified as prior art for the present disclosure. Summary of the invention

[0006] Embodiments are generally directed to a method, such as a computer-implemented method of determining whether to identify an image or video. The method includes detecting a person in an image or video by determining a person image region corresponding to a boundary of the person or a facial image region corresponding to a location of the person's face. The method further includes analyzing pixel values ​​corresponding to the person or the location of the person's face based on the boundary of the person or the facial image region to generate a recognition avoidance signal. The method further includes determining whether the recognition avoidance signal indicates that an action was taken to prevent recognition of the person. The method further includes, in response to the recognition avoidance signal indicating that the action was taken to prevent recognition of the person, refusing to identify the person.

[0007] In some embodiments, refusing to perform identification of the person includes performing at least one of: storing the identification avoidance signal in association with the image or video to indicate that identification of the person is not to be performed, deleting the image or video, archiving the image or video, or demoting the image or video in search results. In some embodiments, the method further includes, in response to the identification avoidance signal not indicating that the action was taken to prevent identification of the person, performing identification of the person by determining the identity of the person and indexing the image or video in association with the identity. In some embodiments, determining that the action was taken to prevent identification of the person further includes determining that the identification avoidance signal corresponds to an avoidance value that does not meet a threshold, and further in response to the avoidance value not meeting the threshold, performing identification of the person. In some embodiments, the person is a first person, determining that the action was taken to prevent identification of the person further includes determining that the identification avoidance signal corresponds to an avoidance value that meets a threshold, and the method further includes, in response to the avoidance value meeting the threshold, detecting a second person in the image and generating the identification avoidance signal for the second person. In some embodiments, the action includes (1) the person using an object to obscure at least a portion of the person's face, or (2) at least a portion of the person's face is digitally obscured. In some embodiments, the identification of the person is based on extracting a recognition template. In some embodiments, the image is a first image, the video is a first video, and further includes: identifying one or more additional images or one or more additional videos associated with an event, and determining not to identify the person within the one or more additional images or one or more additional videos associated with the event. In some embodiments, determining that the identification avoidance signal indicates that the action was taken to prevent identification of the person includes determining an action type, and further includes: identifying the person by identifying the person, and based on the action type, sharing the identification of the person with a group of users in a social network, and not sharing the identification of the person with any user who is not part of the group of users in the social network.

[0008] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon, when the instructions are executed by one or more computers, the instructions cause the one or more computers to perform operations, the operations including: detecting the person in an image or video by determining a person image area corresponding to the boundary of the person or a facial image area corresponding to the position of the person's face, analyzing pixel values ​​corresponding to the person or the position of the person's face based on the boundary of the person or the facial image area to generate an identification avoidance signal, determining whether the identification avoidance signal indicates that an action has been taken to prevent identification of the person, and identifying the person in response to the identification avoidance signal not indicating that the action has been taken to prevent identification of the person.

[0009] In some embodiments, performing the identification of the person includes determining the identity of the person and indexing the image or video in association with the identity. In some embodiments, the operation further includes: in response to the identification avoidance signal indicating that the action has been taken to prevent the identification of the person, performing at least one of the following: storing the identification avoidance signal in association with the image or video to indicate that the person identification is not to be performed, deleting the image or video, archiving the image or video, or demoting the image or video in search results. In some embodiments, the operation further includes: in response to the identification avoidance signal indicating that the action has been taken to prevent the identification of the person, determining whether the identification avoidance signal corresponds to an avoidance value that satisfies a threshold, and in response to the avoidance value failing to satisfy the threshold, performing the identification of the person. In some embodiments, the action includes (1) the person using an object to obscure at least a portion of the face of the person, or (2) at least a portion of the face of the person is digitally obscured. In some embodiments, the operation further includes: determining a permission associated with the person, and wherein performing the identification of the person is based on the permission associated with the person.

[0010] In some embodiments, a system includes one or more processors and a memory storing instructions executed by the one or more processors, the instructions including: detecting a person in an image or video by determining a person image area corresponding to a boundary of the person or a facial image area corresponding to a position of the face of the person, analyzing pixel values ​​corresponding to the person or the position of the face of the person based on the boundary of the person or the facial image area to generate an identification avoidance signal, determining whether the identification avoidance signal indicates that an action has been taken to prevent identification of the person, and in response to the identification avoidance signal not indicating that the action has been taken to prevent identification of the person, identifying the person. In other words, analyzing pixel values ​​of the image or video on and / or within the boundary.

[0011] In some embodiments, refusing to perform identification of the person includes performing at least one of: storing the identification avoidance signal in association with the image or video to indicate that identification of the person is not to be performed, deleting the image or video, archiving the image or video, or demoting the image or video in search results. In some embodiments, the memory stores additional instructions, the additional instructions including: in response to the identification avoidance signal not indicating that the action was taken to prevent identification of the person, performing identification of the person by determining the identity of the person and indexing the image or video in association with the identity. In some embodiments, determining that the action was taken to prevent identification of the person further includes determining that the identification avoidance signal corresponds to an avoidance value that does not meet a threshold, and also in response to the avoidance value not meeting the threshold, performing identification of the person. In some embodiments, the person is a first person, determining that the action was taken to prevent identification of the person further includes determining that the identification avoidance signal corresponds to an avoidance value that meets a threshold, and the memory stores additional instructions, the additional instructions including, in response to the avoidance value meeting the threshold, detecting a second person in the image, and generating the identification avoidance signal for the second person.

[0012] In some embodiments, a machine-readable storage device stores instructions that, when executed, cause a processing device to perform operations including a method according to any embodiment or aspect described herein.

[0013] Various embodiments described below advantageously prevent a person from being identified in an image when the person does not want to be identified. Preventing identification can include preventing the person from being identified, and preventing the image from being indexed in association with the person's identity. In some embodiments, identification can include multiple levels, such as (1) deleting the image; (2) not identifying the person in the image; (3) identifying the person in the image, but only sharing the person's identification with users who have authorization from the person to know the person's identity. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references are used to refer to similar elements.

[0015] Figure 1 A block diagram is shown of an example system for determining whether to identify a person in an image or video in accordance with some embodiments.

[0016] Figure 2 A block diagram is shown of an example computing device that determines whether to identify a person in an image or video in accordance with some embodiments.

[0017] Figures 3A-3D Examples of different actions taken by a person in an image to prevent recognition are shown in accordance with some embodiments.

[0018] Figure 4 Shown is an example digital alteration of a person in an image to hinder recognition in accordance with some embodiments.

[0019] Figure 5 A flow chart illustrating an example method of determining whether a person has taken action to prevent recognition in an image in accordance with some embodiments. DETAILED DESCRIPTION

[0020] Example System

[0021] Figure 1 A block diagram of an example system 100 for determining whether to identify a person in an image or video is shown. The illustrated system 100 includes an identification server 101, user devices 115a, 115n, a second server 120, and a network 105. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the system 100 may include Figure 1 Other servers or devices not shown. Figure 1 In the illustrative drawings and the remaining drawings, a letter following a reference number, e.g., "115a", indicates a reference to the element having that specific reference number. A reference number without a following letter in the text, e.g., "115", indicates a general reference to the embodiment of the element with that reference number.

[0022] The recognition server 101 may include a processor, a memory, and network communication capabilities. In some embodiments, the recognition server 101 is a hardware server. The recognition server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection, such as Ethernet, a coaxial cable, a fiber optic cable, etc., or a wireless connection, such as In some embodiments, the recognition server 101 sends and receives data to and from one or more of the user devices 115a, 115n and the second server 120 via the network 105. The recognition server 101 may include a recognition application 103a and a database 199.

[0023] The recognition application 103a can be a code and routine operable to determine whether to recognize a person in an image or video. In some embodiments, the recognition application 103a can be implemented using hardware including a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In some embodiments, the recognition application 103a can be implemented using a combination of hardware and software.

[0024] Database 199 may store animated objects, message streams, and the like. For example, database 199 may store images and / or videos. In embodiments where people are identified in images and videos, database 199 may store images and videos that are indexed and associated with the identities of people. For example, images may be indexed in association with metadata describing users of a social network, including links to profiles of users of the social network. Database 199 may also store social network data associated with user 125, user preferences of user 125, and the like.

[0025] User device 115 may be a computing device including memory and a hardware processor. For example, a user device may include a desktop computer, a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0026] In the illustrated embodiment, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections, such as or other wireless technologies. The user devices 115a, 115n are accessed by users 125a, 125n, respectively. Figure 1 The user devices 115a, 115n in FIG. 1 are used as examples. Figure 1 Two user devices, 115 a and 115 n , are shown, but the present disclosure applies to system architectures having one or more user devices 115 .

[0027] In some embodiments, user device 115 may be a user device included in a wearable device worn by user 125. For example, user device 115 is included as part of a clip (e.g., a wristband), part of jewelry, or part of a pair of glasses. In another example, user device 115 may be a smartwatch. User 125 may display data associated with recognition application 103 on a display of a device worn by user 125. For example, recognition application 103a may display an image, video, and / or questions about permissions associated with a person identified in an image or video on a display of a smartwatch or smart wristband.

[0028] In some embodiments, the recognition application 103b may be stored on the user device 115a. The recognition application 103 may include a thin client recognition application 103b stored on the user device 115a and a recognition application 103a stored on the recognition server 101. For example, the recognition application 103b stored on the user device 115a may display a message stream including an animated object. The user device 115a may include a camera that captures an image or video, and the recognition application 103b sends the image or video to the recognition server 101 for processing. The recognition application 103a stored on the recognition server 101 may receive the image or video and determine whether the recognition avoidance signal associated with the image or video indicates that an action has been taken to prevent the recognition of a person.

[0029] The second server 120 may include a processor, a memory, and a network communication capability. The second server 120 may access the network 105 via a signal line 109. The second server 120 may include an application that uses information about people identified in images and videos by the recognition application 103a. For example, the second server 120 may include a social networking application that receives recognition information about people in images and videos from the recognition application 103a and associates the recognition information with users of the social networking application. In another example, the second server 120 may include a photo editing application for digitally blocking images or videos, for example, by blurring or pixelating the faces of people in the images. In yet another example, the second server 120 may include a photo application, a calendar application, an event organizer application, and the like.

[0030] In the illustrated embodiment, the entities of system 100 are communicatively coupled via network 105. Network 105 can be of a conventional type, either wired or wireless, and can have a number of different configurations, including a star configuration, a token ring configuration, or other configurations. In addition, network 105 can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and / or other interconnected data paths over which multiple devices can communicate. In some embodiments, network 105 can be a peer-to-peer network. Network 105 can also be coupled to or include a portion of a telecommunications network for transmitting data in a variety of different communication protocols. In some embodiments, network 105 includes Communication Network, Wireless Local Area Network (WLAN) computer communications specified by IEEE 902.11, or cellular communications networks for sending and receiving data, including via Short Message Service (SMS), Multimedia Message Service (MMS), Hypertext Transfer Protocol (HTTP), direct data connections, email, etc. Figure 1One network 105 is shown coupled to the user device 115 and the identification server 101, but in practice one or more networks 105 may be coupled to these entities.

[0031] In the case where the systems and methods discussed herein can collect or use personal information about a user (e.g., user data, information about a user's social network, a user's location, a user's biometric information, a user's activities and / or demographic information, storage and analysis of images or videos by the recognition server 101 or the recognition application 103, etc.), the user is provided with an opportunity to control whether to collect personal information, whether to store personal information, whether to use personal information, whether to analyze images or videos, and how to collect, store and use information about the user. That is, the systems and methods discussed herein can only collect, store and / or use user personal information when explicit authorization is received from the relevant user to collect, store and / or use user personal information. For example, the user is provided with control over whether a program or feature collects user information about the particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is collected to allow control over the collection of information related to the user, so as to provide permission or authorization as to whether to collect information and as to which parts of the information to be collected. For example, one or more such control options may be provided to the user via a communication network. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the user's identifying information can be processed, for example, anonymized, so that personally identifiable information cannot be determined from the video. As another example, the user's geographic location can be summarized into a larger area so that the user's specific location cannot be determined.

[0032] Example computing device

[0033] Figure 2A block diagram of an example computing device 200 that determines whether to recognize a person in an image or video is shown. The computing device 200 may be the recognition server 101, the user device 115, or a combination of the recognition server 101 and the user device 115. The computing device 200 may include a processor 235, a memory 237, a communication unit 239, a display 241, and a storage device 247. Depending on the type of computing device 200, additional components may be present, or some of the previous components may be omitted. For example, if the computing device 200 is the recognition server 101, the computing device 200 may not include the display 241. The recognition application 103 may be stored in the memory 237. In an embodiment where the computing device 200 is a wearable device, the computing device 200 may not include the storage device 247. In some embodiments, the computing device 200 may include other components not listed here, such as a battery, etc. The components of the computing device 200 may be communicatively coupled by the bus 220.

[0034] Processor 235 includes an arithmetic logic unit, a microprocessor, a general purpose controller, or some other processor array to perform calculations and provide instructions to the display device. Processor 235 processes data and may include a variety of computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture that implements a combination of instruction sets. Although Figure 2 A single processor 235 is included, but multiple processors 235 may be included. Other processors, operating systems, sensors, displays, and physical configurations may be part of computing device 200. Processor 235 is coupled to bus 220 for communication with other components via signal line 222.

[0035] Memory 237 stores instructions and / or data that can be executed by processor 235. Instructions can include codes for performing the technology described herein. Memory 237 can be a dynamic random access memory (DRAM) device, a static RAM, or some other memory devices. In some embodiments, memory 237 also includes non-volatile memory, such as a static random access memory (SRAM) device or flash memory, or similar permanent storage devices and media, including hard drives, compact disc read-only memory (CD-ROM) devices, DVD-ROM devices, DVD-RAM devices, DVD-RW devices, flash memory devices, or some other large-capacity storage devices for storing information on a more permanent basis. Memory 237 includes codes and routines that can be operated to perform identification applications 103, which are described in more detail below. Memory 237 is coupled to bus 220 for communicating with other components via signal line 224.

[0036] The communication unit 239 transmits data to and receives data from at least one of the user device 115 and the recognition server 101, depending on where the recognition application 103 may be stored. In some embodiments, the communication unit 239 includes a port for direct physical connection to the network 105 or another communication channel. For example, depending on where the recognition application 103 may be stored, the communication unit 239 includes a universal serial bus (USB), secure digital (SD), category 5 cable (CAT-5), or similar port for wired communication with the user device 115 or the recognition server 101. In some embodiments, the communication unit 239 includes a wireless transceiver for using one or more wireless communication methods, including IEEE 802.11, IEEE 802.16, Or another suitable wireless communication method to exchange data with the user device 115, the identification server 101 or other communication channels. The communication unit 239 is coupled to the bus 220 for communicating with other components via the signal line 226.

[0037] In some embodiments, the communication unit 239 includes a cellular communication transceiver for sending and receiving data over a cellular communication network, including via a short message service (SMS), a multimedia message service (MMS), a hypertext transfer protocol (HTTP), a direct data connection, email, or another suitable type of electronic communication. In some embodiments, the communication unit 239 includes a wired port and a wireless transceiver. The communication unit 239 also provides other conventional connections to the network 105 for distributing files and / or media objects using standard network protocols, including but not limited to the User Datagram Protocol (UDP), TCP / IP, HTTP, HTTP Security (HTTPS), Simple Mail Transfer Protocol (SMTP), SPDY, Quick UDP Internet Connection (QUIC), etc.

[0038] The display 241 may include hardware operable to display graphical data received from the recognition application 103. For example, the display 241 may render graphics to display an image or questions about permissions associated with an image. The display 241 is coupled to the bus 220 for communication with other components via signal line 228.

[0039] Storage device 247 may be a non-transitory computer-readable storage medium that stores data that provides the functionality described herein. In an embodiment where computing device 200 is recognition server 101, storage device 247 may include Figure 1199 in the database 199. Storage device 247 can be a DRAM device, an SRAM device, a flash memory, or some other memory device. In some embodiments, storage device 247 also includes non-volatile memory or similar permanent storage devices and media, including hard drives, CD-ROM devices, DVD-ROM devices, DVD-RAM devices, DVD-RW devices, flash memory devices, or some other large-capacity storage devices for storing information on a permanent basis. Storage device 247 is coupled to bus 220 for communicating with other components via signal line 230.

[0040] The recognition application 103 may include a detector 202 , an analyzer 204 , a recognition module 206 , and a user interface module 208 .

[0041] Detector 202 detects a person in an image or video. In some embodiments, detector 202 includes a set of instructions that can be executed by processor 235 to detect a person. In some embodiments, detector 202 is stored in memory 237 of computing device 200 and can be accessed and executed by processor 235.

[0042] In some embodiments, the detector 202 uses a computer vision algorithm to process the image or video. The computer vision algorithm may include a person detector and a face detector for detecting people and / or faces. The person detector and the face detector may use a person template and a face template, which respectively describe data associated with a person and a face. For example, the person template and the face template may correspond to object priors describing different types of people and faces. The face template and the person template may be based on expected points corresponding to an expected face or person. For example, the person template may include expected locations of a head, shoulders, arms, chest, legs, feet, etc. and distances between locations. The detector 202 may also determine whether the location of a point on a person corresponds to a person's posture. For example, the point may correspond to a person standing, sitting, running, etc.

[0043] The detector 202 may determine a human image region corresponding to a boundary of a person or a facial image region corresponding to a location of a person's face. For example, the detector 202 may generate a bounding box surrounding a person's face and / or body. In some embodiments, the detector 202 may detect a face based on facial points and distances between facial points. For example, the detector 202 may identify eyes, eyebrows, noses, and mouths, as well as hair, and if facial points correspond to expected distances between facial points, then confirm that these facial points correspond to a face. The detector 202 may determine a face based on a convex hull by generating a shape (e.g., a convex polygon) from the facial points and confirming that the shape corresponds to an expected shape of a face. In some embodiments where an image includes a pixel mask instead of a face, the detector 202 may identify the pixel mask and determine that the pixel mask corresponds to a face. In some embodiments, the detector 202 may use clustering to identify a person. In an example where the detector 202 is processing a video, the detector 202 may process each image frame of the video to detect a face and / or a person.

[0044] The analyzer 204 generates the identification avoidance signal. In some embodiments, the analyzer 204 includes a set of instructions that can be executed by the processor 235 to generate the identification avoidance signal. In some embodiments, the analyzer 204 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0045] Analyzer 204 processes each detected person and / or face in an image or video to determine an identification avoidance signal that indicates whether the person appears to not wish to be photographed, or whether the photograph has been edited to mask or alter the person's identity.

[0046] The analyzer 204 may generate a recognition avoidance signal by analyzing pixel values ​​corresponding to the location of a person or a person's face based on the boundaries of the person or the facial image region. The analyzer 204 may determine whether the recognition avoidance signal indicates that an action has been taken to prevent recognition of the person. For example, the action may include a person in the image using an object to block at least a portion of the person's face, where the object is a body part such as a hand, clothing, etc. Figures 3A-3D Examples of different actions taken by a person in an image to prevent recognition according to some embodiments are shown.Alternatively or additionally, the action may include digitally obscuring at least a portion of the person's face.

[0047] Go to Figure 3A , image 300 includes a woman raising her hand, which may indicate that the woman does not want to be identified in the image. Because the woman's hand does not completely cover her face, it is unclear whether she is trying to cover her face, moving her hand to grab something, or waving to someone.

[0048] In some embodiments, the analyzer 204 may generate a recognition avoidance signal based on the boundaries of the person or facial image region by analyzing pixel values ​​corresponding to the location of the person or the person's face. The analyzer 204 may classify the person's expression based on template expressions, such as happy, sad, angry, worried, etc. The template expression may be based on whether the person's mouth is upturned, straight, drooping, or open; the person's eyes are open, closed, or clenched; the person's eyebrows are raised or at an angle; the person's wrinkles are visible, etc. For example, in Figure 3A , analyzer 204 may determine that the woman does not want to be recognized because a portion of her hand overlaps her face and she has a worried expression as indicated by her mouth forming an O, her eyes open, and her eyebrows raised. Conversely, if the hand is in the same position but the woman is smiling, analyzer 204 may determine that the woman is fine with being recognized in the image.

[0049] In some embodiments, the analyzer 204 scores the recognition avoidance signal by determining an avoidance value corresponding to the recognition avoidance signal. In some embodiments, if the avoidance value meets a threshold, the analyzer 204 determines that an action has been taken to prevent the recognition of the person. Other schemes are also possible, such as if the avoidance value fails to meet the threshold, it is determined that an action has been taken to prevent the recognition of the person. The analyzer 204 can determine the avoidance value based on the percentage of the face that is occluded. For example, if more than 50% of the face is occluded, the analyzer 204 can determine that the person has taken an action to prevent the recognition of the person. In another example, because the image of the person includes an occluded face and the person is in a scrunched-up position, the analyzer 204 determines that the avoidance value is 90 out of a ratio of 100.

[0050] Go to Figure 3B , image 325 includes a woman showing an indication that she does not want to be identified in the image. In this example, a larger portion of the woman's face is obscured by her hand, which is a stronger indication that the woman does not want to be identified in the image. In some embodiments, analyzer 204 may process a series of images associated with the event and generate an identification avoidance signal based on the series of images. For example, Figure 3A and Figure 3B may be part of the same event. Because both images of the female include at least a portion of her face obscured by her hand, analyzer 204 may determine that the female indicates that the female does not want to be identified in the image. Analyzer 204 may generate an avoidance value based on the series of images. For example, analyzer 204 may be based on Figure 3A and 3BTo determine that the woman does not want to be recognized because the average (or mean) of the avoidance values ​​of the two images corresponds to an avoidance value that meets (or exceeds, or is below, etc.) a threshold. In another example, the analyzer 204 may be based on Figure 3A , Figure 3B and one A series of images is used to determine that in the series of images, the female does not place her hands on her face, indicating that the female does not object to being identified.

[0051] Go to Figure 3C , image 350 includes a male whose hand almost completely obscures his face. This is an example of a person who does not want to be recognized. In some embodiments, no other actions can offset this direct indication that the person does not want to be recognized. For this example, analyzer 204 generates a recognition avoidance signal indicating that the person does not want to be recognized.

[0052] Go to Figure 3D , image 375 includes a person wearing a hat that is pulled down over the person's face and with a hand in front of the bottom of the person's face. This is another example of a person who does not want to be recognized. For this example, analyzer 204 generates a recognition avoidance signal indicating that the person does not want to be recognized.

[0053] The analyzer 204 may determine that the recognition avoidance signal indicates that an action was taken to prevent recognition of the person, wherein the action includes at least a portion of the person's face being digitally obscured. Digitally obscuring the face may include blurring, pixelating, or masking the face. The user may digitally obscure the face using editing software that is part of the camera or photo editing software on the user's phone. The user may also digitally obscure the face using third-party software, such as software provided by the second server 120.

[0054] Go to Figure 4 , an example image 400 including digital alterations of people to prevent recognition is shown. In this example, the two people on the right are shown as having pixelated faces. The user may have post-processed the image to add pixelation to those faces. Based on the user digitally blocking the two people on the right, the analyzer 204 can determine that an action has been taken to prevent the two people on the right from being recognized. As a result, the analyzer 204 can generate a recognition avoidance signal for each of those two people to indicate that they should not be recognized. As discussed in more detail below, the recognition module 206 can determine not to perform recognition of those two people. However, the recognition module 206 can determine to perform recognition of the three people in the left portion of the image 400 because the faces of those three people are not blocked.

[0055] In some embodiments, the analyzer 204 determines the identity of sensitive objects within the image or video and generates an identification avoidance signal based on the identity of the sensitive objects. Sensitive objects may include drug paraphernalia, illegal objects, nudity, etc. For example, the analyzer 204 may determine that an image of a person with a hand covering 10% of the person's face combined with a sensitive object in the background results in the generation of an identification avoidance signal indicating that the person should not be identified.

[0056] In some embodiments, the analyzer 204 identifies the type of action and determines based on the type of action whether the recognition avoidance signal indicates that an action was taken to prevent recognition of the person. For example, the type of action may include a person raising a hand to cover the person's face or pinching the brim of the person's hat. The analyzer 204 may determine that raising a hand upward indicates that the person does not want to be recognized, and pinching the brim of the person's hat indicates that the image or video may be shared with a group of users in the social network (e.g., users to whom the person is connected in the social network).

[0057] In some embodiments, the analyzer 204 can implement machine learning, which can generate and / or enhance a classifier for generating a recognition avoidance signal. In some embodiments, machine learning can be implemented in one or more components of the recognition application 103, such as using a trained model. Without using user information, a trained model can be trained using synthetic data, such as data automatically generated by a computer.

[0058] In some embodiments, the trained model may be trained, for example, based on training data for which permission has been explicitly obtained from the user to train using user data. The training data may include any data that is permitted for training, such as synthetic or computer-generated data, data permitted for training, and the like, such as videos and corresponding metadata. The training data may include images and videos that include actions that have been taken to prevent identification of a person. The images and videos may come from internally generated videos, such as those stored on the database 199 of the recognition server 101, or from images and videos received from the second server 120. For example, the second server 120 may be a media server that provides metadata to images and videos indicating that an action has been taken to prevent identification of a person.

[0059] In some embodiments, a trained model can be obtained based on supervised learning, for example, based on training data including images and videos and corresponding metadata. For example, a trained model can include a model form or structure (e.g., describing the number and organization of multiple nodes to layers of a neural network, with associated weights). In some embodiments, a trained model can be trained so that the analyzer 204 applies the trained model to generate an identification avoidance signal, which indicates that an action has been taken to prevent the identification of a person. In this way, a signal is extracted from the content of the image or video that controls the processing and further analysis of the image or video itself, such as identifying the person shown in the image or video. This is advantageous because the processing and further analysis are controlled without requiring additional information other than the video or image itself. In addition, when shooting an image or video, the person shown in the image of the video can control whether subsequent processing and further analysis of the image or video should be allowed, so that, for example, the identification of the person is prevented and not performed, thereby increasing the confidentiality and privacy of the person shown in the image or video even when the image or video is shared in a social network.

[0060] In some embodiments, users help create training data by providing user input. Users may be asked to identify images and videos with actions taken to prevent identification of a person. As a result of the user input, the training data may have accurate identification of the types of actions taken to prevent identification of a person.

[0061] Based on the training data, the analyzer 204 can generate a trained model that can generate a recognition avoidance signal based on the image and video, which indicates that an action was taken to prevent the recognition of a person. The trained model can be a temporal action localization model. In various embodiments, the analyzer 204 can use a Bayesian classifier, a support vector machine, a neural network, or other learning techniques to generate a trained model.

[0062] In some embodiments, the trained model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or divides the input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of the processing from each tile), a sequence-to-sequence neural network (e.g., a network that takes sequence data as input and produces a sequence of results as output, such as words in a sentence, frames in a video, etc.), etc. The model form or structure may specify the connections between various nodes and the organization of nodes to layers. For example, the nodes of the first layer (e.g., the input layer) may receive data as input data or application data. For example, when the trained model is used for image analysis, such data may include, for example, one or more pixels of each node. Subsequent intermediate layers may receive the output of the nodes of the previous layer as input according to the connections specified in the model form or structure. These layers may also be referred to as hidden layers. The last layer (e.g., the output layer) produces the output of the machine learning application. For example, depending on the particular trained model, the output may be a set of labels for the image, a representation of the image that permits the image to be compared to other images (e.g., a feature vector for the image), an output sentence in response to an input sentence, one or more categories of input data, etc. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0063] In various embodiments, the trained model may include multiple nodes that are arranged into layers according to the model structure or form. In some embodiments, the node may be a computing node without memory, for example, configured to process the input of a unit to produce the output of a unit. The calculation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to produce a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, for example, multiple processor cores of a multi-core processor, a separate processing unit using a general processing unit, or a dedicated neural circuit may be used to perform calculations of multiple nodes in parallel. In some embodiments, the node may include a memory, for example, capable of storing one or more earlier inputs and using one or more earlier inputs when processing subsequent inputs. For example, a node with a memory may include a long short-term memory (LSTM) node. The LSTM node may use a memory to maintain a "state" that permits the node to act like a finite state machine (FSM). Models with such nodes may be useful when processing sequence data, such as words in a sentence or paragraph, frames in a video, speech or other audio, etc.

[0064] In some embodiments, the trained model may include embeddings or weights for each node. For example, the trained model may be initialized as a plurality of nodes organized into layers specified by the model form or structure. Upon initialization, corresponding weights may be applied to connections between each pair of nodes connected in the model form, such as nodes in successive layers of a neural network. For example, corresponding weights may be randomly assigned or initialized to default values. The trained model may then be trained, for example, using data to produce results.

[0065] The recognition module 206 performs recognition of people in the image or video. In some embodiments, the recognition module 206 includes a set of instructions that can be executed by the processor 235 to perform recognition. In some embodiments, the recognition module 206 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0066] In some embodiments, recognition module 206 performs recognition of the person in response to the recognition avoidance signal indicating that action has not been taken to prevent recognition of the person and / or the recognition avoidance signal corresponds to an avoidance value below a threshold.

[0067] In some embodiments, the recognition module 206 performs recognition of a person by extracting a recognition template. If it is difficult to recognize a person in an image, for example, because a portion of the person's face is blocked, the lighting is too low to be recognized, the person's face is in profile, etc., the recognition module 206 can use other images from the same event or other image frames in the same video to recognize the person. In some embodiments, the recognition module 206 can determine the identity of the person based on identifiable information, such as clothing patterns, logos, personal items, unique hairstyles, jewelry, tattoos, postures, etc. In some embodiments, the recognition module 206 can use a machine learning classification system to recognize a person. In some embodiments, the recognition module 206 can use machine learning to recognize a person. The above reference generates a recognition avoidance signal to describe the processing using machine learning, and will not be repeated here with reference to identifying a person. The recognition module 206 can index images associated with the identity of the person.

[0068] If the recognition avoidance signal indicates that an action was taken to prevent recognition of the person and / or the recognition avoidance signal corresponds to an avoidance value that satisfies a threshold, the recognition module 206 may determine that the person is not recognized. If there are other people in the image, the recognition module 206 may instruct the detector 202 to detect another person in the image until all people in the image have been processed.

[0069] Determining not to identify a person in an image or video can take a number of forms. For example, in some embodiments, determining not to identify a person in an image can include deleting the image or video from storage device 247, archiving the image or video, or demoting the image or video in search results. In other embodiments, determining not to identify a person can include indexing images that are not associated with the identity of the person.

[0070] In some embodiments, a person may specify permissions associated with an image, and the recognition module 206 may recognize the person based on the permissions. For example, a user may provide permissions that allow images of the user to be recognized using people associated with the user in a social network (e.g., connected to, followed, friends, etc.), but not the user in public images. As a result, the recognition module 206 may determine that the recognition avoidance signal indicates that an action has been taken to prevent recognition of the person and to perform recognition of the person. The recognition module 206 may determine the permissions associated with the person, and recognize the person in the image based on the permissions associated with the person. For example, the recognition module 206 may determine that the user does not want to be recognized in any image based on the permissions. In another example, the recognition module 206 may determine that the user may be recognized in images and videos shared with friends rather than the public based on the permissions, and as a result, recognition is provided when the images and videos are shared with friends.

[0071] In some embodiments, recognition module 206 can instruct user interface module 208 to generate graphical data for displaying a user interface that requires the person being recognized to confirm that the person does not want to be recognized. For example, recognition module 206 can perform recognition of the person, determine that an image is to be deleted based on their permission, and instruct user interface module 208 to require confirmation before deleting the image.

[0072] The user interface module 208 generates graphical data for displaying a user interface. In some embodiments, the recognition module 206 includes a set of instructions that can be executed by the processor 235 to generate graphical data. In some embodiments, the user interface module 208 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0073] In some embodiments, user interface module 208 generates graphical data for displaying a user interface for a user to establish permissions. Permissions relate to how an image or video should be processed. For example, a user may provide preferences in which the user has indicated that the user does not want to be identified in an image or video, resulting in the user's identity being associated with the image or video for people related to the user in the social network, but not for a version of the image or video shared with people not related to the user in the social network.

[0074] In some embodiments, the user interface module 208 receives instructions from the analyzer 204 or the recognition module 206 to generate a user interface including an image or video. The user interface may also include a request for the user to confirm an action. For example, the user interface may include a request for the user to confirm that the user does or does not want to be identified in the image or video, that the user wants to share the image or video with a group of people, that the user wants to delete the image or video, archive the image or video, or downgrade the image or video in the search results, etc.

[0075] Example Method

[0076] Figure 5 A flow chart of an example method 500 of determining whether a person has taken action to prevent recognition in an image according to some embodiments is shown. The method 500 is performed by a recognition application 103 stored on a computing device 200 (such as a user device 115, a recognition server 101, or part of a user device 115 and a recognition server 101).

[0077] At block 502, a person is detected in an image or video by determining a person image region corresponding to a boundary of the person or a facial image region corresponding to a location of a face of the person. At block 504, pixel values ​​corresponding to a location of the person or the face of the person are analyzed based on the boundary of the person or the facial image region to generate a recognition avoidance signal.

[0078] At block 506, a determination is made as to whether the identification avoidance signal indicates that an action was taken to prevent identification of the person. If the identification avoidance signal does not indicate that an action was taken to prevent identification of the person, then at block 508, identification of the person is performed. If the identification avoidance signal indicates that an action was taken to prevent identification of the person, then at block 510, a determination is made as to whether the identification avoidance signal corresponds to an avoidance value that satisfies a threshold. If the avoidance value does not satisfy the threshold, the method 500 moves to block 508, and identification of the person is performed. If the avoidance value satisfies the threshold, the method 500 proceeds to block 502, and another person in the image or video is detected, and the method 500 is repeated until all persons in the image or video have been analyzed.

[0079] In the above description, for the purpose of explanation, many specific details are set forth in order to provide a comprehensive understanding of the specification. However, it is obvious to those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid blocking the description. For example, the above embodiments may be described primarily with reference to user interfaces and specific hardware. However, the embodiments may be applied to any type of computing device that can receive data and commands, and any peripheral device that provides services.

[0080] References in the specification to "some embodiments" or "some examples" mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one embodiment described. The phrase "in some embodiments" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0081] Some parts of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing use the essence of their work to be most effectively communicated to other persons skilled in the art. An algorithm is generally considered here to be a self-consistent sequence of steps leading to a desired result. The steps are those steps requiring physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic data that can be stored, transmitted, combined, compared, and otherwise manipulated. Mainly for reasons of common use, it has been shown that sometimes it is convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0082] It should be remembered, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, it will be apparent from the following discussion that discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like throughout the description refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0083] Embodiments of the specification may also relate to a processor for performing one or more steps of the method described above. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0084] The specification may take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments containing hardware and software elements. In some embodiments, the specification is implemented with software, which includes but is not limited to firmware, resident software, microcode, etc.

[0085] Furthermore, the description may take the form of a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer usable or computer readable medium may be any means that can contain, store, communicate, propagate, or transport the program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0086] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory employed during actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

[0087] In the situations discussed above where the system collects or uses personal information, the system provides the user with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, the user's preferences, or the user's current location), or to control whether and / or how content that may be more relevant to the user is received from a server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information cannot be determined for the user, or the geographic location of the user for which location information is obtained may be summarized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control how information about the user is collected and how the information is used by the server.

Claims

1. A computer-implemented method for preventing identification of a person, characterized in that The method comprises: Detecting a person in an image or video by determining a person image region corresponding to a boundary of the person or a face image region corresponding to a position of the face of the person; analyzing pixel values ​​corresponding to the location of the person or the face of the person based on the boundary of the person or the facial image region to generate an identification avoidance signal, wherein the identification avoidance signal indicates whether the person does not appear to wish to be photographed or whether the image or video has been edited to mask or change the identity of the person; determining whether the recognition avoidance signal indicates that an action was taken to prevent recognition of the person, wherein the action includes the person using an object to obscure at least a portion of the person's face or at least a portion of the person's face being digitally obscured; and Responsive to the identification avoidance signal indicating that the action was taken to prevent identification of the person, identification of the person is denied.

2. The method according to claim 1, characterized in that Denying identification of the person includes at least one of storing the identification avoidance signal in association with the image or video to indicate that identification of the person is not to be performed, deleting the image or video, archiving the image or video, or downgrading the image or video in search results.

3. The method according to claim 1, characterized in that Further including: In response to the identification avoidance signal not indicating that the action was taken to prevent identification of the person, identifying the person is performed by determining an identity of the person and indexing the image or the video in association with the identity.

4. The method according to claim 1, characterized in that: Determining that the action was taken to prevent recognition of the person further comprises determining that the recognition avoidance signal corresponds to an avoidance value that does not meet a threshold, wherein the avoidance value is determined based on a percentage of the face that is occluded, or the face is occluded and the person is in a huddled position; and Also in response to the avoidance value not satisfying the threshold, identifying the person is performed.

5. The method according to claim 1, characterized in that: The person in question is the first person; Determining that the action was taken to prevent identification of the person further comprises determining that the identification avoidance signal corresponds to an avoidance value that satisfies a threshold; and The method further includes, in response to the avoidance value satisfying the threshold, detecting a second person in the image and generating the identification avoidance signal of the second person.

6. The method according to claim 1, characterized in that The object is a body part of the person.

7. The method according to claim 1, characterized in that The identification of the person is performed based on extracting an identification template.

8. The method according to claim 1, characterized in that The image is a first image, the video is a first video, and further comprises: identifying one or more additional images or one or more additional videos associated with the event; and A determination is made not to recognize the person within the one or more additional images or the one or more additional videos associated with the event.

9. The method according to claim 1, characterized in that: Determining that the identification avoidance signal indicates that the action was taken to prevent identification of the person includes determining a type of action, and further includes: The person is identified by identifying the person and sharing the identification of the person with a group of users in a social network based on the action type and not sharing the identification of the person with any user who is not part of the group of users in the social network.

10. A non-transitory computer-readable medium having instructions stored thereon, characterized in that: When the instructions are executed by one or more computers, the instructions cause the one or more computers to perform operations including: Detecting a person in an image or video by determining a person image region corresponding to a boundary of the person or a face image region corresponding to a position of the face of the person; analyzing pixel values ​​corresponding to the location of the person or the face of the person based on the boundary of the person or the facial image region to generate an identification avoidance signal, wherein the identification avoidance signal indicates whether the person does not appear to wish to be photographed or whether the image or video has been edited to mask or change the identity of the person; determining whether the recognition avoidance signal indicates that an action was taken to prevent recognition of the person, wherein the action includes the person using an object to obscure at least a portion of the person's face or at least a portion of the person's face being digitally obscured; and Responsive to the identification avoidance signal not indicating that the action was taken to prevent identification of the person, identifying the person is performed.

11. The computer-readable medium of claim 10, wherein: Performing the recognition of the person includes determining the identity of the person and indexing the image or the video in association with the identity.

12. The computer-readable medium of claim 10, wherein: The operations further include: In response to the recognition avoidance signal indicating that the action has been taken to prevent recognition of the person, at least one of the following is performed: storing the recognition avoidance signal in association with the image or video to indicate that no recognition of the person is to be performed, deleting the image or video, archiving the image or video, or downgrading the image or video in search results.

13. The computer-readable medium of claim 10, wherein: The operations further include: In response to the recognition avoidance signal indicating that the action has been taken to prevent recognition of the person, determining whether the recognition avoidance signal corresponds to an avoidance value that satisfies a threshold, wherein the avoidance value is determined based on a percentage of the face that is occluded, or the face is occluded and the person is in a shrunken position; and In response to the avoidance value failing to satisfy the threshold, identification of the person is performed.

14. The computer-readable medium of claim 10, wherein: The object is a body part of the person.

15. The computer-readable medium of claim 10, wherein: The operations further include: determining permissions associated with the person; and The identifying of the person is performed based on the permission associated with the person.

16. A system for preventing identification of a person, characterized in that The system comprises: one or more processors; and A memory coupled to the one or more processors storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: Detecting a person in an image or video by determining a person image region corresponding to a boundary of the person or a face image region corresponding to a position of the face of the person; analyzing pixel values ​​corresponding to the location of the person or the face of the person based on the boundary of the person or the facial image region to generate an identification avoidance signal, wherein the identification avoidance signal indicates whether the person does not appear to wish to be photographed or whether the image or video has been edited to mask or change the identity of the person; determining whether the recognition avoidance signal indicates that an action was taken to prevent recognition of the person, wherein the action includes the person using an object to obscure at least a portion of the person's face or at least a portion of the person's face being digitally obscured; and Responsive to the identification avoidance signal indicating that the action was taken to prevent identification of the person, identification of the person is denied.

17. The system according to claim 16, characterized in that Denying identification of the person includes at least one of storing the identification avoidance signal in association with the image or video to indicate that identification of the person is not to be performed, deleting the image or video, archiving the image or video, or downgrading the image or video in search results.

18. The system according to claim 16, characterized in that The memory stores additional instructions, the additional instructions comprising: In response to the identification avoidance signal not indicating that the action was taken to prevent identification of the person, identifying the person is performed by determining an identity of the person and indexing the image or the video in association with the identity.

19. The system according to claim 16, characterized in that: Determining that the action was taken to prevent recognition of the person further comprises determining that the recognition avoidance signal corresponds to an avoidance value that does not meet a threshold, wherein the avoidance value is determined based on a percentage of the face that is occluded, or the face is occluded and the person is in a huddled position; and Also in response to the avoidance value not satisfying the threshold, identifying the person is performed.

20. The system according to claim 16, characterized in that: The person in question is the first person; Determining that the action was taken to prevent identification of the person further comprises determining that the identification avoidance signal corresponds to an avoidance value that satisfies a threshold; as well as The memory stores additional instructions including, in response to the avoidance value satisfying the threshold, detecting a second person in the image and generating the identification avoidance signal of the second person.

Images