Method and apparatus for reducing false positives in facial recognition

Abstract

An apparatus may include a memory, a communication interface, and a processor. The processor is configured to receive image data and first context data associated with the image data from an imaging device. The image data includes at least one image of a field of view. The processor is also configured to receive second context data associated with a user of a user device. The second context data is generated in response to the user device receiving a wireless signal transmitted by an antenna operably coupled to the imaging device. The processor is also configured to determine a potential presence of a user in the image data based on comparing the first context data with the second context data, analyze the image data to identify the user in the image data, and send the image data to the user.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and is a continuation of U.S. patent application Ser. No. 16 / 118,780, filed on Aug. 31, 2018, entitled “Methods and Apparatus for Reducing False Positives in Facial Recognition,” which is incorporated herein by reference in its entirety. Background Art

[0003] Embodiments described herein relate generally to facial recognition and video analytics, and more particularly to apparatus and methods for reducing false positives in facial recognition applications.

[0004] The increase in the availability and capabilities of electronic devices (such as cameras, tablets, smart phones, etc.) has allowed some people to take pictures of their experiences and / or capture videos of their experiences. For example, the inclusion and improvement of cameras in smart phones, tablets, and / or other similar devices has led to an increase in those devices being used to take pictures (e.g., photo data, image data, etc.) and videos (e.g., video streaming data). While it has become easier for some people to take pictures and / or videos of their experiences, in some cases, it remains a challenge to include desired parties (including people who would otherwise take pictures or videos). Moreover, generally speaking, the person must remember and / or have the opportunity to take the pictures and / or video, otherwise the opportunity may be lost.

[0005] In some cases, venues and / or events (such as sporting events, concerts, assemblies, graduations, etc.) have cameras that can capture pictures and / or video of attendees. However, in some cases, analyzing, parsing, and / or otherwise making available pictures and / or video streams may use relatively large amounts of resources, may be inaccurate, and / or may fail to provide associated contextual data, etc. More specifically, in some cases, it may be difficult to verify whether a particular person detected in a picture is actually at the location captured in the picture due to false positives obtained from using facial recognition alone to identify a person in the picture.

[0006] Therefore, there is a need for an improved apparatus and method for using context and location data to reduce false alarms, such as at public events. Summary of the invention

[0007] In some embodiments, an apparatus may include a memory, a communication interface in communication with the memory and configured to communicate via a network, and a processor in communication with the memory and the communication interface. The processor is configured to receive image data from an imaging device via the network and the communication interface and to receive first context data associated with the image data. The image data includes at least one image of a field of view. The processor is also configured to receive second context data associated with a user of a user device via the network and the communication interface. The second context data is generated in response to the user device receiving a wireless signal, which (1) is sent by an antenna operably coupled to the imaging device in response to the imaging device generating the image data and (2) covers at least a portion of the field of view. The processor is also configured to determine the potential presence of a user in the image data based on comparing the first context data with the second context data, analyze the image data based on at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data, and send the image data to the user.

[0008] In some embodiments, an apparatus may include: an imaging device configured to generate image data representing a field of view; a communication interface that communicates with the imaging device and is configured to communicate via a network; and a processor that communicates with the imaging device and the communication interface. In response to the imaging device generating the image data, the processor is configured to control the communication interface to broadcast a wireless signal covering at least a portion of the field of view. The wireless signal includes at least a portion of the first context data associated with the image data. The processor is also configured to control the communication interface to send the image data and the first context data associated with the image data to a server via the network. In response to receiving the image data, the server determines the potential presence of the user in the image data based on comparing the first context data with the second context data associated with the user of the user device and generated in response to the user device receiving the wireless signal, and analyzes the image data based on at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data.

[0009] In some embodiments, an apparatus may include a communication interface and a processor in communication with the communication interface. The processor is configured to control the communication interface to receive a wireless signal from a first device, the wireless signal including first context data associated with image data generated by an imaging device associated with the first device. The first device is also configured to send the image data and the first context data to a server. The processor is also configured to control the communication interface to send second context data to the server in response to receiving the first context data, so that in response to receiving the second context data, the server determines the potential presence of a user in the image data based on comparing the first context data with the second context data, and analyzes the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data. The processor is configured to receive image data from the server in response to the server identifying the user in the image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a schematic diagram of a recognition system according to an embodiment.

[0011] Figure 1B is a schematic diagram of an identification system according to another embodiment.

[0012] Figure 2 is included in Figure 1A and Figure 1B Schematic diagram of the host device in the identification system.

[0013] Figure 3 is included in Figure 1A and Figure 1B Schematic diagram of a client device in an identification system.

[0014] Figure 4 is a flowchart illustrating a method of using a host device in a video recognition system according to an embodiment.

[0015] Figure 5 is a flowchart illustrating a method of using a client device including an imaging device in a video recognition system according to an embodiment.

[0016] Figure 6 is a flow chart illustrating a method of using a client device in a video recognition system according to an embodiment. DETAILED DESCRIPTION

[0017] In some embodiments, an apparatus may include a memory, a communication interface in communication with the memory and configured to communicate via a network, and a processor in communication with the memory and the communication interface. The processor is configured to receive image data from an imaging device via the network and the communication interface and first context data associated with the image data. The image data includes at least one image of a field of view. The processor is also configured to receive second context data associated with a user of a user device via the network and the communication interface. The second context data is generated in response to the user device receiving a wireless signal, which (1) is sent by an antenna operably coupled to the imaging device in response to the imaging device generating the image data, and (2) covers at least a portion of the field of view. The processor is also configured to determine the potential presence of the user in the image data based on comparing the first context data with the second context data, analyze the image data based on at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data, and send the image data to the user.

[0018] Embodiments described herein relate to detecting and / or confirming the presence of a user in media based on facial recognition data and contextual data associated with the media. In some embodiments, a method of image analysis includes receiving image data generated by a first user device (also referred to as a first client device) including an imaging device (e.g., configured to capture an image of a field of view) at a host device (also referred to as a server) via a network. The host device also receives first contextual data associated with the image data. In some embodiments, the first contextual data is also transmitted by the first user device. In some other embodiments, the first contextual data is transmitted by another device (e.g., an antenna, a server, a communication interface, and / or any other suitable device) operably coupled to the first user device.

[0019] The host device is also configured to receive, via the network, second context data associated with a user of a second user device (also referred to as a second client device). In some embodiments, the second user device generates the second context data in response to receiving a wireless signal from the first user device, and the first user device generates the wireless signal in response to the imaging device generating image data. The first user device is configured to broadcast the wireless signal so that the wireless signal covers at least a portion of the field of view. In other cases, the first user device does not send the wireless signal to the second user device, and the second user device may send the second context data periodically, in response to a request from the host device, and / or based on any other event or schedule.

[0020] The host device may then determine the potential presence of the user in the image data (i.e., associated with a second user device) based on comparing the first context data to the second context data. The host device also analyzes the image data based on at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data, and sends the image data to the user.

[0021] The comparison between the first context data and the second context data can be performed in a variety of embodiments. In some embodiments, the second user device generates the second context data spontaneously. In other words, the second user device generates the second context data without involving a wireless signal from the first user device, and the first user device can be configured to send a wireless signal without responding to generating image data. For example, the second user device can send the second location information of the second user device to the host device (as part of the second context data), and the first context data includes the first location information of the first user device. The first / second location information can include the first / second geographic location of the first / second user. Then, the host device can compare the first location information with the second location information. If the two locations are within a predetermined value and / or range of each other (e.g., 1 meter, 2 meters, 5 meters, 10 meters or any other suitable value), then the host device can determine that the second user device is potentially present in the image data generated by the first user device. In these embodiments, the second user device can periodically (e.g., every 0.1 second, every 0.5 second, every 1 second, every 2 seconds or at any other suitable frequency) send the second location information to the host device.

[0022] The first context data may also include first timing information associated with the image data (e.g., a timing point at which the image was captured), and the second context data may include second timing information indicating a timing point at which the second context data was sent out. The host device may also compare this first timing information with the second timing information. If the two timing points are within a threshold value (e.g., 3 minutes, 2 minutes, 1 minute, 30 seconds, 15 seconds, or less, including any values ​​and sub-ranges therebetween), then the host device may determine that the second user is potentially present in the image data.

[0023] In some embodiments, the second user device generates second context data in response to receiving a wireless signal from the first user device. For example, in response to receiving a wireless signal from the first user device, the second user device sends second context data including identification (ID) information associated with the user to the host device and sends a second timestamp. The second timestamp can be the time when the wireless signal is received or the time when the second context data is sent. The first context data sent by the first user device includes a first timestamp indicating the time when the image data was generated. By comparing the two timestamps, the host device can determine whether the user is potentially present in the image data.

[0024] In another example, the first context data and the second context data may include first location information and second location information indicating the locations of the first user device and the second user device, respectively. By comparing the two location information, the host device may also determine the possibility that the user exists in the image data.

[0025] In yet another example, the wireless signal sent by the first user device may include ID information (e.g., a serial number) associated with the image data and / or the first user device. In response to receiving the wireless signal including the ID information, the second user device may forward at least a portion of the ID information to the host device. The first user device also sends the same ID information (e.g., included in the first context data) to the host device. The host device may then determine the likelihood that the user is present in the image data based on whether the ID information in the second context data matches the ID information in the first context data.

[0026] In some embodiments, both the first context data and the second context data may be sent by the second user device to the host device. For example, the wireless signal may include the first context data, and the second user device then forwards this first context data (together with the second context data, or as part of the second context data) to the host device. The first context data may include ID information associated with the first user device and / or the image data. The host device may then determine that the second user device is near the first user device based on the fact that the ID information associated with the first user device and / or the image data is included in the data from the second user device.

[0027] The facial recognition performed by the host device can be before or after comparing the first context data to the second context data. In some embodiments, the host device compares the first context data to the second context data to determine the potential presence of the user in the image data (e.g., whether the user is close to the first client device). If the host device determines that the user is potentially present in the image data (e.g., based on the comparison of the first context data to the second context data), the host device will perform facial recognition on the image data to search for this particular user, and confirm the presence of this particular user if the facial recognition also returns a positive match. In some cases, if the host device determines that the user is not potentially present in the image data, the host device does not search for the user in the image data. In this way, the host device can reduce the number of users searched in a particular media file and reduce false positives by linking the user's context data and the user's appearance with data obtained from the media.

[0028] In some embodiments, the host device may perform facial recognition to identify all possible users in the image data. The host device then compares the second context data to the first context data to determine the potential presence of a user associated with the second user device, after which the host device may determine whether the user associated with the second user device is within the possible users identified from the facial recognition. If so, the host device may determine that the user is indeed present in the image data.

[0029] Once the host device determines that the user associated with the second user device is present in the image data, the host device can send the image data to the user. In some embodiments, the image data includes one or more images that the user is present in. In some embodiments, the image data includes a video stream, wherein the user is present in at least one frame of the video stream.

[0030] As used in this specification, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, the term "module" is intended to mean a single module or a combination of modules, and "network" is intended to mean one or more networks or a combination thereof.

[0031] As used herein, the term "module" refers to any component and / or collection of operatively coupled electrical components that may include, for example, memory, processors, electrical traces, optical connectors, software (executed in hardware), etc. For example, a module executed in a processor may be a hardware-based module (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP)) and / or a software-based module (e.g., a module of computer code stored in a memory and / or executed at a processor) capable of performing one or more specific functions associated with the module.

[0032] Embodiments and methods described herein may use facial recognition data to (1) search for one or more images of a registered user (e.g., a person whose facial recognition data is predetermined), and (2) provide images (or a video stream including images) to a client device associated with the user (e.g., a smart phone, tablet, computer, wearable electronic device, etc.). Facial recognition generally involves analyzing one or more images of a person's face to determine, for example, salient features of his or her facial structure (e.g., cheekbones, chin, ears, eyes, jaw, nose, hairline, etc.), and then defining qualitative and / or quantitative data sets associated with and / or otherwise representing these salient features. For example, one approach includes extracting data associated with salient features of a person's face and defining a data set including geometric and / or coordinate-based information (e.g., three-dimensional (3-D) analysis of facial recognition data). For example, another approach includes extracting image data into qualitative values ​​and comparing those values ​​to a template, etc. (e.g., two-dimensional (2-D) analysis of facial recognition data). In some cases, another approach may include any suitable combination of 3-D analysis and 2-D analysis.

[0033] Some facial recognition methods and / or algorithms include principal component analysis using eigenfaces (e.g., eigenvectors associated with facial recognition), linear discriminant analysis, elastic bundle graph matching using Fisherface algorithm, hidden Markov models, multilinear subspace learning using tensor representation, neuron motivated dynamic link matching, convolutional neural network (CNN), etc. or a combination thereof. Any embodiments and / or methods described herein may use and / or implement any suitable facial recognition method and / or algorithm or a combination thereof, such as those described above.

[0034] Figure 1A1 is a schematic diagram of an identification system 100 (e.g., image recognition and / or video recognition) according to an embodiment. In some cases, the identification system 100 (also referred to herein as a "system") can be used to present an image or video stream of a user based at least in part on facial recognition data. At least a portion of the system 100 can be represented and / or described, for example, by an instruction set or code stored in a memory and executed in a processor of an electronic device (e.g., a host device, a server or server group, a personal computer (PC), a network device, etc.). For example, in some embodiments, the host device can receive a signal associated with a request to register facial recognition data associated with a user, and in response, the facial recognition data can be stored in a database. Similarly, the host device can receive a signal associated with image data (including video stream data). In some cases, one or more processors of the host device can then execute an instruction set or code associated with analyzing image data stored in a memory of the host device to determine whether one or more images of the user are present in the image data based at least in part on facial recognition data and / or contextual information (e.g., location). If an image is found in the image data, then the one or more processors can isolate the associated portion of the image data. Furthermore, one or more processors may execute instruction sets or codes to (1) associate contextual data (e.g., time, location, event, etc.) with the image data and (2) define a contextual image for the user. The one or more processors may then send a signal to a client device associated with the user indicating instructions to present a contextual image for the user on a display of the client device (e.g., by graphically rendering the contextual image in an interface instantiated on the client device). The image may be a contextual image based on contextual data that is (1) associated with the image and / or (2) used to help identify the user in the image.

[0035] The system 100 includes a host device 110 in communication with a database 140, a first client device 150a and a second client device 150b (collectively referred to as client devices 150, also referred to as user devices 150), and an image capture system 160. The host device 110 may be any suitable host device in electronic communication with the database 140, the client device 150, and the image capture system 160, such as a server or a group of servers, a network management device, a personal computer (PC), a processing unit, etc. For example, in this embodiment, the host device 110 may be a server or a group of servers (deployed in substantially the same location and / or facility or distributed in more than one location) in electronic communication with the database 140, the client device 150, and the image capture system 160 via a network 105, as described in further detail herein.

[0036] The client device 150 may be any suitable device, such as a PC, a laptop, a convertible laptop, a tablet, a personal digital assistant (PDA), a smart phone, a wearable electronic device (e.g., a smart watch, etc.), etc. In some embodiments, the client device 150 includes a consumer electronic product. In some embodiments, the client device 150 may include any other electronic device that includes at least a memory, a processor, a communication interface, a display, and one or more input terminals (e.g., see below). Figure 3 ). For example, the client device 150a / b may include a stadium camera that communicates with a computing device located remotely from the stadium camera. The memory, processor, communication interface, display, and (one or more) input terminals may be connected and / or electrically coupled to each other, such as to allow signals to be sent therebetween. For example, in some embodiments, the memory may be a random access memory (RAM), a memory buffer, a hard drive, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), etc. The processor may be any suitable processing device configured to run or execute an instruction set or code (e.g., stored in a memory), such as a general purpose processor (GPP), a central processing unit (CPU), an accelerated processing unit (APU), a graphics processor unit (GPU), an application specific integrated circuit (ASIC), etc. Such a processor may run or execute an instruction set or code stored in the memory associated with using a PC application, a mobile application, an Internet web browser, a cellular and / or wireless communication (via a network), etc. More specifically, the processor may execute instruction sets or code stored in the memory associated with sending facial recognition data to and / or receiving facial recognition data and / or contextual video stream data from host device 110, as described in further detail herein.

[0037] The communication interface of the client device 150 may be any suitable module and / or device that can configure resources to communicate with the host device 110, such as one or more network interface cards, etc. Such a network interface card may include, for example, an Ethernet port that can configure the client device 150 to communicate with the host device 110 via a network (e.g., network 105), etc. Radio equipment, Radio devices (e.g. antenna), a near field communication (NFC) radio, and / or a cellular radio. As such, the communication interface can send signals to and / or receive signals from a processor associated with electronically communicating with the host device 110 via the network 105. In addition, the communication interfaces of the client devices 150 can also send and / or receive signals to each other, i.e., a first client device 150a communicating with a second client device 150b. In some cases, the communication interfaces of the client device 150 can include multiple communication interfaces (e.g., a plurality of communication interfaces for communicating with the host device 110). Communication interface and for sending and / or broadcasting signals to other client devices 150 communication interface).

[0038] The display of client device 150 may be, for example, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor, etc., which may graphically represent any suitable portion of system 100 (e.g., a graphical user interface (GUI) associated with a web page, a PC application, a mobile application, etc.). In some embodiments, such a display may be and / or may include a touch screen configured to receive tactile user input. In some cases, the display may be configured to graphically represent data associated with facial recognition processing and / or data associated with an image and / or video stream, as described in further detail herein.

[0039] The input(s) of client device 150 may be any suitable module and / or device that may receive one or more inputs (e.g., user inputs) and may send signals to and / or receive signals from a processor associated with the one or more inputs. In some embodiments, the input(s) may be and / or may include a port, plug, and / or other interface configured to be placed in electronic communication with a device. For example, such an input may be a Universal Serial Bus (USB) port, an Institute of Electrical and Electronics Engineers (IEEE) 1394 (Firewire) port, a Thunderbolt port, a Lightning port, or the like. In some embodiments, a display may be included in a touch screen, etc., configured to receive tactile user inputs.

[0040] In some embodiments, the input end may be a camera and / or other imaging device. For example, in some embodiments, such a camera may be integrated into the client device 150 (e.g., as in a smartphone, tablet, laptop, etc.) and / or may communicate with the client device 150 via a port, etc. (e.g., such as described above). The camera may be any suitable imaging device, such as, for example, a webcam or a forward-facing camera (e.g., a camera substantially pointing in the same direction as the display) included in a smartphone or tablet. In this way, the user may manipulate the client device 150 so that the camera captures an image (e.g., a photo) or a video. Moreover, in some cases, the display may be configured to graphically render data associated with the image and / or video captured by the camera. For example, in some embodiments, the client device 150 may be a smartphone, tablet, or wearable electronic device including a forward-facing camera. In some cases, the user may manipulate the client device 150 to take a picture or video of himself or herself (e.g., also referred to as a "selfie") via the camera.

[0041] In some cases, a camera (e.g., an input) included in client device 150 can be used to capture an image of a user's face, which can then be used to register facial recognition data associated with the user. Specifically, the user can manipulate client device 150a / b so that the camera captures an image of the user's face. In some cases, the display can be configured to graphically render instructions, frames, boundaries, guidance, and / or any other suitable graphical representation of data that can provide the user with instructions associated with the desired alignment of the image of the user's face. Once the camera captures the desired image, the processor can receive and / or retrieve data associated with the image of the user's face, and can then execute (e.g., stored in a memory) an instruction set or code associated with at least a portion of the facial recognition processing. For example, in some cases, the processor can execute an instruction set or code associated with verifying the alignment between the instructions, frames, boundaries, etc. graphically rendered on the display and the captured image of the user's face. In some cases, client device 150a / b may be configured to send a signal associated with data representing an image of the user to host device 110 via network 105 when alignment is verified, and in response, host device 110 may perform any suitable one or more facial recognition processes on the data, as further described in detail herein.

[0042] Image capture system 160 may be and / or may include any suitable one or more devices configured to capture image data. For example, image capture system 160 may be and / or may include one or more cameras and / or image recording devices configured to capture images (e.g., photographs) and / or record video streams. In some embodiments, image capture system 160 may include multiple cameras that communicate with a central computing device such as a server, a personal computer, a data storage device (e.g., a network attached (NAS) storage device, a database, etc.). In such embodiments, the cameras may be autonomous (e.g., may capture image data without user prompting and / or input) and may each send image data to a central computing device (e.g., via a wired or wireless connection, port, serial bus, network, etc.), which in turn may store the image data in a memory and / or other data storage device. Moreover, the central computing device may communicate with host device 110 (e.g., via network 105) and may be configured to send at least a portion of the image data to host device 110. Although in Figure 1A 105, but in other embodiments, such a central computing device may be included in, part of, and / or otherwise coupled to host device 110. In still other embodiments, the camera may communicate with host device 110 (e.g., via network 105) without such a central computing device.

[0043] In some embodiments, the image capture system 160 may be associated with and / or owned by a venue, etc., such as, for example, a sports arena, a theme park, a theater, and / or any other suitable venue. In other embodiments, the image capture system 160 may be used in or at the venue, but owned by a different entity (e.g., an entity licensed and / or otherwise authorized to use the image capture system 160 in or at the venue, such as, for example, a television camera at a sporting event). In still other embodiments, the image capture system 160 may include any number of client devices (e.g., user devices), etc., such as smartphones, tablets, etc., which may be used as cameras or recorders. In such embodiments, at least some of the client devices may communicate with the host device 110 and / or a central computing device associated with the venue (e.g., as described above).

[0044] For example, in some embodiments, a camera integrated into the first client device 150a may form and / or be part of at least a portion of the image capture system 160, such as Figure 1BAs shown in . In this manner, a first user of the first client device 150a can manipulate the first client device 150a to capture pictures and / or video recordings, and in response, the first client device 150a can upload and / or otherwise send pictures (e.g., image data, photo data, etc.) and / or video recording data to the host device 110. In some cases, the picture and / or video recording data can be stored on the first client device 150a for any suitable time and uploaded and / or sent to the host device 110 at a later time. Moreover, after the picture and / or video recording data is sent to the host device 110, the picture and / or video recording data can be stored on the first client device 150a. That is, sending the picture and / or video recording data does not delete and / or remove the picture and / or video recording data from the first client device 150a (e.g., a copy of the data is sent to the host device 110). Therefore, as Figure 1B As shown in , the image capture system 160 need not be associated with a specific event and / or venue. In this case, the first user can manipulate the first client device 150a (e.g., an application of the first client device 150a) to capture user-generated content (e.g., pictures, image data, photo data, video stream data, etc.) via a camera and / or recording device (e.g., image capture system 160) integrated into the first client device 150a.

[0045] In some cases, image capture system 160 is configured to capture image data associated with a venue and / or an event. In other words, image capture system 160 is configured to capture image data within a predetermined, known and / or given context. For example, in some cases, image capture system 160 may include one or more image capture devices (e.g., cameras and / or video recorders) installed at an arena, etc. and configured to capture image data associated with patrons, guests, performers, etc. at the arena. In this way, image capture system 160 is configured to capture image data within the context of the arena and / or an event occurring at the arena. Thus, the captured image data may be, for example, "contextual image data." That is, the image data is associated with at least some contextual data. As non-limiting examples, host device 110 may receive image data and / or video stream data from image capture system 160 and data associated with context (e.g., "context data" associated with the arena and / or events occurring at the arena, and / or any other suitable context and / or metadata) from any suitable data source, etc.; may associate context data with, for example, image data; may define a user-specific context image and / or a user-specific context video stream associated with, for example, a user of client device 150a / b; and may send a user-specific context image and / or a user-specific context video stream associated with the user to client device 150a / b. Alternatively and / or additionally, host device 110 may also receive context data from one or more of user devices 150 as described herein.

[0046] As described above, client device 150 and image capture system 160 may communicate with host device 110 via one or more networks. Figure 1A As shown in , client device 150 and image capture system 160 can communicate with host device 110 via a communication interface of host device 110 and network 105. Network 105 can be any type of network, such as, for example, a local area network (LAN), a virtual network such as a virtual local area network (VLAN), a wide area network (WAN), a metropolitan area network (MAN), World Interoperability for Microwave Access (WiMAX), a cellular network, the Internet, and / or any other suitable network implemented as a wired and / or wireless network. As an example, network 105 can be implemented as a wireless network based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard (also known as ) of a wireless local area network (WLAN). Moreover, network 105 may include a combination of any type of network, such as, for example, a LAN or WLAN and the Internet. In some embodiments, client device 150 may communicate with host device 110 and network 105 via an intermediate network and / or an alternative network (not shown), which may be similar to or different from network 105. As such, client device 150 may send data to host device 110 and / or receive data from host device 110 using a variety of communication modes (e.g., associated with any of the networks described above), which may or may not be sent to host device 110 using a common network. For example, client device 150 may be a mobile phone (e.g., a smart phone) connected to host device 110 via a cellular network and the Internet (e.g., network 105).

[0047] In some cases, the network may facilitate, for example, peer-to-peer networking sessions, etc. In this case, the peer-to-peer networking session may include, for example, a client device and / or any other suitable electronic device, each of which shares common characteristics. For example, in some cases, a peer-to-peer networking session may include any suitable client device (e.g., an electronic device registered in database 140, etc.) within a predetermined proximity of a venue, event, location, etc. For example, in some cases, such a peer-to-peer networking session may include any number of registered client devices present at a venue (e.g., a sporting event). In some cases, a peer-to-peer networking session may be automatically established based on contextual data associated with a user and / or client device. In other cases, a peer-to-peer networking session may be automatically established based on one or more users "logging in" and / or otherwise announcing his or her presence at a venue, etc. (e.g., "squawking" the presence of a user). In some cases, a user may "log in" when the user arrives at an event, etc. (e.g., a sporting event, a concert, a wedding, a birthday party, a rally, etc.), when registering, when capturing an image or video stream, etc. Additionally, a "check-in" may include identifying information, such as, for example, geographic location data, date and time data, personal or user identification data, and the like. In some embodiments, a user may also search, via an application on their client device 150, for events and / or locations for which video stream data has been captured. A user may "check-in" to an event and / or location returned from a search. As described herein, checking-in to an event and / or location may initiate processing of video stream data associated with that event and / or location, such as to determine whether a user may be matched to an image or video stream data.

[0048] In other cases, a user may manually establish a peer-to-peer networking session that includes, for example, a predetermined set or group of users. In some cases, such a peer-to-peer networking session may be a public network, a private network, and / or an otherwise restricted access network. For example, in some cases, a user may request to join a networking session and / or may receive an invitation to join a networking session, etc. In some cases, establishing a peer-to-peer networking session may, for example, facilitate communication (e.g., a group chat session, etc.) and / or sharing of image and / or video data between users included in the peer-to-peer networking session.

[0049] The host device 110 may be any suitable device configured to send data to and / or receive data from the database 140, the client device 150, and / or the image capture system 160. In some embodiments, the host device 110 may be used as, for example, a server device (e.g., a web server device), a network management device, an administrator device, etc. In some embodiments, the host device 110 may be a group of servers or devices housed together in or on the same blade, rack, and / or facility or distributed in or on multiple blades, racks, and / or facilities. The host device 110 includes at least a memory 115, a processor 120, and a communication interface 125 (see, for example, Figure 2 ). In some embodiments, memory 115, processor 120, and communication interface 125 are connected and / or electrically coupled so that signals can be sent between memory 115, processor 120, and communication interface 125. Host device 110 may also include and / or may otherwise be operably coupled to a database 140 configured to store user data, facial recognition data, contextual data (e.g., associated with time, location, venue, event, etc.), video streams, etc.

[0050] Memory 115 may be, for example, RAM, a memory buffer, a hard drive, a database, ROM, EPROM, EEPROM, etc. In some cases, memory 115 of host device 110 includes instruction sets or code for performing one or more facial recognition actions and / or for communicating (e.g., sending and / or receiving) data with at least one device (e.g., client device 150) using one or more suitable communication modes.

[0051] Processor 120 may be any suitable processor, such as, for example, a GPP, a CPU, an APU, a GPU, a network processor, a front-end processor, an ASIC, an FPGA, etc. Thus, processor 120 may be configured to perform and / or execute instruction sets, processes, modules, and / or code stored in memory 115. For example, processor 120 may be configured to execute instruction sets, processes, and / or modules associated with, among other things, receiving facial recognition data (e.g., from client device 150), analyzing facial recognition data, registering and / or storing facial recognition data, receiving video stream data (e.g., from image capture system 160), analyzing video stream data and comparing the video stream data to facial recognition data, receiving and comparing contextual data, sending video stream data (e.g., to client device 150a / b), receiving and / or analyzing characteristics of video stream data (e.g., based on location information determined such as background landmarks and / or background scenery data included in the video stream data), and / or any other suitable processing, as further described herein.

[0052] Communication interface 125 may be any suitable device (e.g., any device configured to collect and / or at least temporarily store data (such as facial recognition data, video streams, etc.) that can place host device 110 in communication with database 140, client device 150, image capture device 160, and / or any other suitable device and / or service that communicates with network 105). In some embodiments, communication interface 125 may include one or more wired and / or wireless interfaces, such as, for example, a network interface card (NIC), an Ethernet interface, an optical carrier (OC) interface, an asynchronous transfer mode (ATM) interface, and / or a wireless interface (e.g., Radio equipment, radio device, NFC radio device, etc.).

[0053] Return to Figure 1A, the database 140 associated with the host device 110 can be any suitable database, such as, for example, a relational database, an object database, an object-relational database, a hierarchical database, a network database, an entity-relational database, a structured query language (SQL) database, an extensible markup language (XML) database, a digital repository, a media library, a cloud server or storage device, etc. In some embodiments, the host device 110 can communicate with the database 140 via the communication interface 125 through any suitable network (e.g., the network 105). In such embodiments, the database 140 can be included in or stored by a network attached storage (NAS) device that can communicate with the host device 110 through the network 105 and / or (one or more) any other network. In other embodiments, the database can be stored in the memory 115 of the host device 110. In still other embodiments, the database 140 can be operably coupled to the host device 110 via a cable, a bus, a server rack, etc.

[0054] Database 140 may store and / or at least temporarily retain data associated with recognition system 100. For example, in some cases, database 140 may store data associated with and / or otherwise represent user profiles, resource lists, facial recognition data, patterns and / or methods, contextual data (e.g., associated with time, location, venue, event, etc.), video streams or portions thereof, images, location information (such as landmark data), etc. In other words, database 140 may store data associated with users whose facial image data has been registered with system 100 (e.g., "registered users"). In some embodiments, database 140 may be and / or include a relational database in which data may be stored, for example, in tables, matrices, vectors, etc., according to a relational model. For example, in some cases, host device 110 may be configured to store image data and / or video stream data received from a video or image source (e.g., image capture system 160) and contextual data associated with the video stream data in database 140. In some cases, as described in further detail herein, the image data and / or video stream data and the contextual data associated therewith may collectively define a contextual video stream, etc. In other cases, the video stream data may be stored in database 140 without the contextual data, etc. In some cases, the contextual data may be used to reduce false positives associated with facial recognition processing, as described herein.

[0055] In some embodiments, the user profile can be a user profile data structure that includes information related to the user's access to the image data. For example, the user profile data structure can include a user profile identifier, facial recognition data (e.g., data obtained from an image of the user that can be used to match the user to an image from the image data (e.g., facial feature data)), a list of identifiers associated with an image data structure stored in database 140 and associated with the user, a list of identifiers associated with user profile data structures of other users associated with the user (e.g., as friends and / or contacts), user location data, etc.

[0056] In some embodiments, users can add each other as friends within the application through which they access image data. For example, when a user associated with a first user profile is a contact of another user associated with a second user profile, users can also be automatically associated with each other. For example, a user operating a client device can have a list of contacts and / or other contact information stored at the client device. The application can retrieve and import contact information, can match contact information with information in at least one user profile in a database, and can automatically associate the at least one user profile with that user.

[0057] In some embodiments, users can be associated with each other by storing a list of friends and / or contacts within each user profile of each user (e.g., a list of identifiers of user profiles to be added as friends of a particular user). When a user adds friends and / or contacts, the user can be automatically notified when the friends and / or contacts record and / or receive image data, video stream data, etc. In some embodiments, the host device 110 can also use the stored relationships between users to automatically process image data and / or video stream data associated with the user (e.g., to determine whether the user's friends and / or contacts can be found in the image data and / or video stream data). For example, when image data is received, when friends and / or contacts are associated with a user, etc., the host device 110 can automatically process the image data to determine whether facial image data associated with the user's friends and / or contacts can be matched to the image data.

[0058] Although host device 110 is shown and described with reference to FIG. 1 as including and / or otherwise operably coupled to database 140 (e.g., a single database), in some embodiments, host device 110 may be operably coupled to any number of databases. Such databases may be configured to store at least a portion of a data set associated with system 100. For example, in some embodiments, host device 110 may be operably coupled to and / or otherwise communicate with a first database and operably coupled to and / or otherwise communicate with a second database, the first database being configured to receive and at least temporarily store user data, user profiles, etc., the second database being configured to receive and at least temporarily store image data and / or video stream data and context data associated with the image data and / or video stream data. In some embodiments, host device 110 may be operably coupled to and / or otherwise communicate with a database stored in or on client device 150 and / or image capture system 160. Similarly described, at least a portion of the database may be implemented in and / or stored by client device 150 and / or image capture system 160. In this manner, host device 110 may communicate with any number of databases that may be physically deployed at a different location than host device 110, and in some cases database 140 may communicate with any number of databases that may be physically deployed at a different location than host device 110 while communicating with host device 110 (e.g., via network 105).

[0059] In some embodiments, database 140 can be a searchable database and / or repository. For example, in some cases, database 140 can store image data and / or video stream data (e.g., contextual video stream data) associated with a user. In some cases, a user can search database 140 to retrieve and / or view one or more images and / or video streams associated with a user stored in database 140. In some cases, a user can have limited access and / or privileges to update, edit, delete and / or add images and / or video streams associated with his or her user profile (e.g., contextual video streams specific to a user, etc.). In some cases, a user can, for example, update and / or modify permissions and / or access associated with user-specific images and / or video streams associated with that user. For example, in some cases, a user can redistribute, share and / or save data associated with the user. In other cases, a user can block access to user-specific data, etc. In some cases, a user can redistribute and / or share content, data, images and / or video streams (e.g., may or may not be associated with a user) that are otherwise shared with a user.

[0060] Return to Figure 2 As described above, the processor 120 of the host device 110 may be configured to perform certain functions or instructions. These functions may be implemented, for example, in hardware, software stored in the memory 115 and / or executed in the processor 120, and / or any combination thereof. For example, Figure 2 As shown in , the processor 120 includes an analyzer 121 (e.g., to perform an analysis function), a database interface 122 (e.g., to perform a database function), a renderer 123 (e.g., to perform a rendering function), and a location analyzer 124 (e.g., to perform a location function). Figure 2 As shown in , the analyzer 121, the database interface 122, the renderer 123 and the location module can be connected and / or electrically coupled. As such, signals can be sent between the analyzer 121, the database interface 122, the renderer 123 and the location analyzer 124.

[0061] The analyzer 121 includes an instruction set associated with receiving and / or collecting data, images, and / or video streams associated with facial recognition of a user that can be executed by the processor 120 (or a portion thereof). More particularly, the analyzer 121 can be operably coupled to the communication interface 125 and / or otherwise communicate with the communication interface 125 and can receive data therefrom. Such data can, for example, be associated with a user (e.g., facial recognition information, profile information, preferences, activity logs, location information, contact information, calendar information, social media activity information, etc.), associated with a site (e.g., location data, resource data, event schedules), associated with an event, etc. As further described in detail herein, the analyzer 121 can receive signals associated with requests and / or instructions from the communication interface 125 to perform and / or conduct any number of processes associated with facial recognition.

[0062] In some cases, the analyzer 121 may receive data from the communication interface 125 in substantially real time. That is, in some cases, an electronic device (e.g., client device 150) included in the system 100 may be manipulated by a user to define and / or update data associated with the user's facial recognition, and once defined and / or updated, the data may be sent to the host device 110 via the network 105. Thus, the communication interface 125 may send a signal to the analyzer 121 upon receipt of the data, which receives the data within a very short period of time after being defined and / or updated by the electronic device. In other embodiments, the analyzer 121 may receive data from the communication interface 125 at a predetermined rate, etc., based on, for example, aggregate processing, current and / or predicted processor, memory and / or network loads, etc.

[0063] As described above, analyzer 121 may be configured to receive, aggregate, analyze, sort, parse, modify, and / or update data associated with facial recognition processing, etc. More particularly, in some cases, a user may manipulate client device 150a / b to capture one or more images or video streams of his or her face (as described in further detail herein), and in turn may transmit signals associated with and / or representative of the image data to host device 110, e.g., via network 105. In some cases, communication interface 125 may receive the image data and may transmit the associated signals to analyzer 121. Upon receipt, analyzer 121 may execute instruction sets or codes (e.g., stored in analyzer 121 and / or in memory 115) associated with aggregating, analyzing, sorting, updating, parsing, and / or otherwise processing the image data. More specifically, the analyzer 121 may perform any suitable facial recognition processing and / or algorithm, such as, for example, principal component analysis using eigenfaces (e.g., eigenvectors associated with facial recognition), linear discriminant analysis, elastic bundle graph matching using Fisherface algorithm, hidden Markov model, multilinear subspace learning using tensor representation, neuron-motivated dynamic link matching, convolutional neural network (CNN), etc. or a combination thereof. In some embodiments, image data provided by the user to the host device 110 may be used in a subsequent facial recognition process via the analysis portion 121 to identify the user.

[0064] Analyzer 121 may define a user profile, among other things, including image data of the user and any other suitable information or data associated with the user, such as, for example, pictures, video recordings and / or audio recordings, personal and / or identification information (e.g., name, age, gender, birthday, hobbies, etc.), calendar information, contact information (e.g., associated with the user and / or the user's friends, family, partners, etc.), device information (e.g., media access control (MAC) addresses, Internet Protocol (IP) addresses, etc.), location information (e.g., current location data and / or historical location data), social media information (e.g., profile information, usernames, passwords, lists of friends or contacts, etc.), and / or any other suitable information or data. As such, analyzer 121 may send a signal to database interface 122 indicating an instruction to store the user profile data in database 140, as described in further detail herein.

[0065] In some cases, analyzer 121 may receive video stream data (or image data from a photo, for example) and may be configured to analyze and / or process the image data and / or video stream data to determine whether a portion of the image data and / or video stream data matches any appropriate portion of the user's image data. That is, analyzer 121 may use previously stored user image data as a template to compare the data included in the image and / or video stream with. In other words, analyzer 121 performs facial recognition processing and / or analysis on the image data and / or video stream data based at least in part on previously stored user image data. In some embodiments, host device 110 (and more specifically communication interface 125) receives image data and / or video stream data from image capture system 160 either directly (e.g., from one or more cameras via network 105) or indirectly (e.g., from a computing device via network 105, which in turn communicates with one or more cameras). In some embodiments, analyzer 121 may be configured to analyze and / or process video stream data based at least in part on separating, parsing, classifying, and / or otherwise deconstructing the video stream data into its individual frames (e.g., static pictures at predetermined times during a video stream). As such, analyzer 121 may compare and / or analyze data included in a video stream frame relative to previously stored user image data.

[0066] In some cases, the analyzer 121 may also analyze the image data and / or video stream data to determine context information associated with the image data and / or video stream, such as, for example, location, venue, time, coincident events (e.g., a team scoring a goal, for example, captured at a "kissing shot", etc.) and / or any other suitable context information. In some cases, the analyzer 121 may be configured to match, aggregate at least a portion of the image data and / or video stream and / or otherwise associate at least a portion of the image data and / or video stream with context data. For example, in some cases, the image data and / or video stream data may represent, for example, a user at a sports event. In this case, the context data may be, for example, a video stream of a sports event or a game, and may include data associated with time, location, venue, team, etc. As such, the analysis portion 121 may be configured to aggregate the image data and / or video stream data and context data so that the video stream data and context data are substantially consistent (e.g., data associated with substantially the same time occurs and / or is captured). In other cases, the context data may include data associated with any other suitable context. In some cases, analyzer 121 may be configured to use contextual information associated with the image data and / or video stream and data related to the user's location to further connect the image data and / or video stream to a specific user. Analyzer 121 may be configured to compare the contextual information with the user's location before comparing the data included in the video stream with previously stored user image data.

[0067] If the analyzer 121 determines that at least a portion of the image data and / or data in the video stream meets the criteria (e.g., matches previously stored user image data to a predetermined and / or acceptable probability), then the analyzer 121 may send one or more signals to the database interface 122 indicating instructions to store at least a portion of the image and / or video stream data in the database 140 and to associate that data with previously stored user image data and / or otherwise store that data. In some cases, the analyzer 121 may send a signal to the database interface 122 such that the individual frames are stored in the database 140, which in turn may be subsequently retrieved and processed to define the video stream. In other cases, the analyzer 121 may send one or more signals to the database interface 122 such that the portion of the video stream data is stored in the database 140. That is, the analyzer 121 may re-define and / or reconstruct the video stream at least in part from the individual frames (separated or deconstructed as described above).

[0068] In some cases, host device 110 may receive image data (e.g., from image capture system 160 and via network 105 and communication interface 125) and analyzer 121 and / or any other suitable non-hosted Figure 2 The modules shown in can perform one or more pre-processing and / or pre-sorting processes before performing facial recognition processing (described above). For example, in some embodiments, analyzer 121 (or other modules) can analyze video stream data to determine and / or define a data set that includes, for example, identification information and / or contextual information (such as location, time, event, image and / or video identifiers, etc.). Once defined, analyzer 121 can analyze user data stored in database 140 (e.g., via sending a signal to database interface 122 indicating an instruction to query database 140, etc.) to determine that a portion of data associated with the user meets (one or more) criteria, such as matching a data set including contextual information associated with image data.

[0069] In some cases, the criterion(s) may be associated with a confidence level and / or match threshold expressed in any suitable manner (e.g., a value such as a decimal, a percentage, etc.). For example, in some cases, the criterion(s) may be a threshold, etc., such as a 70% match between the image data and at least a portion of the data stored in the database, a 75% match between the image data and at least a portion of the data stored in the database, an 80% match between the image data and at least a portion of the data stored in the database, an 85% match between the image data and at least a portion of the data stored in the database, a 90% match between the image data and at least a portion of the data stored in the database, a 95% match between the image data and at least a portion of the data stored in the database, a 97.5% match between the image data and at least a portion of the data stored in the database, a 99% match between the image data and at least a portion of the data stored in the database, or any percentage therebetween.

[0070] In some cases, the data associated with the user may include, for example, calendar data, location data, preference data, etc. For example, if the data does not meet the criteria, then the analyzer 121 may define an indication that the data associated with the user can be excluded from, for example, facial recognition processing. In this way, pre-processing and / or pre-sorting can reduce the amount of processing load during facial recognition processing, etc. Although described above as querying the database 140 to obtain user data, in some embodiments, the host device 110 may send a signal indicating a request for location data, etc. associated with that device to a device associated with the user (e.g., client device 150a / b). As described above, upon receiving location data (e.g., global positioning service (GPS) data of the device, using location information and / or features (such as landmarks and / or background scenery) within an image or video, etc.), etc., the analyzer 121 may determine whether the location data matches the location data associated with the video stream.

[0071] For example, in some cases, analyzer 121 may receive image data (including video stream data) from a sporting event that also includes location data associated with, for example, an arena. In response, analyzer 121 may send a request for location data from a client device associated with a user (e.g., client device 150). For example, if the location data associated with the image data and the location data associated with the client device are substantially similar (e.g., the location data associated with the image data and the location data associated with the client device indicate a source of the image, and the client device is within a predetermined distance of each other) and / or the location data associated with the client device is within a predetermined range of location data values, etc., then analyzer 121 may increase the confidence score and / or otherwise treat the result as contributing to satisfying a threshold and / or otherwise satisfying the criteria(s).

[0072] The location data may be, for example, geolocation data based on GPS, network location and / or data (e.g., via NFC authentication, Bluetooth authentication, cellular triangulation, cognitive network exchanges and / or protocols, etc.), social network data (such as "check-ins"), etc. For example, location analyzer 124 may process the location data to identify the location of a video stream and / or a user, and provide the data to analyzer 121 to allow analyzer 121 to modify the confidence score. In this manner, a confidence score may be calculated based on the location data.

[0073] In other embodiments, when the location data associated with the image and the location data associated with the user are substantially similar (e.g., the location data associated with the image and the location data associated with the client device indicate that the source of the video stream and the client device are within a predetermined distance of each other), the location analyzer 124 can process the location data and can provide the processed location data to the analyzer 121. The analyzer 121 can then generate and / or modify a confidence score based on the location data and the facial recognition analysis of the image. In this manner, when the location data associated with the image and the location data associated with the user are determined to be substantially similar, a confidence score can be generated and / or modified, and when the location data associated with the image and the location data associated with the user are substantially dissimilar, a confidence score cannot be generated and / or modified. In addition, in this manner, the confidence score can be calculated as a result of both the location data analysis and the facial recognition analysis. In this manner, the host device 110 (e.g., via the analyzer 121) can determine, for example, the proximity of the client device to the location where the video stream data was captured.

[0074] In some embodiments, the analysis portion 121 may generate and / or modify a confidence score based on ID information associated with the image data received by the host device 110. For example, the analysis portion 121 may compare the ID information contained in the second context data (e.g., from the second user device 150b) with the ID information contained in the first context data (e.g., from the first user device 150a). If the ID information matches each other, the host device 110 may determine that the second context data is generated in response to receiving a wireless signal from the first user device, and the second context data is sent by the first user device in response to generating the image data, and further determine the potential presence of the second user in the image data. The host device 110 may further modify the confidence score using facial recognition. For example, the analysis portion 121 may compare the image data with the facial recognition data of the second user. If the host device 110 finds a match in the image data, the host device 110 may increase the confidence score. If the host device 110 does not find a match in the image data, the host device may reduce the confidence score.

[0075] Although described as analyzing location data, in other cases, the analyzer 121 may analyze data associated with any suitable source, activity, location, pattern, purchase, etc. For example, in some cases, the analyzer 121 may analyze ticket sales associated with a venue. In other cases, the analyzer 121 may analyze social media posts, comments, likes, etc. In some cases, the analyzer 121 may collect and / or analyze data associated with a user (as described above), and may define a user profile that may include, for example, user identification data, facial recognition data, client device data, purchase data, Internet web browsing data, location data, social media data, preference data, etc., among others. Thus, the user's profile data may be analyzed to determine a confidence score, value, and / or indicator that may be evaluated relative to a threshold score, value, and / or indicator to determine whether the user data and / or video stream data meets the criteria(s). Thus, in such embodiments, non-facial recognition data (e.g., ticket sales data, social media posts and / or features such as a person's wardrobe in a video or image, location data such as landmarks within an image, background scenery data, etc.) may be used to corroborate facial recognition data and / or increase / decrease confidence scores.

[0076] Although analyzer 121 is described above as analyzing image data to define facial recognition data and contextual data associated with the image, in other embodiments, facial recognition processing and contextual data processing can be performed separately and / or independently. For example, in some embodiments, analyzer 121 can be configured to perform facial recognition processing, while different modules, processors, devices, servers, etc. can be configured to perform contextual data processing. For example, location analyzer 124 can analyze images and / or video streams based on location data, characteristics of the image, etc. Therefore, when compared to facial recognition processing and contextual data processing performed by the same module, the time used to analyze video stream data can be reduced and / or the processing load can be distributed.

[0077] As described above, the database interface 122 includes a set of instructions executed by the processor 120 (or a portion thereof) associated with monitoring the database 140 and / or updating the data stored in the database 140. For example, the database interface 122 may include instructions to cause the processor 120 to update the data stored in the database 140 using at least a portion of the facial recognition data received from the analyzer 121. More specifically, the database interface 122 may receive, for example, user image data associated with a user from the analyzer 121, and in response, the user image data may be stored in the database 140. In some cases, as described above, the database interface 122 may receive a signal from the analyzer 121 indicating a request to query the database 140 to determine whether the data stored in the database 140 and associated with the user image data of the user matches any suitable portion of the video stream data and / or the image data. For example, if at least a portion of the video stream data and / or the image data meets (one or more) criteria (hereinafter referred to as "criteria" for simplicity, but multiple "criteria" are not excluded), then the database interface 122 may be configured to update the data stored in the database 140 associated with that user. That is, if at least a portion of the video stream data and / or the image data matches the previously provided user image data within a predetermined probability, etc. However, if the video stream data and / or the image data does not match the previously provided user image data stored in the database 140, then the database interface 122 may, for example, query the database 140 for the next entry (e.g., data associated with the next user), and / or may otherwise not update the database 140. Moreover, the database interface 122 may be configured to store data in the database 140 in a relation-based manner and / or in any other suitable manner (e.g., the database 140 may be a relational database, etc.).

[0078] The renderer 123 includes an instruction set executed by the processor (or a portion thereof) that is associated with defining a context image and / or presentation that represents at least a portion of the image data that meets the criteria during facial recognition processing, as described above. More specifically, the renderer 123 can be configured to define a context image and / or presentation that represents a user identified (e.g., via facial recognition) at an event, venue, location, etc. Once the context image is defined, the renderer 123 can send a signal associated with the context image to the communication interface 125, which in turn can send a signal (e.g., via the network 105) to the client device 150a / b indicating instructions to graphically render the context image and / or video stream on a display of the client device 150a / b.

[0079] Although renderer 123 and / or other portions of host device 110 are described above as sending signals to client device 150a / b indicating instructions to present a context image on a display of client device 150a / b, in other cases renderer 123 may define a context image and may send signals to database interface 122 indicating instructions to store the context image in database 140. In such cases, data associated with the context image may be stored and / or otherwise associated with user data stored in database 140.

[0080] In some cases, host device 110 may retrieve context images from database 140 in response to a request from client device 150a / b (and / or any other suitable device). More specifically, in some embodiments, a user may manipulate client device 150a / b to access a web page on the Internet. After being authenticated (e.g., entering credentials, etc.), the user may interact with the web page so that a request to access the context image is sent from client device 150a / b to host device 110. Thus, host device 110 (e.g., database interface 122) may retrieve the context image from database 140 and may send a signal to client device 150a / b, which may be operable to present the context image on a display (e.g., via the Internet and the web page by rendering the context image). In other words, the context image may be stored on the "cloud" and may be accessed via a web browser and the Internet.

[0081] Although the analyzer 121, database interface 122, and renderer 123 are described above as being stored and / or executed in the host device 110, in other embodiments, any of the modules may be stored and / or executed in, for example, the client device 150 and / or the image capture system 160. For example, in some embodiments, the client device 150 may include, define, and / or store a rendering module (e.g., as a local application). The rendering module may be substantially similar or identical to the renderer 123 of the host device 110. In such embodiments, the rendering module of the client device 150 may replace the functionality of the renderer 123 otherwise included and / or executed in the host device 110. Thus, the rendering module of the client device 150 may receive, for example, a data set associated with a context image and upon receipt may define a rendering to be presented on a display of the client device 150.

[0082] Figure 3 is included in Figure 1A and Figure 1BSchematic diagram of a client device 150 in an identification system. The client device 150 includes a processor 152 operably coupled to a memory 154, a communication interface 156, an optional imaging device 158, and an optional display 159. The client device 150 can be configured as either a first client device 150a or a second client device 150b.

[0083] In some embodiments, the first client device 150a and the second client device 150b may be interchangeable. For example, a first user using the first client device 150a may use the second client device 150b to capture an image including the second user's field of view, and the host device 110 may identify the second user in the image data and send the image data to the second user based on the methods described herein. On the other hand, a second user using the second client device 150b may also capture an image including the first user's field of view, and the host device 110 may identify the first user in the image data and send the image data to the first user.

[0084] When configured as a first client device in the recognition system 100, the client device 150 includes an imaging device 158 configured to generate image data representing a field of view. The communication interface 156 also communicates with the imaging device 158 and is configured to communicate via a network (e.g., network 105). In some cases, in response to the imaging device 158 generating image data, the processor 120 is configured to control the communication interface 156 to broadcast a wireless signal covering at least a portion of the field of view. The wireless signal includes at least a portion of the first context data associated with the image data. The processor 152 is also configured to control the communication interface 156 to send the image data and the first context data associated with the image data to a server (e.g., host device 110) via the network, so that in response to receiving the image data, the server determines the potential presence of the user in the image data based on comparing the first context data with the second context data, which is associated with the user of the second user device and generated in response to the second user device receiving the wireless signal. The server also analyzes the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data.

[0085] When configured as a second client device in the recognition system 100, the client device 150 includes a communication interface 156 and a processor 152. The processor is configured to control the communication interface 156 to receive a wireless signal from the first device, the wireless signal including first context data associated with image data generated by an imaging device associated with the first device. The first device is also configured to send the image data and the first context data to the server. In response to receiving the first context data, the processor 152 is configured to control the communication interface 156 to send the second context data to the server, so that in response to receiving the second context data, the server determines the potential presence of the user in the image data based on comparing the first context data with the second context data. The server also analyzes the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data. The processor 152 in the client device 150 is also configured to receive the image data from the server in response to the server identifying the user in the image data. The processor 152 can also control the display 159 to display the received image data.

[0086] use Figure 1A-Figure 3 The device and system shown can implement various methods of image analysis. Figure 4-Figure 6 Some example methods are described.

[0087] Figure 4 is a diagram illustrating a method according to an embodiment of the present invention. Figure 1A and Figure 1B In the method 400, at 410, the host device (e.g., host device 110) receives image data from a first user device, which may include an image capture system, such as the one described above with reference to FIG. Figure 1A and Figure 1B The host device also receives first context data associated with the image data. More specifically, the host device can receive the image data either directly (e.g., from one or more cameras via a network) or indirectly (e.g., from a computing device via a network that in turn communicates with one or more cameras).

[0088] The image data includes at least one image of a field of view. The field of view (e.g., a scene captured by an image capture system in at least one image) may include various types of venues, such as sporting events, concerts, assemblies, graduations, summer camps, or any other suitable situations. In some cases, for example, the image data may be captured and / or photographed by a user of a first user device (e.g., 150a). The user may be a participant in an event. In other cases, the image data may be captured and / or acquired by the organizer of the event. For example, the image data may be captured and / or photographed by one or more fixed cameras installed and / or controlled by the organizer of the event. In another example, the image data may be captured and / or photographed by one or more photographers associated with the event (e.g., employed by the organizer). In some cases, the image data may be captured and / or photographed by a third party (such as a camera associated with a television station).

[0089] In one example, the camera may record image data and may send the image data to the host device. In another example, the user may record (one or more) images through an application running on a client device operated by the user (e.g., the first client device 150a) (e.g., via a user generated content (UGC) interface within the application running on the client device). By initiating the recording through the application (e.g., by clicking a "capture", "record" and / or similar button in the UGC interface), the user may capture and / or record one or more images, and the client device may associate the first context data with the image. The context data may include, for example, location data (e.g., geolocation data, data from near field communication (NFC), data from Bluetooth communication with other devices, cellular triangulation, event and / or location login data, and / or network Wi-Fi connection information), timing data (e.g., from a clock operably coupled to the image capture system), ID data (e.g., a serial number generated by the client device for each image or video), etc. As an example, the host device may tag the image with location data, timing data, and / or ID data, and / or may associate the image with a data structure that encapsulates location data, timing data, and / or ID data.

[0090] At 420, the host device receives second context data from a second user device (e.g., 150b in system 100). The second context data may be associated with a user of the second user device or the second user device itself. The second context data is generated in response to the user device receiving a wireless signal, which is transmitted by an antenna operably coupled to the imaging device in response to the imaging device generating image data. The wireless signal may be based on any suitable wireless technology, such as, for example, Radio frequency (RF), 3G, 4G, 5G, optical communication and / or infrared communication, etc.

[0091] In some embodiments, the second user device receives the wireless signal via one protocol (e.g., Bluetooth) and sends the second context data to the host device via another protocol (e.g., WiFi). In some embodiments, the second device uses the same protocol to receive the wireless signal and send the second context data.

[0092] The wireless signal may cover at least a portion of the field of view of the image data. Similarly as described, the wireless signal may be broadcast so that a user device (such as a second user device) near the first user device can receive the wireless signal. In some embodiments, the wireless signal is broadcast by the first user device so that the wireless signal is sent to the area around the first user device. In some other embodiments, the wireless signal is directed toward the field of view. Directed wireless signals can reduce the number of user devices that may receive the wireless signal and therefore send context data to the host device, and accordingly improve the accuracy of subsequent facial recognition by reducing the number of possible users in the image data. Similarly as described, because the wireless signal that triggers the second user device to send the second context data to the host device is directed toward the same field of view of the image data, as long as the host device receives the second context data from the user device, the chance that the user of the user device is located within the image data is high.

[0093] In some embodiments, the first user device includes a camera associated with the venue. In these cases, the camera can be operably coupled to an antenna (or a computing device including an antenna). When the camera captures an image or video, the antenna is configured to send out a wireless signal. For example, a controller such as a control station or console can be used to control both the camera and the antenna, in which case the instructions sent to the camera to capture the image or video are also used to control the antenna to send out a wireless signal. In another example, the camera may include a communication interface that communicates with the antenna (or computing device). When the camera captures an image or video, the communication interface is configured to send a signal to the antenna (or computing device), causing the antenna to send out a wireless signal.

[0094] At 430, the host device determines a potential presence of the user in the image data based on comparing the first context data to the second context data. The comparison between the first context data and the second context data can be performed in various implementations.

[0095] In some embodiments, for example, in response to receiving a wireless signal from a first user device, the second user device sends second context data including identification (ID) information associated with the image data, the user and / or the second user device to the host device and sends a second timestamp. The second timestamp can be, for example, the time when the wireless signal is received or the time when the second context data is sent. The first context data sent by the first user device includes a first timestamp indicating the time when the image data was generated. By comparing the two timestamps, the host device can determine whether the user is potentially present in the image data.

[0096] In other embodiments, for example, the first context data and the second context data may include first location information and second location information indicating the locations of the first user device and the second user device, respectively. By comparing the first location information with the second location information, the host device may also determine the likelihood that the user is present in the image data.

[0097] In yet another example, the wireless signal sent by the first user device may include ID information (e.g., a serial number) associated with the image data and / or the first user device. In response to receiving the wireless signal, the second user device may forward at least a portion of the ID information to the host device. The first user device also sends the same ID information (e.g., included in the first context data) to the host device. The host device may then determine the likelihood that the user is present in the image data based on whether the ID information in the second context data matches the ID information in the first context data.

[0098] In some embodiments, both the first context data and the second context data may be sent by the second user device to the host device. For example, the wireless signal may include the first context data, and the second user device then forwards this first context data (together with the second context data, or as part of the second context data) to the host device. The first context data may include ID information associated with the first user device and / or the image data. Then, based on the fact that the ID information associated with the first user device and / or the image data is included in the data from the second user device, the host device can determine that the second user device is near the first user device.

[0099] At 440, the host device analyzes the image data to identify the user in the image data. In some cases, the host device may analyze the image data based on and / or using at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data. For example, the host device may receive the image data, and may analyze and / or process the image data to determine whether a portion of the image data matches any suitable portion of facial image data stored in a database (e.g., database 140). That is, the host device may use the facial image data as a template for comparison with the image data.

[0100] In some cases, the criteria for positive facial recognition can be associated, for example, with a match of the image data to the facial image data with a predetermined and / or acceptable probability. In some embodiments, the image data includes a video stream, and the host device can be configured to analyze and / or process the video stream based at least in part on separating, parsing, classifying, and / or otherwise deconstructing the video stream into its individual frames (e.g., static images at predetermined times during the video stream). As such, the host device can compare and / or analyze data included in the video stream frames relative to the facial image data.

[0101] The facial recognition at 440 performed by the host device may be before or after the comparison of the first context data with the second context data at 430. In other words, the order of 430 and 440 may be freely arranged.

[0102] In some embodiments, the host device compares the first context data with the second context data to determine the potential presence of the user in the image data (e.g., if the user is close to the first client device). The host device then performs facial recognition on the image data to search for this particular user. If the facial recognition also returns a positive match, the host device can confirm the presence of this particular user. In this way, the host device can reduce the number of users searched in a particular media file and reduce false positives by linking the user's context data and the user's appearance with data obtained from the media. In some embodiments, the host device does not perform facial recognition to search for users (or anyone) who are not identified based on the comparison of the first context data with the second context data (e.g., from step 430). In such an embodiment, processing at the host device can be reduced. False positives associated with facial recognition analysis can also be reduced.

[0103] In other embodiments, the host device may perform facial recognition to identify possible users in the image data before comparing the context data. The host device then compares the second context data to the first context data to determine the potential presence of a user associated with the second user device, after which the host device may determine whether the user associated with the second user device is within the possible users identified from the facial recognition. If so, the host device may determine that the user is indeed present in the image data.

[0104] At 450, the host device sends image data to the user. As described herein, because the user is identified at least in part based on the context data, the image data sent to the user is also referred to as a "context image". Similarly, if the image data includes a video stream, the video stream sent to the user is referred to as a "context video stream" because the user is identified at least in part based on the context data as described herein. When the criteria associated with facial recognition of facial image data in the image data are met, the context image of the user is defined. For example, when the host device determines that at least a portion of the data in the image data meets the criteria (e.g., matching the facial image data to a predetermined and / or acceptable probability), the host device can define the context image of the user and can store the context image of the user in a database. In the case where the context image of the user is defined, the host device sends the image data to the user at 450. For example, the host device can send a signal indicating an instruction to present the context image of the user on a display of the client device (e.g., by graphically rendering the context image in an interface instantiated on the client device 150b). In some cases, the host device can send a signal to the client device via a network, which is operable to present the context image of the user on a display of the client device.

[0105] In some cases, the host device may also send a message to the first user who retrieved the image data indicating that one or more other users are identified in the image data. The host device may also suggest that the second user add the identified users as friends on one or more social media platforms.

[0106] In some cases, the host device may also request the second user to identify and / or confirm the presence of the user in the image data. For example, in the case where the host device cannot identify a person in the image data, the host device may ask the first user for suggestions of possible identities of the person. In another example, in the case where the host device identifies a person in the image data but the confidence level does not meet a criterion (e.g., is not above a threshold), the host device may request the first user to confirm the attempted identification.

[0107] In other cases, the host device may store the context image (e.g., in a database, etc.) and may be configured to retrieve the user's context image from the database in response to a request from a second user device (and / or any other suitable device). More specifically, in some embodiments, the user may manipulate the client device to access a web page on the Internet and / or log into an application. After being authenticated (e.g., entering credentials, etc.), the user may interact with the web page and / or application so that a request for accessing the context image is sent from the client device to the host device. Thus, the host device may retrieve the context image from the database and may send a signal to the client device, which may be operable to present the context image on a display (e.g., by graphically rendering the context image via the Internet and the web page) and / or otherwise allow the client device to download the image. Similar to the above, the context image may be stored on the "cloud" and accessed via a web browser and / or application and the Internet.

[0108] In other embodiments, when the context image meets the criteria (e.g., when the context image matches the user's facial image data to a predetermined probability, etc.), the host device can automatically send the context image to the user. For example, in some embodiments, the user can also operate a second user device to instantiate an application that is tracking the user's user location data. When an image capture device (e.g., such as an autonomous camera and / or another user device) records image data, the host device can determine that the image data matches the user based on facial analysis of the image data and facial image data associated with the user. The user's client device can also send location data associated with the user and the client device to the host device. The host device can use both facial analysis and location information to refine the probability that the user appears in the image data. If the probability that the user appears in the image data meets the criteria (e.g., exceeds a predetermined threshold, etc.), the host device can send the context image to the user. Alternatively, the host device can pre-filter the image data based on location information, so that the probability is calculated when the user's location information is substantially similar to the location information of the image data, and the probability is not calculated when the location data of the image data is substantially dissimilar to the location information of the user.

[0109] In other embodiments, when the image data meets the criteria (e.g., when the image data matches the user's facial image data up to a predetermined probability, etc.), the host device can store the image data and associate the image data with the user based on the user's interaction with the image data. For example, in some embodiments, the user can access an application instantiated on a client device associated with the user to search for and / or access the image data. The user can view the image data, for example, in a user profile of another user associated with that user, and / or can search for the image data to view within the interface of the application. When the user accesses the image data within the application, the application can send a signal to the host device indicating that the user is accessing the image data. The host device can automatically determine whether facial analysis of the image data has been performed based on the facial image data associated with that user, and, if the user's facial image data has not been previously compared with the image data, facial analysis of the image data can be automatically performed based on the facial image data of that user. In this way, the host device can delay processing the image data to identify the user within the image data until the user attempts to access the image data.

[0110] In some cases, a user can search for an event and "check in" to the event after the event. For example, the user can identify the event (e.g., by viewing a list of events, by viewing the location of the event on a map, etc.) and can select the event. Based on the user's selection of the event, the host device can perform facial analysis on the image data associated with that event based on the facial image data of that user. If the host device identifies (e.g., with a predetermined probability) that an image includes the user, the host device can provide the image to the user.

[0111] Although the above method 400 is described as sending and / or receiving image data, context data, etc. and presenting user-specific image data and / or sharing user-specific image data with one or more users, it should be understood that the system can be arranged so that video stream data and / or image data can be captured in any suitable manner, analyzed by any suitable device and sent to any suitable user or user device and / or shared with any suitable user or user device. For example, in some cases, a user can manipulate a user device (e.g., a client device, such as client device 150) to capture a facial image of the user. For example, a user can open a mobile application (e.g., when the user or client device is a smart phone or other mobile or wearable electronic device) and can capture a facial image (e.g., "selfie") via the camera of the client device. In other words, the user can control the camera of the client device via the application to capture a selfie. This selfie can be provided to register a user so that the application can identify the user's facial recognition data (e.g., facial features). This facial recognition data can be used to identify the user in a subsequently received video and / or image.

[0112] In some cases, a user may capture content (e.g., image data and / or video streams) via an application. As described above, the content may be an image and / or video stream of one or more people in a given context (such as, for example, one or more people at a sporting event, etc.). In some cases, a user capturing (e.g., generating) the content may be associated with contextual data (such as time, date, location, venue, event, etc.) and / or may be otherwise tagged with data and / or metadata.

[0113] In other cases, user-generated content does not need to be associated with contextual data. User-generated content (e.g., video stream data, etc.) can be analyzed via facial recognition and / or other image analysis via a client device or a host device to determine the presence of any registered user (e.g., any user whose user profile is stored in a database). If a registered user is identified in the video stream, the user, client device, and / or host device can define a user-specific video stream associated with one or more of the identified users. The user, client device, and / or host device can then determine whether to share the user-specific video stream with each identified user.

[0114] In some cases, based on user profiles and / or preferences and / or based on settings within a mobile application or account, sharing of user-specific (one or more) video streams can be automatic. In other cases, sharing of user-specific (one or more) video streams can be based on manual or other inputs from the user (e.g., based on selection, etc.). In still other cases, sharing of user-specific (one or more) video streams can be based on a peer-to-peer networking session in which each user (or each client device used in the peer-to-peer networking session) receives a user-specific video stream. In this way, user-generated content (e.g., video streams and / or image data captured by a user) can be captured, analyzed, and / or shared in a manner similar to that described herein.

[0115] Method 400 may also include a registration process for registering a user in a facial recognition database. The registration process may include receiving, at a host device and via a network from a client device (e.g., a first client device or a second client device) a signal indicating a request to register facial image data associated with a user. For example, in some embodiments, the network may be any suitable network or combination of networks, such as, for example, network 105 described above with reference to FIG. 1 . The host device may be connected to the host device 105 described above with reference to FIG. 1 . Figure 1A , Figure 1B and Figure 2 The host device 110 described above is substantially similar or identical to the host device 110 described above. Figure 3 The client device 150 described above may be substantially similar or identical to the client device 150 described above. In some cases, the client device may be configured to capture initial facial image data and may send the initial facial image data to the host device. Specifically, in some embodiments, the client device may be configured to capture one or more facial images of the user in any suitable manner. Thus, the host device may receive the facial image data from the client device and may perform any suitable processing associated with registering the user and / or the user's facial image data, etc.

[0116] The registration process also includes registering facial recognition data associated with the user and storing the facial recognition data in a database in communication with the host device. The database can be any suitable database, such as, for example, database 140 described above with reference to FIG. 1. Registration of facial recognition data can include any suitable processing, methods, and / or algorithms associated with facial recognition, such as those described above. In some cases, the host device can be configured to define user image data based on facial recognition, etc., and can store at least a portion of the user image data in a database.

[0117] In some implementations, the registration process can be performed concurrently with other steps in method 400. For example, a user of the second user device can include an image of the second user (e.g., a selfie) in the second context data. The host device can first register the second user or update the registration of the second user, and then perform the remaining steps in method 400, such as comparing the first context data to the second context data, facial recognition, and sending a context image to the second user, if applicable.

[0118] In some embodiments, method 400 may be suitable for performing image analysis without the need for the first user device to send a wireless signal and the second user device to receive the wireless signal. More specifically, the second user device may generate the second context data periodically and / or on a schedule. For example, the second user device may send the location information of the second user device to the host device (as part of the second context data), and the first context data includes the location information of the first user device. Then, the host device may compare the location information of the first user device with the location information of the second user device. If the two locations are within a predetermined value (e.g., 1 meter, 2 meters, 5 meters, 10 meters or any other suitable value), the host device may determine that the user associated with the second user device is potentially present in the image data generated by the first user device. In these embodiments, the second user device may, for example, periodically send the second location information to the host device every 0.1 second, every 0.5 second, every 1 second, every 2 seconds or any other suitable frequency.

[0119] In some embodiments, the first context data may also include first timing information associated with the image data (e.g., the time when the image was taken), and the second context data may include second timing information indicating the time when the second context data was sent to the host device. The host device may also compare this first timing information with the second timing information. If the two times are within a threshold of each other (e.g., 3 minutes, 2 minutes, 1 minute, 30 seconds, 15 seconds or less, including any values ​​and sub-ranges therebetween), then the host device may determine that the second user is potentially present in the image data.

[0120] Figure 5 is a flow chart illustrating a method 500 of using a client device including an imaging device in a video recognition system according to an embodiment. Figure 1A and Figure 1B, the method 500 is implemented by the first user device 150a in the system 100 shown in . In this method 500, at 510, the first user device 150a generates image data using an imaging device (e.g., representing a field of view). In some cases, the image data can be generated using an imaging device included in the first user device. In other cases, the image data can be generated using an imaging device operably coupled to the first user device. For example, the imaging device may include a camera at a stadium. The camera can communicate with a computing device (e.g., a computer or a phone) and transmit the acquired image data (including video) to the computing device, which in turn sends the image data to a host device.

[0121] At 520, in response to the imaging device generating image data, the first user device may control the communication interface to broadcast a wireless signal covering at least a portion of the field of view of the image data. The wireless signal includes at least a portion of the first context data associated with the image data. As described above, the first context data may include information indicating the location of the first user device, the time when the image data was acquired, ID information assigned to the image data (e.g., a serial number), ID information associated with the first user device, and / or ID information associated with an operator of the first user device, etc.

[0122] In some embodiments, the wireless signal is automatically sent when the image data is captured. In other embodiments, the operator of the first user device can control the first user device to retain the wireless signal. For example, the operator can capture an image of a scene without any person in the image, in which case the operator of the first user device can prevent and / or limit the first user device from broadcasting the wireless signal to reduce the computing load of the first computing device and / or the server.

[0123] At 530, the first user device sends the image data and the first context data associated with the image data to a server (e.g., a host device 100 in system 100) via a network, so that the server performs image analysis. The image analysis includes determining the potential presence of a user in the image data based on comparing the first context data with the second context data. The second context data is associated with a user of the second user device and is generated in response to the second user device receiving the wireless signal. The image analysis also includes analyzing the image data based on at least one of a two-dimensional facial recognition analysis, a three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data.

[0124] As described above, the second context data may include, for example, location information of the second user device, the time when the wireless signal is received from the first user device, the time when the second context data is sent, and ID information associated with the second user device and / or the second user, etc. The second context data may also include at least a portion of the information contained in the wireless signal, such as ID information of image data.

[0125] Figure 6 is a flow chart illustrating a method 600 of using a client device in a video recognition system according to an embodiment. Figure 1A-1B The second user device 150b in the system 100 shown in FIG. 6 implements the method 600. In this method 600, at 610, the second user device receives a wireless signal from the first user device, the wireless signal including first context data associated with image data generated by an imaging device associated with the first device. The first device is also configured to send the image data and the first context data to the server.

[0126] At 620, in response to receiving the first context data, the second user device sends second context data to the server, so that in response to receiving the second context data, the server performs image analysis. The image analysis includes determining a potential presence of the user in the image data based on comparing the first context data to the second context data. The image analysis also includes analyzing the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data.

[0127] In some embodiments, the second context data is automatically sent in response to receiving a wireless signal from the first user device. In other embodiments, the second user may have the option to turn off the transmission of the second context data (e.g., to protect privacy) and / or the user may provide approval before sending the second context data.

[0128] At 630, in response to the server identifying the user in the image data, the second user device receives the image data from the server. Alternatively, the second user device receives an indication that the second user is identified in one or more images (or video streams), thereby prompting the second user to review the images stored on the host device (or a database associated with the host device).

[0129] In some embodiments, the second user device receives a request to confirm the recognition performed by the host device. The second user may have the option to "unmark" the recognition (e.g., to protect privacy). In some embodiments, the second user may have the option to suggest a new recognition. For example, the second user may provide identity information of the person in the image data, and the host device may perform facial recognition again based on this identity information.

[0130] Although various embodiments have been described above, it should be understood that they are only given as examples rather than limitations. For example, although embodiments and methods are described herein as context images and / or video streams of users defined at events, etc. and context images and / or video streams are sent to client devices and / or otherwise allowed to access context images and / or video streams via, for example, web browsers and the Internet, in other embodiments, host devices can store any number of context images and / or video streams associated with users in a database. In some cases, host devices can be configured to define user profiles of any number of context images and / or video streams that can include users. In some cases, users can access his or her user profile via mobile applications, computer applications, web browsers, the Internet, etc. Moreover, in some cases, users can share or otherwise request host devices to share any number of context images and / or video streams of users with different users and / or via social media sites. In some cases, users can allow access to a portion of his or her user profile so that other users can view the context images and / or video streams included therein.

[0131] Although specific examples have been described in detail above, the embodiments and methods described herein can be used in any suitable manner. For example, although system 100 is described above as a context image and / or video stream limited to a user at a sporting event, in other embodiments, the methods described herein can be used to identify individuals using, for example, facial recognition and video analysis in any suitable setting, venue, arena, event, etc. For example, in some embodiments, the above method can be used to capture context images and / or video streams at concerts, assemblies, graduation ceremonies, gatherings, shopping malls, commercial venues, etc. In one example, a host device can receive a context video stream from, for example, a graduation ceremony. In some cases, as described above, the host device can perform any suitable facial recognition and / or video analysis to identify graduates (and / or any individual and / or user). Moreover, the host device can be configured to analyze context information, such as a user profile associated with a graduate, the order of students crossing the stage, location data associated with a client device of a graduate, and / or any other suitable data. As such, the host device may analyze the data to verify the identity of the graduate (e.g., when the data meets the criteria(s)) and may define a contextual video stream of the graduate, such as as he or she walks across the stage to receive a diploma, etc. In other cases, the host device may identify a family member or friend of the graduate and may define his or her contextual video stream in a similar manner.

[0132] Although the embodiments have been described above as being performed on a specific device and / or in a specific part of a device, in other embodiments, any embodiment and / or method described herein may be performed on any suitable device. For example, although the context image and / or video stream has been described as being sent to a host device (e.g., host device 110) for facial recognition and / or image analysis, in other embodiments, any suitable analysis may be performed on or at a client device. For example, in some cases, a user may capture a video stream (e.g., a context video stream) via a camera of a client device, and in response, the client device may analyze the video to identify any number of registered users, etc. in the video stream. In some cases, the analysis may be via a convolutional neural network sent to a client device and / or stored on a client device (e.g., stored in a memory and associated with a system application). In some cases, the analysis may be preprocessed and / or pre-sorted based on, for example, a user's contact list, a friend list, an established connection, context data, etc., as described above. In some cases, a client device may send a user-specific video stream to any identified user, as described above. In other embodiments, the client device may upload and / or send the analyzed video stream and / or the user-specific video stream(s) to host device 110 and / or database 140 .

[0133] Although the video stream and / or image data are described as "context" above, it should be understood that the video stream data and / or image data can be independent of the "context data" and / or not associated with the "context data". For example, in some cases, a user can capture a video stream and / or an image, and the video stream and / or image can be uploaded for processing without defining and / or sending context data associated with the video stream and / or image data. In some cases, a host device, etc. (e.g., host device 110) can receive a video stream and / or image data generated by a user, and in response, one or more facial recognition processes and / or any other suitable analysis can be performed on the data to define, for example, a user-specific video stream or a user-specific image that is independent of the context data. As used herein, a user-specific video stream or a user-specific image can be defined with or without the use of context data. For example, a host device can be configured to identify a user in an image by analyzing an image against the user's facial recognition data stored in a database without using context data.

[0134] Although embodiments have been particularly shown and described, it will be understood that various changes in form and detail may be made. Although various embodiments have been described as having specific features and / or combinations of components, other embodiments are possible having any combination of features and / or components from any embodiment discussed above.

[0135] In the case where the methods and / or events described above indicate specific events and / or processes that occur in a specific order, the ordering of specific events and / or processes can be modified. In addition, specific events and / or processes can be performed simultaneously with parallel processes where possible, and sequentially as described above. Although the specific method of facial recognition is described above according to a specific embodiment, in some cases, any method of facial recognition can be combined, expanded, enhanced and / or otherwise performed together on a collection of facial recognition data. For example, in some cases, the method of facial recognition can include using feature vectors, feature faces and / or other 2-D analysis and any suitable 3-D analysis (such as 3-D reconstruction of multiple 2-D images) to analyze facial recognition data. In some cases, the use of 2-D analysis methods and 3-D analysis methods can, for example, produce more accurate results with less load on resources (e.g., processing equipment) than would otherwise be derived from only 3-D analysis or only 2-D analysis. In some cases, facial recognition can be performed via convolutional neural networks (CNN) and / or via CNN in combination with any suitable two-dimensional (2-D) and / or three-dimensional (3-D) facial recognition analysis methods. Furthermore, using multiple analysis methods may be used, for example, for redundancy, error checking, load balancing, etc. In some cases, using multiple analysis methods may allow the system to selectively analyze facial recognition data sets based at least in part on the specific data included therein.

[0136] Some embodiments described herein relate to computer storage products with non-transient computer-readable media (also referred to as non-transient processor-readable media) having instructions or computer codes for performing various computer-implemented operations. Computer-readable media (or processor-readable media) are non-transient in the sense that they do not themselves include transient propagating signals (e.g., propagating electromagnetic waves that carry information on a transmission medium (such as space or cables)). The media and computer code (also referred to as code) can be codes designed and constructed for one or more specific purposes. Examples of non-transient computer-readable media include, but are not limited to, magnetic storage media (such as hard disks, floppy disks, and tapes); optical storage media (such as compact disks / digital video disks (CD / DVDs), compact disk read-only memories (CD-ROMs), and holographic devices); magneto-optical storage media (such as optical disks); carrier signal processing modules; and hardware devices (such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), read-only memories (ROMs), and random access memory (RAM) devices) that are specifically configured to store and execute program codes. Other embodiments described herein relate to computer program products, which may include, for example, instructions and / or computer codes discussed herein.

[0137] Some embodiments and / or methods described herein may be implemented by software (implemented on hardware), hardware, or a combination thereof. The hardware portion may include, for example, a general purpose processor, a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC). The software portion (implemented on hardware) may be expressed in various software languages ​​(e.g., computer code), including C, C++, Java, and the like. TM , Ruby, Visual Basic TM And / or other object-oriented, procedural or other programming languages ​​and development tools. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions (such as generated by a compiler), codes for generating web services, and files containing more advanced instructions executed by a computer using an interpreter. For example, embodiments may be implemented using imperative programming languages ​​(e.g., C, Fortran, etc.), functional programming languages ​​(Haskell, Erlang, etc.), logical programming languages ​​(e.g., Prolog), object-oriented programming languages ​​(e.g., Java, C++, etc.), or other suitable programming languages ​​and / or development tools. Other examples of computer code include, but are not limited to, control signals, encrypted codes, and compressed codes.

Claims

1. A device comprising: Memory; a communication interface in communication with the memory and configured to communicate via a network; as well as a processor in communication with the memory and the communication interface, the processor configured to receive image data from an imaging device via the network and the communication interface and to receive first context data associated with the image data, the image data comprising at least one image of a field of view of the imaging device, The processor is configured to receive, via the network and the communication interface, second context data from a user device different from the imaging device, the second context data being associated with a user of the user device, the second context data being automatically generated in response to the user device receiving a wireless signal within the field of view, the wireless signal being transmitted by an antenna operatively coupled to the imaging device in response to the imaging device generating image data, The processor is configured as: determining a potential presence of a user associated with the user device in the image data based on comparing the first context data to the second context data; analyzing the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data, and Image data depicting the user is sent to a user device.

2. The device according to claim 1, wherein: Wireless signals include Bluetooth signals.

3. The device according to claim 1, wherein: The wireless signals include directional wireless signals toward the field of view.

4. The device according to claim 1, wherein: The image data includes video stream data.

5. The device according to claim 1, wherein: The processor is configured as: analyzing the image data in response to receiving the second contextual data to search the image data for the user, and Based on comparing the first context data with the second context data, a presence of the user in the image data is confirmed.

6. The device of claim 1, wherein: The first context data includes a first timestamp associated with the image data, The second context data includes a second timestamp associated with receipt of the wireless signal by the user equipment, and The processor is configured to determine a potential presence of the user when a difference between the first timestamp and the second timestamp is within a threshold.

7. The device of claim 1, wherein: The first context data includes a first geographic location associated with the image data, The second context data includes a second geographic location associated with the user device, and The processor is configured to determine a potential presence of the user if a difference between the first geographic location and the second geographic location is within a threshold.

8. The apparatus of claim 1, wherein: The first context data includes first identification information associated with the image data and / or the imaging device; The second context data includes second identification information associated with the image data, the user and / or the user device, and The processor is configured to determine the potential presence of the user by comparing the first identification information with the second identification information.

9. The apparatus of claim 1, wherein: a memory configured to store a plurality of user profile data structures, each user profile data structure from the plurality of user profile data structures comprising facial recognition data for an associated user; The second context data includes identification information associated with the user device and / or the user, and The processor is configured to: (i) retrieve a user profile data structure based on identification information associated with a user device, the user profile data structure from the plurality of user profile data structures and associated with the user; and (ii) identifying the user in the image data based on comparing the facial recognition data in the user profile data structure with the image data.

10. The device of claim 1, wherein: The processor is also configured to: receiving a facial image from the user, generating facial recognition data using the facial image based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or CNN, The user is identified in the image data based on comparing the facial recognition data to the image data.

11. An apparatus comprising: an imaging device configured to generate image data representative of a field of view of the imaging device; a communication interface in communication with the imaging device and configured to communicate via a network; as well as a processor in communication with the imaging device and the communication interface, In response to the imaging device generating image data, the processor is configured to: controlling the communication interface to automatically broadcast a wireless signal covering at least a portion of the field of view, the wireless signal including at least a portion of the first context data associated with the image data, and controlling the communication interface to send the image data and the first context data associated with the image data to the server via the network, so that in response to receiving the image data, the server: determining a potential presence of a user associated with the user device in the image data based on comparing the first context data with second context data associated with a user of the user device received from a user device different from the imaging device, the second context data automatically generated in response to receipt of a wireless signal by the user device within the field of view; as well as The image data is analyzed based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data.

12. The device of claim 11, wherein: The image data includes video stream data.

13. The device of claim 11, wherein: Wireless signals include Bluetooth signals.

14. The device of claim 11, wherein: The wireless signal is a directional wireless signal towards the field of view.

15. The apparatus of claim 11, wherein: The first context data includes a first timestamp associated with the image data, The second context data includes a second timestamp associated with receipt of the wireless signal by the user equipment, and The processor is configured to control the communication interface to send the image data and first context data associated with the image data to the server, so that when a difference between the first timestamp and the second timestamp is within a threshold, the server determines a potential presence of the user.

16. The apparatus of claim 11, wherein: The first context data includes a first geographic location associated with the image data, The second context data includes a second geographic location associated with the user device, and The processor is configured to control the communication interface to send the image data and first context data associated with the image data to the server, such that if a difference between the first geographic location and the second geographic location is within a threshold, the server determines a potential presence of the user.

17. The apparatus of claim 11, wherein: The first context data includes first identification information associated with the image data and / or the imaging device; The second context data includes second identification information associated with the image data, the user and / or the user device, and The processor is configured to control the communication interface to send the image data and first context data associated with the image data to the server, so that the server determines the potential presence of the user by comparing the second identification information with the first identification information.

18. The apparatus of claim 11, wherein: The second context data includes identification information associated with the user device and / or the user, and The processor is configured to control the communication interface to send the image data and first context data associated with the image data to the server, so that the server: (i) retrieves a user profile data structure based on the identification information, the user profile data structure being from a plurality of user profile data structures and including facial recognition data associated with the user; and (ii) identifying the user in the image data based on comparing the facial recognition data in the user profile data structure with the image data.

19. An apparatus comprising: Communication interface; as well as A processor, the processor being in communication with the communication interface, the processor being configured to: controlling the communication interface to receive a wireless signal from a user device, the wireless signal comprising first context data associated with image data automatically generated by an imaging device associated with the user device, the user device being further configured to send the image data and the first context data to a server, the user device broadcasting the wireless signal covering at least a portion of a field of view of the imaging device, and In response to receiving the wireless signal, automatically generate second context data associated with a user of the user device and control the communication interface to send the second context data to the server, so that in response to receiving the second context data, the server: determining a potential presence of a user associated with the user device in the image data based on comparing the first context data to the second context data; analyzing the image data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or a convolutional neural network (CNN) to identify the user in the image data, and The processor is configured to receive image data from the server and via the communication interface in response to the server identifying the user in the image data.

20. The device of claim 19, wherein: Wireless signals include Bluetooth signals.

21. The apparatus of claim 19, wherein: The first context data includes a first timestamp associated with the image data, The second context data includes a second timestamp associated with receipt of the wireless signal from the user equipment, and The processor is configured to control the communication interface to send second context data to the server, so that in response to receiving the second context data, the server determines a potential presence of the user when a difference between the first timestamp and the second timestamp is within a threshold.

22. The apparatus of claim 19, wherein: The first context data includes a first geographic location associated with the image data, The second context data includes a second geographic location associated with the user device, and The processor is configured to control the communication interface to send second context data to the server, such that in response to receiving the second context data, the server determines a potential presence of the user if a difference between the first geographic location and the second geographic location is within a threshold.

23. The apparatus of claim 19, wherein: The first context data includes first identification information associated with the image data and / or the user device; The second context data includes second identification information associated with the image data, the user and / or the user device, and The processor is configured to control the communication interface to send second context data to the server, so that in response to receiving the second context data, the server determines the potential presence of the user by comparing the second identification information with the first identification information.

24. The apparatus of claim 19, wherein: The second context data includes identification information associated with the user and / or the user device, The processor is configured to control the communication interface to send second context data to the server, so that in response to receiving the second context data, the server (i) retrieves a user profile data structure associated with the user from a memory storing a plurality of user profile data structures based on the identification information, each user profile data structure including facial recognition data of the associated user, and (ii) identifies the user in the image data based on comparing the facial recognition data in the user profile data structure with the image data.

25. The apparatus of claim 19, wherein: The processor is configured to send a facial image of the user to a server, so that the server (i) generates facial recognition data based on at least one of two-dimensional facial recognition analysis, three-dimensional facial recognition analysis, or CNN, and (ii) identifies the user in the image data based on comparing the facial recognition data with the image data.

Images