Using segmented image security

Abstract

Systems and methods can be used for access control. These systems and methods can include: using a data processing system to access a video stream, the video stream including an image, the image including a virtual background; segmenting the image into a foreground portion and a background portion to determine whether the foreground portion or the background portion of the image meets a threshold requirement; and outputting an alert in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement.

Description

[0001] Priority Declaration

[0002] This application claims priority from Luxembourg patent application No. LU102082, filed on September 25, 2020, which is incorporated herein by reference in its entirety. Background Art

[0003] Security requirements are constantly evolving. Whether driven by increased data or granularity requirements or physical security needs, access authorization is a crucial component of security. Physical barriers (e.g., locked doors), usernames, passwords, and keycodes have varying degrees of success in preventing security breaches. Attempts to circumvent or override these security measures can result in unauthorized access to data or physical spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In the accompanying drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals with different letter suffixes may represent different instances of similar components. The accompanying drawings generally illustrate various embodiments discussed in this document by way of example and not limitation.

[0005] Figure 1 An exemplary access control system according to some examples of the present disclosure is illustrated.

[0006] Figure 2 An access control device and user interface according to some examples of the present disclosure are illustrated.

[0007] Figure 3 Illustrated are exemplary techniques for background replacement of an image or video according to some examples of the present disclosure.

[0008] Figure 4 Illustrated are exemplary techniques for video or image replacement according to some examples of the present disclosure.

[0009] Figure 5 A flow chart illustrating a technique for detecting an image or video that is substantially free of foreground regions, according to some examples of the present disclosure.

[0010] Figure 6 A flow chart illustrating a technique for detecting pre-recorded images or videos according to some examples of the present disclosure is illustrated.

[0011] Figure 7 Illustrated is a block diagram of an example machine that can implement one or more of the techniques discussed herein, according to some examples of the present disclosure. DETAILED DESCRIPTION

[0012] Systems and methods for providing access control via a security system are described herein. Access to digital or physical resources can be controlled using the security techniques and systems described herein. For example, digital access to data, services, applications, real-time online events, and the like can be controlled. Physical access to spaces, rooms, buildings, filing cabinets, safes, and the like can be controlled. In some examples, access to a combination of digital and physical assets can be controlled, such as physical access to a computer and digital access to data on the computer.

[0013] In an example, these systems and methods can be used to determine whether a live person is present in a video or image submitted for access. For example, the systems and methods described herein can determine whether a video stream is live or pre-recorded. In another example, the systems and methods described herein can determine whether a foreground (e.g., including a person) is present in an image. In various examples, an alert can be provided indicating that an unauthorized access attempt has occurred.

[0014] In an example, the systems and methods described herein can be used to prevent access to controlled assets when the access attempt is unauthorized. In response to detecting an unauthorized access attempt, various actions can be automatically performed. For example, the automatic action can include an automated phone call, an automated text message, an automated email message, blocking incoming video at a user device, blocking outgoing video at a user device, disabling voting entry from a user device, blocking access to a secure physical location, blocking access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, requiring additional information (e.g., a password to be entered), etc.

[0015] The systems and methods described herein address technical problems related to security verification, including detecting unauthorized access to secure information or secure locations. Access control and security are critical to modern infrastructure, and attempts to circumvent current security measures are ongoing. Some circumvention efforts rely on manipulating video streams or captured images. One exemplary technical solution described herein provides a method for determining whether a video stream is pre-recorded or live based on identifying the source type of the video stream. Another exemplary technical solution described herein provides a technique for determining whether a foreground portion of an image meets a threshold requirement based on segmentation of the image or a portion of the image.

[0016] Figure 1Exemplary access control systems 104 and 108 according to some examples of the present disclosure are illustrated. Access control system 104 is used for the security of physical assets 102 (e.g., doors, rooms, safes, file cabinets, buildings, etc.), and access control system 108 is used for the security of digital assets 106 (e.g., data, services, applications, real-time online events, video conferences, etc.). User 110 may attempt to access digital assets 106 or physical assets 102. In an example, user 110 may attempt to circumvent the access controls of system 104 or 108.

[0017] Security systems such as access control systems 104 or 108 can use the security techniques described herein to prevent unauthorized access to physical assets 102 or digital assets 106, respectively. The security techniques can include determining whether a video stream is pre-recorded or live based on identifying the source type of the video stream, or determining whether a foreground portion of an image meets a threshold requirement based on segmentation of the image or a portion of the image.

[0018] In an example, a video stream or image is used to verify the identity of user 110 to allow access when authorized. A video stream or image can be used to verify that user 110 exists, is a specific person, is performing a gesture, etc. However, a video stream or image may be corrupted by an unauthorized user. For example, user 110 can pre-record a video and submit the pre-recorded video to replace the real-time video. In this example, unauthorized attempted access can be identified based on the source type of the pre-recorded video (e.g., not a camera or hardware component, but a video file, application source, or software component). In another example, a video stream or image can include a background portion of an image that appears to include user 110. The background portion can replace the entire or most of the video stream or image in which user 110 appears to be present. In this way, user 110 can appear to be in a real-time video or image with a replaced background, such as in a real-time video conferencing session, but can be missing from the foreground portion of the video stream or image.

[0019] Figure 2 An access control device 200 having a user interface 202 is illustrated according to some examples of the present disclosure. The access control device 200 may include or be communicatively coupled to a camera 204. The access control device 200 may store or access a video or background source 210 (e.g., a video or image stored on a storage device such as a database, long-term storage, short-term storage, etc.). The camera 204 may be used to capture the video or image.

[0020] User interface 202 shows an image (which may be an image of a video stream) of user 206 (who may be in the foreground or background portion of the image) and a reference image 208 as part of a virtual background. In the case of authorized use or access, user 206 is in the foreground portion of the image, and the image or video stream is captured and output in real time by camera 204. In the case of unauthorized use or access, user 206 is in the background portion of the image or the image is pre-recorded (including optionally pre-recorded by camera 204 rather than in real time). In the case of unauthorized use or access, pre-recorded video can be accessed from video or background source 210 to replace or substitute the real-time image or video stream from camera 204. The pre-recorded video in the case of unauthorized use or access can be used as a virtual background to replace the background of the real-time image or video captured by camera 204.

[0021] In an exemplary scenario, a student participating in a live video conferencing session may be user 206. Unauthorized use or access includes a student attempting to skip an online course or test without the teacher noticing. The student may pre-record a video that includes a representation of user 206 and inject the pre-recorded video into a later live video conferencing session, either replacing the live feed from camera 204 or as a virtual background. In the virtual background example, the user may point camera 204 at a wall or any background that does not feature movement, people, animals, or other foreground objects. In this example, the virtual background replaces all or substantially all of the live video stream, making it appear to viewers of the live video conferencing session that user 206 is in the image, even though user 206 is not present in the image.

[0022] Such unauthorized use or access can be detected based on the source type of the video stream (e.g., a virtual web camera appears in a device list, a connection port is identified, a tag is included in the video stream metadata, etc.) or based on segmentation of the image in the video stream. The techniques for detecting unauthorized use or access can be run together, in parallel or serially, or individually. The video conferencing application can perform one or more of the techniques described.

[0023] Upon detecting unauthorized use or access, an alert can be issued (e.g., to the teacher in the example above). For example, the video conferencing application can mark the video stream as potentially compromised or untrusted, such as on the receiver device. For example, the marking can be an alert triangle icon with explanatory tooltip text or a pop-up window that appears next to the video stream on the receiver device.

[0024] In another example, unauthorized use or access can be detected based on segmentation. Background replacement functionality is used in many real-time video conferencing applications, which allows users to select an image or video as a background. AI-driven segmentation algorithms can analyze images (e.g., frames of a video stream) to look for people on the scene, for example, in a separate stand-alone security application or in a local real-time video conferencing application. The foreground and background portions are segmented and compared to a threshold requirement. The threshold requirement may include a requirement for a minimum percentage of pixels or area of the image in the foreground portion or a maximum percentage of pixels or area of the image in the background portion. In another example, the threshold requirement may require the presence of a foreground, or a minimum percentage or number of pixels or area in a particular portion of the image (e.g., in the center portion of the image, such as the middle third of the image).

[0025] In this example, segmentation can be performed by a deep neural network. The exemplary algorithm takes a color image as input and outputs a black and white segmentation mask of the same resolution. Each pixel is classified as foreground or background. The foreground is composited on top of the background image.

[0026] Returning to the student example, the student can fake the presence of user 206 by pre-recording a photo or video to use as a virtual background. Then, by pointing the camera 204 at an empty area, such as a wall, the virtual background is used to replace the entire captured video. In this example, segmentation techniques can be used to determine that there is no foreground or substantially no foreground (e.g., 90% background) in the image of the output video, or when the segmentation mask is empty or substantially empty, for example.

[0027] When the segmentation technology detects that the threshold requirement is not met, an alarm can be output or another mitigation action can be automatically performed. For example, a sign or an alarm can indicate that the segmentation algorithm has not detected a person. The sign or an alarm can be sent to the user interface of the teacher. The mitigation action can include the action of the automatic service configured by the host, which can be personalized for the host or include default settings. In an example, the automatic service configured by the host can include an automatic phone call, an automatic text message or an automatic email message (for example, to a security officer, to a teacher, to a boss, to a parent, to a user 206, to a student, etc.). The automatic service configured by the host can include blocking incoming video at the access control device 200 (which can include the user device of the user 206 or the security device used by the user 206 to attempt unauthorized access). In an example, the automatic service configured by the host can include blocking outgoing video (for example, blocking video from leaving the access control device 200) or blocking other outputs from the access control device 200 (for example, to prevent further attacks). In some examples, the automatic service configured by the host can include prohibiting voting input from the user 206, for example, by blocking the voting component on the access control device 200 or blocking the certificate of the user 206.

[0028] In the event of an unauthorized attempt to access a physical resource, the host-configured automated services may include preventing access to a secure physical location (e.g., a room, building, filing cabinet, safe, closet, locker, vehicle, garage, etc.), blocking access to the access control device 200 (e.g., physically, such as with a door or covering), access to all or part of the access control device 200, or revoking access to the secure physical location. The user 206 may be physically prevented from entering or engaging with the secure physical resource, or the user 206 may be prevented from using or accessing the secure physical resource (e.g., starting a vehicle). In the event of an unauthorized attempt to access a digital resource, the host-configured automated services may include preventing access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, etc. In the case of physical or digital resources, the host-configured automated services may include requiring additional information, such as a password, two-factor authentication, a one-time code, restarting a security protocol, etc.

[0029] Figure 3An exemplary block diagram illustrating authorized and unauthorized background replacement for an image or video according to some examples of the present disclosure is illustrated. Authorization technique 300A includes capturing an image 302 (which may include a frame of a video stream), segmenting the video into a captured background portion 304 and a captured foreground portion 306. Virtual background portions 308 are then identified, and these portions are used along with the previously segmented captured foreground portion 306 to create a composite output image 312. When segmented, the composite output image 312 will identify the foreground portion 306 with the user, as well as the background portion 308. Alternatively, the previous steps with the background portion 308 and the foreground portion 306 can be replicated from the composite output image 312.

[0030] Unauthorized technique 300B may include capturing image 302 (or omitting this step), but then replacing the entire image 302 with a virtual background image 314 that includes the user. A composite output image 316 is generated, but does not include the foreground portion because the entire image 302 is replaced with virtual background image 314, not just the background portion of image 302. Although composite output image 312 and composite output image 316 appear identical (e.g., to a viewer or user of an access control system), they differ when segmented. For example, when composite output image 312 is segmented, background portion 308 and foreground portion 306 can be identified. However, when composite output image 316 is segmented, the entire image (or substantially the entire image, subject to potential artifacts) appears as background (or foreground, depending on the segmentation algorithm used). Therefore, composite output image 316 can be distinguished from composite output image 312 to identify unauthorized access attempts in technique 300B and authorized access attempts in technique 300A, respectively.

[0031] Figure 4 An exemplary block diagram illustrating authorized and unauthorized video or image replacement according to some examples of the present disclosure is illustrated. Authorized technique 400A includes capturing video or images 402 using a physical camera and optionally replacing a background image 404 before output. However, in unauthorized technique 400B, no captured video output 406 is sent. Instead, unauthorized technique 400B includes accessing a pre-recorded image or video 408 from a database or other storage device, optionally replacing the captured video or captured image 410 with the pre-recorded video or image (in other examples, the camera does not capture an image or video), and outputting the pre-recorded video or image 412.

[0032] Unauthorized technology 400B can be identified as including real-time or pre-recorded images or videos based on the source (e.g., camera 402 or storage device 408). When the source is camera 402 (which can include a phone or computer's built-in camera, a camera accessed via a connection port such as USB, or a remote camera), the image or video can be identified as real-time. When the source is storage device 408 (or otherwise relies on a software source, such as a video playback application), the image or video can be identified as pre-recorded.

[0033] A source can be identified based on its type (e.g., hardware or software, camera or storage device, etc.). The source type can be identified in the metadata of an image or video. The source type can be identified based on the source's connection type (e.g., a physical connection port can correspond to a live image or video, while a software or internal connection can correspond to a pre-recorded image or video). The metadata can include the source's driver information, a camera name or logo identifying the source, and the source's connection type. In an example, a video can be identified as live or pre-recorded based on a comparison of the source (e.g., identified in the metadata) with a list of allowed devices or connection types. When the source device or connection type is on the allowed list, the image or video can be identified as live (e.g., authorized). When the source device or connection type is not on the allowed list, the image or video can be identified as pre-recorded (e.g., unauthorized). In another example, instead of an allowed list, a forbidden list can be used, where a device or connection type appearing on the forbidden list indicates that the image or video is pre-recorded. The source type can be identified based on detected patterns of disk or processor activity.

[0034] Figure 5 A flow chart illustrating a technique for detecting an image or video that is substantially free of foreground regions according to some examples of the present disclosure is shown. The technique 500 may be performed using processing circuitry, for example, one or more processors of a device such as a computer, laptop, mobile device, or the like (e.g., as described below with respect to Figure 7 Detailed discussion further below.) For example, technique 500 can be performed by a data processing system, such as a user device (e.g., a phone, laptop, desktop computer, etc.) or a server. Technique 500 can be used to prevent or alert on unauthorized access or use of assets such as physical locations (e.g., rooms, filing cabinets, safes, areas, etc.), digital locations (e.g., websites, secure servers, etc.), services, data, etc.

[0035] Technique 500 includes an operation 510 of accessing or providing a video stream including an image including a virtual background. The video stream can be recorded or output during a real-time video conferencing session. The real-time video conferencing session can include an online video call with multiple participants, including video or audio connections.

[0036] Technique 500 includes an operation 520 for segmenting an image into a foreground portion and a background portion. Segmenting the image may include using a segmentation technique used by an application running a real-time video conferencing application, which is also used to apply a virtual background to a video stream for use in a real-time video conferencing application. In an example, segmenting the image includes using a machine learning model, such as a deep learning model, a neural network, etc. Operation 520 may include generating a segmentation mask including the foreground portion and the background portion.

[0037] Technique 500 includes a decision operation 530 to determine whether a portion of an image (e.g., a foreground portion or a background portion) meets a threshold requirement. When the portion of the image meets the threshold requirement, technique 500 can return to operation 510 or can end. When the portion of the image does not meet the threshold requirement, technique 500 can proceed to operation 540. The threshold requirement can be associated with the classification of pixels in the image. For example, pixels in the image can be classified as foreground pixels or background pixels. The threshold requirement can correspond to a percentage, number, or sampling of pixels in the foreground portion or background portion of the image. For example, the threshold requirement may not be met when the sampled pixels or 95% of the pixels in the image (sampling can occur randomly or according to a sampling scheme, such as one in every 4 or 16 square pixels in the image) are background pixels.

[0038] In an example, technique 500 may include determining that a participant viewing in a video stream is looking away from a camera capturing the video stream, and turning off the camera in response to determining that the participant is looking away from the camera.

[0039] Technique 500 includes an operation 540 of outputting an alert in response to determining that a foreground portion or a background portion of the image does not meet the threshold requirement. The alert may indicate unauthorized access or use. In an example, segmenting the image includes segmenting multiple images of the video stream, and the alert may be output in response to determining that a corresponding foreground portion or a background portion of at least a portion of the multiple images does not meet the threshold requirement. Operation 540 may include generating a warning indication on a user interface of the user device, the warning indication indicating that a participant associated with generating the video stream is not present in the real-time video conference. In an example, the alert may be provided for display on a tenant dashboard.

[0040] In response to determining that the portion of the image does not meet the threshold requirement, technique 500 may include an automatic action. The automatic action may include activating a host-configured automatic service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automatic service including at least one of the following: an automatic phone call, an automatic text message, an automatic email message, blocking incoming video at the user device, blocking outgoing video at the user device, disabling voting input from the user device, preventing access to a secure physical location, preventing access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, requesting additional information, and the like.

[0041] Figure 6 A flow chart illustrating a technique for detecting pre-recorded images or videos according to some examples of the present disclosure is shown. The technique 600 may be performed using one or more processors of a device such as a computer, laptop, mobile device, etc. (e.g., as described below with respect to Figure 7 Detailed discussion further below.) For example, technique 600 can be performed by a data processing system, such as a user device (e.g., a phone, laptop, desktop computer, etc.) or a server. Technique 600 can be used to prevent or alert on unauthorized access or use of assets such as physical locations (e.g., rooms, filing cabinets, safes, areas, etc.), digital locations (e.g., websites, secure servers, etc.), services, data, etc.

[0042] Technique 600 includes an operation 610 of identifying a video stream. Identifying the video stream may include accessing or providing the video stream. The video stream may be recorded or output during a real-time video conferencing session. The real-time video conferencing session may include an online video call with multiple participants, including video or audio connections. In other examples, the video stream may be used in conjunction with a security access requirement system.

[0043] Technique 600 includes an operation 620 of determining an identification of a source type for a video stream, the source type corresponding to the type of source generating the video stream. The identification can be based on information extracted from metadata associated with the video stream. In an example, when the source type corresponds to a camera, the source type can indicate that the video stream is live, while when the source type corresponds to a video playback application, the source type can indicate that the video stream is pre-recorded. The source type identification can be determined based on at least one of driver information in the metadata of the video stream, a camera name or logo in the metadata of the video stream, and a connection type of the source, by comparing the information extracted from the metadata with a list of allowed devices, etc.

[0044] Technique 600 includes a decision operation 630 to determine whether the video stream is real-time or pre-recorded based on the type of source type determined. When the type of source type determined indicates that the video stream is real-time, technique 600 may return to operation 610 or may end. When the type of source type determined indicates that the video stream is pre-recorded, technique 600 may proceed to operation 640. In an example, when the connection type of the source is a physical connection port (e.g., via a USB connection, an HDMI connection, etc.), it may be determined that the video stream is real-time.

[0045] Technique 600 includes an operation 640 of outputting an alert to a device in response to determining that the source type indicates that the video stream is pre-recorded. The device providing the alert may include a host device of a live video conferencing session, a tenant dashboard, a server, a service, etc. The alert may indicate unauthorized access or use. In an example, technique 600 may include determining that a participant viewing the video stream is looking away from a camera capturing the video stream, and turning off the camera in response to determining that the participant is looking away from the camera.

[0046] Operation 640 may include generating a warning indication on a user interface of the user device. The warning indication may indicate that the video stream is pre-recorded. In another example, the warning indication may indicate that the participant associated with generating the video stream is not present in the real-time video conference.

[0047] In response to determining that the portion of the image does not meet the threshold requirement, technique 600 may include an automatic action. The automatic action may include activating a host-configured automatic service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automatic service including at least one of the following: an automatic phone call, an automatic text message, an automatic email message, blocking incoming video at the user device, blocking outgoing video at the user device, disabling voting input from the user device, preventing access to a secure physical location, preventing access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, requesting additional information, and the like.

[0048] Figure 7 The block diagram illustrates an exemplary machine 700 that can implement one or more of the techniques (e.g., methodologies) discussed herein according to some examples of the present disclosure. In some embodiments, the machine 700 may operate as a standalone device or may be connected (e.g., using a network) to other machines. In a networked deployment, the machine 700 may operate in the capacity of a server machine, a client machine, or both in server-client network environment. The machine 700 may be configured to perform Figure 5 or Figure 6 The machine 700 may be configured to provide Figure 1 or Figure 2 In one embodiment, the machine 700 can be a user interface. In an example, the machine 700 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 700 can be a user device, a remote device, a second remote device, or other device, and can take the form of a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or a bridge, or any machine capable of executing instructions (sequential or otherwise) specifying actions to be taken by the machine. Furthermore, while a single machine is illustrated, the term “machine” should also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), or other computer cluster configurations.

[0049] Examples as described herein may include or operate on logic or multiple components, modules, or mechanisms (hereinafter referred to as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In an example, circuits may be arranged as modules in a specified manner (e.g., internally or relative to external entities such as other circuits). In an example, all or part of one or more computer systems (e.g., standalone machines, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) to operate as modules that perform specified operations. In an example, the software may reside on a machine-readable medium. In an example, when executed by the underlying hardware of the module, the software causes the hardware to perform the specified operations.

[0050] Thus, the term "module" is understood to include a tangible entity, i.e., an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., provisionally) configured (e.g., programmed) to operate in a particular manner or to perform some or all of any of the operations described herein. With reference to examples of temporarily configured modules, each module need not be instantiated at any one time. For example, where the modules include a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one instance in time and to constitute a different module at a different instance in time.

[0051] The machine (e.g., a computer system) 700 may include a hardware processor 702 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 704, and a static memory 706, some or all of which may communicate with each other via an interconnect (e.g., a bus) 708. The machine 700 may also include a display unit 710, an alphanumeric input device 712 (e.g., a keyboard), and a user interface (UI) navigation device 714 (e.g., a mouse). In an example, the display unit 710, the input device 712, and the UI navigation device 714 may be a touch screen display. The machine 700 may also include a storage device (e.g., a drive unit) 716, a signal generating device 718 (e.g., a speaker), a network interface device 720, and one or more sensors 721, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 700 may include an output controller 728, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), high-definition multimedia interface (HDMI), etc.) connection, to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0052] The storage device 716 may include a machine-readable medium 722 on which is stored one or more sets of data structures or instructions 724 (e.g., software) used by any one or more of the techniques or functions described herein. The instructions 724 may also reside, completely or at least partially, within the main memory 704, static storage 706, or hardware processor 702 during execution by the machine 700. In an example, one or any combination of the hardware processor 702, main memory 704, static storage 706, or storage device 716 may constitute a machine-readable medium.

[0053] Although the machine-readable medium 722 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (eg, a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 724 .

[0054] The term "machine-readable medium" may include any medium that can store, encode, or carry instructions for execution by the machine 700 and cause the machine 700 to perform any one or more of the techniques of the present disclosure, or any medium that can store, encode, or carry data structures used by or associated with these instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, the machine-readable medium may be a non-transitory machine-readable medium. In some examples, the machine-readable medium may include a machine-readable medium that is not a transitory propagation signal.

[0055] The instructions 724 may also be sent or received over the communication network 726 via the network interface device 720 using a transmission medium. The machine 700 may communicate with one or more other machines using any of a number of transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network called The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as 16 family of standards), IEEE 802.15.4 family of standards, Long Term Evolution (LTE) family of standards, Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, and the like. In an example, the network interface device 720 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the communication network 726. In an example, the network interface device 720 may include multiple antennas to communicate wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. In some examples, the network interface device 720 may use multi-user MIMO technology for wireless communication.

[0056] Example 1 is a method performed by a data processing system, comprising: using processing circuitry of the data processing system to access a video stream, the video stream including an image, the image including a virtual background; segmenting the image into a foreground portion and a background portion to determine whether the foreground portion or the background portion of the image meets a threshold requirement, the threshold requirement being associated with a classification of pixels in the image; and outputting an alert in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the alert indicating unauthorized access or use.

[0057] In Example 2, the subject matter of Example 1 includes: wherein segmenting the image includes using segmentation technology of an application running a real-time video conference, the segmentation technology also used to apply the virtual background to the video stream for use in the real-time video conference.

[0058] In Example 3, the subject matter of Examples 1-2 includes: wherein segmenting the image includes using a deep learning model.

[0059] In Example 4, the subject matter of Examples 1-3 includes: wherein segmenting the image includes determining that a participant capable of viewing in the video stream is looking away from a camera capturing the video stream, and turning off the camera in response to determining that the participant is looking away from the camera.

[0060] In Example 5, the subject matter of Examples 1-4 includes: wherein segmenting the image comprises generating a segmentation mask including the foreground portion and the background portion.

[0061] In Example 6, the subject matter of Examples 1-5 includes: wherein segmenting the image includes segmenting multiple images of the video stream, and wherein the alarm is output in response to determining that the corresponding foreground portion or background portion of at least a portion of the multiple images does not meet the threshold requirement.

[0062] In Example 7, the subject matter of Examples 1-6 includes: wherein outputting the alert includes generating a warning indication on a user interface of a user device, the warning indication indicating that a participant associated with generating the video stream is not present within the live video conference.

[0063] In Example 8, the subject matter of Examples 1-7 includes: activating a host-configured automatic service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automatic service including at least one of the following: an automatic telephone call, an automatic text message, an automatic email message, blocking incoming video at a user device, blocking outgoing video at a user device, disabling voting input from a user device, blocking access to a secure physical location, blocking access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, or requesting additional information.

[0064] In Example 9, the subject matter of Examples 1-8 includes: wherein outputting the alert comprises providing the alert for display on a tenant dashboard.

[0065] In Example 10, the subject matter of Examples 1-9 includes: wherein the video stream is pre-recorded and output during a real-time video conferencing session.

[0066] Example 11 is a data processing system comprising: a processing circuit; and a memory comprising instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for: accessing a video stream, the video stream comprising an image, the image comprising a virtual background; segmenting the image into a foreground portion and a background portion to determine whether the foreground portion or the background portion of the image meets a threshold requirement, the threshold requirement being associated with a classification of pixels in the image; and outputting an alert in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the alert indicating unauthorized access or use.

[0067] In Example 12, the subject matter of Example 11 includes: wherein, for said segmenting the image, said processing circuit is further caused to use a segmentation technique of an application running a real-time video conference, said segmentation technique also being used to apply said virtual background to said video stream for use in said real-time video conference.

[0068] In Example 13, the subject matter of Examples 11-12 includes: wherein, to segment the image, the processing circuit is further caused to use a deep learning model.

[0069] In Example 14, the subject matter of Examples 11-13 includes: wherein, in order to segment the image, the processing circuit is further caused to determine that a participant who can be viewed in the video stream is looking away from a camera capturing the video stream, and turn off the camera in response to determining that the participant is looking away from the camera.

[0070] In Example 15, the subject matter of Examples 11-14 includes wherein, to segment the image, the processing circuitry is further caused to generate a segmentation mask comprising the foreground portion and the background portion.

[0071] In Example 16, the subject matter of Examples 11-15 includes: wherein, in order to segment the image, the processing circuit is also caused to segment multiple images of the video stream, and wherein the alarm is output in response to determining that the corresponding foreground portion or background portion of at least a portion of the multiple images does not meet the threshold requirement.

[0072] In Example 17, the subject matter of Examples 11-16 includes: wherein, to output the alert, the processing circuit is further caused to generate a warning indication on a user interface of the user device, the warning indication indicating that a participant associated with generating the video stream is not present in the live video conference.

[0073] In Example 18, the subject matter of Examples 11-17 includes: wherein the processing circuit is further caused to activate a host-configured automatic service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automatic service including at least one of the following: an automatic telephone call, an automatic text message, an automatic email message, blocking incoming video at the user device, blocking outgoing video at the user device, disabling voting input from the user device, blocking access to a secure physical location, blocking access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, or requesting additional information.

[0074] In Example 19, the subject matter of Examples 11-18 includes, wherein, to output the alert, the processing circuitry is further caused to provide the alert for display on a tenant dashboard.

[0075] In Example 20, the subject matter of Examples 11-19 includes: wherein the video stream is pre-recorded and output during a real-time video conferencing session.

[0076] Example 21 is a method performed by a data processing system, comprising: identifying a video stream; determining, using processing circuitry of the data processing system, an identification of a source type of the video stream based on information extracted from metadata associated with the video stream, the source type corresponding to a type of source generating the video stream; determining, using the processing circuitry of the processing system, whether the video stream is real-time or pre-recorded based on the determined type of source type; and in response to determining that the source type indicates that the video stream is pre-recorded, outputting an alert to a device, the alert indicating unauthorized access or use.

[0077] In Example 22, the subject matter of Example 21 includes: wherein, when the source type corresponds to a camera, the source type indicates that the video stream is real-time, and when the source type corresponds to a video playback application, the source type indicates that the video stream is pre-recorded.

[0078] In Example 23, the subject matter of Examples 21-22 includes: wherein outputting the alert comprises generating a warning indication on a user interface of the user device indicating that the video stream is pre-recorded.

[0079] In Example 24, the subject matter of Examples 21-23 includes: activating a host-configured automatic service in response to determining that the video is pre-recorded, the host-configured automatic service comprising at least one of: an automatic telephone call, an automatic text message, an automatic email message, blocking incoming video at the user device, blocking outgoing video at the user device, disabling voting input from the user device, preventing access to a secure physical location, preventing access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, or requesting additional information.

[0080] In Example 25, the subject matter of Examples 21-24 includes: wherein outputting the alert comprises providing the alert for display on a tenant dashboard.

[0081] In Example 26, the subject matter of Examples 21-25 includes: wherein the video stream is output during a real-time video conferencing session.

[0082] In Example 27, the subject matter of Examples 21-26 includes: wherein the source type identification is determined based on at least one of driver information in the metadata of the video stream, a camera name or logo in the metadata of the video stream, a connection type of the source, or is determined by comparing the information extracted from the metadata with a list of allowed devices.

[0083] In Example 28, the subject matter of Example 27 includes: wherein when the connection type of the source is a physical connection port, the video stream is determined to be real-time.

[0084] In Example 29, the subject matter of Examples 21-28 includes: wherein the data processing system includes a user device for generating the video stream, and wherein the device receiving the alert is a device remote from the user device.

[0085] Example 30 is at least one machine-readable medium comprising instructions for operation of a computing system, the instructions, when executed by a machine, causing the machine to perform the operations of any one of the methods of Examples 21-29.

[0086] Example 31 is an apparatus comprising means for performing any one of the methods of Examples 21-29.

[0087] Example 32 is a data processing system comprising: a processing circuit; and a memory comprising instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for: identifying a video stream; determining, using the processing circuit of the data processing system, an identification of a source type of the video stream based on information extracted from metadata associated with the video stream, the source type corresponding to the type of source that generated the video stream; determining, using the processing circuit of the processing system, whether the video stream is real-time or pre-recorded based on the determined type of the source type; and in response to determining that the source type indicates that the video stream is pre-recorded, outputting an alert to a device, the alert indicating unauthorized access or use.

[0088] In Example 33, the subject matter of Example 32 includes: wherein, when the source type corresponds to a camera, the source type indicates that the video stream is real-time, and when the source type corresponds to a video playback application, the source type indicates that the video stream is pre-recorded.

[0089] In Example 34, the subject matter of Examples 32-33 includes: wherein, to output the alert, the processing circuit is further caused to generate a warning indication on a user interface of a user device indicating that the video stream is pre-recorded.

[0090] In Example 35, the subject matter of Examples 32-34 includes: wherein the processing circuit is further caused to activate a host-configured automatic service in response to determining that the video is pre-recorded, the host-configured automatic service comprising at least one of: an automatic telephone call, an automatic text message, an automatic email message, blocking incoming video at the user device, blocking outgoing video at the user device, disabling voting input from the user device, preventing access to a secure physical location, preventing access to secure digital information, terminating access to secure digital information, presenting a different set of information for display, or requesting additional information.

[0091] In Example 36, the subject matter of Examples 32-35 includes: wherein, to output the alert, the processing circuit is further caused to provide the alert for display on a tenant dashboard.

[0092] In Example 37, the subject matter of Examples 32-36 includes: wherein the video stream is output during a real-time video conferencing session.

[0093] In Example 38, the subject matter of Examples 32-37 includes: wherein the source type identification is determined based on at least one of driver information in the metadata of the video stream, a camera name or logo in the metadata of the video stream, a connection type of the source, or is determined by comparing the information extracted from the metadata with a list of allowed devices.

[0094] In Example 39, the subject matter of Example 38 includes: wherein when the connection type of the source is a physical connection port, the video stream is determined to be real-time.

[0095] In Example 40, the subject matter of Examples 32-39 includes: wherein the data processing system includes a user device for generating the video stream, and wherein the device receiving the alert is a device remote from the user device.

[0096] Example 41 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-40.

[0097] Example 42 is an apparatus comprising means for implementing any of Examples 1-40.

[0098] Example 43 is a system for implementing any of Examples 1-40.

[0099] Example 44 is a method for implementing any of Examples 1-40.

Claims

1. A method performed by a data processing system, the method comprising: accessing, using processing circuitry of the data processing system, a video stream, the video stream comprising an image, the image comprising a virtual background, the image depicting a user; Segmenting the image into a foreground portion and a background portion, wherein the background portion is the virtual background; determining whether the foreground portion or the background portion of the image satisfies a threshold requirement, the threshold requirement corresponding to a percentage, number, or sampling of pixels in the foreground portion or the background portion of the image; outputting an alert in response to determining that the foreground portion or the background portion of the image does not satisfy the threshold requirement, the alert indicating unauthorized access or use corresponding to the user located in the background portion of the image; as well as and activating a host-configured automated service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automated service comprising at least one of: blocking incoming video at an access control device, blocking outgoing video at an access control device, preventing access to secure digital information, and terminating access to secure digital information.

2. The method according to claim 1, wherein Segmenting the image includes using segmentation techniques of an application running the real-time video conferencing application.

3. The method according to any one of the preceding claims, wherein Segmenting the image includes using a deep learning model.

4. The method according to any one of the preceding claims, wherein Segmenting the image includes determining that a participant viewing in the video stream is looking away from a camera capturing the video stream, and turning off the camera in response to determining that the participant is looking away from the camera.

5. The method according to any one of the preceding claims, wherein Segmenting the image includes generating a segmentation mask including the foreground portion and the background portion.

6. The method according to any one of the preceding claims, wherein Segmenting the image includes segmenting a plurality of images of the video stream, and wherein the alarm is output in response to determining that a corresponding foreground portion or background portion of at least a portion of the plurality of images fails to meet the threshold requirement.

7. The method according to any one of the preceding claims, wherein Outputting the alert includes generating a warning indication on a user interface of a user device, the warning indication indicating that a participant associated with generating the video stream is not present within the real-time video conference.

8. The method according to any one of the preceding claims, wherein The automated service configured by the host further includes at least one of: an automated phone call, an automated text message, an automated email message, disabling voting entry from a user device, preventing access to a secure physical location, presenting a different set of information for display, or requiring additional information.

9. The method according to any one of claims 1 to 8, wherein: Outputting the alert includes providing the alert for display on a tenant dashboard.

10. The method according to any one of claims 1 to 8, wherein: The video stream is output during a real-time video conferencing session.

11. A data processing system, comprising: processing circuit; as well as a memory comprising instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations for: accessing a video stream, the video stream including an image, the image including a virtual background, the image depicting a user; Segmenting the image into a foreground portion and a background portion, wherein the background portion is the virtual background; determining whether the foreground portion or the background portion of the image satisfies a threshold requirement, the threshold requirement corresponding to a percentage, number, or sampling of pixels in the foreground portion or the background portion of the image; outputting an alert in response to determining that the foreground portion or the background portion of the image does not satisfy the threshold requirement, the alert indicating unauthorized access or use corresponding to the user located in the background portion of the image; as well as and activating a host-configured automated service in response to determining that the foreground portion or the background portion of the image does not meet the threshold requirement, the host-configured automated service comprising at least one of: blocking incoming video at an access control device, blocking outgoing video at an access control device, preventing access to secure digital information, and terminating access to secure digital information.

12. The data processing system according to claim 11, wherein: To segment the image, the processing circuitry is further caused to use a segmentation technique of an application running the real-time video conference, the segmentation technique also being used to apply the virtual background to the video stream for use in the real-time video conference.

13. The data processing system according to claim 11, wherein: To segment the image, the processing circuitry is further caused to use a deep learning model.

14. The data processing system according to claim 11, wherein: To segment the image, the processing circuitry is further caused to determine that a participant viewing the video stream is looking away from a camera capturing the video stream, and to turn off the camera in response to determining that the participant is looking away from the camera.

15. The data processing system according to claim 11, wherein: To segment the image, the processing circuit is further caused to generate a segmentation mask comprising the foreground portion and the background portion.

Images