Audio privacy protection for monitoring systems

Abstract

Systems and methods of audio privacy in a networked video surveillance system are described. A video camera can include an image sensor and a microphone to generate a video stream. In response to detecting a person speaking condition in the video stream, audio data can be selectively modified to mask a person's speech component of the audio data for storing and / or displaying the surveillance video stream.

Description

Technical Field

[0001] This disclosure relates in general to video surveillance systems, and more specifically to network video surveillance systems with audio privacy protection. Background Technology

[0002] Network-based video surveillance systems are a growing computing application in both commercial and personal markets. Some video surveillance systems may include one or more cameras communicatively connected to a server (such as a network video recorder) via a wired interface, wired or wireless LAN, or wired or wireless WAN (such as the Internet). As the cameras record video, the video is forwarded to a server system where it is stored and / or analyzed for later retrieval. Client or user systems communicatively connect to the server system to request, receive, and display streams of recorded video data and / or associated alarms and analyses.

[0003] Some video surveillance systems can be configured to capture both image and audio data from a video stream. For example, each camera may include an image sensor and at least one microphone (or microphone array), and a synchronized audio track is encoded with the image data from the video stream. Under normal operation, the video stream, including the audio data, can be stored by a server system and provided for display on a client or user system, where the captured audio is intact. In some surveillance systems, analysis can be applied to the audio data for audio event detection and related alarms or triggers. For example, the detection of gunshots, baby cries, speech, fire alarms, etc., can trigger a system response such as an alarm or contact with emergency services. In some surveillance systems, audio can also be used for person-to-person communication, such as when a floor security guard can speak within the microphone range of one or more cameras to communicate with another security guard at a monitoring station displaying the video stream. Video streams with audio tracks can be stored, analyzed, and displayed for both their image and audio content.

[0004] In some applications and / or for some users, audio data may be considered unnecessary, distracting, or raise privacy concerns. Some system configurations may not include audio capabilities, or may allow users to disable audio capture and / or display, for example, to ensure the privacy of conversations in monitored areas. In some surveillance systems, audio capabilities can be enabled or disabled based on rule-based systems, such as selectively enabling or disabling audio capture at specific locations or at predefined times.

[0005] A more robust and automated system for managing audio privacy while still capturing and using other audio data within the video surveillance system may be advantageous. A more flexible and reliable approach may be needed to provide selectively enabled audio privacy protection within video surveillance systems. Summary of the Invention

[0006] This paper describes various aspects of audio privacy protection in video surveillance systems, particularly selective audio modification when human speech is detected.

[0007] One general aspect includes a system comprising a camera and a controller. The camera may include: an image sensor configured to capture image data of a video stream, and a microphone configured to capture audio data of the video stream. The controller may be configured to: receive the video stream from the camera; determine a person speaking condition based on the video stream; selectively modify the audio data in the video stream during the person speaking condition in response to determining the person speaking condition; and store the video stream having the modified audio data.

[0008] Implementations may include one or more of the following features. Selectively modifying audio data in a video stream during a human speaking condition may include processing the audio data to mask the human speech component of the audio data. The controller may be further configured to determine the human speech component of the audio data during the human speaking condition. Processing the audio data to mask the human speech component may include: separating the human speech component of the audio data from the remaining non-speech components of the audio data; selectively masking the human speech component of the audio data; and keeping the remaining non-speech components of the audio data unchanged. Processing the audio data to mask the human speech component may further include: separating the human speech component of the audio data into multiple human speech components corresponding to individual speakers among a plurality of speakers; identifying at least one individual speaker among the plurality of speakers; and selectively masking a portion of the human speech component of the audio data corresponding to the at least one individual speaker in response to identifying the at least one individual speaker. The controller may be further configured to: detect audio events from audio data in the video stream; and process the audio data in the video stream during a human speaking condition using a speech recognition algorithm to identify a protected speaker. Determining a human speaking condition from a video stream may be in response to the detection of an audio event, and selectively modifying the audio data in the video stream during the human speaking condition may further be in response to the identification of a protected speaker. The controller may be further configured to process the audio data in the video stream using a speech recognition algorithm to identify a consent pattern, and selectively modifying the audio data in the video stream during the human speaking condition may further be in response to the absence of a consent pattern in the audio data. The controller may be further configured to detect video events from image data in the video stream, and determining a human speaking condition from the video stream may be in response to the detection of the video event. The controller may be further configured to process the image data in the video stream using a facial recognition algorithm to identify a protected speaker, and selectively modifying the audio data in the video stream during the human speaking condition may further be in response to the identification of the protected speaker. The system may include a storage device, an analytics engine, and a user device. The storage device may be configured to store an unmodified video stream from a camera and a video stream from the controller with modified audio data. The analytics engine may be configured to: process the unmodified video stream to determine a human speaking condition from the video stream; and notify the controller of the human speaking condition. The user equipment can be configured to: display a video stream with modified audio data to a user using the user equipment's graphical user interface and speakers; determine the user's security credentials; verify audio access permissions corresponding to the security credentials; and display an unmodified video stream to the user using the user equipment's graphical user interface and speakers.

[0009] Another general aspect includes a computer-implemented method comprising: receiving a video stream from a camera, wherein the camera may include an image sensor configured to capture image data of the video stream; and a microphone configured to capture audio data of the video stream; determining a person speaking condition based on the video stream; selectively modifying the audio data in the video stream during the person speaking condition in response to determining the person speaking condition; and storing the video stream having the modified audio data.

[0010] Implementations may include one or more of the following features. Selectively modifying audio data in a video stream during a human speaking condition may include processing the audio data to mask the human speech component of the audio data. A computer-implemented method may include determining the human speech component of the audio data during a human speaking condition. Processing the audio data to mask the human speech component of the audio data may include: separating the human speech component of the audio data from the remaining non-speech components of the audio data; selectively masking the human speech component of the audio data; and keeping the remaining non-speech components of the audio data unchanged. A computer-implemented method may include detecting an audio event from audio data in a video stream, wherein determining a human speaking condition from the video stream is in response to detecting the audio event. A computer-implemented method may include processing audio data in a video stream using a speech recognition algorithm to identify a protected speaker, wherein selectively modifying the audio data in the video stream during a human speaking condition further responds to identifying the protected speaker. A computer-implemented method may include processing audio data in a video stream using a speech recognition algorithm to identify a consent pattern, wherein selectively modifying the audio data in the video stream during a human speaking condition further responds to the absence of a consent pattern in the audio data. A computer-implemented method may include: detecting a video event from image data in a video stream, wherein determining a human speaking condition from the video stream in response to detecting the video event; and processing the image data in the video stream using a facial recognition algorithm to identify a protected speaker, wherein selectively modifying audio data in the video stream during the human speaking condition further in response to identifying the protected speaker. A computer-implemented method may also include: storing an unmodified video stream from a camera and a video stream having modified audio data in a storage device; processing the unmodified video stream to determine a human speaking condition from the video stream; displaying the video stream with modified audio data to a user using a graphical user interface and a speaker of a user device; determining the user's security credentials; verifying audio access permissions corresponding to the security credentials; and displaying the unmodified video stream to the user using the graphical user interface and the speaker of the user device.

[0011] Another general aspect includes a surveillance system comprising: a camera including an image sensor configured to capture image data of a video stream and a microphone configured to capture audio data of the video stream; a processor; a memory; means for receiving the video stream from the camera; means for automatically determining a person speaking condition based on the video stream; means for selectively modifying the audio data in the video stream during the person speaking condition in response to determining the person speaking condition; and means for displaying the video stream using the modified audio data.

[0012] Various implementations advantageously apply the teachings of computer-based surveillance systems to improve the functionality of such systems. These implementations include operations that overcome or at least reduce problems previously encountered in video surveillance systems, and are therefore more efficient and / or more flexible than other video surveillance systems. That is, the various implementations disclosed herein include hardware and / or software with features that improve audio privacy protection in video surveillance systems, for example, by using selective audio modification when conditions for human speech and / or other event triggers are met. Therefore, the implementations disclosed herein provide various improvements to network-based video surveillance systems.

[0013] It should be understood that the language used in this disclosure is chosen primarily for readability and guidance purposes and does not limit the scope of the topics disclosed herein. Attached Figure Description

[0014] Figure 1 A computer-based monitoring system is illustrated schematically.

[0015] Figure 2 It schematically shows that it can be generated by Figure 1 The computer-based monitoring system uses an audio privacy mode for processing.

[0016] Figure 3 schematically shown Figure 1 Some components of a computer-based monitoring system.

[0017] Figure 4 This is a flowchart of an exemplary method for providing an audio privacy mode.

[0018] Figure 5 This is a flowchart of an exemplary method for selectively modifying audio data in an audio privacy mode.

[0019] Figure 6 This is a flowchart of an exemplary method for triggering audio privacy mode.

[0020] Figure 7 This is a flowchart of an exemplary method for displaying a video stream in an audio privacy mode on a user device. Detailed Implementation

[0021] Figure 1 An embodiment of an exemplary video surveillance system 100 is illustrated, wherein multiple cameras 110 are interconnected with a network video recorder 130 to display surveillance video on a user equipment 170. While some exemplary features are shown, various other features are not shown for brevity and to avoid obscuring relevant aspects of the exemplary embodiments disclosed herein. In some embodiments, the cameras 110, network video recorder 130, and user equipment 170 are computer-based components interconnected via a network 102. Additional components (such as a network video server 160 and / or a network video storage device 162) may also be connected to the network 102. In some embodiments, one or more cameras (such as...) Figure 1 Cameras 110.5-110.n can be directly connected to network recorder 130 without communicating through network 102. Similarly, in an alternative embodiment (not shown), user equipment 170 can be directly connected to network recorder 130.

[0022] In some implementations, one or more networks 102 may be used to communicatively interconnect various components of the monitoring system 100. For example, each component (such as camera 110, network video recorder 130, external storage device 140, network video server 160, network video storage device 162, and / or user equipment 170) may include one or more network interfaces and corresponding network protocols for communicating via network 102. Network 102 may include wired and / or wireless networks (e.g., any number and / or configuration of public and / or private computer networks) that can be suitably coupled for data transmission. For example, network 102 may include any device of a conventional data communication network, such as a local area network (LAN), a wide area network (WAN), a telephone network such as the public switched telephone network (PSTN), an intranet, the Internet, or any other suitable communication network or combination of communication networks. In some implementations, network 102 may include multiple different networks, subnetworks, and / or virtual private networks (VPNs) that may be used to restrict communication between specific components. For example, camera 110 may be located on a limited access network, so that video data and control data can only be transmitted between camera 110 and network video recorder 130, thereby enabling network video recorder 130 to control access to camera 110 and its video data.

[0023] Camera 110 may include an analog or digital camera connected to an encoder that generates an encoded video stream with defined resolution, aspect ratio, and video encoding format. Camera 110 may be configured to capture both image and audio data from the video stream. In some embodiments, camera 110 may include an Internet Protocol (IP) camera configured to encode its respective video stream and stream these streams to network video recorder 130 via network 102. In some embodiments (not shown), the encoder may reside in network video recorder 130. In some embodiments, camera 110 may be configured to receive audio data via an integrated or connected microphone and include embedded and / or synchronized audio streams in its respective video stream. In some embodiments, camera 110 may include an image sensor 112, a processor or central processing unit (CPU) 114, memory 116, an encoder 118, a microphone 120, control circuitry 122, and / or a network interface 126. In some embodiments, camera 110 may include onboard analytics, such as a video analytics subsystem 124.

[0024] For example, image sensor 112 may include a solid-state device configured to capture light waves and / or other electromagnetic waves and convert the light into an image typically composed of colored pixels. Image sensor 112 can determine the underlying image size, resolution, bandwidth, depth of field, dynamic range, and other parameters of captured video image frames. Image sensor 112 may include an electro-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), and / or other image sensor devices of various sensor sizes and aspect ratios.

[0025] Digital video data from image sensor 112 may be received by processor 114 for (temporary) storage and processing in memory 116 and / or encoded by encoder 118. Processor 114 may include any type of conventional processor or microprocessor for interpreting and executing instructions. Memory 116 may include random access memory (RAM) or another type of dynamic storage device storing information and instructions for execution by processor 114, and / or read-only memory (ROM) or another type of static storage device storing static information and instructions for use by processor 114, and / or any suitable storage element such as solid-state storage. Memory 116 may store basic input / output system (BIOS), firmware, and / or operating system instructions for initializing and executing instructions and processes of camera 110. Encoder 118 may encode the video stream received by image sensor 112 using a variety of possible digital encoding and / or compression formats. In some embodiments, encoder 118 may use compressed video formats to reduce the storage size and network bandwidth necessary for storing and transmitting the initial video stream. For example, encoder 118 can be configured to encode video data into Joint Picture Experts Group (JPEG), Moving Picture Experts Group (MPEG)-2, MPEG-4, Advanced Video Coding (AVC) / H.264 and / or other video coding standards or proprietary formats.

[0026] In some implementations, one or more microphones 120 may be selectively enabled to capture audio data in parallel with the image data captured by image sensor 112. For example, microphone 120 may be configured with an audio sensor that captures sound waves and converts them into a time-based audio data stream. In some implementations, encoder 118 may include an audio encoder that operates in conjunction with a video encoder to encode a synchronized audio data stream within the video stream. For example, the video format used by encoder 118 may include one or more audio tracks for encoding audio data to accompany image data during video stream playback.

[0027] Control circuitry 122 may include control circuitry for managing the physical position of camera 110. In some embodiments, camera 110 may be a pan-tilt-zoom (PTZ) camera capable of remote orientation and zoom control. Control circuitry 122 may be configured to receive motion commands via network interface 126 and / or via another interface such as a dedicated remote control interface, such short-range infrared signals, Bluetooth, etc. For example, network video recorder 130 and / or user equipment 170 may be configured to send PTZ commands to control circuitry 122, which translates these commands into motion control signals for controlling the position of camera 110 via multiple actuators. In some embodiments, control circuitry 122 may include logic components for automatically responding to movement detected by image sensor 112 to redirect camera 110 toward the source of movement. For example, automatic tracking features can be embodied in the firmware, enabling the camera to estimate the size and position of an object based on pixel changes in the raw video stream from image sensor 112, adjust the camera's position to follow the moving object, and return to a default position when no more movement is detected. In some embodiments, control circuitry 122 may include logic for virtual PTZ or ePTZ, enabling the high-resolution camera to digitally zoom and pan to portions of the image collected by image sensor 112 without physical camera movement. In some embodiments, control circuitry 122 may include software and one or more application programming interface (API) interfaces for enabling remote devices to control additional features and capabilities of camera 110. For example, control circuitry 122 may enable network video recorder 130 and / or user equipment 170 to configure video formats, enable and disable microphone 120, set motion detection, automatic tracking, and similar features, and / or initiate video data streaming.

[0028] In some implementations, camera 110 may include a video analytics subsystem 124 configured for onboard video analysis. For example, video analytics subsystem 124 may be configured to use CPU 114 and memory 116 to perform at least a portion of video analysis on video data captured by camera 110. In some implementations, video analytics subsystem 124 may be configured to operate similarly to video analytics subsystem 156 in network video recorder 130, as further described below, and specifically embody one or more analytics engines and / or analytics model libraries.

[0029] Network interface 126 may include one or more wired or wireless connections to network 102 and / or a dedicated camera interface of network recorder 130. For example, network interface 126 may include an Ethernet port and a corresponding protocol for IP communication with network recorder 130. In some embodiments, network interface 126 may include a Power over Ethernet (PoE) connection to network recorder 130 or another camera access point. PoE allows power and network data for camera 110 to travel on the same wire. In some embodiments, network interface 126 may enable the IP camera to be configured to have a network resource with an IP address accessible on a LAN, WAN, or the Internet. For example, network recorder 130 and / or user equipment 170 may be configured to selectively receive video from camera 110 from any location connected to the Internet using Internet addressing and security protocols.

[0030] Network video recorder 130 may include a computer system configured to record video streams from camera 110. For example, network video recorder 130 may be configured to receive video streams from each of the cameras 110 for storage, analysis, and / or display via user equipment 170. In some embodiments, camera 110 may transmit encoded video streams based on raw image data collected from its respective image sensor 112, with or without video data compression. A single video stream may be received from each camera 110, and network video recorder 130 may be configured to receive video streams from all connected parallel cameras, provided network bandwidth and processing resources permit.

[0031] The network video recorder 130 may include a housing and a bus that interconnects at least one processor or CPU 132, at least one memory 134, at least one storage device 140, and at least one interface (such as a camera interface 136, a network interface 138, and / or a storage interface 142). The housing (not shown) may include various sub-components for mounting the network video recorder 130, housing any physical connectors for the interfaces, and protecting the sub-components. Some housings may be configured for mounting within a rack system. The bus (not shown) may include one or more conductors that allow communication between components of the network video recorder 130. The processor 132 may include any type of processor or microprocessor that interprets and executes instructions or operations. The memory 134 may include random access memory (RAM) or another type of dynamic storage device storing information and instructions for execution by the processor 132 and / or read-only memory (ROM) or another type of static storage device storing static information and instructions for use by the processor 132 and / or any suitable storage element.

[0032] In some embodiments, the network video recorder 130 may include a camera interface 136 configured for connection to one or more cameras 110. For example, camera interface 136 may include multiple Ethernet ports and a PoE-compliant protocol for connection to cameras 110.5-110.n. In some embodiments, camera interface 136 may include a PoE network switch for providing power to the connected cameras and routing data packets (such as control and video data) to and from the cameras 110.5-110.n. In some embodiments, the network video recorder 130 may not include a dedicated camera interface 136 and may use network interface 138 to communicate with the cameras 110 via network 102.

[0033] Network interface 138 may include one or more wired or wireless network connections to network 102. Network interface 138 may include physical interfaces (such as Ethernet ports) and associated hardware and software protocols (such as network interface cards) for communicating via network 102.

[0034] Storage device 140 may include one or more non-volatile memory devices, such as hard disk drives (HDDs) or solid-state drives (SSDs), configured to store video data. In some embodiments, storage device 140 is or includes multiple solid-state drives (SSDs). In some embodiments, network video recorder 130 may include internal storage device 140.1 and scalable storage devices that connect additional storage devices 140.n via storage interface 142. Each storage device 140 may include non-volatile memory (NVM) or device controller 144 based on computing resources (processor and memory) and multiple NVMs or media devices 146 for data storage (e.g., one or more NVM devices, such as one or more flash memory devices). In some embodiments, a respective data storage device 140 in one or more data storage devices includes one or more NVM controllers, such as flash controllers or channel controllers (e.g., storage devices having NVM devices in multiple memory channels). In some embodiments, storage devices 140 may each be encapsulated in a housing, such as a multi-part sealed housing with a defined form factor and ports and / or connectors for interconnection with storage interface 142. Storage device 140.1 and each extended storage device 140.n can be the same storage device type or different storage device types.

[0035] In some embodiments, the corresponding data storage device 140 may include a single media device, while in other embodiments, the corresponding data storage device 140 may include multiple media devices. In some embodiments, the media device includes NAND flash memory or NOR flash memory. In some embodiments, the storage device 140 may include one or more hard disk drives (HDDs). In some embodiments, the storage device 140 may include a flash memory device, which in turn includes one or more flash memory dies, one or more flash memory packages, one or more flash memory channels, etc. However, in some embodiments, one or more data storage devices in the data storage device 140 may have other types of non-volatile data storage media (e.g., phase-change random access memory (PCRAM), resistive random access memory (ReRAM), spin-transfer torque random access memory (STT-RAM), magnetoresistive random access memory (MRAM), etc.).

[0036] In some embodiments, each storage device 140 includes a device controller 144, which includes one or more processing units (sometimes also referred to as CPUs, processors, microprocessors, or microcontrollers) configured to execute instructions in one or more programs. In some embodiments, the one or more processors are shared by one or more components within the functionality of the device controller and, in some cases, beyond its functionality. A media device 146 is coupled to the device controller 144 via a connector that typically transmits commands in addition to data, and optionally transmits metadata, error correction information, and / or other information, in addition to transmitting data values ​​to be stored in the media device and data values ​​read from the media device 146. The media device 146 may include any number (i.e., one or more) of memory devices, including but not limited to non-volatile semiconductor memory devices such as flash memory devices. In some embodiments, the media device 146 may include NAND or NOR flash memory devices composed of single-level cells (SLC), multi-level cells (MLC), or tertiary cells.

[0037] In some implementations, the media device 146 in storage device 140 is divided into multiple addressable and individually selectable blocks, sometimes referred to as erase blocks. In some implementations, an individually selectable block is the smallest erasable cell in the flash memory device. In other words, each block contains the minimum number of memory cells that can be erased simultaneously (i.e., in a single erase operation). Each block is typically further divided into multiple pages and / or word lines, where each page or word line is typically an instance of the smallest individually accessible (readable) portion of the block. However, in some implementations (e.g., using certain types of flash memory), the smallest individually accessible unit of data set is a sector or codeword, which is a subunit of a page. That is, a block comprises multiple pages, each page contains multiple sectors or codewords, and each sector or codeword is the smallest unit of data for reading data from the flash memory device.

[0038] Data units can describe any size allocation of data, such as host blocks, data objects, sectors, pages, multi-page blocks, erase / program blocks, media devices / packages, etc. Storage locations can include physical and / or logical locations on storage device 140 and can be described and / or allocated at different granularity levels depending on the storage medium, storage device / system configuration, and / or context. For example, for host read / write purposes, storage locations may be allocated based on host logical block address (LBA) data unit size and addressability, but in other contexts, they may be managed as pages with storage device addressing managed in the media flash translation layer (FTL). Media segments can include physical storage locations on storage device 140, which may also correspond to one or more logical storage locations. In some embodiments, a media segment may include a series of contiguous physical storage locations, such as adjacent data units on a storage medium, and for flash memory devices, the media segment may correspond to one or more media erase or program blocks. Logical data groups can include multiple logical data units that can be logically grouped regardless of storage location, such as data objects, video media files, or other logical data constructs consisting of multiple host blocks. In some implementations, storage device 140 may be specifically configured to manage the storage and overwriting of video data in continuous surveillance applications used for video surveillance.

[0039] Storage interface 142 may include a physical interface for connecting to one or more external storage devices using an interface protocol that supports storage device access. For example, storage interface 142 may include a Peripheral Component Interconnect (PCIe), Serial Advanced Technology Attachment (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Universal Serial Bus (USB), FireWire, or a similar storage interface connector that supports storage protocol access to storage device 140.n. In some embodiments, storage interface 142 may include a wireless data connection with sufficient bandwidth for video data transmission. Depending on the configuration and protocol used by storage interface 142, storage device 140.n may include a corresponding interface adapter, firmware, and / or protocol for receiving, managing, and responding to storage commands from network recorder 130.

[0040] The network video recorder 130 may include multiple modules or subsystems stored and / or instantiated in memory 134 for execution as instructions or operations by processor 132. For example, memory 134 may include a camera control subsystem 150 configured to control camera 110. Memory 134 may include a video capture subsystem 152 configured to receive video streams from camera 110. Memory 134 may include a video storage subsystem 154 configured to store received video data in storage device 140 and / or network video storage device 162. Memory 134 may include a video analysis subsystem configured to analyze video streams and / or video data for defining events such as motion, identified objects, identified faces, and combinations thereof. Memory 134 may include a video display subsystem configured to selectively display video streams on user equipment 170, which may be attached to the network video recorder 130 or remotely connected via network 102.

[0041] In some implementations, the camera control subsystem 150 may include an interface protocol and a set of functions and parameters for using the camera 110, configuring the camera, communicating with the camera, and providing command messages to the camera. For example, the camera control subsystem 150 may include an API and command set for interacting with control circuitry 122 to access one or more camera functions. In some implementations, the camera control subsystem 150 may be configured to: set video configuration parameters for image sensor 112 and / or video encoder 118; manage microphone 120 and related parameters; access pan-tilt-zoom characteristics of control circuitry 122; set or modify camera-based motion detection, tripwire detection, and / or low-light detection parameters in memory 116; and / or additionally manage the operation of camera 110. For example, the camera control subsystem 150 may maintain a camera configuration table, page, or similar data structure that includes entries for each managed camera and its corresponding camera-specific configuration parameters, active control features (such as PTZ control), and other configuration and control information for managing camera 110. In some implementations, each camera 110 may be assigned a unique camera identifier, which may be used by the camera control subsystem 150, the video capture subsystem 152, and / or other subsystems to associate video data with the camera to which it is received.

[0042] In some implementations, the video capture subsystem 152 may include an interface protocol and a set of functions and parameters for receiving video streams from camera 110. For example, the video capture subsystem 152 may include video data channels and associated data buffers for managing multiple camera video data streams. In some implementations, each camera 110 may be assigned a dedicated video channel for continuously and / or selectively sending its video stream to network recorder 130. The video capture subsystem 152 may be configured to pass each received video stream to video storage subsystem 154, video analytics subsystem 156, and / or video display subsystem 158. For example, the received video stream may be buffered by the video capture subsystem 152 and then streamed to video storage subsystem 154, video analytics subsystem 156, and / or video display subsystem 158. The video capture subsystem 152 may receive camera video streams that include embedded audio data synchronized with the image data. In some implementations, the video capture subsystem 152 may be configured to process combined audio / image video streams and / or separate audio tracks into individual audio streams for storage, analysis, and / or display.

[0043] In some implementations, the video storage subsystem 154 may include an interface protocol and a set of functions and parameters for managing the storage of video data in storage device 140 and / or network video storage device 162 for later retrieval and use by video analytics subsystem 156 and / or video display subsystem 158. For example, the video storage subsystem 154 may write camera video stream data from a video data buffer to non-volatile storage in storage device 140, and the video analytics subsystem 156 and / or video display subsystem 158 may be configured to selectively read video data from storage device 140. In some implementations, the video storage subsystem 154 may include managing the video storage space in storage device 140 and / or network video storage device 162 according to one or more data retention and / or data archiving schemes. For example, the monitoring system 100 may support continuous and / or triggered recording of video data from the camera 110, and the video storage subsystem 154 may include logic components for implementing a data retention and overwrite strategy in which a fixed storage space of the storage device 140 is cyclically used to: store video and video data captured in the most recent time period that meets specific retention criteria and / or delete or archive video data after one or more time periods defined in the data retention strategy. In some embodiments, the video storage subsystem 154 may include or access video decoders and / or encoders for storing video data in a storage video format different from the camera video format, such as using different codecs, compression factors, frame rates, resolutions, image sizes, etc.

[0044] In some implementations, the video analytics subsystem 156 may include an interface protocol and a set of functions and parameters for analyzing video data from camera 110 to generate metadata, detect events and associated triggers, and / or further process the camera video stream. For example, the video analytics subsystem 156 may be configured to run one or more event detection algorithms to determine, flag, and / or initiate alarms or other actions in response to detected video events. In some implementations, the video analytics subsystem 156 may be configured to flag or construct a metadata structure that maps detected events to time and image location markers in the video stream from which those events were detected. For example, the video analytics subsystem 156 may use motion, tripwire detection, object recognition, face recognition, audio detection, speech recognition, verbal recognition, and / or other algorithms to determine events occurring in the video stream and flag these events in corresponding metadata tracks and / or separate metadata tables associated with video data objects. In some implementations, the video stream and / or portions thereof may be flagged with conditional flags or tags. For example, the video analytics subsystem 156 may determine human speaking conditions based on audio detection and / or speech recognition, or determine low-light conditions based on the light, color, and / or wavelength bandwidth present in the video data, and add appropriate markers. In some embodiments, the video analytics subsystem 156 may include event processing logic for determining a response to the detection of one or more detected events, such as issuing an alarm to user equipment 170, initiating further video processing, and / or triggering the selective display of a video stream including the detected event through the video display subsystem 158. In some embodiments, the video analytics subsystem 156 may operate on the video data received by the video capture subsystem 152, the video data stored by the video storage subsystem 154, and / or combinations thereof in real-time or near real-time based on the nature of the video event (and processing requirements), the amount of video to be processed, and other factors. In some embodiments, the video analytics subsystem 156 may include one or more analytics engines configured for specific types of events and corresponding event detection algorithms or models, such as a speech detection analytics engine, a speech recognition analytics engine, an object recognition analytics engine, a face recognition analytics engine, etc.

[0045] In some implementations, the video display subsystem 158 may include an interface protocol and a set of functions and parameters for displaying video from the video capture subsystem 152 and / or the video storage subsystem 154 on the user equipment 170. For example, the video display subsystem 158 may include monitoring or display configurations for displaying one or more video streams in real-time or near real-time on a graphical user display of the user equipment 170, and / or for receiving video navigation commands from the user equipment 170 to selectively display stored video data from the video storage subsystem 154. In some implementations, the video display subsystem 158 may support video streams with or without audio data and / or enable audio channels in the video streams to be selectively provided to the user equipment 170. For example, some user equipment may be configured not to receive audio data, and other user equipment may selectively receive one or more audio channels based on user identification and / or security credentials. In some implementations, the video display subsystem 158 may maintain an index of real-time / near real-time video streams and / or stored or archived video streams that can be accessed by the user equipment 170. In some implementations, the video index may include a corresponding metadata index, which includes video data parameters (e.g., time, location, camera identifier, format, low light / normal light, audio channels, etc.), metadata of detected video events (event time, location, type, parameters, event conditions, etc.), and / or video management parameters for displaying and managing video data (expiration, activity / archiving, access control, etc.). The video display subsystem 158 may be configured to support user equipment 170 when directly attached to the network recorder 130 and / or via a network 102 within a LAN, WAN, VPN, or the Internet.

[0046] In some implementations, the monitoring system 100 may include one or more remote and / or cloud-based resources to support the functionality of the network video recorder 130 and / or user equipment 170. For example, the monitoring system 100 may include a network video server 160, such as a cloud-based server system, configured to host some, all, or selected functions of the network video recorder 130. As another example, the monitoring system 100 may include a network video storage device 162, such as a cloud-based attached storage system or distributed storage system, for storing active and / or archived video data, supplementing and / or replacing storage device 140. In some implementations, most of the functions described above for the network video recorder 130 may reside in the network video recorder 130, and selected functions may be configured to utilize additional resources in the network video server 160 and / or network video storage device 162. For example, network video server 160 may be configured to support dedicated and / or processing-intensive event detection algorithms to complement video analytics subsystem 156, and / or network video storage device 162 may be configured to support archiving of inactive video data for long-term storage.

[0047] User device 170 can be any suitable computer device, such as a computer, computer server, laptop computer, tablet computer, netbook, internet kiosk, personal digital assistant, mobile phone, smartphone, gaming device, or any other computing device. User device 170 is sometimes referred to as a host, client, or client system. In some embodiments, user device 170 can host or instantiate one or more applications for interfacing with monitoring system 100. For example, user device 170 can be a personal computer or mobile device running a monitoring and management application configured to provide a user interface for network video recorder 130. In some embodiments, user device 170 can be configured to access camera 110 and / or its corresponding video stream via network video recorder 130 and / or directly via network 102. In some embodiments, one or more functions of network video recorder 130 can be instantiated in user device 170, and / or one or more functions of user device 170 can be instantiated in network video recorder 130.

[0048] User equipment 170 may include one or more processors or CPUs 172 for executing computational operations or instructions stored in memory 174 to access video data and other functions of network video recorder 130 via network 102. In some embodiments, processor 172 may be associated with memory 174 and input / output devices 176 for performing both video display operations and monitoring system management operations. Processor 172 may include any type of processor or microprocessor that interprets and executes instructions or operations. Memory 174 may include random access memory (RAM) or another type of dynamic storage device storing information and instructions for execution by processor 172 and / or read-only memory (ROM) or another type of static storage device storing static information and instructions for use by processor 172 and / or any suitable storage element. In some embodiments, user equipment 170 may allocate a portion of memory 174 and / or another local storage device (located in or attached to user equipment 170) for storing selected video data for user equipment 170. In some embodiments, user equipment 170 may include one or more I / O devices 176. For example, a graphic display (such as a monitor and / or touchscreen display) and / or other user interface components (such as a keyboard, mouse, function buttons, speaker, microphone, vibration motor, touchpad, pen, voice recognition, biometric mechanisms, and / or any number of supplementary devices for adding functionality to user equipment 170). Network interface 178 may include one or more wired or wireless network connections to network 102. Network interface 178 may include physical interfaces (such as an Ethernet port) and / or associated hardware and software protocols (such as network interface cards, wireless network adapters, and / or cellular data interfaces) for communication via network 102.

[0049] User equipment 170 may include multiple modules or subsystems stored and / or instantiated in memory 174 for execution by processor 172 as instructions or operations. For example, memory 174 may include a video manager 180 configured to provide a user interface for selectively navigating and displaying live, near-live, and / or stored video streams. Memory 174 may include an alarm manager 182 configured to provide a user interface for setting, monitoring, and displaying alarms based on video events. Memory 174 may include a camera manager 184 configured to provide a user interface for identifying, configuring, and managing cameras 110. Memory 174 may include a configuration manager 186 to provide a user interface for setting and managing system settings, user access control, storage options, and other configuration settings for the monitoring system 100. Memory 174 may include a network video recorder (NVR) manager 188 configured to provide a user interface for identifying, configuring, and managing network video recorders 130 and / or multiple network video recorders. The memory 174 may include an analysis manager configured to provide a user interface for selecting, training, and managing event detection algorithms for the monitoring system 100.

[0050] In some implementations, video manager 180 may include an interface protocol and a set of functions and parameters for navigating and displaying video streams from camera 110. For example, video manager 180 may include a graphical user interface and interactive controls for displaying lists, tables, thumbnails, or similar interface elements to select and display video streams for specific cameras, times, locations, and / or events. In some implementations, video manager 180 may enable split-screen display of multiple camera video streams. For example, near real-time video streams from all active cameras (with predetermined delays based on network latency, storage, and processing times) may be displayed on a monitoring interface, or a set of video streams corresponding to detected events may be displayed in an event viewing interface. In some implementations, video manager 180 may display or play audio data from the video streams via speakers in user equipment 170 or speakers connected to that user equipment. For example, the video stream may have an accompanying audio channel or track for providing audio data synchronized with the video stream displayed on user equipment 170. In some implementations, video manager 180 may include a data structure that aggregates all video data stored in monitoring system 100 to enable users to locate and view older surveillance videos. For example, a video management log or database may include entries for stored video data indexed by relevant metadata, such as video data parameters (e.g., time, location, camera identifier, format, low light / normal light, etc.), metadata of detected video events (event time, location, type, parameters, etc.), and / or video management parameters (expiration, activity / archiving, access control, etc.). In some embodiments, the video manager 180 may be configured to interface with the video display subsystem 158 and / or the storage subsystem 154 to identify and retrieve selected video data. In some embodiments, access to the video stream via the video display subsystem 158 may depend on user identification, security credentials, and / or access permissions.

[0051] In some implementations, the alarm manager 182 may include an interface protocol and a set of functions and parameters for setting, monitoring, and displaying alarms based on video events. For example, a user may define a set of trigger events that generate visual, auditory, tactile, and / or notification-based (email, text message, automated call, etc.) alarms for user device 170. In some implementations, the alarm manager 182 may include multiple preset alarm conditions with associated event parameters and allow the user to enable and disable alarm types and / or change associated event parameters. In some implementations, the alarm manager 182 may be configured to overlay graphical elements or event indicators representing detected events onto the video stream displayed through the video manager 180. For example, detected motion, objects, or faces may be framed or highlighted, marked with relevant identifiers, or otherwise indicated in video playback on user device 170. In some implementations, the alarm manager 182 may be configured to interface with the video analytics subsystem 156, the video capture subsystem 152, and / or directly with the camera 110 to receive event notifications or parameters.

[0052] In some implementations, camera manager 184 may include an interface protocol and a set of functions and parameters for identifying, configuring, and managing camera 110. Configuration manager 186 may include an interface protocol and a set of functions and parameters for setting and managing system settings, user access control, user identification and security credentials, storage options, and other configuration settings. NVR manager 188 may include an interface protocol and a set of functions and parameters for identifying, configuring, and managing network video recorder 130. For example, each of camera manager 184, configuration manager 186, and / or NVR manager 188 may include a series of graphical user interfaces for displaying the manager's corresponding component identification and related configuration parameters and enabling users to view and / or change these parameters to manage surveillance system 100 and its component systems. In some implementations, the camera manager 184, configuration manager 186, and / or NVR manager 188 may provide changed parameters to the affected components, such as the camera manager 184 sending camera configuration parameter changes to the selected camera 110, the NVR manager 188 sending NVR configuration parameter changes to the network recorder 130, and / or the configuration manager 186 sending system configuration parameter changes to all affected components.

[0053] In some implementations, the analysis manager 190 may include an interface protocol and a set of functions and parameters for selecting, training, and managing event detection algorithms. For example, the analysis manager 190 may include a library of event detection algorithms for different event types. In some implementations, the event detection algorithms may include a pre-configured set of parameters and / or model weights based on a training dataset processed independently of the monitoring system 100. For example, the analysis manager 190 may include object detection algorithms configured for common objects, situations, and cameras. In some implementations, the analysis manager 190 may include a pre-configured training dataset and / or allow users to define a training dataset based on a predefined base algorithm or model for determining or refining the event detection algorithm parameters and / or model weights. In some implementations, the analysis manager 190 may interface with the analysis subsystem 156 to: process video data received by the network video recorder 130 using event detection algorithms configured through the analysis manager 190; and / or select, train, and manage these algorithms.

[0054] Figure 2 This illustrates a computer-based monitoring system 200 (such as...) Figure 1 This is a schematic diagram illustrating the audio privacy mode processing implemented in the monitoring system 100. Similar to... Figure 1 Camera 210 of camera 110 can be configured to capture image data from light waves impacting an image sensor and sound waves received by a microphone. Camera 210 can generate an audio / video stream 212 (e.g., a video stream including at least one audio channel with synchronized audio data).

[0055] Audio / video stream 212 can be streamed to audio event detector 214 and / or video event detector 216 to determine if a human speaking condition exists in one or more frames of the audio and / or video data. For example, camera 210 may include automated control circuitry that recognizes human speech events from the audio data. As another example, event detection of a human speaking condition can be performed by a network video recorder (such as...). Figure 1The network video recorder (130) receives audio / video data 212 from camera 210. The network video recorder can process the real-time video stream from camera 210 to determine the presence of a human speaking condition based on audio data, video data, or a combination thereof. For example, an audio event detector 214 may include a human voice detector algorithm or a speech recognition algorithm to determine the presence of one or more people speaking in the audio data. Similarly, a video event detector 216 may be based on facial features or body language behavior recognition that suggests one or more people are speaking in a video image. At decision box 218, if an audio or video event corresponding to human speech is detected, further processing 228 can be performed using the human voice detector 230, speech recognition 234, facial recognition 250, and / or behavior recognition 252 to determine whether an audio protection mode should be activated for human speech, or whether the event trigger is sufficient to activate an audio privacy mode 240 without further processing 228. Otherwise, normal processing of the audio / video stream 212 without modified audio can be performed at boxes 220, 222, and 224.

[0056] If no event triggering indicating human speech is detected at box 218, then the audio / video stream 212 does not have the condition of human speech. At box 220, the audio / video stream 212 may be stored in a persistent storage device. For example, a network video recorder may store the encoded audio / video stream in one or more storage devices configured for monitoring video storage. At box 222, the audio / video stream 212 may be displayed to a user as raw audio data. For example, the network video recorder may stream the audio / video stream 212 to a user display, such as a graphical user display and speakers on user equipment 170, for monitoring or viewing. At box 224, the audio / video stream 212 may be analyzed for additional events and / or further processing unrelated to the audio privacy mode. For example, the network video recorder may include one or more analysis engines for detecting various event types, adding / modifying relevant metadata, issuing alarms or notifications, and / or initiating further processing of the image and / or audio data.

[0057] In some implementations, further processing 228 may be triggered by event trigger 218. In some implementations, some or all of the human speech detector 230, speech recognition 234, facial recognition 250, and / or behavior recognition 252 may be included in the audio event detector 214 and / or video event detector 216. Further processing 228 may represent exemplary additional criteria and / or event evaluation logic for determining whether audio privacy mode 240 should be activated, and different specific implementations may use some or all of them to enable audio privacy mode 240 after initially determining the human speaking condition.

[0058] Human speech detector 230 may include a processor configured to detect the presence of human speech within an audio data stream. For example, human speech detector 230 may process audio data from audio / video stream 212 to detect audio data signatures indicating human speech within a wider range of audio artifacts, including background, environment, and / or other non-speech sources. In some embodiments, the analysis engine may execute a speech activity detection (VAD) algorithm for audio / video stream 212 and / or audio channels from that audio / video stream to detect the presence or absence of human speech. For example, the algorithm may focus on the presence of frequencies common to signal patterns indicating speech pitch and mode. At block 232, if no human speech is detected, an audio privacy mode may not be required, and processing of audio / video stream 212 may continue to block 220. If human speech is detected, a human speaking condition may be flagged. In some embodiments, a human speaking condition is sufficient to trigger audio privacy mode 240. In some embodiments, additional processing of the audio data containing human speech may be required before initiating audio privacy mode.

[0059] Speech recognition 234 may include a processor configured to detect the presence or absence of individual human voices. For example, speech recognition 234 may process audio data from audio / video stream 212 and compare human voice patterns in the audio data with one or more known voiceprints. In some embodiments, the analysis engine may perform speech recognition, speaker identification, or voice authentication against a database of identified users' voices. By comparing a unique set of audio factors constituting a human voice (voiceprint) with the human voice in the audio data, speech recognition 234 may identify whether the speaker is known, and if so, whether the speaker is a protected speaker for enabling an audio privacy mode. For example, monitoring system 200 may include a list of protected speakers that identifies individuals for whom an audio privacy mode should be enabled and includes their voiceprints for use by speech recognition 234. In some embodiments, speech recognition 234 may support speech binarization 235, where multiple individual speakers are detected and / or identified in the same human voice audio data. For example, in cases where the audio / video stream includes multiple speakers, such as conversations between multiple parties, speech binarization 235 can support the identification and tracking of multiple individual speakers and their speech contributions to the collective audio stream. At box 236, if speaker protection is not required, an audio privacy mode may not be necessary, and processing of the audio / video stream 212 can continue to box 220. If a protected speaker is detected, a protected speaker identifier can be flagged. In some embodiments, the protected speaker identifier is sufficient to trigger audio privacy mode 240. In some embodiments, additional processing of the audio data containing the identified speaker may be required before initiating audio privacy mode.

[0060] In some implementations, speech recognition 234 may further include speech recognition for identifying specific phrases indicating consent to be recorded. For example, speech recognition 234 may process audio data from audio / video stream 212 to extract specific words or phrases from human speech. In some implementations, the detection of consent patterns may allow audio privacy modes to be bypassed or disabled when a speaker on a protected speaker list wishes to be recorded and their speech displayed by monitoring system 200. For example, the analysis engine may include a library of consent patterns representing phrases that a speaker can use to indicate consent to be recorded. At box 238, if a speaker indicates consent to be recorded, audio privacy mode may not be required, and processing of audio / video stream 212 may continue to box 220. If no consent is given, audio privacy mode 240 may be activated. In some implementations, speech recognition for determining consent to be recorded may be a default response, with or without speaker recognition, when a human speaking condition is detected. For example, monitoring system 200 may default to enabling audio privacy mode for all speakers to protect their privacy unless consent is actively given and detected by speech recognition 234.

[0061] Audio privacy mode 240 may include metadata tags and / or operational status for processing, storing, and displaying audio / video stream 212 when audio privacy mode is enabled. For example, audio privacy mode 240 may include operational modes that mask human speech or speech components of the audio channels of audio / video stream 212. Audio privacy mode 240 can be implemented by masking the modified audio / video stream 242 of human speech components. For example, the audio data can be modified by separating and removing the human speech component or replacing it with white noise, replacing the tone, or another audio feature that prevents the user from hearing the speech when displaying the video stream. In some embodiments, the modified audio / video stream 242 may be tagged or accompanied by metadata indicating that audio privacy mode is enabled.

[0062] Once modified, further use of the modified audio / video stream 212 will protect the audio privacy of human speech in the video stream. At box 244, the modified audio / video stream 242 may be stored in a persistent storage device. For example, a network video recorder may store the encoded audio / video stream in one or more storage devices configured for monitoring video storage. At box 246, the modified audio / video stream 242 may be displayed to a user, wherein the modified audio data protects the human speech component. For example, the network video recorder may stream the modified audio / video stream 242 to a user display, such as a graphical user display and speakers on user equipment 170, for monitoring or viewing. At box 248, the modified audio / video stream 212 may be analyzed for additional events and / or further processing unrelated to the audio privacy mode. For example, the network video recorder may include one or more analysis engines for detecting various event types, adding / modifying relevant metadata, issuing alarms or notifications, and / or initiating further processing of the image and / or audio data.

[0063] In some embodiments, the video event detector 216 and / or additional processing 228 may include facial recognition 250 and / or behavior recognition 252 for processing image data of the audio / video stream 212 to determine speaker identity and / or consent patterns. In some embodiments, facial recognition 250 may operate similarly to speech recognition 234 for identifying a protected speaker, but based on facial vectors in the image data rather than voiceprints in the audio data. In some embodiments, behavior recognition 252 may operate similarly to speech recognition 234 for identifying the consent of the speaker to be recorded, but based on visual consent patterns in body language or gestures rather than the speech content.

[0064] Figure 3 The illustration schematically shows selected modules of a monitoring system 300 configured for selective audio privacy mode. The monitoring system 300 can be combined with... Figures 1 to 2 The same components and configurations are shown. For example, the monitoring system 300 can be configured in a network video recorder similar to network video recorder 130. In some embodiments, one or more selected modules can access or be instantiated therein: the processor, memory, and other resources of a camera configured for video capture (similar to camera 110); and / or user equipment configured for video surveillance, similar to user equipment 170.

[0065] The monitoring system 300 may include a bus 310 that interconnects at least one processor 312, at least one memory 314, and at least one interface (such as a camera interface 316 and a network interface 318). The bus 310 may include one or more conductors that allow communication between components of the monitoring system 300. The processor 312 may include any type of processor or microprocessor that interprets and executes instructions or operations. The memory 314 may include random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by the processor 312, and / or read-only memory (ROM) or another type of static storage device that stores static information and instructions for use by the processor 312, and / or any suitable storage element such as a hard disk or solid-state storage element.

[0066] Camera interface 316 can be configured to connect to one or more cameras. For example, camera interface 316 may include multiple Ethernet ports and a PoE-compliant protocol for connecting to multiple cameras. In some implementations, camera interface 316 may include a PoE network switching device for providing power to connected cameras and routing data packets (such as control data and video data) to and from connected cameras.

[0067] Network interface 318 may include one or more wired or wireless network connections to a network (similar to network 102). Network interface 318 may include physical interfaces (such as Ethernet ports) and associated hardware and software protocols (such as network interface cards or wireless adapters) for communicating over the network.

[0068] The surveillance system 300 may include one or more non-volatile memory devices 320 configured to store video data. For example, the non-volatile memory device 320 may include multiple flash memory packages organized as an addressable memory array and / or one or more solid-state drives or hard disk drives. In some embodiments, the non-volatile memory device 320 may include multiple storage devices located within, attached to, or accessible by a network video recorder for storing and accessing video data.

[0069] The monitoring system 300 may include multiple modules or subsystems stored and / or instantiated in memory 314 for execution by processor 312 as instructions or operations. For example, memory 314 may include a monitoring controller 330 configured to control multiple cameras, capture and store video streams from those cameras, and enable user access, such as through a monitoring application 350. Memory 314 may include an analytics engine configured to analyze video data to detect events and / or initiate further processing for use by the monitoring controller 330 and / or the monitoring application 350. Memory 314 may also include a monitoring application configured to provide a user interface for monitoring, viewing, and managing surveillance video and / or the monitoring system 300.

[0070] The monitoring controller 330 may include interface protocols, functions, parameters, and data structures for connecting to and controlling cameras, capturing and storing video data from those cameras, and communicating with the analysis engine 340 and the monitoring application 350. For example, the monitoring controller 330 may be an embedded firmware application and corresponding hardware located in a network video recorder configured to communicate network- and / or directly with a group of associated cameras. The monitoring controller 330 may be configured as a central acquisition point for video streams from associated cameras, enabling the captured video data to be analyzed by the analysis engine 340 and allowing video streams and video event alarms to be presented to the user via the monitoring application 350.

[0071] In some implementations, the monitoring controller 330 may include multiple hardware and / or software modules configured to use the processor 312 and memory 314 to process or manage defined operations of the monitoring controller 330. For example, the monitoring controller 330 may include a camera control interface 332, a video capture interface 334, a video storage interface 336, and an access and display manager 338.

[0072] Camera control interface 332 may include a camera interface protocol and a set of functions, parameters, and data structures for using, configuring, communicating with, and providing command messages to cameras via camera interface 316 and / or network interface 318. For example, camera control interface 332 may include APIs and command sets for interacting with control circuitry in each camera to access one or more camera functions. In some embodiments, camera control interface 332 may be configured to: set video configuration parameters for camera image sensors, microphones, and / or video encoders; access pan-tilt-zoom features; set or modify camera-based motion detection, tripwire detection, and / or low-light detection parameters; and / or additionally manage camera operation. For example, camera control interface 332 may maintain a camera configuration table, page, or similar data structure that includes entries for each managed camera and its corresponding camera-specific configuration parameters, active control features (such as PTZ control), and other configuration and control information for managing the cameras. In some implementations, each camera can be assigned a unique camera identifier, which can be used by the monitoring controller 330, the analytics engine 340, and / or the monitoring application 350 to associate video data with the camera that receives it.

[0073] In some embodiments, the camera control interface 332 may include control signals, commands, or messages for manipulating specific features of one or more cameras. For example, the camera control interface 332 may include control signals for audio control 332.1. In some embodiments, audio control 332.1 may include an API and corresponding logic components for enabling and / or disabling microphones in selected cameras by sending command messages or signals to selected cameras. In some embodiments, the camera control interface 332 may include a human speech detector 332.2 configured to detect human speaking conditions. For example, human speaking conditions may be determined based on frequency ranges and other audio signal patterns associated with human speech. Audio signals or data may be compared to a set of human speech audio characteristics used to determine human speaking conditions. In some embodiments, audio control 332.1 may disable the camera's microphone in response to human speech detector 332.2 detecting human speaking conditions. For example, a simple audio privacy mode may be enabled via the camera control interface 332, which is configured to disable microphones or filter human speech frequencies in response to detecting human speaking conditions, such that encoded video streams available for other components do not contain or mask human speech components. In some implementations, the ability to enable and disable the microphone in response to the detection of a person speaking may be embodied in similar hardware and / or software functions within each camera, and the camera control interface 332 may implement a simple audio privacy mode response feature simply through camera configuration parameters.

[0074] Video capture interface 334 may include a camera interface protocol and a set of functions, parameters, and data structures for receiving video streams from associated cameras. For example, video capture interface 334 may include video data channels and associated data buffers for managing multiple camera video data streams. In some embodiments, each camera may be assigned a dedicated video channel for continuously and / or selectively sending its video stream to video capture interface 334. Video capture interface 334 may be configured to pass each received video stream to video storage interface 336, analysis engine 340, and / or access / display manager 338. For example, the received video stream may be buffered by video capture interface and then streamed to video storage interface 336, analysis engine 340, and / or access / display manager 338. In some embodiments, video capture interface 334 may receive camera video metadata describing the camera video format, time and location information, and event or condition markers based on onboard camera analysis. Video capture interface 334 may generate additional video metadata for video format changes and provide video metadata to video storage interface 336 and / or other components.

[0075] In some embodiments, the video capture interface 334 may include a video stream manager 336.1 configured to identify and manage multiple video streams received from the camera. For example, the video stream manager 336.1 may manage: video buffer allocation and space; processing of video streams from one camera video format to another; refreshing buffered video to a storage device via the video storage interface 336; and / or refreshing the buffered video for display via the access / display manager 338. In some embodiments, the video stream manager 336.1 may send video streams to the analysis engine 340 for analysis and / or provide the analysis engine 340 with notifications regarding the availability and storage location of video data for analysis in the non-volatile memory 320 (as determined by the video storage interface 336). In some embodiments, the video stream manager 334.1 may include configurable video paths. For example, storage paths (through the video storage interface 336), display paths (through the access / display manager 338), and / or analysis paths (through the analysis engine 340) may each be configured for specific processing, priorities, and timings. In some implementations, the video stream manager 334.1 may be configured to use an encoder / decoder 334.2 to decode camera video streams in a first (camera) video format and re-encode these camera video streams in one or more other formats. For example, the video stream manager 334.1 may use the encoder / decoder 334.2 to change resolution, image size, frame rate, codec, compression factor, color / grayscale, or other video format parameters. In some implementations, the video capture interface 334 may be configured to identify audio channels in the received video stream, and the encoder / decoder 334.2 may be configured to encode and decode synchronized audio data in the audio channels of the audio / video stream.

[0076] Video storage interface 336 may include a storage interface protocol and a set of functions, parameters, and data structures for managing the storage of video data in non-volatile memory 320 (such as storage devices and / or network video storage devices) for later retrieval and use by access / display manager 338 and / or analysis engine 340. For example, video storage interface 336 may write camera video stream data 320.1 from a video data buffer and / or storage path video data from video capture interface 334 to non-volatile memory 320. In some embodiments, video storage interface 336 may include storage manager 336.1 configured to manage video storage space in non-volatile memory 320 according to one or more data retention and / or data archiving schemes. For example, the monitoring system 300 may support continuous and / or triggered recording of video data from associated cameras, and the storage manager 336.1 may include logic components for implementing a data retention and overwrite strategy in which the fixed storage space of the non-volatile memory 320 is cyclically used to: store video and video data captured in the most recent time period that meets specific retention criteria and / or delete or archive video data after one or more time periods defined in the data retention strategy. The video storage interface 336 may also receive and store video metadata 320.7 as tags or metadata tracks in the video data, or receive and store it in an associated metadata table, file, or similar data structure associated with the corresponding video data object.

[0077] In some implementations, the video storage interface 336 may be configured to store video streams in multiple formats. For example, the video storage interface 336 may support normal video formats such as camera video stream data 320.1, and audio privacy mode formats such as privacy video stream data 320.4. In some implementations, the camera video stream data 320.1 may include image data 320.2 and audio data 320.3 encoded in a synchronous video stream format. For example, the camera video stream data 320.1 may be a camera video stream received from an attached camera, and is in a format encoded by the camera and / or a standard video storage format supported by the video storage interface 336 for storing video streams. In some implementations, the privacy video stream data 320.4 may include image data 320.5 and masked audio data 320.6 encoded in a synchronous video stream format. For example, privacy video stream data 320.4 may be based on camera video stream data 320.1, wherein audio data 320.3 has been modified into masked audio data 320.6 to remove or obscure the speech of one or more people when an audio privacy mode is enabled and a human speaking condition is detected. In some embodiments, image data 320.2 and image data 320.5 may be the same. In some embodiments, storage manager 336.1 allocates or receives storage locations for stored video data and provides storage locations corresponding to different video paths to the respective consumable components. For example, storage manager 336.1 may provide storage locations for privacy video stream data 320.4 to access / display manager 338 and provide camera video stream data 320.1 to analysis engine 340. In some embodiments, storage manager 336.1 may initially provide camera video stream data 320.1 to analysis engine 340 to determine and process the video stream for voice privacy mode, and then receive privacy video stream data 320.4 from analysis engine 340 for storage.

[0078] Access / Display Manager 338 may include APIs and a set of functions, parameters, and data structures for displaying video from video capture interface 334 and / or video storage interface 336 to a user display application (such as monitoring application 350). For example, Access / Display Manager 338 may include monitoring or display configurations for displaying one or more video streams in real-time or near real-time on a graphical user display of a user device and / or receiving video navigation commands from the user device to selectively display stored video data from non-volatile memory 320. In some embodiments, Access / Display Manager 338 may maintain an index of real-time / near real-time video streams and / or stored or archived video streams available for access by monitoring application 350. In some embodiments, the video index may include a corresponding metadata index that includes video data parameters (e.g., time, location, camera identifier, format, low light / normal light, etc.), detected video event metadata (event time, location, type, parameters, etc.), and / or video management parameters (expiration, activity / archiving, access control, etc.) for displaying and managing video data. Access / Display Manager 338 can be configured to support monitoring application 350 when instantiated in the same computing device as monitoring controller 330, when directly attached to a computing device hosting monitoring controller 330, and / or via a network within a LAN, WAN, VPN, or Internet.

[0079] In some implementations, the access / display manager 338 may include controls for determining which monitoring applications and their users have access to specific video streams, characteristics, and / or configuration parameters. For example, the access / display manager 338 may include an access control service 338.1 for verifying one or more security credentials used to authenticate users. In some implementations, the access control service 338.1 may include a username or similar user identifier coupled to one or more security credentials, such as a password, biometric authentication, security questions, CAPTCHA, etc., used to authenticate users. In some implementations, user authentication, user identifiers, or profiles may be associated with a set of access permissions for the video streams, characteristics, and configuration parameters of the monitoring system 300. For example, each user profile may be associated with user settings that determine their access, such as different user levels (e.g., system administrator, owner / super user, monitor / user, etc.) with different access permissions and / or customized user settings, which may be configured during system setup or by a user with system administrator credentials. In some implementations, user settings may include audio access permissions 338.2, which may determine whether a user can bypass audio privacy modes. For example, if audio privacy mode is enabled for the video, a normal user may have general audio access permissions that only allow access to a privacy-protected version of the video stream (such as privacy video stream data 320.4). System administrators, super users, and / or designated user profiles may be configured to access privacy video stream data 320.4 and camera video stream data 320.1 for videos used in audio privacy mode to hear the original audio data 320.3 or masked audio data 320.6. In some implementations, user profiles may also be used to determine and associate voiceprints for speaker identification features. For example, some or all users may provide sample audio data for generating and storing voiceprints, and may configure corresponding protected speaker and / or default consent parameters to enable or disable audio privacy mode when their voice is detected.

[0080] The analysis engine 340 may include interface protocols, functions, parameters, and data structures for analyzing video data to detect video events, adding them to video metadata, such as issuing alerts via the monitoring application 350, and / or providing additional processing based on detected events or conditions, such as masking speaker data for audio privacy modes. For example, the analysis engine 340 may be an embedded firmware application and corresponding hardware located in a network video recorder configured to perform local analysis of video data captured from associated cameras, and may be integrated into or accessible by the monitoring controller 330. In some embodiments, the analysis engine 340 may run on a computing device separate from the monitoring controller 330, such as a camera with onboard analysis, a dedicated analysis device, a data storage system with analysis capabilities, or a cloud-based analysis service. In some embodiments, depending on the nature of the video event (and processing requirements), the amount of video to be processed, and other factors, the analysis engine 340 may operate in real-time or near real-time on the video data received by the video capture interface 334, the video data stored by the video storage interface 336, and / or combinations thereof. In some implementations, the monitoring system 200 may include multiple analysis engines configured for specific types of events and corresponding event detection algorithms or models.

[0081] In some implementations, the analysis engine 340 may include multiple hardware and / or software modules configured to use processor 312 and memory 314 to process or manage defined operations of the analysis engine 340. For example, the analysis engine 340 may include an event manager 342, an analysis model library 344, and a training service 346. The analysis engine 340 may be configured to run one or more event detection algorithms to determine, tag, and / or initiate alarms or other actions in response to detected video events. In some implementations, the analysis engine 340 may be configured to tag or construct a metadata structure that maps detected events to time and image location markers in the video stream from which those events were detected. For example, the analysis engine 340 may use motion, tripwire detection, object recognition, face recognition, audio detection, speech recognition, and / or other algorithms to determine events occurring in the video stream and tag these events in corresponding metadata tracks and / or separate metadata tables associated with video data objects.

[0082] Event Manager 342 may include a storage interface and / or buffer interface protocol, as well as a set of functions, parameters, and data structures for processing target video streams for predefined event types, updating or adding descriptions of detected video events, and / or triggering additional actions such as alarms, notifications, and / or additional processing. For example, Event Manager 342 may be configured to process all input video streams for Monitoring Controller 330; and / or selectively process video data based on user selection (via Monitoring Application 350) or metadata criteria received from cameras or video capture interfaces 334. In some implementations, Event Manager 342 may include a video selector 342.1 configured to select target video streams or video datasets for analysis. For example, Video Selector 342.1 may identify real-time video streams or bounded video datasets for near-real-time analysis, such as videos with a specified camera identifier and timestamps between start and end time markers and / or including a defined set of metadata parameters.

[0083] Event manager 342 may include event type selector 342.2 configured to determine one or more event types to be detected from selected video data. For example, the analysis configuration may be configured to analyze the selected video stream against a predefined set of motion detection, tripwire detection, object recognition, face recognition, audio detection, speech recognition, and / or similar video event types. Each event type may be mapped to or correspond to an analysis model type, parameter set, and one or more model weights used to define an event detection algorithm stored in analysis model library 344 for use by analysis engine 340 to detect possible video events. In some implementations, event manager 342 may be configured to detect human speaking conditions and initiate logic for determining an audio privacy mode. For example, event type selector 342.2 may be configured with a sequence of audio and / or video events related to determining the audio privacy mode and related logic for determining whether the audio privacy mode should be flagged for the video stream (or a portion thereof) and forwarded to audio data processor 346 for audio privacy processing.

[0084] In some implementations, the event manager 342 may include detection event processing logic components for determining a response to the detection of one or more detected events. For example, the metadata / alarm manager 342.3 may add one or more detection event tags to video metadata 320.7, send alarms or notifications to the access / display manager 338 for communication with the monitoring application 350, and / or initiate additional sequences of event detection and / or processing. In some implementations, the metadata / alarm manager 342.3 may include a set of logical rules in response to multiple sequential and / or parallel detection events, where associated tags, alarm triggering, and processing steps are defined within this set of logical rules. For example, the set of audio privacy modes for the rules in the metadata / alarm manager may be configured to resemble... Figure 2 Additional processing 228 in the system is used to logically navigate a series of event detections using different analysis models, such as human speech detection, speech (speaker) recognition, speech binarization (multiple speaker identification), speech (consent) recognition, facial recognition and / or behavior recognition, and related results. Each event detection may result in one or more metadata tags being added to the video metadata 320.7, and in some cases, trigger additional event detection processes, alarms, and / or video processing. In some embodiments, the metadata / alarm manager 342.3 may include logic for additional conditions or data structures beyond the analysis model library 344 and the target video data and metadata. For example, an audio privacy mode may be logically determined using a protected speaker list 342.4 or another data structure, such as whether audio protection mode processing should be initiated for an identified speaker. In some embodiments, such additional condition information may be received or accessed from other system components, such as the protected speaker list 342.4 configured via user profile parameters in the access / display manager 338.

[0085] The analytical model library 344 may include APIs and a set of functions, parameters, and data structures for storing multiple analytical models for the analytical engine 340 to use in processing video data. For example, the analytical model library 344 may include multiple trained analytical models and corresponding event detection algorithms for different event types, target parameters (e.g., car, license plate, equipment, person, voiceprint, consent pattern, etc.), and / or training conditions. Each analytical model includes a set of basic equations for the analytical model type, a set of target parameters, and one or more weights that determine the event detection algorithm to be used for event detection processing. In some embodiments, the analytical model library 344 may include different sets of analytical models for different video conditions. In some embodiments, the analytical model library 344 may include a set of analytical models associated with a defined processing sequence (e.g., determining an audio privacy pattern). In some embodiments, each analytical model may include or be supported by corresponding APIs and functions, parameters, and data structures for processing a target video stream using one or more corresponding detection algorithms.

[0086] In some implementations, the analysis engine 340 may include or access a human speech detector 344.1 for determining the presence of human speech in audio data (such as audio data 320.3) and adding a human speech condition tag to video metadata 320.7. The human speech detector 344.1 may include a processor configured to detect the presence of human speech within the audio data stream. For example, the human speech detector 344.1 may process the audio data 320.3 to detect audio data signatures indicating human speech within a wider range of audio artifacts, including background, environment, and / or other non-speech sources. In some implementations, the human speech detector 344.1 may execute a speech activity detection (VAD) algorithm on the audio data 320.3 to detect the presence or absence of human speech. For example, the algorithm may focus on the presence of human speech and / or frequencies common to signal patterns indicating speech tone and mode.

[0087] In some implementations, the analysis model library 344 may include or access a speech recognition engine 344.2 for processing the human speech components of audio data 320.3 for specific voiceprints, speech, semantic or emotional content and / or other audio patterns indicating speech content. In some implementations, the speech recognition engine 344.2 may include a processor configured to detect the presence or absence of individual human speech. For example, the speech recognition engine 344.2 may include speaker identification 344.3 to process audio data 320.3 and compare human speech patterns in the audio data with one or more known voiceprints. In some implementations, the speech recognition engine 344.2 may perform speaker identification 344.3 against a voice library of an identified user (e.g., a user configured in access / display manager 338). By comparing a unique set of audio factors constituting human speech (voiceprint) with human speech in the audio data, the speech recognition engine 344.2 may identify whether the speaker is known, and if so, whether the speaker is a protected speaker for enabling an audio privacy mode. For example, event manager 342 may include or access a protected speaker list 342.4 that identifies individuals who should enable audio privacy modes and includes their voiceprints for use by speech recognition engine 344.2. In some embodiments, speaker recognition 344.3 may support speech binarization, wherein multiple individual speakers are detected, identified, and / or tracked through speech components of audio data 320.3. For example, multiple speech patterns and corresponding individual speakers (known or unknown) may be identified in individual human speech components, and decision logic may be applied to protected speakers, unprotected speakers, and combinations thereof, as well as to changes in speakers over time for different portions of the audio data segmented into human speech components. In some embodiments, speech recognition engine 344.2 may also include speech recognition for identifying specific phrases indicating consent that have been recorded. For example, consent recognition 344.4 may process audio data 320.3 to extract specific words or phrases from human speech components. In some implementations, when a speaker on a protected speaker list wishes to have their voice recorded and displayed via monitoring system 300, the detection of consent patterns can enable audio privacy patterns to be bypassed or disabled. For example, consent recognition 344.4 may include a library of consent patterns representing phrases that a speaker can use to indicate consent to be recorded.

[0088] In some embodiments, the analysis model library 344 may include or access a facial recognition engine 344.5 for processing image data 320.2 for additional visual markers targeting a specific face or individual identity. In some embodiments, the facial recognition engine 344.5 may operate similarly to speech recognition 344.3 for identifying a protected speaker, but based on a facial recognition algorithm using facial vectors in the image data instead of voiceprints in the audio data. In some embodiments, the analysis model library 344 may include or access a behavior recognition engine 344.6 for processing image data 320.2 for specific consent patterns or behaviors indicating consent or dissent. In some embodiments, the behavior recognition engine 344.6 may operate similarly to consent recognition 344.4 for identifying the consent of the speaker to be recorded, but based on visual consent patterns in body language or gestures rather than speech content.

[0089] Audio data processor 346 may include an API and a set of functions, parameters, and data structures for processing audio data from a video stream to further enhance audio privacy. For example, audio data processor 346 may be configured to receive audio data 320.3 for camera video stream data 320.1 and return masked audio data 320.6 and / or privacy video stream data 320.4. In some embodiments, audio data processor 346 may be invoked by event manager 342 in response to logic indicating that an audio privacy mode should be triggered for a target video stream or portion of the video stream. Event manager 342 may add metadata tags and / or trigger operation states for processing, storing, and displaying privacy video stream data 320.4 when audio privacy mode is enabled. In some embodiments, audio data processor 346 may first use human speech separator 346.1 to separate human speech or human speech components from audio data 320.3. For example, an algorithm similar to the human speech detector 344.1 can be used to separate the human speech component of audio data 320.3 from the remaining components, which include ambient noise and background noise and / or other sound events that are not human speech. In some embodiments, the human speech separator 346.1 may also support speech binarization and further separate the human speech component of audio data 320.3 into individual speaker speech components among multiple speakers. In some embodiments, a speech mask 346.2 may be applied to the separated human speech component and / or injected into the audio data within the frequency range in which human speech is detected. For example, the human speech component of the data may be removed or canceled with an inverse sound pattern to effectively remove the human speech component from the audio data 320.3. Alternatively, the audio data may be modified by replacing it with white noise, replacing the pitch, or another audio feature that prevents the user from hearing the speech while the video stream is displayed.

[0090] The monitoring application 350 may include interface protocols, functions, parameters, and data structures for providing a user interface for monitoring and viewing surveillance video and / or managing the monitoring system 300, such as through the monitoring controller 330. For example, the monitoring application 350 may be a software application running on a user device integrated with, connected to, or network-communicating with the monitoring controller 330 and / or a hosted network video recorder. In some embodiments, the monitoring application 350 may run on a computing device (such as a personal computer, mobile device, or other user device) separate from the monitoring controller 330. In some embodiments, the monitoring application 350 may be configured to interact with an API presented by the access / display manager 338.

[0091] In some implementations, the monitoring application 350 may include multiple hardware and / or software modules configured to use processor 312 and memory 314 to process or manage defined operations of the monitoring application 350. For example, the monitoring application 350 may include a video manager 352, an alarm manager 354, and an analysis manager 356.

[0092] Video Manager 352 may include APIs and a set of functions, parameters, and data structures for navigating and displaying video streams from cameras accessed via Monitoring Controller 330. For example, Video Manager 352 may include a graphical user interface and interactive controls for displaying lists, tables, thumbnails, or similar interface elements to select and display video streams for specific cameras, times, locations, and / or events. In some implementations, Video Manager 352 may enable split-screen display of multiple camera video streams. For example, near real-time video streams from all active cameras (with predetermined delays based on network latency, storage, and processing times) may be displayed on a monitoring interface, or a set of video streams corresponding to a detected event may be displayed in an event viewing interface. In some implementations, Video Manager 352 may include data structures that aggregate all video data stored in Monitoring System 300 to enable users to locate and view older monitoring videos. For example, a video management log or database may include entries for stored video data indexed by relevant metadata, such as video data parameters (e.g., time, location, camera identifier, format, low light / normal light, etc.), metadata of detected video events (event time, location, type, parameters, etc.), and / or video management parameters (expiration, activity / archiving, access control, etc.).

[0093] In some implementations, the video manager 352 may be configured to selectively provide an audio-private video stream based on the privacy video stream data 320.4 when an audio privacy mode is enabled (e.g., when a human speaking condition is detected). For example, the video manager 352 may receive the privacy video stream data 320.4 from the access / display manager 338 as part of a display path triggered by the audio privacy mode. The video manager 352 may enable playback of the audio-private video stream, where image data 320.4 is displayed on a graphical user display and masked audio data 320.6 is displayed (or played) through speakers. In some implementations, the video manager 352 may receive a user identifier and / or security credentials for authentication via the access control service 338.1. For example, a user may log in to the monitoring application 350 via a user interface on a user device hosting the monitoring application 350. In some implementations, some users may be able to selectively access the audio-private video stream and the camera video stream without modifying the audio data. For example, a video stream with an audio privacy mode marker may initially be displayed using privacy video stream data 320.4, and a user with audio access permissions, including audio privacy bypass, may choose to access or display the video stream based on camera video stream data 320.1 to hear the original audio data in which human speech components are not masked or otherwise modified. In some embodiments, video manager 352 may include an event overlay function 352.2 configured to overlay metadata markers and / or video event indication identifiers onto the video stream displayed to the user. For example, video manager 352 may read video metadata provided with the video stream to be displayed and determine one or more visual indication identifiers based on the video metadata, such as alphanumeric markers, graphic frames or blocks, static or video image interpolations, etc. This overlay of the video stream with masked audio data can indicate an audio privacy mode to the user.

[0094] Alarm Manager 354 may include an API and a set of functions, parameters, and data structures for setting, monitoring, and displaying alarms based on detected video events. For example, a user can define a set of trigger events that generate visual, auditory, tactile, and / or notification-based (email, text message, automated call, etc.) alarms on a user device. In some implementations, Alarm Manager 354 may include multiple preset alarm conditions with associated event parameters and allow the user to enable and disable alarm types and / or change associated event parameters. In some implementations, Alarm Manager 354 may be configured to operate in conjunction with Event Overlay Function 352.2 to overlay graphical elements or event indicators representing detected events onto the video stream displayed through Video Manager 352. For example, detected motion, objects, or faces may be bound or highlighted, marked with relevant identifiers, or otherwise indicated in video playback on the user device.

[0095] Analysis Manager 356 may include an API and a set of functions, parameters, and data structures for selecting, training, and managing event detection algorithms. For example, Analysis Manager 356 may include a user interface to Analysis Model Library 344 for use with one or more Analysis Engines 340. In some embodiments, the event detection algorithm may include a pre-configured set of parameters and / or model weights based on a training dataset processed independently of Monitoring System 300. For example, Analysis Manager 356 may include object detection algorithms configured for common objects, situations, and cameras. In some embodiments, Analysis Manager 356 may include access to a training service and / or a pre-configured training dataset. For example, Analysis Manager 356 may enable users to define a training dataset for determining or refining event detection algorithm parameters and model weights based on a predefined base algorithm or model. In some embodiments, Analysis Manager 356 may interface directly with Analysis Engine 340 for selecting, training, managing, and using event detection algorithms configured through Analysis Manager 356. In some embodiments, Analysis Manager 356 may interface with Access / Display Manager 338 for accessing and managing one or more Analysis Engines 340.

[0096] like Figure 4 As shown, an exemplary method for providing an audio privacy mode can be used, namely according to Figure 4 Method 400, shown in boxes 410 to 424, operates the monitoring system 300.

[0097] At box 410, light waves can be received at the camera's image sensor. For example, the camera can be activated to capture video continuously or in response to triggering conditions.

[0098] At box 412, sound waves can be received at the camera microphone. For example, the camera can be configured to capture audio synchronized with its video image capture when the microphone is enabled.

[0099] At box 414, a surveillance video stream can be generated in response to capturing image and audio data. For example, a camera can convert image sensor and microphone data into synchronized image and audio data, encode the video stream with at least one audio channel, and provide the camera video stream to the surveillance controller.

[0100] At box 416, the condition for human speech can be determined. For example, a human speech detector in a camera of a surveillance controller can detect at least a portion of the video stream as including human speech.

[0101] At box 418, audio data can be determined based on the video stream. For example, an audio data processor can read and / or separate audio data, such as audio channels, from the image data used for processing.

[0102] At box 420, the audio data can be selectively modified. For example, during a time period that includes human speech conditions or a video segment, the audio data processor can at least mask the human speech component of the audio data.

[0103] At box 422, a video stream with modified audio data can be stored. For example, a monitoring controller can store audio privacy video stream data (including modified audio channels or tracks) to one or more storage devices.

[0104] At box 424, a video stream with modified audio data can be displayed to the user. For example, a monitoring application running on the user's device can receive private video stream data from a monitoring controller and selectively display it on the user's device's graphical user interface and speakers.

[0105] like Figure 5 As shown, an exemplary method for selectively modifying audio data for an audio privacy mode can be used, i.e., according to... Figure 5 Method 500 shown in boxes 510 to 532, operation monitoring system 300.

[0106] At box 510, a surveillance video stream can be received. For example, the surveillance controller can receive an encoded camera video stream that includes a synchronized audio channel.

[0107] At box 512, a human speech condition can be detected. For example, a human speech detector can determine that at least one human voice is present in one or more portions of the camera video stream. If no human speech condition is detected, method 500 can proceed to box 530 to process the unmodified video stream. If a human speech condition is detected, method 500 can proceed to box 514 to process according to an audio privacy mode, thereby protecting the privacy of the human speech component of the video stream.

[0108] At box 514, an audio data track can be selected. For example, the audio processor can select an audio track or channel from a video stream that includes human speech components.

[0109] At box 516, human speech components can be determined. For example, an audio processor can use a human speech detector or similar algorithm to identify frequencies or signal patterns containing human speech components.

[0110] At box 518, human speech components can be separated. For example, an audio processor can separate the identified frequencies or signal patterns from the rest of the audio data, such as background noise and non-human speech sources or events.

[0111] At box 520, the human speech component can be masked. For example, the audio processor can inject white noise, masked pitch, or other audio signals to cancel or obscure the human speech component in the audio data.

[0112] At box 522, the masked human speech component can be merged with the remaining non-speech components. For example, after separating or removing the human speech component, the masked human speech component can be combined with the remaining audio data.

[0113] At box 524, a selected audio track from the video stream can be replaced with a modified audio track. For example, the audio processor can provide modified audio data with masked human speech components to synchronize with and / or re-encode with the image data from the video stream.

[0114] In box 526, a video stream with modified audio data can be stored. For example, a monitoring controller can store audio privacy video stream data (including modified audio channels or tracks) to one or more storage devices.

[0115] At box 528, a video stream with modified audio data can be displayed to the user. For example, a monitoring application running on the user's device can receive private video stream data from a monitoring controller and selectively display it on the user's device's graphical user interface and speakers.

[0116] In box 530, unmodified camera video streams can be stored. For example, a monitoring controller can store camera video stream data (including raw audio channels or tracks) to one or more storage devices.

[0117] At box 532, an unmodified camera video stream can be displayed to the user. For example, a monitoring application running on the user's device can receive camera video stream data from the monitoring controller and selectively display it on the user's device's graphical user interface and speakers, with human voice intact.

[0118] like Figure 6 As shown, an exemplary method for triggering audio privacy mode can be used, namely according to... Figure 6 Method 500 shown in boxes 610 to 638, operation monitoring system 300.

[0119] At box 610, a surveillance video stream can be received. For example, the surveillance controller can receive an encoded camera video stream that includes a synchronized audio channel.

[0120] At box 612, audio data can be processed for speech detection. For example, a human speech detector can process audio data from a surveillance video stream to determine whether human speech is present in the audio data.

[0121] At box 614, parallel to or as an alternative to box 612, video data can be processed for behavior recognition. For example, the behavior recognition engine can process image data from a surveillance video stream to determine whether one or more people in the image might be speaking, for example, based on mouth movements or other body language.

[0122] At box 616, it can be determined whether a person is speaking based on boxes 612 and / or 614. For example, the event manager can evaluate whether the human voice detector or behavior recognition engine returns a positive human voice indicator. If not, audio privacy mode can be disabled at box 638. If yes, method 600 can proceed to boxes 618 and / or 620.

[0123] At box 618, audio data can be processed for speaker recognition. For example, a speech recognition engine can process audio data from a surveillance video stream to identify one or more speakers in the audio data.

[0124] At box 620, parallel to or as an alternative to box 618, video data can be processed for facial recognition. For example, a facial recognition engine can process image data from a surveillance video stream to determine whether one or more people in the image are likely to be identifiable speakers.

[0125] At box 622, one or more speaker identities from boxes 618 and / or 620 can be compared with a list of protected speakers. For example, the event manager can compare identified speakers with a list of protected speakers to determine whether one or more speakers are on the list of protected speakers.

[0126] At box 624, it can be determined whether a protected speaker has been detected from the surveillance video stream based on boxes 628 and / or 630 and 632. If a protected speaker is detected, method 600 can enable an audio privacy mode at box 626. If no protected speaker is detected, method 600 can continue to boxes 628 and / or 630.

[0127] At box 626, an audio privacy mode can be enabled. For example, the audio processor can manipulate the audio data, such as targeting... Figure 5 The method described in 500.

[0128] At box 628, audio data can be processed for speech content recognition. For example, a speech recognition engine can process audio data from a surveillance video stream to determine if any consent patterns appear in the speech.

[0129] At box 630, in parallel with or as an alternative to box 628, video data can be processed for behavior recognition. For example, a behavior recognition engine can process image data from a surveillance video stream to determine whether all speakers have expressed consent through patterns of consent in their behavior.

[0130] At box 632, one or more patterns from boxes 628 and / or 630 can be compared to a list of consent patterns. For example, the speech recognition engine and the behavior recognition engine can maintain a set of consent patterns mapped to specific phrases or body language that can be detected from their respective data.

[0131] At box 634, it can be determined whether consent was detected from the monitored video stream based on boxes 628 and / or 630 and 632. For example, the event manager can evaluate whether the speech recognition engine or behavior recognition engine returned positive consent indicators for all speakers. If yes, audio privacy mode can be disabled at box 638. If no, audio privacy mode can be enabled at box 636.

[0132] At box 636, an audio privacy mode can be enabled. For example, the audio processor can manipulate the audio data, such as targeting... Figure 5 The method described in 500.

[0133] At box 638, audio privacy mode can be disabled. For example, a monitoring controller can process the raw video stream without modifying the audio data.

[0134] like Figure 7 As shown, an exemplary method for displaying a video stream with an audio privacy mode on a user device can be used, i.e., according to... Figure 7 Method 700, shown in boxes 710 to 722, operates the monitoring system 300.

[0135] At box 710, available video can be navigated via a graphical user interface. For example, a monitoring application running on the user's device can provide a video navigation interface that enables the selection, display, and navigation of video streams, such as play, pause, stop, forward playback, reverse playback, fast forward, slow down, etc.

[0136] At box 712, you can select a monitoring video stream for display. For example, a monitoring application can allow users to select video stream segments to view.

[0137] At box 714, a video stream with audio privacy enabled can be displayed. For example, a monitoring application can retrieve selected video stream segments from stored or streamed video data managed by a monitoring controller, the selected video stream being marked with an audio privacy mode tag or other indicator, the monitoring controller providing an audio privacy-enabled version of the video stream with modified audio data, and the monitoring application displaying the privacy-enabled version of the received video stream on the user's graphical interface display and speakers.

[0138] At box 716, security credentials can be determined for the user. For example, a monitoring application may require the user to log in with security credentials before displaying any video, or it may prompt for security credentials in response to an audio privacy mode flag, allowing the user to bypass the voice privacy mode if their credentials permit.

[0139] At box 718, audio access permissions associated with the user and security credentials can be verified. For example, a monitoring controller may include an access control list associated with a user profile and determine whether the user has permission to bypass the audio privacy of a target video stream.

[0140] At box 720, video streams with audio privacy enabled are accessible. For example, in response to security credentials and audio access permissions, the monitoring controller can provide the monitoring application with raw camera video streams containing unmodified audio data.

[0141] At box 722, a video stream with audio privacy enabled can be displayed. For example, a surveillance application can display the received raw camera video stream with unmodified audio data on the user's device's graphical interface display and speakers.

[0142] While at least one exemplary embodiment has been presented in the foregoing detailed description of the present technology, it should be understood that numerous variations are possible. It should also be understood that one or more exemplary embodiments are examples and are not intended to limit the scope, applicability, or configuration of the technology in any way. Rather, the foregoing detailed description will provide those skilled in the art with convenient guidance for implementing exemplary embodiments of the present technology, and it should be understood that various modifications can be made to the function and / or arrangement of the elements described in the exemplary embodiments without departing from the scope of the technology set forth in the appended claims and their legal equivalents.

[0143] As will be understood by those skilled in the art, various aspects of this technology can be embodied as a system, method, or computer program product. Therefore, some aspects of this technology can take the form of a wholly hardware implementation, a wholly software implementation (including firmware, resident software, microcode, etc.), or a combination of hardware and software aspects, which are generally referred to herein as circuits, modules, systems, and / or networks. Furthermore, various aspects of this technology can take the form of a computer program product embodied in one or more computer-readable media, including computer-readable program code embodied thereon.

[0144] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a physical computer-readable storage medium. For example, a physical computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, crystal, polymer, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. Non-limiting examples of physical computer-readable storage media include, but are not limited to, electrical connections including one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, optical fiber, compact optical disc read-only memory (CD-ROM), optical processors, magnetic processors, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store programs or data for use by or in connection with an instruction execution system, apparatus, and / or device.

[0145] Computer code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, radio frequency (RF), or any suitable combination thereof. The computer code used to perform various aspects of this technology may be written in any statically typed language, such as the C programming language or other similar programming languages. The computer code may execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the latter case, the remote computing device may be connected to the user's computing device via any type of network or communication system, including but not limited to establishing a local area network (LAN) or wide area network (WAN), network aggregation, or connection to an external computer (e.g., via the Internet provided by an Internet service provider).

[0146] The various aspects of this technology can be described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing apparatus (processor) of a general-purpose computer, special-purpose computer, or other programmable data processing device to produce a machine, such that instructions executable by the processing apparatus or other programmable data processing device create means for implementing the operations / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0147] Some computer program instructions may also be stored in a computer-readable medium that may instruct a computer, other programmable data processing apparatus, or one or more other devices to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing including instructions that implement the operations / actions specified in one or more boxes of a flowchart and / or block diagram. Some computer program instructions may also be loaded onto a computing device, other programmable data processing apparatus, or one or more other devices to cause a series of operational steps to be performed on the computing device, other programmable apparatus, or one or more other devices to produce a computer-implemented process, such that the instructions, which are executable by the computer or other programmable apparatus, provide one or more processes for implementing one or more operations / actions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0148] The flowcharts and / or block diagrams in the above figures illustrate the architecture, functionality, and / or operation of possible specific implementations of apparatus, systems, methods, and / or computer program products according to various aspects of the present technology. In this regard, boxes in flowcharts or block diagrams may represent modules, segments, or portions of code, which may include one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative aspects, some functions shown in the boxes may occur in a different order than shown in the figures. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order, depending on the operations involved. It should also be noted that boxes or combinations of boxes in block diagrams and / or flowcharts may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, capable of performing one or more specified operations or actions.

[0149] Although one or more aspects of the present technology have been described and discussed in detail, those skilled in the art will understand that modifications and / or adjustments may be made to various aspects without departing from the scope of the present technology as set forth in the appended claims.

Claims

1. A monitoring system, comprising: The camera includes: An image sensor, configured to capture image data from a video stream; and A microphone configured to capture audio data from the video stream; and The controller is configured to: Receive the video stream from the camera; Determine the conditions for a person to speak based on the video stream; Determine the human speech component of the audio data during the human speaking condition; In response to the human speaking condition, and using a facial recognition algorithm, the image data in the video stream is processed to identify the speaker identity corresponding to the human speech component; Based on comparing the speaker identity with the protected speaker data structure, it is determined that the speaker identity has a protected speaker status; In response to determining that the speaker identity has the protected speaker state and using a speech recognition algorithm, the audio data in the video stream corresponding to the speaker identity is processed to identify an audio consent pattern, wherein identifying the audio consent pattern includes identifying at least one word in the human speech component of the audio data, the at least one word being configured to indicate that consent is recorded; In response to determining that the speaker identity has the protected speaker state and that the audio consent pattern is not present in the audio data corresponding to the speaker identity, the audio data in the video stream is modified during the speaker speaking condition to mask the speaker's speech components; and The video stream containing the modified audio data is stored.

2. The monitoring system according to claim 1, wherein: The controller is also configured to process the image data in the video stream corresponding to the speaker identity in response to determining that the speaker identity has the protected speaker state, in order to identify a visual consent pattern; The visual consent pattern includes a speaker's gesture corresponding to the speaker's identity, and indicates that consent is recorded; The controller processes the audio data for the audio consent mode and the image data for the visual consent mode in parallel. and The audio data is modified in response to the absence of the visual consent mode.

3. The monitoring system of claim 1, wherein identifying the audio consent pattern includes identifying a phrase selected from a plurality of phrases indicating that consent has been recorded.

4. The monitoring system according to claim 1, wherein processing the audio data to mask the human speech component of the audio data comprises: The human speech component of the audio data is separated from the other non-speech components of the audio data; The human speech component of the audio data is masked; as well as The remaining non-speech components of the audio data remain unchanged.

5. The monitoring system according to claim 1, wherein processing the audio data to mask the human speech component of the audio data further comprises: The human speech component of the audio data is separated into multiple human speech components corresponding to individual speakers among multiple speakers; Identify at least one individual speaker among the plurality of speakers; as well as In response to identifying the at least one individual speaker, a portion of the human speech component corresponding to the audio data of the at least one individual speaker is masked.

6. The monitoring system according to claim 1, wherein: The controller is further configured to: The human speech component of the audio data is separated into multiple human speech components corresponding to individual speakers among multiple speakers; and The speech recognition algorithm is used to process the audio data in the video stream to identify the separated human speech components corresponding to the speaker identity with a protected speaker status. and Modifying the audio data in the video stream during the human speaking condition includes: Masking the separated human speech components corresponding to the speaker identity with a protected speaker state; as well as Other speech components among the multiple human speech components in the audio data remain unchanged.

7. The monitoring system according to claim 1, wherein: The controller is further configured to process the image data in the video stream to identify visual consent patterns; The visual consent pattern includes body language or gestures that indicate consent; and The audio data in the video stream is modified during the person speaking condition in response to the absence of the visual consent pattern in the image data.

8. The monitoring system according to claim 1, wherein: The controller is further configured to detect video events from the image data in the video stream; and The condition of determining the person speaking based on the video stream is in response to the detection of the video event.

9. The monitoring system according to claim 1, wherein: The controller is also configured to process the audio data in the video stream using the speech recognition algorithm to identify the speaker identity corresponding to the human speech component; and Determining that the speaker's identity has a protected speaker status includes: A set of audio factors from the audio data is compared with a known voiceprint in the protected speaker data structure; as well as A set of facial vectors from the image data is compared with known facial vectors in the protected speaker data structure.

10. The monitoring system according to claim 1, further comprising: Storage device, the storage device being configured to store: An unmodified video stream from the camera; and The video stream with modified audio data from the controller; Analysis engine, which is configured as follows: Process the unmodified video stream to determine the person's speaking conditions based on the video stream; as well as The controller is notified of the person's speaking conditions; and User equipment, the user equipment being configured to: The user equipment's graphical user interface and speakers are used to display the video stream with modified audio data to the user; Determine the user's security credentials; Verify the audio access permissions corresponding to the security credentials; as well as The user device uses its graphical user interface and speaker to display the unmodified video stream to the user.

11. A computer-implemented method, comprising: Receives a video stream from a camera, wherein the camera includes: An image sensor, configured to capture image data from the video stream; and A microphone configured to capture audio data from the video stream; Determine the conditions for a person to speak based on the video stream; Determine the human speech component of the audio data during the human speaking condition; In response to the human speaking condition, and using a facial recognition algorithm, the image data in the video stream is processed to identify the speaker identity corresponding to the human speech component; Based on comparing the speaker identity with the protected speaker data structure, it is determined that the speaker identity has a protected speaker status; In response to determining that the speaker identity has the protected speaker state and using a speech recognition algorithm, the audio data in the video stream corresponding to the speaker identity is processed to identify an audio consent pattern, wherein identifying the audio consent pattern includes identifying at least one word in the human speech component of the audio data, the at least one word being configured to indicate that consent is recorded; In response to determining that the speaker identity has the protected speaker state and that the audio consent pattern is not present in the audio data corresponding to the speaker identity, the audio data in the video stream is modified during the speaker speaking condition to mask the speaker's speech components; and The video stream containing the modified audio data is stored.

12. The computer-implemented method according to claim 11, further comprising: In response to determining that the speaker identity has the protected speaker state, the image data in the video stream corresponding to the speaker identity is processed to identify a visual consent pattern; in The visual consent pattern includes a speaker's gesture corresponding to the speaker's identity, and indicates that consent is recorded; The processing of the audio data for the audio consent mode and the image data for the visual consent mode occurs in parallel; and The audio data is modified in response to the absence of the visual consent mode.

13. The computer-implemented method of claim 11, wherein identifying the audio consent pattern includes identifying a phrase selected from a plurality of phrases indicating that consent has been recorded.

14. The computer-implemented method of claim 11, wherein modifying the audio data to mask the human speech component of the audio data comprises: The human speech component of the audio data is separated from the other non-speech components of the audio data; The human speech component of the audio data is masked; as well as The remaining non-speech components of the audio data remain unchanged.

15. The computer-implemented method of claim 11, wherein modifying the audio data to mask the human speech component of the audio data comprises: The human speech component of the audio data is separated into multiple human speech components corresponding to individual speakers among multiple speakers; Identify at least one individual speaker among the plurality of speakers; as well as In response to identifying the at least one individual speaker, a portion of the human speech component corresponding to the audio data of the at least one individual speaker is masked.

16. The computer-implemented method according to claim 11, further comprising: The human speech component of the audio data is separated into multiple human speech components corresponding to individual speakers among multiple speakers; as well as The speech recognition algorithm is used to process the audio data in the video stream to identify the separated human speech components corresponding to the speaker identity having a protected speaker status, wherein modifying the audio data in the video stream during the human speaking condition includes: Masking the separated human speech components corresponding to the speaker identity with a protected speaker state; as well as Other speech components among the multiple human speech components in the audio data remain unchanged.

17. The computer-implemented method according to claim 11, further comprising: Process the image data in the video stream to identify visual consent patterns; in The visual consent pattern includes body language or gestures that indicate consent; and The audio data in the video stream is modified during the person speaking condition in response to the absence of the visual consent pattern in the image data.

18. The computer-implemented method according to claim 11, further comprising: The speech recognition algorithm is used to process the audio data in the video stream to identify the speaker identity corresponding to the human speech component, wherein determining that the speaker identity has a protected speaker status includes: A set of audio factors from the audio data is compared with a known voiceprint in the protected speaker data structure; and A set of facial vectors from the image data is compared with known facial vectors in the protected speaker data structure.

19. The computer-implemented method according to claim 18, further comprising: Stored in a storage device: An unmodified video stream from the camera; and The video stream containing the modified audio data; Process the unmodified video stream to determine the person's speaking conditions based on the video stream; The user equipment's graphical user interface and speakers are used to display the video stream with modified audio data to the user; Determine the user's security credentials; Verify the audio access permissions corresponding to the security credentials; as well as The user device uses its graphical user interface and speaker to display the unmodified video stream to the user.

20. A monitoring system, comprising: The camera includes: An image sensor, configured to capture image data from a video stream; and A microphone configured to capture audio data from the video stream; processor; Memory; A means for receiving the video stream from the camera; A device for automatically determining the conditions under which a person is speaking based on the video stream; A means for determining human speech components of audio data during the human speaking condition; A device for processing the image data in the video stream in response to the human speaking condition and using a facial recognition algorithm to identify the speaker identity corresponding to the human speech component; A means for determining that a speaker identity has a protected speaker status by comparing the speaker identity with a protected speaker data structure; A means for processing audio data in the video stream corresponding to the speaker identity in response to determining that the speaker identity has the protected speaker state and using a speech recognition algorithm to identify a consent pattern, wherein identifying the consent pattern includes identifying at least one word in the human speech component of the audio data, the at least one word being configured to indicate that consent is recorded; A means for modifying the audio data in the video stream to mask the human speech components during a person speaking condition in response to determining that the speaker identity has the protected speaker state and that the consent pattern is not present in the audio data corresponding to the speaker identity; and A means for displaying the video stream using modified audio data.

Images