Systems and methods for automatically extracting and matching labels derived from a camera feed for moving sound sources recorded with a microphone array

Abstract

A method for labeling audio data comprises receiving video stream data and audio stream data corresponding to at least a portion of the video stream data. The method also comprises labeling at least some objects within the video stream data. Furthermore, the method comprises calculating at least one offset value for at least one portion of the audio stream data corresponding to at least one labeled object within the video stream data. The method also comprises synchronizing at least one portion of the video stream data with the corresponding portion of the audio stream data. Finally, the method comprises labeling at least the portion of the audio stream data corresponding to the at least one labeled object within the video stream data and generating training data using at least some of the labeled portions of the audio stream data.

Description

Technical area

[0001] The present disclosure relates to computer systems having artificial intelligence capabilities, including neural networks. In embodiments, this disclosure relates to the automatic extraction and matching of labels derived from video data for moving sound sources recorded with an audio acquisition array. background

[0002] When developing data for training machine learning models, collecting and labeling data, especially sound data, is a laborious, costly, and time-consuming undertaking that can represent a significant bottleneck in most current machine learning pipelines. People use a variety of sound signals from an environment when making decisions in everyday life. Developers are increasingly attempting to integrate such decision-making into various machine learning models. While techniques for using a machine learning model to understand human speech are relatively widespread, using a machine learning model to understand non-speech environmental sounds is a comparatively new and rapidly growing area of ​​interest. Brief description

[0003] One aspect of the disclosed embodiments includes a method for generating training data. The method includes receiving audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment, receiving video stream data associated with the data acquisition environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image acquisition array disposed in the data acquisition environment at a distance from the at least one audio acquisition array, and identifying one or more objects in the video stream data using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions.The method further comprises, for at least one of the one or more objects identified in the video stream data, identifying a portion of the audio stream data corresponding to the at least one object using outputs of a probabilistic audio event detector. The method further comprises calculating at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition feature, and calculating a shift value for the at least one object based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object.The method further comprises synchronizing the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object using at least the at least one offset value and the shift value, and labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value. The method further comprises generating training data using at least one of the labeled portion of the audio stream data.

[0004] Another aspect of the disclosed embodiments features a system for training a machine learning model for audio recognition. The system includes a processor and a memory. The memory includes instructions that, when executed by the processor, cause the processor to: receive audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment; receive video stream data associated with the data acquisition environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image acquisition array disposed in the data acquisition environment at a distance from the at least one audio acquisition array;Identifying, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data; for at least one object of the one or more objects identified in the video stream data, identifying, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; calculating, based on at least one data capture feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data;Calculating, for the at least one object, a shift value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object; synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object; labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; generating training data using at least one of the labeled portions of the audio stream data;and training a machine learning model for audio recognition using the training data.;

[0005] Another aspect of the disclosed embodiments includes an apparatus for training a machine learning model for audio recognition. The apparatus includes a processor and a memory. The memory includes instructions that, when executed by the processor, cause the processor to: receive audio stream data associated with at least one audio capture device located in a data capture environment; receive video stream data associated with the data capture environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image capture device located in the data capture environment at a distance from the at least one audio capture device;Identifying, using outputs from at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data; for at least one object of the one or more objects identified in the video stream data, identifying, using outputs from a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; calculating, based on at least one data capture feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data;Calculating, for the at least one object, a shift value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object; synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object; labeling, using at least one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; generating training data using at least one of the labeled portions of the audio stream data;and training a machine learning model for audio recognition using the training data, wherein the machine learning model for audio recognition, trained using the training data, is configured to recognize one or more events in audio stream data provided as input to the machine learning model for audio recognition; Short description of the drawings Fig. 1 generally shows a system for training a neural network according to the principles of the present disclosure. Fig. 2 generally shows a computer-implemented method for training and using a neural network according to the principles of the present disclosure. The Fig. 3A-3B generally illustrate an audio data labeling system according to the principles of the present disclosure. Fig. 4 is a flowchart generally illustrating an audio data labeling method according to the principles of the present disclosure. Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine and a control system according to the principles of the present disclosure. Fig. Figure 6 shows a schematic diagram of the control system of Fig. 5, which is configured to control a vehicle, which may be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot, according to the principles of the present disclosure. Fig. Figure 7 shows a schematic diagram of the control system of Fig. 5, which is designed to control a manufacturing machine, such as a punch cutter, a cutting device or a deep hole drill, of a manufacturing system, for example a part of a production line. Fig. Figure 8 shows a schematic diagram of the control system of Fig. 5, which is designed to control a power tool, such as a drill or a screwdriver, which has an at least partially autonomous mode. Fig. Figure 9 shows a schematic diagram of the control system of Fig. 5, which is designed to control an automated personal assistant. Fig. Figure 10 shows a schematic diagram of the control system of Fig. 5, which is designed to control a surveillance system, such as an access control system or an observation system. Fig. Figure 11 shows a schematic diagram of the control system of Fig. 5, which is designed to control an imaging system, for example an MRI device, an X-ray imaging device or an ultrasound device. Detailed description

[0006] Embodiments of the present disclosure are described herein. It should be understood, however, that the disclosed embodiments are merely examples, and that other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the embodiments. As will be understood by those skilled in the art, various features illustrated and described with reference to one of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described.The combinations of features presented provide representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0007] As described, when developing data for training machine learning models, the collection and labeling of data, especially sound data, is a laborious, costly, and time-consuming undertaking that can represent a significant bottleneck in most current machine learning pipelines. Humans use a variety of sound signals from an environment when making decisions in everyday life. Developers are increasingly attempting to integrate such decisions into various machine learning models. While techniques for using a machine learning model to understand human speech are relatively widespread, using a machine learning model to understand non-speech environmental sounds is a comparatively newer field and a rapidly growing topic.

[0008] Recent advances in deep learning technology for images and text result from access to a large amount of labeled data. However, collecting and strongly labeling audio data (e.g., labeling the event type, as well as the start and end times of the event in an audio sample) can be relatively difficult. Collecting audio data is generally a challenging task because, in real-world scenarios, an essentially unlimited audio vocabulary is available, which can make predicting the vocabulary for a given task essentially impossible.

[0009] Unlike spoken words, which use a limited number of alphabets, the variations of ambient sounds are unlimited. Consequently, collecting specific sounds in a predefined environment is not realistic. Therefore, most audio data is acquired in a continuous environment, and later, human annotators extract and label the desired audio events. Such labeling can be relatively time-consuming (e.g., because annotators must listen to data for hours to locate the events of interest or target events). Furthermore, precisely labeling the beginning and end of an audio event after locating the target events can be noisy and subjective due to the transient nature of the signal, which typically lacks sharp boundaries.

[0010] Another common method for collecting audio data is to use available data (e.g., from various public sources). However, such data is typically of low quality, and associated labels for such data can often be automatically generated using the title of the audio or video file associated with the data. These labels are sometimes noisy and are usually weak labels (e.g., a section of the audio file is labeled as an audio event, but the exact time boundaries of the event are not specified). This typically presents a number of challenges for training traditional machine learning models and deep learning models.

[0011] Accordingly, systems and methods, such as those described herein, may be desirable that are configured to couple audio data capture with a video stream and integrate the advancements in the image domain to automatically label audio data. In some embodiments, the systems and methods described herein may be configured to capture and annotate moving sound sources using an audio capture array, such as a microphone array or other suitable audio capture array or device.

[0012] In some embodiments, the systems and methods described herein may be configured to use a tandem of a video capture device (e.g., a camera or other suitable image capture device) and an audio capture array to collect data. The systems and methods described herein may be configured to automatically label the moving sources appearing in the video data. The systems and methods described herein may be configured to incorporate a signal processing-based approach to synchronize the collected audio and video data. The systems and methods described herein may be configured to extract the corresponding audio labels. The systems and methods described herein may be configured to use any suitable combination of a microphone array and other sensors that enable automatic labeling for moving sound sources.

[0013] In some embodiments, the systems and methods described herein may be configured to collect both audio and video stream data from a data capture and collection environment. The systems and methods described herein may be configured to use pre-trained object detection models to automatically label and track objects in the video stream data and label corresponding audio data.

[0014] The systems and methods described herein may be configured to identify a misalignment between audio and video stream data based on a positioning of the image capture device relative to the audio capture array and based on a field of view of the image capture device. The systems and methods described herein may be configured to map the video labels to the audio data based on a time shift between the image capture device and the audio capture array. In some embodiments, the systems and methods described herein may be configured to manually observe one or more samples of the audio data and the video data to identify the time shift (e.g., assuming the time shift is constant).

[0015] Additionally or alternatively, the systems and methods described herein may be configured to use a probabilistic approach to automatically match object labels extracted from the video stream data to the audio stream data in response to the time shift between the image capture device and the audio capture array not being constant (e.g., due to the view angle of the image capture device, the speed of the object, the direction of movement of the object, a variance between asynchronous audio and video capture systems, and the like).

[0016] The systems and methods described herein may be configured to label moving sound events associated with visual cues / actions. The systems and methods described herein may be configured to extract the direction of a moving object from the video stream data. The systems and methods described herein may be configured to use the extracted direction of the moving object as a label for audio event localization and arrival direction estimation (e.g., the task of detecting the location of the sound and the relative direction of the sound). In some embodiments, the systems and methods described herein may be configured to provide an initial labeling prediction and receive further annotations from a user.

[0017] As described, the systems and methods described herein can be configured to use a probabilistic approach to automatically match object labels extracted from the video stream data with the audio stream data. Since image capture devices and audio capture devices typically do not share the same clock (e.g., as two separate devices), the systems and methods described herein can be configured to perform synchronization between the object labels extracted from the video stream data and the sound events recorded by the audio capture array.Such synchronization can be particularly difficult when the field of view of the image capture device is such that the maximum sound intensity for a given event occurs when the object is not in the center of a video frame, because objects moving in different directions have different offsets between the label extracted from the video stream data and the sound event recorded in the audio stream data.

[0018] In some embodiments, the systems and methods described herein may, when given a list of automatically detected objects from the video stream data, where each object v = (t s , t e , d, l) by a start time t s , an end time t e , a label l and a direction d (from left to right, from right to left) labeled, for each object, be designed to calculate a central time as follows: tc=ts+te−ts2

[0019] The systems and methods described herein may be designed to a (t), t ∈ [t c - δ, t c + δ], ie the audio stream data between t c - δ and t c + δ, where δ is a sufficiently large margin for the sound of the particular object to be audible.

[0020] The systems and methods described herein may be designed to a (t, l, d), the probability over time that an object with the label l and direction d in y a (t). For example, the systems and methods described herein may be configured to calculate the probability function L a (t, l, d) if y a (t) is a collection of signals from the audio acquisition array, by: filtering y a(t) to obtain only the bandwidth of interest for the particular type of sound event l; extracting a potentially overlapping time window from y a (t); and, for each window, calculating the response of a beamforming algorithm oriented to a predefined set of angles. For N a Angles and N t Time windows this leads to a direction-of-arrival (DoA) map D ∈ R Na×Nt , where each element D(i,j), i ∈ [1, N a ],j ∈ [1, N t ] represents the energy from the i-th angle in the j-th time window.

[0021] The systems and methods described herein may be configured to execute a beamforming algorithm that receives as input the signals recorded by the audio detection array and a steering direction. The systems and methods described herein may be configured to calculate the acoustic energy associated with that direction. For example, a beamforming algorithm that is steered in a direction α (e.g., an angle between the audio detection array 304 and a first object, such as a first vehicle 308) or β (e.g., an angle between the audio detection array 304 and a second object, such as a second vehicle 310) will exhibit higher energy than when steered in other directions, as in Fig. 3B.

[0022] The systems and methods described herein may be configured to select a predefined DoA template based on a direction d (e.g., a matrix P ∈ R Na×Np , where N p is the number of time windows spanned by the template). Each element P(i, j), i ∈ [1, N a ],j ∈ [1, N p ] represents the energy from the i-th angle in the j-th time window. L a (t,l, d) ∈ R Nt-Np+1 is the result of the 2D cross-correlation (pattern matching) between D and P, which can be defined as follows: La(t,l,d)=∑i=1Na∑j=1NpP(i,j)⋅D(i,t+j),t∈[1,Nt−Np+1]

[0023] The systems and methods described herein may be configured to provide pattern matching, where a minimal family of patterns P comprises two patterns, one for sound objects moving from left to right and one for sound objects moving from right to left. The systems and methods described herein may be configured to determine the time t p to determine at which L a (t, l, d) is maximum. The offset between the video object and the corresponding sound event is t p - t c .

[0024] In some embodiments, the systems and methods described herein can be configured to integrate a video object detection model into the video stream data to automatically label the corresponding moving sound sources accompanying visual cues / actions. The systems and methods described herein can be configured to integrate the video object detection model to extract a trajectory of moving sound sources. The systems and methods described herein can be configured to use a probabilistic approach to automatically synchronize isolated moving sound events with the corresponding labels from the video object detection model. The systems and methods described herein can be configured to use any other sensors to automatically label moving sound sources.The systems and methods described herein may be configured to provide data synchronization to match labels generated by different sensors.

[0025] In some embodiments, the systems and methods described herein may be configured to receive video stream data from at least one image capture device (e.g., a camera or other suitable image capture device) associated with a data capture environment. It is understood that the systems and methods described herein may be configured to utilize any number of image capture devices.

[0026] The systems and methods described herein may be configured to receive audio stream data corresponding to at least a portion of the video stream data from at least one audio capture array (e.g., one or more microphones or one or more other suitable audio capture arrays or devices). The at least one audio capture array is located remotely from the at least one image capture device or proximate to the at least one image capture device. The systems and methods described herein may be configured to label at least some objects of the video stream data using outputs of at least one first machine learning model configured to provide outputs comprising one or more object detection predictions.

[0027] The systems and methods described herein may be configured to calculate (e.g., using at least one likelihood function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition feature. The systems and methods described herein may be configured to synchronize (e.g., using at least one offset value) at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data.The at least one data acquisition feature may include one or more features of the at least one image acquisition device, one or more features of the at least one audio acquisition array, one or more features corresponding to a position of the at least one image acquisition device relative to the at least one audio acquisition array, one or more features corresponding to a movement of an object in the video stream data, one or more other suitable data acquisition features, or a combination thereof.

[0028] The systems and methods described herein may be configured to label at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value. Each respective label may include an event type, an event start indicator, and an event end indicator. The systems and methods described herein may be configured to generate training data using at least a portion of the labeled portion of the audio stream data. The systems and methods described herein may be configured to train a second machine learning model using the training data.The systems and methods described herein may be configured to recognize, using the second machine learning model, one or more tones / sounds associated with the audio data provided as input to the second machine learning model.

[0029] Fig. 1 shows a system 100 for training a neural network. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, the input interface may be as shown in Fig. 1, may consist of a data storage interface 104 that can access the training data 102 from a data store 106. For example, the data storage interface 104 may be a memory interface or a persistent storage interface, e.g., a hard disk or SSD interface, but may also be a personal, local, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data store 106 may be an internal data store of the system 100, such as a hard disk or SSD, but may also be an external data store, such as a network-accessible data store.

[0030] In some embodiments, the data store 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 may access from the data store 106. However, it is understood that the training data 102 and the data representation 108 of the untrained neural network may each be accessed from a different data store, e.g., via a different subsystem of the data store interface 104. Each subsystem may be of a type as described above for the data store interface 104.

[0031] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on design parameters for the neural network and therefore may not be explicitly stored on the data store 106. The system 100 may further include a processor subsystem 110 that may be configured to provide an iterative function as a replacement for a stack of layers of the neural network to be trained during operation of the system 100. The respective layers of the stack of layers to be replaced may have common weights and receive as input and output of a previous layer, or, for a first layer of the stack of layers, an initial activation and a portion of the input of the stack of layers.

[0032] The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. An iteration of training by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by determining, among other operations defining the forward propagation portion, an equilibrium point of the iterative function at which the iterative function converges to a fixed point. Determining the equilibrium point includes using a numerical root-finding algorithm to find a root solution for the iterative function minus its input, and providing the equilibrium point as a substitute for an output of the stack of layers in the neural network.

[0033] The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network; this data may also be referred to as trained model data 112. As also in Fig. 1, the output interface may, for example, be formed by the data storage interface 104, which in these embodiments is an input / output ('IO') interface via which the trained model data 112 may be stored in the data storage 106. For example, the data representation 108 defining the 'untrained' neural network may be at least partially replaced by the data representation 112 of the trained neural network during or after training by adjusting the parameters of the neural network, such as weights, hyperparameters, and other types of neural network parameters, to reflect the training on the training data 102. This is also shown in Fig. 1 by reference numerals 108, 112, which refer to the same data set in data storage 106. In some embodiments, data representation 112 may be stored separately from data representation 108 that defines the 'untrained' neural network. In some embodiments, the output interface may be separate from data storage interface 104, but generally, it may be of the type described above for data storage interface 104.

[0034] Fig. 2 shows a data annotation / enhancement system 200 for implementing a system for annotating and / or enhancing data. The data annotation system 200 may include at least one computer system 202. The computer system 202 may include at least one processor 204 operatively connected to a memory unit 208. The processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit implementing an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families.

[0035] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the process described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, a network interface, and input / output interfaces into a single integrated device. Computer system 202 may implement an operating system to manage various aspects of operation.

[0036] The storage unit 208 may include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computer system 202 is disabled or loses power. The volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the storage unit 208 may store a machine learning model 210 (e.g., in Fig. 2 as an ML model 210) or an algorithm, a training dataset 212 for the machine learning model 210 and a raw source dataset 216.

[0037] The computer system 202 may include a network interface device 222 configured to enable communication with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface as defined in the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may further be configured to provide a communication interface to an external network 224 or a cloud.

[0038] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may enable easy information and data exchange between computing devices and networks. One or more servers 230 may communicate with the external network 224.

[0039] Computer system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. I / O interface 220 may include additional serial interfaces for communication with external devices (e.g., a USB (Universal Serial Bus) interface).

[0040] Computer system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables system 200 to receive control inputs. Examples of input devices may include human-machine interface input devices such as keyboards, mice, touchscreens, voice input devices, and other similar devices. Computer system 202 may include a display device 232. Computer system 202 may include hardware and software for outputting graphic and text information to display device 232. Display device 232 may include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user or operator.The computer system 202 may be further configured to enable interaction with remote HMI and remote display devices via the network interface device 222.

[0041] System 200 may be implemented using one or more computer systems. While the example shows a single computer system 202 implementing all of the described functions, it is intended that various features and functions may be separated and implemented by multiple computer units communicating with each other. The selected system architecture may depend on a variety of factors.

[0042] The system 200 may implement a machine learning algorithm 210 configured to analyze the raw source dataset 216. The raw source dataset 216 may include raw or unprocessed sensor data, which may represent an input dataset for a machine learning system. The raw source dataset 216 may include video, video segments, images, text-based information, and raw or partially processed sensor data (e.g., a radar map of objects). In some embodiments, the machine learning algorithm 210 may be a neural network algorithm configured to perform a predetermined function. For example, in automotive applications, the neural network algorithm may be configured to identify pedestrians in video images.

[0043] The computer system 200 may store a training dataset 212 for the machine learning algorithm 210. The training dataset 212 may represent a set of previously created data for training the machine learning algorithm 210. The training dataset 212 may be used by the machine learning algorithm 210 to learn weighting factors associated with a neural network algorithm. The training dataset 212 may include a set of source data having corresponding results or outcomes that the machine learning algorithm 210 attempts to duplicate through the learning process. In this example, the training dataset 212 may include source videos with and without pedestrians and corresponding presence and location information. The source videos may include various scenarios in which pedestrians are identified.

[0044] The machine learning algorithm 210 may operate in a learning mode using the training dataset 212 as input. The machine learning algorithm 210 may be executed for a number of iterations using the data from the training dataset 212. At each iteration, the machine learning algorithm 210 may update internal weighting factors based on the results obtained. For example, the machine learning algorithm 210 may compare output results (e.g., annotations) with those contained in the training dataset 212. Because the training dataset 212 has the expected results, the machine learning algorithm 210 may determine when performance is acceptable. After the machine learning algorithm 210 has reached a predetermined level of performance (e.g.,100% agreement with the results associated with the training dataset 212), the machine learning algorithm 210 may be executed using data not included in the training dataset 212. The trained machine learning algorithm 210 may be applied to new datasets to generate annotated data.

[0045] The machine learning algorithm 210 may be configured to detect a specific feature in the raw source data 216. The raw source data 216 may include a plurality of instances or input data sets for which annotation results are desired. For example, the machine learning algorithm 210 may be configured to detect the presence of a pedestrian in video images and annotate the occurrences. The machine learning algorithm 210 may be programmed to process the raw source data 216 to detect the presence of the specific features. The machine learning algorithm 210 may be configured to detect a feature in the raw source data 216 as a predetermined feature (e.g., pedestrian). The raw source data 216 may originate from a variety of sources. For example, the raw source data 216 may be actual input data collected by a machine learning system.The raw source data 216 may be machine-generated to test the system. For example, the raw source data 216 may include raw video images from a camera.

[0046] In the example, machine learning algorithm 210 may process raw source data 216 and output a display of a representation of an image. The output may also include an augmented representation of the image. A machine learning algorithm 210 may generate a confidence level or confidence factor for each generated output. For example, a confidence value that exceeds a predetermined high confidence threshold may indicate that machine learning algorithm 210 is confident that the identified feature corresponds to the particular feature. A confidence value that is below a low confidence threshold may indicate that machine learning algorithm 210 has some uncertainty about whether the particular feature is present.

[0047] As is generally the case in the Fig. 3A and Fig. 3B, a system 300 may include an image capture device 302, an audio capture array 304, and the computer system 202. The system 202 may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video object detection to identify one or more objects in corresponding images of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture array 304. The audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to perform video object detection using the output of at least a first machine learning model (e.g.,the machine learning model 210 or another suitable machine learning model) configured to provide an output comprising one or more object detection predictions to label at least some objects of the video stream data.

[0048] The system 202 may calculate (e.g., using at least one probabilistic function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition characteristic. The system 202 may synchronize at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value.The at least one data acquisition feature may include one or more features of the at least one image acquisition device, one or more features of the at least one audio acquisition array, one or more features corresponding to a position of the at least one image acquisition device relative to the at least one audio acquisition array, one or more features corresponding to a movement of an object in the video stream data, one or more other suitable data acquisition features, or a combination thereof.

[0049] The system 202 may label at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value. Each respective label may include an event type, an event start indicator, and an event end indicator. The system 202 may generate training data using at least a portion of the labeled portion of the audio stream data. The system 202 may train a second machine learning model using the training data. The system 202 may recognize one or more tones / sounds associated with the audio data provided as input to the second machine learning model using the second machine learning model.The second machine learning model may comprise any suitable machine learning model, and may be configured to perform any suitable function as described herein with respect to the . Fig. 6-11 is described.

[0050] Fig. 4 is a flowchart generally illustrating an audio data labeling method 400 according to the principles of the present disclosure. At 402, the method 400 receives video stream data associated with a data capture environment from at least one image capture device. For example, system 202 may receive the video stream data from image capture device 302.

[0051] At 404, method 400 receives audio stream data corresponding to at least a portion of the video stream data from at least one audio capture array. For example, system 202 may receive the audio stream data from audio capture array 304.

[0052] At 406, method 400 labels at least some objects of the video stream data using outputs from at least a first machine learning model configured to provide outputs including one or more object detection predictions. For example, system 202 may label at least some objects of the video stream data using outputs from machine learning model 210.

[0053] At 408, based on at least one data acquisition characteristic, method 400 calculates at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data. For example, system 202 may calculate the at least one offset value.

[0054] At 410, the method 400 synchronizes at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value. For example, the system 202 may synchronize at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value.

[0055] At 412, the method 400 labels at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value. For example, the system 202 may label at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value.

[0056] At 414, the method 400 generates training data using at least a portion of the labeled portion of the audio stream data. For example, the system 202 may generate training data using at least a portion of the labeled portion of the audio stream data.

[0057] At 416, method 400 trains a second machine learning model using the training data. For example, system 202 may train the second machine learning model using the training data.

[0058] Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine 500 and the control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a state of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed state into sensor signals 508 and transmit the sensor signals 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, LiDAR, ultrasonic, and motion sensors. In some embodiments, the sensor 506 is an optical sensor configured to capture optical images of an environment proximate the computer-controlled machine 500.

[0059] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As explained below, the control system 502 may be further configured to calculate actuator control commands 510 depending on the sensor signals and to transmit the actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0060] As in Fig. 5, the control system 502 includes a receiving unit 512. The receiving unit 512 may be configured to receive sensor signals 508 from the sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x, without the presence of the receiving unit 512. Each input signal x may be a part of each sensor signal 508. The receiving unit 512 may be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to an image recorded by the sensor 506.

[0061] The control system 502 includes a classifier 514. The classifier 514 may be configured to classify input signals x into one or more labels using a machine learning (ML) algorithm, such as a neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameters θ may be stored in and provided by the non-volatile memory 516. The classifier 514 is configured to determine output signals y from input signals x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 may transmit output signals y to the conversion unit 518. The conversion unit 518 is designed to convert output signals y into actuator control commands 510.The control system 502 is configured to transmit actuator control commands 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control commands 510. In some embodiments, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on output signals y.

[0062] The actuator 504 is configured to perform an action corresponding to the associated actuator control command 510 upon receipt by the actuator 504. The actuator 504 may include control logic configured to convert actuator control commands 510 into a second actuator control command used to control the actuator 504. In one or more embodiments, actuator control commands 510 may be used to control a display instead of, or in addition to, an actuator.

[0063] In some embodiments, the control system 502 includes a sensor 506 instead of or in addition to the computer-controlled machine 500, which includes the sensor 506. The control system 502 may also include an actuator 504 instead of or in addition to the computer-controlled machine 500, which includes the actuator 504.

[0064] As in Fig. 5, control system 502 also includes a processor 520 and a memory 522. Processor 520 may include one or more processors. Memory 522 may include one or more storage devices. Classifier 514 (e.g., ML algorithms) of one or more embodiments may be implemented by control system 502, which includes non-volatile memory 516, processor 520, and memory 522.

[0065] Non-volatile memory 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of permanently storing information. Processor 520 may include one or more devices selected from high-performance computing (HPC) systems, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that manipulate signals (analog or digital) based on computer-executable instructions in memory 522.Memory 522 may comprise a single memory device or a number of memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0066] Processor 520 may be configured to read into memory 522 or execute computer-executable instructions residing in non-volatile memory 516 that embody one or more machine learning algorithms and / or methods of one or more embodiments. Non-volatile memory 516 may include one or more operating systems and applications. Non-volatile memory 516 may store compiled and / or interpreted computer programs created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL, either alone or in combination.

[0067] When executed by processor 520, the computer-executable instructions of non-volatile memory 516 may cause control system 502 to implement one or more of the ML algorithms and / or methods described herein. Non-volatile memory 516 may also include ML data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0068] The program code embodying the algorithms and / or methods described herein may be distributed individually or collectively as a program product in a variety of different forms. The program code may be distributed using a computer-readable storage medium containing computer-readable program instructions for causing a processor to perform aspects of one or more embodiments. Computer-readable storage media, which are inherently non-transitory, may include volatile and non-volatile, as well as removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer. Computer-readable program instructions can be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or over a network to an external computer or external storage device.

[0069] Computer-readable program instructions stored in a computer-readable medium may be used to instruct a computer, other types of programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture having instructions that implement the functions, acts, and / or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, acts, and / or operations specified in the flowcharts and diagrams may be rearranged, processed serially, and / or processed concurrently, consistent with one or more embodiments. Furthermore, any flowcharts and / or diagrams may have more or fewer nodes or blocks than those consistent with one or more embodiments.

[0070] The processes, methods, or algorithms may be embodied, in whole or in part, using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0071] Fig. 6 shows a schematic diagram of the control system 502 configured to control the vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively, or in addition to one or more of the above-mentioned specific sensors, the sensor 506 may include a software module configured, when executed, to determine a state of the actuator 504.A non-limiting example of a software module includes a weather information software module configured to determine current or future weather conditions near the vehicle 600 or another location.

[0072] The classifier 514 of the control system 502 of the vehicle 600 can be configured to detect objects in the vicinity of the vehicle 600 based on input signals x. In such an embodiment, the output signal y can include information characterizing the proximity of objects to the vehicle 600. The actuator control command 510 can be determined according to this information. The actuator control command 510 can be used to avoid collisions with the detected objects.

[0073] In some embodiments, vehicle 600 is an at least partially autonomous vehicle, where actuator 504 may be embodied in a brake, drive system, engine, drivetrain, or steering system of vehicle 600. Actuator control commands 510 may be determined to control actuator 504 such that vehicle 600 avoids collisions with detected objects. Detected objects may also be classified according to what classifier 514 considers most likely, such as pedestrians or trees. Actuator control commands 510 may be determined depending on the classification. In a scenario where an adversary attack may occur, the system described above may be further trained to better detect objects or to identify a change in lighting conditions or angle for a sensor or camera on vehicle 600.

[0074] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control command 510 may be determined such that a drive unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot can avoid collisions with identified objects.

[0075] In some embodiments, vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, vehicle 600 may use an optical sensor as sensor 506 to determine a condition of plants in an environment near vehicle 600. Actuator 504 may be a nozzle designed to spray chemicals. Depending on an identified type and / or condition of the plants, actuator control command 510 may be determined to cause actuator 504 to spray the plants with an appropriate amount of suitable chemicals.

[0076] Vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, sensor 506 may be an optical sensor configured to detect a condition of an object to be processed by the household appliance. For example, if the household appliance is a washing machine, sensor 506 may detect a condition of the laundry in the washing machine. Actuator control command 510 may be determined based on the detected condition of the laundry.

[0077] Fig. Figure 7 shows a schematic diagram of control system 502 configured to control system 700 (e.g., a manufacturing machine), such as a punch cutter, a milling cutter, or a deep hole drill, of a manufacturing system 702, such as a portion of a production line. Control system 502 may be configured to control actuator 504, which, in turn, is configured to control system 700 (e.g., the manufacturing machine).

[0078] The sensor 506 of the system 700 (e.g., the manufacturing machine) may be an optical sensor configured to detect one or more features of the product of manufacture 704. The classifier 514 may be configured to determine a state of the product of manufacture 704 from one or more of the detected features. The actuator 504 may be configured to control the system 700 (e.g., the manufacturing machine) for a subsequent manufacturing step of the product of manufacture 704 depending on the determined state of the product of manufacture 704. The actuator 504 may be configured to control functions of the system 700 (e.g., the manufacturing machine) for the subsequent product of manufacture 706 of the system 700 (e.g., the manufacturing machine) depending on the determined state of the product of manufacture 704.

[0079] Fig. Figure 8 shows a schematic diagram of control system 502 configured to control a power tool 800, such as a drill or screwdriver, having an at least partially autonomous mode. Control system 502 may be configured to control an actuator 504 configured to control power tool 800.

[0080] The sensor 506 of the power tool 800 may be an optical sensor configured to detect one or more features of the work surface 802 and / or the fastener 804 being driven into the work surface 802. The classifier 514 may be configured to determine a condition of the work surface 802 and / or the fastener 804 relative to the work surface 802 from one or more of the detected features. The condition may be that the fastener 804 is flush with the work surface 802. Alternatively, the condition may be the hardness of the work surface 802. Actuator 504 may be configured to control the power tool 800 such that the drive function of the power tool 800 is adjusted depending on the determined condition of the fastener 804 relative to the work surface 802 or one or more detected features of the work surface 802.For example, actuator 504 may interrupt the drive function when the condition of fastener 804 is flush relative to work surface 802. As another non-limiting example, actuator 504 may apply additional or less torque depending on the hardness of work surface 802.

[0081] Fig. Figure 9 shows a schematic diagram of the control system 502 configured to control the automated personal assistant 900. The control system 502 may be configured to control the actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, a stove, an oven, a microwave, or a dishwasher.

[0082] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of gestures 904 from user 902. The audio sensor may be configured to receive a voice command from user 902.

[0083] The control system 502 of the automated personal assistant 900 may be configured to determine actuator control commands 510 configured to control the control system 502. The control system 502 may be configured to determine actuator control commands 510 in accordance with sensor signals 508 from the sensor 506. The automated personal assistant 900 is configured to transmit sensor signals 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify the gesture 904 performed by the user 902, determine actuator control commands 510, and transmit the actuator control commands 510 to the actuator 504. The classifier 514 may be configured to retrieve information from the non-volatile memory in response to the gesture 904 and to output the retrieved information in a form suitable for receipt by user 902.

[0084] Fig. Figure 10 shows a schematic diagram of control system 502 configured to control surveillance system 1000. Surveillance system 1000 may be configured to physically control access through door 1002. Sensor 506 may be configured to detect a scene relevant to deciding whether to grant access. Sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. Such data may be used by control system 502 to detect a person's face.

[0085] Classifier 514 of control system 502 of surveillance system 1000 may be configured to interpret the image and / or video data by comparing identities of known individuals stored in non-volatile memory 516, thereby determining the identity of an individual. Classifier 514 may be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. Control system 502 is configured to transmit actuator control command 510 to actuator 504. In this embodiment, actuator 504 may be configured to lock or unlock door 1002 in response to actuator control command 510. In some embodiments, non-physical, logical access control is also possible.

[0086] The surveillance system 1000 may also be an observation system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene being monitored, and the control system 502 is configured to control a display 1004. The classifier 514 is configured to determine a classification of a scene, e.g., whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that the classifier 514 determines to be suspicious. Using one embodiment of the disclosed system, the surveillance system may predict that objects will appear at certain times in the future.

[0087] Fig. 11 shows a schematic diagram of control system 502 configured to control imaging system 1100, such as an MRI scanner, an X-ray imaging scanner, or an ultrasound scanner. Sensor 506 may be, for example, an imaging sensor. Classifier 514 may be configured to determine a classification of all or a portion of the acquired image. Classifier 514 may be configured to determine or select an actuator control command 510 in response to the classification obtained from the trained neural network. For example, classifier 514 may interpret a region of an acquired image as potentially anomalous. In this case, actuator control command 510 may be determined or selected to cause display 1102 to display the image and highlight the potentially anomalous region.

[0088] In some embodiments, a method for labeling audio data comprises receiving video stream data associated with a data capture environment from at least one image capture device. The method further comprises receiving audio stream data corresponding to at least a portion of the video stream data from at least one audio capture array. The method further comprises labeling at least some objects of the video stream data using outputs from at least one first machine learning model configured to provide outputs comprising one or more object detection predictions.The method further comprises calculating at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition feature, and synchronizing at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value. The method further comprises labeling at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value.The method further comprises generating training data using at least a portion of the labeled portion of the audio stream data and training a second machine learning model using the training data.

[0089] In some embodiments, the at least one audio capture array comprises a plurality of audio capture devices. In some embodiments, the at least one audio capture array is remotely located from the at least one image capture device. In some embodiments, labeling the at least some objects of the video stream data using the output of at least the first machine learning model comprises labeling the at least some objects of the video stream data with at least one event type, an event start indicator, and an event end indicator. In some embodiments, the at least one data capture feature comprises one or more features of the at least one image capture device. In some embodiments, the at least one data capture feature comprises one or more features of the at least one audio capture array.In some embodiments, the at least one data acquisition feature comprises one or more features corresponding to a location of the at least one image capture device relative to the at least one audio capture array. In some embodiments, the at least one data acquisition feature comprises one or more features corresponding to motion of an object in the video stream data. In some embodiments, calculating the at least one offset value for at least a portion of the audio stream data corresponding to the at least one labeled object of the video stream data based on the at least one data acquisition feature comprises using at least one probability-based function.

[0090] In some embodiments, a system for labeling audio data comprises a processor and a memory. The memory comprises instructions that, when executed by the processor, cause the processor to: receive, from at least one image capture device, video stream data associated with a data capture environment; receive, from at least one audio capture array, audio stream data corresponding to at least a portion of the video stream data; label, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, at least some objects of the video stream data; calculate, based on at least one data capture feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data;Synchronizing, using at least the at least one offset value, at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data; labeling, using one or more labels of the labeled objects of the video stream data and the at least one offset value, at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data; generating training data using at least a portion of the labeled portion of the audio stream data; training a second machine learning model using the training data; and recognizing, using the second machine learning model, one or more tones / sounds associated with the audio data provided as input to the second machine learning model.

[0091] In some embodiments, the at least one audio capture array comprises a plurality of audio capture devices. In some embodiments, the at least one audio capture array is remotely located from the at least one image capture device. In some embodiments, the instructions further cause the processor to, using the output of at least the first machine learning model, label at least some objects of the video stream data by labeling the at least some objects of the video stream data with at least one event type, an event start indicator, and an event end indicator. In some embodiments, the at least one data capture feature comprises one or more features of the at least one image capture device. In some embodiments, the at least one data capture feature comprises one or more features of the at least one audio capture array.In some embodiments, the at least one data acquisition feature includes one or more features corresponding to a position of the at least one image capture device relative to the at least one audio capture array. In some embodiments, the at least one data acquisition feature includes one or more features corresponding to motion of an object in the video stream data. In some embodiments, the instructions further cause the processor to, based on the at least one data acquisition feature, calculate the at least one offset value for at least a portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least one probability-based function.

[0092] In some embodiments, an apparatus for labeling audio data comprises a processor and a memory. The memory comprises instructions that, when executed by the processor, cause the processor to: receive, from at least one image capture device, video stream data associated with a data capture environment; receive, from at least one audio capture array, audio stream data corresponding to at least a portion of the video stream data; label, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, at least some objects of the video stream data with at least one event type, an event start indicator, and an event end indicator;Calculating, based on at least one data acquisition feature and using at least one probability-based function, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; synchronizing, using at least the at least one offset value, at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data; labeling, using one or more labels of the labeled objects of the video stream data and the at least one offset value, at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data; generating training data using at least a portion of the labeled portions of the audio stream data; and training, using the training data, a second machine learning model.

[0093] In some embodiments, the at least one audio capture array comprises a plurality of audio capture devices, wherein the plurality of audio capture devices are located remotely from the at least one image capture device.

[0094] In some embodiments, a method for generating training data comprises receiving audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment, receiving video stream data associated with the data acquisition environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image acquisition array disposed in the data acquisition environment at a distance from the at least one audio acquisition array, and identifying one or more objects in the video stream data using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions.The method further comprises, for at least one of the one or more objects identified in the video stream data, identifying, using outputs from a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object. The method further comprises calculating at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition feature, and calculating a shift value for the at least one object based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object.The method further comprises synchronizing the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object using at least the at least one offset value and the shift value, and labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value. The method further comprises generating training data using at least one of the labeled portions of the audio stream data.

[0095] In some embodiments, the at least one audio capture array comprises a plurality of audio capture devices. In some embodiments, the method also comprises training a machine learning model for audio recognition using the training data. The machine learning model for audio recognition, trained using the training data, is configured to receive inputs comprising audio stream data and detect one or more events in the inputs. In some embodiments, the probabilistic audio event detector comprises a mel-frequency cepstral coefficient feature extraction classifier and a logistic regression classifier. In some embodiments, the at least one data capture feature comprises one or more features of the at least one image capture device.In some embodiments, the at least one data capture feature comprises one or more features of the at least one audio capture array. In some embodiments, the at least one data capture feature comprises one or more features corresponding to a position of the at least one image capture device relative to the at least one audio capture array. In some embodiments, the at least one data capture feature comprises one or more features corresponding to motion of an object in the video stream data. In some embodiments, the at least one data capture feature comprises one or more probabilities of an event associated with at least one object of the video stream data occurring over a period of time.

[0096] In some embodiments, a system for training a machine learning model for audio recognition comprises a processor and a memory. The memory comprises instructions that, when executed by the processor, cause the processor to: receive audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment; receive video stream data associated with the data acquisition environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image acquisition array disposed in the data acquisition environment at a distance from the at least one audio acquisition array; identify, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data;for at least one of the one or more objects identified in the video stream data, identifying, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; calculating, based on at least one data acquisition feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; calculating, for the at least one object, a shift value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object;Synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object; labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; generating training data using at least one of the labeled portions of the audio stream data; and training a machine learning model for audio recognition using the training data.

[0097] In some embodiments, the at least one audio capture array comprises a plurality of audio capture devices. In some embodiments, the audio recognition machine learning model trained using the training data is configured to receive inputs comprising audio stream data and detect one or more events in the inputs. In some embodiments, the probabilistic audio event detector comprises a mel-frequency cepstral coefficient feature extraction classifier and a logistic regression classifier. In some embodiments, the at least one data capture feature comprises one or more features of the at least one image capture device. In some embodiments, the at least one data capture feature comprises one or more features of the at least one audio capture array.In some embodiments, the at least one data capture feature comprises one or more features corresponding to a position of the at least one image capture device relative to the at least one audio capture array. In some embodiments, the at least one data capture feature comprises one or more features corresponding to movement of an object in the video stream data. In some embodiments, the at least one data capture feature comprises one or more probabilities of an event associated with at least one object of the video stream data occurring over a period of time.

[0098] In some embodiments, an apparatus for training a machine learning model for audio recognition comprises a processor and a memory. The memory comprises instructions that, when executed by the processor, cause the processor to: receive audio stream data associated with at least one audio capture device disposed in a data capture environment; receive video stream data associated with the data capture environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image capture device disposed in the data capture environment at a distance from the at least one audio capture device; identify, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data;Identifying, for at least one of the one or more objects identified in the video stream data, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; calculating, based on at least one data acquisition feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; calculating, for the at least one object, a shift value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object;Synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object; Labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; Generating training data using at least one of the labeled portions of the audio stream data;and training a machine learning model for audio recognition using the training data, wherein the machine learning model for audio recognition trained using the training data is configured to recognize one or more events in audio stream data provided as input to the machine learning model for audio recognition;

[0099] In some embodiments, the at least one data capture feature comprises one or more probabilities of an event associated with at least one object of the video stream data occurring over a period of time.

[0100] The processes, methods, or algorithms described herein may be provided on / implemented by a processing device, controller, or computer, which may be any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms may be stored as data and instructions executable by a controller or computer in many forms, including, but not limited to, information permanently stored on non-writable storage media such as ROM devices and information modifiably stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM chips, and other magnetic and optical media. The processes, methods, or algorithms may also be implemented in a software-executable object.Alternatively, the processes, methods, or algorithms may be embodied, in whole or in part, using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0101] Although exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The words used in the description are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As described above, the features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated.Although various embodiments may be described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired features, those skilled in the art will recognize that one or more functions or features may be compromised to achieve desired overall system attributes depending on the specific application and implementation. These features may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc.To the extent that embodiments are described as less desirable than other embodiments or prior art implementations with respect to one or more features, these embodiments are not outside the scope of the disclosure and may be desirable for certain applications.

Claims

[1] A method for generating training data, the method comprising: Receiving audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment; Receiving video stream data associated with the data acquisition environment concurrently with the audio stream data, the video stream data associated with at least one image acquisition array located in the data acquisition environment at a distance from the at least one audio acquisition array; identifying, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data; For at least one of the one or more objects identified in the video stream data, identifying, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; calculating, based on at least one data capture feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; calculating, for the at least one object, a shift value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object; Synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data corresponding to the at least one object with the portion of the audio stream data corresponding to the at least one object; Labeling at least the part of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; and Generating training data using at least one of the labeled portions of the audio stream data. [2] The method of claim 1, wherein the at least one audio capture array comprises a plurality of audio capture devices. [3] The method of claim 1, further comprising training a machine learning model for audio recognition using the training data, wherein the machine learning model for audio recognition, trained using the training data, is configured to: Receiving inputs with audio stream data; and Detect one or more events in the inputs. [4] The method of claim 1, wherein the probability-based audio event detector comprises a mel-frequency cepstral coefficient feature extraction classifier or a logical regression classifier. [5] The method of claim 1, wherein the at least one data capture feature comprises one or more features of the at least one image capture device. [6] The method of claim 1, wherein the at least one data acquisition feature comprises one or more features of the at least one audio acquisition array. [7] The method of claim 1, wherein the at least one data capture feature comprises one or more features corresponding to a position of the at least one image capture device relative to the at least one audio capture array. [8] The method of claim 1, wherein the at least one data acquisition feature comprises one or more features corresponding to a movement of an object in the video stream data. [9] The method of claim 1, wherein the at least one data acquisition feature comprises one or more probabilities of an event associated with at least one object of the video stream data occurring over a period of time. [10] A system for training a machine learning model for audio recognition, the system comprising: a processor; and a memory containing instructions that, when executed by the processor, cause the processor to: Receiving audio stream data associated with at least one audio acquisition array disposed in a data acquisition environment; Receiving video stream data associated with the data acquisition environment concurrently with the audio stream data, the video stream data associated with at least one image acquisition array located in the data acquisition environment at a distance from the at least one audio acquisition array; identifying, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or more objects in the video stream data; for at least one of the one or more objects identified in the video stream data, identifying, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; Calculating, based on at least one data acquisition feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; Calculating, for the at least one object, a displacement value based on a cross-correlation between a portion of the video stream data corresponding to the at least one object and the portion of the audio stream data corresponding to the at least one object; Synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data, corresponding to the at least one object, with the part of the audio stream data corresponding to the at least one object; Labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; Generating training data using at least one of the labeled portions of the audio stream data; and Training a machine learning model for audio recognition using the training data. [11] The system of claim 10, wherein the at least one audio capture array comprises a plurality of audio capture devices. [12] The system of claim 10, wherein the machine learning model for audio recognition, trained using the training data, is configured to: Receiving inputs with audio stream data; and Detect one or more events in the inputs. [13] The system of claim 10, wherein the probabilistic audio event detector comprises a mel-frequency cepstral coefficient feature extraction classifier or a logical regression classifier. [14] The system of claim 10, wherein the at least one data capture feature comprises one or more features of the at least one image capture device. [15] The system of claim 10, wherein the at least one data acquisition feature comprises one or more features of the at least one audio acquisition array. [16] The system of claim 10, wherein the at least one data capture feature comprises one or more features corresponding to a position of the at least one image capture device relative to the at least one audio capture array. [17] The system of claim 10, wherein the at least one data capture feature comprises one or more features corresponding to a movement of an object in the video stream data. [18] The system of claim 10, wherein the at least one data capture feature comprises one or more probabilities of an event associated with at least one object of the video stream data occurring over a period of time. [19] Apparatus for training a machine learning model for audio recognition, the apparatus comprising: a processor; and a memory containing instructions that, when executed by the processor, cause the processor to: Receiving audio stream data associated with at least one audio capture device located in a data capture environment; Receiving video stream data associated with the data capture environment concurrently with the audio stream data, wherein the video stream data is associated with at least one image capture device located in the data capture environment at a distance from the at least one audio capture device; Identifying, using outputs of at least a first machine learning model configured to provide outputs comprising one or more object detection predictions, one or multiple objects in the video stream data; for at least one of the one or more objects identified in the video stream data, identifying, using outputs of a probabilistic audio event detector, a portion of the audio stream data corresponding to the at least one object; Calculating, based on at least one data acquisition feature, at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data; Calculating, for the at least one object, a displacement value based on a cross-correlation between a portion of the video stream data, corresponding to the at least one object and the part of the audio stream data corresponding to the at least one object; Synchronizing, using at least the at least one offset value and the shift value, the portion of the video stream data, corresponding to the at least one object, with the part of the audio stream data corresponding to the at least one object; Labeling at least the portion of the audio stream data corresponding to the at least one object using one or more labels extracted for respective objects of the video stream data, the at least one offset value, and the shift value; Generating training data using at least one of the labeled portions of the audio stream data; and Training a machine learning model for audio recognition using the training data, wherein the machine learning model for audio recognition, trained using the training data, is designed to to detect one or more events in audio stream data provided as input to the audio recognition machine learning model. [20] The apparatus of claim 19, wherein the at least one data capture feature comprises one or more probabilities of an event associated with the at least one object of the video stream data occurring over a period of time.

Images