Multi-object detection with single detection per object

Abstract

A multi-object detection utilizing a single detection per object. Systems and methods for data classification include optimizing a neural network by minimizing a rhino loss function, including: receiving a training batch of data samples including a plurality of samples of each of a plurality of classifications; extracting features from the samples to generate a batch of features; processing the batch of features using the neural network to generate a plurality of classifications to distinguish the samples; computing a rhino loss value for the training batch based at least in part on the classifications; and modifying weights of the neural network to reduce the rhino loss value.

Description

Technical Field

[0001] According to one or more embodiments, this application relates generally to classification systems and methods, and more particularly, for example, to systems and methods for training and / or implementing multi-object classification systems and methods. Background Technology

[0002] Object detection is typically implemented as a computer vision technique for locating instances of objects in an image or video. Object detection algorithms often leverage machine learning or deep learning to produce meaningful results. When humans view an image or video, they can identify and locate objects of interest within a few moments. The goal of object detection is to replicate this intelligence using computers. In some systems, objects are detected in an image through an object detection process, and bounding boxes are defined around each detected object using identifiers of the object class. For example, an image of a street block could include dogs, bicycles, and trucks, each detected and classified individually.

[0003] Object detection is used in a variety of real-time systems, such as advanced driver assistance systems that enable cars to detect driving lanes or perform pedestrian detection to improve road safety. Object detection is also useful in applications such as video surveillance, image retrieval, and other systems. Deep learning, machine learning, and other artificial intelligence systems are commonly used to solve object detection problems. Popular deep learning-based methods use convolutional neural networks (CNNs) (such as Region with Convolutional Neural Networks (R-CNN)), You Look Only Once (YOLO), and other methods that automatically learn to detect objects within images.

[0004] In one approach to object detection using deep learning, a custom object detector is created and trained. To train the custom object detector from scratch, the network architecture is designed to learn features of objects of interest, and a large labeled dataset is used to train the CNN. The results of the custom object detector are acceptable for many applications. However, these systems can require significant time and effort to build the layers and weights in the CNN. In a second approach, a pre-trained object detector is used. Many object detection workflows using deep learning leverage transfer learning, a method that allows a system to start with a pre-trained network and then fine-tune it for a specific application. This approach can provide faster results because the object detector has already been trained on thousands or even millions of images, but it has other drawbacks in terms of complexity and accuracy.

[0005] In view of the foregoing, there is a continued need in the art for improved object detection and classification systems and methods. Summary of the Invention

[0006] This disclosure relates to systems and methods for object detection and classification. In various embodiments, improved systems and methods that can be used for a variety of classification problems, including object detection and speech recognition tasks, are described. In some embodiments, the improved training methods include a "rhino" loss function to force the model to activate once for each object. These methods reduce the complexity of the overall system solution by eliminating the need for conventional post-processing typically applied after the classification step in many embodiments. For example, in some object detection systems, a post-processing step called non-maximum suppression is used to reject redundant detections of each object. This post-processing not only increases computational complexity but also degrades performance. The single-detection systems and methods disclosed herein offer advantages over such systems.

[0007] The various embodiments disclosed herein can be used without conventional post-processing, thereby significantly reducing the amount of computational complexity at runtime and improving the effectiveness of accurately estimating small objects. Furthermore, the training system can converge faster than other prior art methods. In speech recognition tasks, for example, the system disclosed herein is configured to apply a re-decoding algorithm to decode speech letters from input data. In practice, decoding may be suboptimal due to the trade-off between processing power and the performance of using search algorithms. The techniques disclosed herein can significantly simplify the decoding portion of speech recognition and can improve performance while reducing computational complexity.

[0008] The scope of this disclosure is defined by the claims, which are incorporated herein by reference. A more complete understanding of this disclosure, and the implementation of its additional advantages, will be provided to those skilled in the art through the following detailed description of one or more embodiments. Reference will be made to the accompanying drawings, which will first be briefly described. Attached Figure Description

[0009] A better understanding of aspects and advantages of this disclosure can be achieved by referring to the following accompanying drawings and the detailed description that follows. It should be understood that the same reference numerals are used to identify the same elements illustrated in one or more of the drawings, wherein the illustrations in the drawings are for the purpose of illustrating embodiments of this disclosure and not for the purpose of limiting embodiments of this disclosure. The components in the drawings are not necessarily to scale, but rather the emphasis is on clearly illustrating the principles of this disclosure.

[0010] Figure 1 An example backbone network for use in an object detection process according to one or more embodiments of the present disclosure is illustrated.

[0011] Figure 2 An example object detection process according to one or more embodiments of the present disclosure is illustrated.

[0012] Figure 3 An example object detection process according to one or more embodiments of the present disclosure is illustrated.

[0013] Figure 4 An example object detection process, including an image of a detected vehicle, is illustrated according to one or more embodiments of the present disclosure.

[0014] Figure 5 The illustration depicts an example object detection process according to one or more embodiments of the present disclosure, including the combination of feature representations to generate activations in a grid cell responsible for detecting a vehicle.

[0015] Figure 6 An example object detection process for an image including a person riding a motorcycle, according to one or more embodiments of the present disclosure, is illustrated.

[0016] Figure 7 Example bounding boxes and cell grids according to one or more embodiments of the present disclosure are illustrated.

[0017] Figure 8 Example bounding boxes and cell grids according to one or more embodiments of the present disclosure are illustrated.

[0018] Figure 9 An example object detection process using bounding boxes and cell meshes according to one or more embodiments of the present disclosure is illustrated.

[0019] Figure 10 The illustration shows an example bounding box and cell grid used in an example object detection process according to one or more embodiments of the present disclosure.

[0020] Figure 11A -C illustrates an example object detection and classification process according to one or more embodiments of the present disclosure.

[0021] Figure 12A -B illustrates an example object detection and classification process according to one or more embodiments of the present disclosure.

[0022] Figure 13 An example neural network according to one or more embodiments of the present disclosure is illustrated.

[0023] Figure 14 An example object detection system according to one or more embodiments of the present disclosure is illustrated. Detailed Implementation

[0024] This disclosure relates to improved systems and methods for object detection and / or classification. The techniques disclosed herein are generally applicable to classification problems, including speech detection and authentication in audio, object detection and classification in images, and / or other classification problems. For example, a two-dimensional classification problem may include an object detection process involving identifying and locating objects of certain classes in an image. Object localization can be accomplished in various ways, including creating bounding boxes around the object. For example, a one-dimensional classification problem may include phoneme recognition. In phoneme recognition, unlike object detection in images, the system receives a sequence of data. When detecting speech, the detection of classes in the sequence is often important. Improved techniques that can be applied to various classification systems are described in this disclosure, including object detection problems (as an example of a 2-D classification problem) and phoneme recognition problems (as an example of a 1-D classification problem utilizing sequential data).

[0025] Regardless of whether the classification system includes a custom object detector or uses a pre-trained object detector, the system designer decides what type of object detection network to use (e.g., a two-level network or a single-level network). The initial level of a two-level network (such as R-CNN and its variants) may contain proposals or subsets of images of objects. The second level classifies the objects within the proposals. Two-level networks can achieve accurate object detection results; however, they are generally slower than single-level networks.

[0026] In single-level networks such as YOLO v2, CNNs use anchor boxes to generate network predictions of regions across an image, and these predictions are decoded to generate the final bounding boxes of objects. Single-level networks can be much faster than two-level networks, but they may not achieve the same level of accuracy, especially for scenes containing small objects. However, single-level networks are simpler, faster, and more memory and computationally efficient object detectors, and are more practically used in many end-user products.

[0027] Many conventional object detector techniques require a post-processing stage, such as non-maximum suppression, to ignore redundant detections for each object. For example, an object detector might detect a single object (e.g., a car) three times at different times and place three different bounding boxes around the object. After applying non-maximum suppression, the highest confidence estimate is retrieved while other estimates are rejected, allowing each object to be identified using a single bounding box. This post-processing stage can impose additional computational complexity, especially when the number of objects per image is high. Embodiments of deep learning-based techniques disclosed herein include single-stage object detectors that do not require post-processing stages such as non-maximum suppression, which improve the performance of estimations for multi-class object detection.

[0028] Referring to the accompanying drawings, embodiments of the present disclosure will now be described. The present disclosure introduces a novel network capable of recognizing multiple object classes and locating the objects using a bounding box for each object. The proposed technique is a pre-trained object detector that utilizes transfer learning to construct a single-level object detector.

[0029] To understand what is in an image, the input image is fed through a convolutional network to construct a rich feature representation of the original image. This part of the architecture, referred to in this paper as the "backbone" network, is pre-trained as an image classifier to learn how to extract features from the image. In this approach, it is recognized that image classification may be easier and cheaper than labeling the entire image, as it only requires a single label, as opposed to defining bounding box annotations for each image. Training can be performed on large labeled datasets (e.g., ImageNet) to learn good feature representations.

[0030] Figure 1 An example of a backbone network is illustrated in the figure, and examples of backbone networks will now be described according to one or more embodiments. A convolutional neural network 100 (e.g., a VGG network) can be implemented using an architecture configured to receive and process input images 110 from a training dataset for image classification. The input image 110 is converted to a fixed size and image format 120 and then passed through multiple convolutional layers 140, which include modified linear activation functions, max pooling layers, softmax output layers 150, and / or other processing steps.

[0031] refer to Figure 2 After pre-training the backbone architecture 100 as an image classifier, the last few layers of the network are removed, causing the backbone network 100 to output a set of stacked feature maps 130, which describe the original image at a low spatial resolution, although at a high feature (channel) resolution. In the illustrated example, the 7×7×512 of the image observation represents 512 feature maps that describe different characteristics of the original image 110.

[0032] refer to Figure 3 The 7×7 grid 130 can be correlated with the original input image 110 to understand what each grid cell represents relative to the original image. Based on this data, the system can also roughly determine the location of an object in the coarse (7×7) feature map by observing which grid cell contains the center of the bounding box annotation. This grid cell can be identified as "responsible" for detecting that specific object. (Reference) Figure 4 For example, a car is identified in bounding box 112, and the center of the identified bounding box and the corresponding grid cell 114 are identified at the cell "responsible" for detecting the car. (See reference) Figure 5 The feature representations from grid 130 are combined to generate the activation responsible for detecting the car in grid cell 114.

[0033] If the input image contains multiple objects, multiple activations can be identified on the grid, representing the object within each activated region. For example, as... Figure 6 As illustrated in the example, two objects, a "person" and a "motorcycle," are detected in an image. In the first image 600A, a first bounding box 610 defines the detected person, and a second bounding box 620 defines the motorcycle. In the next image 600B, the center 610A of bounding box 610 and the center 620A of bounding box 620 are identified. The corresponding grid cells 610B and 620B are illustrated in image 600C. The output of the last layer of the network has two activations 610C and 620C associated with the two objects in image 600D.

[0034] In the various embodiments disclosed herein, the network learns to find the responsible grid cells to be used for object detection. In other words, the network will select all grid cells (such as...) that are inside the object's ground truth bounding box. Figure 7 The grid cells marked with "X" are used as target grid cells and will be used to detect cars in the bounding box 700. The network will then be trained to select one of these target grid cells to activate and use it to detect objects.

[0035] In some embodiments, the last layer generates N*N output probabilities for each class (assuming N=7 in a 7×7 grid). If we assume the number of classes is C, then there will be N*N*C output probabilities. For each of the N*N grid cells, it also generates four coordinates corresponding to the four estimated outputs. , , , The estimated four outputs are related to the x-axis and y-axis positions of the two corners at the top left and bottom right corners of the rectangular bounding box, such as their positions at... Figure 8 As shown in the diagram. The network output is obtained after using a sigmoid function that produces numbers between 0 and 1. The reference point for each grid cell is the center of the grid cell, as indicated by the circle inside grid cell 810. The center of the grid cell corresponds to... . , Move the left corner of the rectangular bounding box to the top left area of ​​the image, and , Move the right corner of the rectangular bounding box to the bottom right area of ​​the image. Therefore, when the value changes from 0 to 1, Move along the horizontal arrow 820 in the image; as the value changes from 0 to 1... Move along the horizontal arrow 830 in the image, and as the value changes from 0 to 1, Move along the vertical arrow 840 in image 800, as the value changes from 0 to 1. Move along the vertical arrow 850 in image 800. Consider the left corner of the image... And the bottom right corner of the image has ,use , , , The estimated coordinates will be mapped to the x and y axes of the image, as shown in the image below. The estimated mapped coordinates for each grid cell will be named... , , , .

[0036] The probability that a grid cell contains an object of class i is defined as And assume the number of classes is C. If all grid cells have... If all values ​​are close to zero, then it is determined that no object was detected in the image.

[0037] The four bounding box descriptors are used to describe the xy coordinates of the top-left corner of the bounding box. and the xy coordinates of the bottom right corner of the bounding box Considering the top left corner of the image and the bottom right corner of the image These values ​​will be mapped to the reference points to obtain the corresponding values. .

[0038] Therefore, the network is configured to learn a convolutional filter for each of the aforementioned attributes, such that it produces 4+C output channels to describe a single bounding box at each grid cell location. This means that the network learns a set of weights to look across all feature maps (assuming 512 in the example above) to evaluate the grid cells.

[0039] The model size can be increased by introducing new parameters to learn each class to estimate the bounding box. In other words, there will be 5*C outputs per grid cell instead of 4+C outputs as shown in the figure below. This will increase the model size at the output layer, and it can improve the performance of models for objects with different aspect ratios or shapes. In this embodiment, we assume 4+C outputs per grid cell unless otherwise stated.

[0040] We will now describe the proposed Rhino loss function, which forces the network to detect each object using only one grid cell activation. Without loss of generality, we assume the number of classes is one (C=1), and the object of interest is "car". Therefore, for the nth grid cell, we have a confidence score for "car". and bounding box coordinates , , , In each image, each object is shown as having a rectangular bounding box surrounding it as its ground truth. All grid cells inside the bounding box are considered as the target grid cells to be used for object detection. For example, Figure 9 Image 900 and Figure 10 The car object in image 1000 has a bounding box with twelve grid cells as the target grid cells. A slice corresponding to the network output for each object is extracted. When all slices have been extracted and there are no more objects in the image, the remaining image belongs to the background objects. For example, in... Figure 10 The image contains only one object, and a slice corresponding to that object is extracted from the image. The remaining image with the background object is shown in image 1020 on the right. For each slice of the image, a mask can be generated to generate the slice. The mask is one when the grid cell is in a slice of the object, and zero elsewhere. For example, as... Figure 10 As illustrated, image 1000 includes a mask for the "car" object, which has a value of one inside the bounding box and a value of zero elsewhere. Figure 10 The diagram illustrates an example mask for a slice, showing slice 1010 representing a car extracted from image 1000 and the remaining image 1020.

[0041] The Rhino loss function for the i-th sample of the data is given below. The total detection loss of batch data of size D .

[0042] Total number of grid cells

[0043] The number of objects or slices of class j in the i-th sample of the data

[0044] Number of classes

[0045] The binary mask of the s-th object of the j-th class in the i-th sample of the data

[0046]

[0047] It is a hyperparameter that needs to be tuned during training.

[0048]

[0049] Now refer to Figure 11A , 11B Example 11C describes an embodiment using a reassigned Rhino loss with overlapping bounding boxes. As illustrated, bounding boxes of the same class (e.g., bounding boxes 1100A and 1100B) can overlap, and therefore the binary masks corresponding to each object can also overlap. For example, in Figure 11A The image in -C contains three classes: "cat", "dog", and "duck", and all bounding boxes (1100A, 1100B, 1100C, and 1100D) of all these objects have overlapping areas. Therefore, in various embodiments, the mask corresponding to these overlapping objects is modified.

[0050] In one embodiment, the mask for each overlapping object is modified during each update of the training. The modified mask is referred to as... Therefore, for each object Calculate the following Rhino soft target scores. .

[0051]

[0052] Rhino soft target scores are computed for each grid cell across all objects. Then, any object of any class with the highest metric value will have its mask set to 1. For example, in... Figure 11B In the image, the two cats have six grid cells in an overlapping region (e.g., region 1120 where bounding boxes 1100A and 1100B overlap). After calculating the Rhino metric, the system determines which grid cells in the overlapping region should be assigned to one of the objects. For example, in the illustrated embodiment, the system determines that the first two black grid cells belong to the right object represented by bounding box 1100B, and the other four white grid cells in the overlapping region belong to the left object represented by bounding box 1100A.

[0053] In one embodiment, the system replaces the mask in (1) with a modified mask calculated using the following method to address the issue of overlapping bounding boxes:

[0054] For each i, compute the following for all j, s, and n:

[0055] For each n, find the one with the following among all s and j. The maximum value and .

[0056] For the other j and s, set as well as .

[0057] If we increase the number of parameters by giving each class a set of coordinates, we don't need to modify the mask for overlapping regions belonging to different classes. In this case, the number of outputs will increase from... Change to This can increase the number of parameters in the object detector model, and it can also improve performance when classes do not have similar shapes or aspect ratios (e.g., people and cars).

[0058] In another embodiment, an alternative method is provided to address the problem of overlapping bounding boxes when two bounding boxes belong to different classes. Note that if the overlapping bounding boxes belong to the same class, the above-mentioned methods can be used. Figure 11A and 11B The proposed method. In this embodiment, we assume that the overlapping class of the s-th object of the j-th class of sample i at grid cell n is... For example, in Figure 11C In the image (which is assumed to be the i-th sample of the data), there are three classes: "cat" (j=0), "dog" (j=1), and "duck" (j=2). If n is a shared grid cell 1130 that overlaps with all three classes, as shown in the figure, then This is because the "duck" object, as the first object of class "duck" (s=0), has two overlaps with the two classes "cat" and "dog" at grid cell n, and therefore... The indices for these two classes are 0 and 1.

[0059] Equation (4) can be modified as follows to solve the problem of overlapping bounding boxes of different classes:

[0060]

[0061] As shown, in (4) an additional item Added to the multiplication to address the issue of overlapping bounding boxes when objects belong to different classes. As mentioned above, if there is a mixture of intra-class and inter-class objects, the mesh cells of intra-class objects can be redistributed using the methods discussed earlier, and inter-class objects will have their Rhino loss function modified as given in (12)-(13).

[0062] The bounding box loss function is designed to estimate the bounding boxes around the estimated object. The total bounding box loss is defined as follows:

[0063]

[0064] in and It is the intersection-over-union (IoU) loss and penalty term defined in [7] for the predicted box B and the target box Bgt of each grid cell n of the j-th object of the j-th class of image i. and Both will use each grid cell n The output is used for calculation. Note that the loss is defined in [7] using the height and width of the bounding box and the center point of the bounding box. Therefore, the xy coordinates of the top left and bottom right of the bounding box are used for calculation. The result will be converted to height / width and center point, and then the loss will be calculated. In one embodiment, the loss can be calculated as described in “Distance-IoU loss: Faster and Better Learning for Bounding BoxRegression” by Zhaohui Zheng1, Ping Wang1, Wei Liu2, Jinze Li3, Rongguang Ye1, and Dongwei Ren at AAAI 2020, which is incorporated herein by reference.

[0065] Total loss function It is the sum of the bounding box loss and the Rhino loss:

[0066]

[0067] It is a hyperparameter that needs to be adjusted to balance the loss values ​​of the two losses (i.e., rhino loss and localization loss).

[0068] Phoneme recognition

[0069] Phoneme recognition involves identifying phonemes (class C) in a sequence of audio data. This is often the initial step in a speech recognition system. The backbone of phoneme recognition can be a recurrent neural network or a CNN. Each output is a confidence score of the probability of detecting the j-th class, obtained after applying a sigmoid function. Similar to object detection, a label window is defined for each phoneme to be classified in the sequence. Note that the label window is a 1-D array, unlike the bounding boxes in object detectors, which are 2-D arrays. Therefore, the rhino loss function for the i-th sample of the data... The total detection loss of batch data of size D It can be obtained as (10).

[0070] An example of applying a Rhino loss with overlapping label windows in a data sequence with redistribution will now be described. As previously stated, if two label windows have overlapping regions, the system will redistribute the overlapping regions to either of the two classes using a Rhino score such as that defined in (11). For example, in Figure 12A In the sequence, classes A and B have an overlapping region in the middle, indicated by the shaded area in audio frame sequence 1200. Unlike object detection, each data frame is not assigned based on a separate Rhino score. Instead, the Rhino scores of the overlapping region for both classes A and B are calculated. The maximum Rhino score over the overlapping region is then obtained. Depending on whether the maximum value belongs to class A or class B, the left region plus the maximum position or the right region plus the maximum position is reassigned to class A (frame 1210) or class B (frame 1220). Note that this reassignment may affect the label window, and therefore it will update the binary mask with each update during training.

[0071] Now refer to Figure 12B This describes an embodiment of the Rhino loss with overlapping labeled windows in a data sequence without reallocation. Overlapping window labeling in a data sequence can be addressed by modifying the Rhino loss function. However, there is at least one difference between the proposed method for object detection and the method applicable to data sequences. The difference lies in the order of each detection in the data sequence. In other words, the order of detection at each time frame of the sequence is important. For example, if a data sequence with labels ABC exists, only the ABC detection order is a correct estimate, and all other estimates including BAC or ACB are incorrect. However, in object detection, the detection order does not differ, and therefore modifications exist in the proposed method in some embodiments.

[0072] Similar to what is discussed in this article , and It is defined as the index of the overlapping class that appears before and after the s-th phoneme. For example, if sequence ABC has overlap, then class B... It will be the index of class A, and the index of class B. This will be the index of class C. Furthermore, we will... Defined as the end time frame of the preceding class (here, class A) and the start time frame of the following class (here, class C) in the overlapping region. This is in Figure 12B The example is shown.

[0073] The modified Rhino loss can be written as follows:

[0074]

[0075]

[0076] Note that, assuming for each time frame n, and If either of these two conditions is not met, then there is no need to calculate the multiplication in (18) or (19).

[0077] The techniques described in this paper provide a general solution to any classification problem, and therefore can be applied to many problems, including object detection, keyword localization, acoustic event detection, and speech recognition. This disclosure offers opportunities to solve many practical problems where high accuracy and low computational complexity are important requirements.

[0078] refer to Figure 13 Hereinafter, example neural networks and training processes will be described, according to one or more embodiments, for use in generating trained artificial intelligence training models for use with the Rhino loss functions disclosed herein for object detection, speaker identification, and other classifications. The neural network 1300 can be implemented as any neural network configured to receive input data samples and generate classifications as taught herein, such as recurrent neural networks, convolutional neural networks (CNNs), or other neural networks.

[0079] The neural network 1300 is trained using a supervised learning process that compares input data with ground truth (e.g., the expected network output). For example, for a speaker verification system, the training dataset 1302 may include sample speech input (e.g., audio samples) labeled with corresponding speaker IDs. Input data 1302 may include other labeled data types, such as multiple images labeled with object classification data, audio data labeled with phoneme recognition, etc. In some embodiments, input data 1302 is provided to a feature extraction process 1304 to generate feature batches for input to the neural network 1300. The input batches are compared with the output of the neural network 1300, and the difference between the generated output data and the ground truth output data is determined using a Rhino loss function 1340 as disclosed herein, and fed back to the neural network 1300 to correct for various trainable weights and biases. Backpropagation techniques (e.g., using stochastic gradient descent or similar algorithms) may be used to feed the loss back to the neural network 1300. In some examples, the training data combination may be presented to the neural network 1300 multiple times until the overall Rhino loss function converges to an acceptable level.

[0080] In some examples, each of the input layer 1310, hidden layer 1320, and / or output layer 1330 includes one or more neurons, wherein each neuron applies a combination of its inputs x (e.g., a weighted sum using a trainable weighted matrix W), adds an optional trainable bias b, and applies an activation function f to generate an output a, as in equation [equation missing]. As shown in the diagram. In some examples, the activation function f can be a linear activation function, an activation function with an upper and / or lower bound, a log-sigmoid function, a hyperbolic tangent function, a modified linear unit function, and / or similar. In some examples, each neuron can have the same or different activation functions.

[0081] After training, the neural network 1300 can be implemented in a runtime environment on a remote device to receive input data and generate associated classifications. It should be understood that the architecture of the neural network 1300 is merely representative, and other architectures are possible, including neural networks with only one hidden layer, neural networks with different numbers of neurons, neural networks without input and / or output layers, neural networks with recurrent layers, and / or the like.

[0082] In other embodiments, training dataset 1302 may include captured sensor data associated with one or more types of sensors, such as speech, visible light images, fingerprint data, and / or other types of biometric information. Training datasets may include images of a user's face for a facial identification system, fingerprint images for a fingerprint identification system, retinal images for a retinal identification system, and / or datasets for training another type of biometric identification system.

[0083] Figure 14 An example system 1400 configured to implement a generalized negative log-likelihood loss for speaker verification, according to one or more embodiments of the present disclosure, is illustrated. However, not all of the components depicted in the example system 1400 may be necessary, and one or more embodiments may include additional components not shown in the figures. Variations in the arrangement and type of components, including additional components, different components, and / or fewer components, may be made without departing from the scope of the present disclosure. Although Figure 14 The example system is configured for speaker verification, but it will be understood that the methods disclosed in this paper can be implemented through other system configurations.

[0084] System 1400 includes an authentication device 1420, which includes a processing unit 1430, an audio input processing unit 1440, a user input / output unit 1446, a communication unit 1448, and a memory 1450. In some embodiments, other sensors and components 1445 may be included to facilitate additional biometric authentication modalities, such as fingerprint recognition, facial recognition, iris recognition, etc. The various components of the authentication device 1420 can interface and communicate via a bus or other electronic communication interface.

[0085] The authentication device 1420 may be implemented as a system-on-a-chip, integrated circuit, or other processing system on a general-purpose computing device, and may be configured to operate as part of the electronic system 1410. In some embodiments, the electronic system 1410 may be or may be coupled to a mobile phone, tablet computer, laptop computer, desktop computer, automobile, personal digital assistant (PDA), television, voice interaction device (e.g., smart speaker, conference speaker system, etc.), network or system access point, and / or other device systems configured to receive user voice input for authentication and / or identification.

[0086] Processing unit 1430 may include one or more of a processor, controller, logic device, microprocessor, single-core processor, multi-core processor, microcontroller, programmable logic device (PLD) (e.g., field-programmable gate array (FPGA)), digital signal processing (DSP) device, application-specific integrated circuit (ASIC), or other devices that can be configured to perform the various operations for audio source enhancement discussed herein by means of hard-wired, software instruction execution, or a combination of both. In the illustrated embodiment, processing unit 1430 includes a central processing unit (CPU) 1432, a neural processing unit (NPU) 1434 configured to implement logic for performing machine learning algorithms, and / or a graphics processing unit (GPU) 1436. Processing unit 1430 is configured to execute instructions stored in memory 1450 and / or other memory components. Processing unit 1430 may perform operations of authentication device 1420 and / or electronic system 1410, including one or more of the processes and / or calculations disclosed herein.

[0087] Memory 1450 may be implemented as one or more memory devices or components configured to store data, including audio data, user data, trained neural networks, authentication data, and program instructions. Memory 1450 may include one or more types of memory devices, including volatile and non-volatile memory devices, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk drive, and / or other types of memory.

[0088] The audio input processing unit 1440 includes circuitry and digital logic components for receiving audio input signals, such as speech sensed by audio sensors (such as one or more microphones 1442) from one or more users 1444. In various embodiments, the audio input processing unit 1440 is configured to process multi-channel input audio streams received from multiple microphones (such as a microphone array) and generate an enhanced target audio signal that includes speech from the users 1444.

[0089] Communication component 1448 is configured to facilitate communication between authentication device 1420 and electronic system 1410 and / or one or more networks and external devices. For example, communication component 1448 may enable Wi-Fi (e.g., IEEE 802.11) or Bluetooth connectivity between electronic system 1410 and one or more local devices, or enable connectivity to a wireless router to provide network access to external computing systems via network 1480. In various embodiments, communication component 1448 may include wired and / or other wireless communication components for facilitating direct or indirect communication between authentication device 1420 and / or other devices and components.

[0090] Depending on the specific implementation, the verification device 1420 may further include other sensors and components 1445. Other sensor components 1445 may include other biometric input sensors (e.g., fingerprint sensors, retinal scanners, video or image capture for facial recognition, etc.), and user input / output components 1446 may include I / O components such as a touchscreen, touchpad display, keypad, one or more buttons, dials, or knobs, speakers, and / or other components operable to enable a user to interact with the electronic system 1410.

[0091] Memory 1450 includes program logic and data configured to facilitate speaker verification and / or perform other functions of authentication device 1420 and / or electronic system 1410 according to one or more embodiments disclosed herein. Memory 1450 includes program logic for instructing processing unit 1430 to perform voice processing 1452 (including speech recognition 1454) on audio input signals received via audio input processing unit 1440. In various embodiments, the voice processing 1452 logic is configured to identify audio samples including one or more spoken expressions for speaker verification processing.

[0092] The memory 1450 may also include program logic for implementing the user authentication control 1462, which may include security protocols for verifying the user 1444 (e.g., for verifying the user's identity for secure transactions, for identifying access to data or programs of the electronic system 1410, etc.). In some embodiments, the user authentication control 1462 includes program logic for a registration and / or enrollment process to identify the user and / or obtain the user's voiceprint information, which may include a unique user identifier and one or more embedding vectors. The memory 1450 may also include program logic for instructing the processing unit 1430 to perform the voice authentication process 1464 as described herein, which may include a neural network trained for speaker verification using a generalized negative log-likelihood loss process, a feature extraction unit for extracting features from input audio samples, a process for identifying embedding vectors and generating centroids or other vectors, and a confidence score for speaker identification.

[0093] The memory 1450 may also include other biometric authentication processes 1466, which may include facial recognition, fingerprint identification, retinal scanning, and / or other biometric processing for a particular implementation. Other biometric authentication processes 1466 may include feature extraction processes, one or more neural networks, statistical analysis modules, and / or other processes. In some embodiments, the user verification control 1462 may process confidence scores or other information from the voice authentication process 1464 and / or one or more other biometric authentication processes 1466 to generate a speaker identification determination. In some embodiments, other biometric authentication processes 1466 include neural networks trained using a process of using batches of biometric input data and a Rhino loss function as described herein.

[0094] The memory 1450 includes program logic for instructing the processing unit 1430 to perform image processing 1456 (including object detection 1456) on images received by one or more units (e.g., other sensors / units 1445, such as image capture unit, communication unit 1448, etc.).

[0095] In various embodiments, authentication device 1420 can operate communicatively with one or more servers across network 1480. For example, neural network server 1490 includes processing components and program logic (e.g., neural network training module 1492) configured to train a neural network for use in speaker verification as described herein. In some embodiments, database 1494 stores training data 1496, including training datasets and verification datasets for use in training one or more neural network models. The trained neural network 1498 can also be stored in database 1494 for download to one or more runtime environments for use in voice authentication process 1464. The trained neural network 1498 can also be provided to one or more verification servers 1482, which provide cloud or other networked speaker identification services. For example, verification server 1482 can receive biometric data, such as voice data or other biometric data, from authentication device 1420 and upload the data to verification server 1482 for further processing. The uploaded data may include received audio samples, extracted features, embedding vectors, and / or other data. Verification server 1482 uses a biometric authentication process 1484, including one or more neural networks trained according to this disclosure (e.g., a trained neural network 1488 stored in database 1486), and system and / or user data 1489, to compare samples with known authentication factors and / or user identifiers to determine whether user 1444 has been verified. In various embodiments, verification server 1482 may be implemented to provide authentication for financial services or transactions, access to the cloud or other online systems, cloud or network authentication services for use by electronic system 1410, etc.

[0096] Where applicable, the various embodiments provided by this disclosure may be implemented using hardware, software, or a combination of hardware and software. Furthermore, where applicable, the various hardware and / or software components described herein may be combined into composite components comprising software, hardware, and / or both, without departing from the scope of this disclosure. Where applicable, the various hardware and / or software components described herein may be divided into sub-components comprising software, hardware, or both, without departing from the scope of this disclosure. Additionally, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0097] According to this disclosure, software such as program code and / or data can be stored on one or more computer-readable media. It is also contemplated that the software identified herein can be implemented using one or more general-purpose or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the ordering of the various steps described herein can be changed, combined into compound steps, and / or divided into sub-steps to provide the features described herein.

[0098] The foregoing disclosure is not intended to limit this disclosure to the precise form disclosed or any particular field of use. Therefore, various alternative embodiments and / or modifications to this disclosure (whether expressly described or implied herein) are expected to be possible. Embodiments of this disclosure have been so described, and those skilled in the art will recognize that changes in form and detail may be made without departing from the scope of this disclosure. Therefore, this disclosure is limited only by the claims.

Claims

1. A method for training a neural network, comprising: receiving a training batch of data samples comprising a plurality of labeled classifications; extracting features from the data samples to generate a batch of features; processing the batch of features using the neural network to generate one or more classifications for each data sample; computing a rhino loss value for a rhino loss function for the training batch, wherein the rhino loss function enforces the neural network to detect each object using only one grid cell; and modifying weights of the neural network to reduce the rhino loss value, wherein the training batch comprises a plurality of speech utterances, or a plurality of audio samples, or a plurality of images.

2. The method of claim 1, wherein computing the rhino loss value further comprises generating the rhino loss value for a plurality of speakers.

3. The method of claim 1, wherein processing the batch of features using the neural network to generate one or more classifications for each data sample further comprises utilizing a single classification per object to identify one or more objects in each sample.

4. The method of claim 1, wherein the plurality of audio samples comprises a first number of speakers and a second number of audio samples per speaker.

5. The method of claim 4, wherein the classifications comprise phoneme recognition in a stream of audio samples.

6. The method of claim 1, further comprising a speaker authentication process, the speaker authentication process comprising: receiving a target audio signal comprising speech from a target speaker; extracting target features from the target audio signal; processing the target features through the neural network to generate one or more user classifications; and determining whether the target speaker is associated with a user identifier based at least in part on the one or more user classifications; wherein determining whether the target speaker is associated with a user identifier comprises computing a confidence score that measures a strength of a classification determination.

7. The method of claim 1, wherein the plurality of images comprises object classification labels.

8. The method of claim 7, wherein processing the batches of features using a neural network to generate one or more classifications for each data sample comprises: generating an object detection classification activation in a grid cell that is determined to be responsible for detecting an object of a classification.

9. The method of claim 7, wherein computing the rhino loss value further comprises generating the rhino loss value for a plurality of object classifications.

10. The method of claim 7, wherein processing the batch of features using the neural network to generate one or more classifications for each data sample comprises using a single stage object detector to detect and localize objects in an image with one bounding box per object.

11. A system for training a neural network, comprising: a logic device configured to train the neural network using a rhino loss function, the logic device configured to perform logic, the logic comprising: receiving a training batch of labeled data samples; extracting features from the data samples to generate a batch of features; processing the batch of features using a neural network to generate a classification configured to classify the data samples; computing a rhino loss value for the training batch based at least in part on the classification; and modify weights of the neural network to reduce the rhino loss value, wherein the rhino loss function forces the neural network to use only one grid cell to detect each object, wherein the training batch comprises a plurality of speech utterances, or a plurality of audio samples, or a plurality of images.

12. The system of claim 11, wherein computing the rhino loss value further comprises computing the rhino loss value for a plurality of speakers based at least in part on the classification.

13. The system of claim 11, wherein processing the batch of features using a neural network to generate one or more classifications for each data sample further comprises utilizing a single classification for each object to identify one or more objects in each sample.

14. The system of claim 11, wherein the logic device is further configured to execute logic comprising a backbone network comprising a pre-trained image classifier configured to learn how to extract features from the images.

15. The system of claim 11, wherein the logic device is further configured to execute logic comprising a backbone network configured for phoneme recognition; wherein each output is a confidence score of a probability of a class being detected, and which is obtained after applying a sigmoid function.

16. A system for training a neural network, comprising: a logic device configured to train the neural network for a classification task using a rhino loss function by executing logic comprising: receiving a training dataset comprising labeled training data samples; pre-training a backbone architecture as a classifier using the training dataset; extracting feature maps from an intermediate layer of the backbone architecture; and identifying a portion of each data sample that is relevant to the extracted feature maps, wherein the rhino loss function forces the neural network to use only one grid cell to detect each object, wherein the training dataset comprises a plurality of images or a plurality of audio samples.

17. The system of claim 16, wherein the logic device is further configured to execute logic comprising subdividing each image into a plurality of grid cells and identifying which grid cell of the plurality of grid cells is relevant to a center of a bounding box annotation of the image.

18. The system of claim 17, wherein the image comprises a plurality of objects, and wherein the logic device is further configured to execute logic comprising generating a single activation for each of the detected objects.

19. The system of claim 16, wherein the plurality of audio samples comprises a plurality of frames, and wherein the logic device is further configured to execute logic comprising identifying phonemes by identifying frames that are relevant to phoneme activations.

20. The system of claim 16, wherein identifying a portion of each data sample that is relevant to the extracted feature maps comprises generating one or more classifications for each data sample using a neural network, including using a single-stage object detector to detect and localize objects in an image using one bounding box per object.

Images