Multi-object detection with single detection per object

Abstract

Systems and methods for classification of data include optimizing a neural network by minimizing an rhino loss value, including receiving a training batch of data samples including a plurality of samples for each of a plurality of classifications, extracting features from the samples to generate a batch of features, processing the batch of features using the neural network to generate the plurality of classifications to distinguish the samples, and outputting the plurality of classifications to the neural network. An rhino loss value for the training batch is calculated based at least in part on the classification, and a weight of the neural network is modified to reduce the rhino loss value.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation-in-part of U.S. patent application No. 17 / 138,889, filed on December 30, 2020, entitled “MULTI-OBJECT DETECTION WITH SINGLEDETECTION PER OBJECT,” which is incorporated herein by reference in its entirety. Technical Field

[0002] According to one or more embodiments, the present application generally relates to classification systems and methods, and more particularly, to systems and methods, for example, for training and / or implementing multi-object classification systems and methods. Background Art

[0003] Object detection is often implemented as a computer vision technique for locating instances of objects in an image or video. Object detection algorithms often leverage machine learning or deep learning to produce meaningful results. When humans view an image or video, they can identify and locate objects of interest in a matter of seconds. The goal of object detection is to replicate this intelligence using computers. In some systems, objects are detected in an image through an object detection process, and a bounding box is defined around each detected object with an identification of the object class. For example, an image of a neighborhood may include a dog, a bicycle, and a truck, each of which is detected and classified.

[0004] Object detection is used in various real-time systems, such as advanced driver assistance systems that enable cars to detect driving lanes or perform pedestrian detection to improve road safety. Object detection is also useful in applications such as video surveillance, image retrieval, and other systems. Deep learning, machine learning, and other artificial intelligence systems are often used to solve object detection problems. Popular deep learning-based methods use convolutional neural networks (CNNs), such as Regions with Convolutional Neural Networks (R-CNN), You Only Look Once (YOLO), and other methods that automatically learn to detect objects within an image.

[0005] In one approach to object detection via deep learning, a custom object detector is created and trained. To train a custom object detector from scratch, a network architecture is designed to train a CNN using a large labeled dataset to learn the features of the object of interest. The results of the custom object detector are acceptable for many applications. However, these systems can require a lot of time and effort to set up the layers and weights in the CNN. In the second approach, a pre-trained object detector is used. Many object detection workflows using deep learning utilize transfer learning, a method that enables a system to start with a pre-trained network and then fine-tune the pre-trained network for a specific application. This approach can provide faster results because the object detector has been trained on thousands or even millions of images, but has other disadvantages in terms of complexity and accuracy.

[0006] In view of the foregoing, there is a continuing need in the art for improved object detection and classification systems and methods. Summary of the invention

[0007] The present disclosure relates to systems and methods for object detection and classification. In various embodiments, improved systems and methods that can be used for various classification problems including object detection and speech recognition tasks are described. In some embodiments, the improved training method includes a "rhino" loss function to force the machine learning model to activate only once for each corresponding object (or phoneme) detected. These methods reduce the complexity of the full system solution, including eliminating the need for conventional post-processing that is usually applied after the classification step in many embodiments. For example, in some object detection systems, a post-processing step called non-maximum suppression is used to reject redundant detections of each object. This post-processing not only increases computational complexity, but also reduces performance. The single detection system and method disclosed herein provide advantages over such systems.

[0008] The various embodiments disclosed herein can be used without conventional post-processing, greatly reducing the amount of computational complexity at run time and improving the effectiveness of accurately estimating small objects. In addition, the training system can converge faster than other prior art methods. In a speech recognition task, for example, the system of the present disclosure is configured to apply a re-decoding algorithm to decode phonetic letters from input data. In practice, the decoding may not be optimal due to the trade-off between the amount of processing and the performance of using a search algorithm. The technology disclosed herein can greatly simplify the decoding portion of speech recognition, and it can improve performance while reducing computational complexity.

[0009] The scope of the present disclosure is defined by the claims, which are incorporated into this section by reference. A more complete understanding of the present disclosure, as well as the realization of its additional advantages, will be provided to those skilled in the art by considering the following specific implementation of one or more embodiments. Reference will be made to the accompanying drawings, which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various aspects of the present disclosure and its advantages may be better understood with reference to the following drawings and the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the drawings, where the purpose of the illustration is to illustrate the embodiments of the present disclosure, not to limit the embodiments of the present disclosure. The components in the drawings are not necessarily to scale, but the emphasis is placed on clearly illustrating the principles of the present disclosure.

[0011] Figure 1 An example backbone network for use with an object detection process according to one or more embodiments of the present disclosure is shown.

[0012] Figure 2 An example object detection process is shown in accordance with one or more embodiments of the present disclosure.

[0013] Figure 3 An example object detection process is shown in accordance with one or more embodiments of the present disclosure.

[0014] Figure 4 An example object detection process is shown for an image including a detected car in accordance with one or more embodiments of the present disclosure.

[0015] Figure 5 An example object detection process is shown including the combination of feature representations to produce activations in grid cells responsible for detecting a car in accordance with one or more embodiments of the present disclosure.

[0016] Figure 6 An example object detection process for an image including a person riding a motorcycle is shown in accordance with one or more embodiments of the present disclosure.

[0017] Figure 7 An example bounding box and cell grid are shown in accordance with one or more embodiments of the present disclosure.

[0018] Figure 8 An example bounding box and cell grid are shown in accordance with one or more embodiments of the present disclosure.

[0019] Fig. 9 An example object detection process using bounding boxes and cell grids is shown in accordance with one or more embodiments of the present disclosure.

[0020] Fig.10An example bounding box and cell grid used in an example object detection process is shown in accordance with one or more embodiments of the present disclosure.

[0021] Fig.11A -C shows an example object detection and classification process according to one or more embodiments of the present disclosure.

[0022] Fig. 12A -B shows an example object detection and classification process according to one or more embodiments of the present disclosure.

[0023] Fig.13 An example neural network in accordance with one or more embodiments of the present disclosure is shown.

[0024] Fig.14 An example object detection system is shown in accordance with one or more embodiments of the present disclosure.

[0025] Fig.15 is a flow diagram depicting example operations for training a neural network in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure relates to improved systems and methods for object detection and / or classification. The technology disclosed herein can generally be applied to classification problems, including speech detection and authentication in audio, object detection and classification in images, and / or other classification problems. For example, a two-dimensional classification problem can include an object detection process involving identifying and locating certain categories of objects in an image. Object localization can be accomplished in various ways, including creating a bounding box around an object. For example, a one-dimensional classification problem can include phoneme recognition. In phoneme recognition, unlike object detection in an image, the system receives a sequence of data. When detecting speech, the detection of categories in the sequence is usually important. In the present disclosure, improved techniques that can be applied to various classification systems are described, including object detection problems (as an example of a 2-D classification problem) and phoneme recognition problems (as an example of a 1-D classification problem on sequential data).

[0027] Whether the classification system includes a custom object detector or uses a pre-trained object detector, the system designer decides what type of object detection network to use (e.g., a two-stage network or a single-stage network). The initial stage of a two-stage network (such as R-CNN and its variants) identifies region proposals, or subsets of an image that may contain objects. The second stage classifies the objects within the region proposals. Two-stage networks can achieve accurate object detection results, but are typically slower than single-stage networks.

[0028] In a single-stage network such as YOLO v2, the CNN uses anchor boxes to produce network predictions for regions across the image, and the predictions are decoded to produce the final bounding boxes of the objects. Single-stage networks can be much faster than two-stage networks, but may not achieve the same level of accuracy, especially for scenes containing small objects. However, single-stage networks are simpler and more computationally efficient object detectors and are more practical in many end-user products.

[0029] Many conventional object detector techniques require the use of a post-processing stage, such as non-maximum suppression, in order to ignore redundant detections of each object. For example, an object detector may detect a single object (e.g., a car) at three different times and place three different bounding boxes around the object. After using non-maximum suppression, the highest confidence estimate will be retrieved, while the other estimates will be rejected, allowing each object to be identified using a single bounding box. This post-processing stage can impose additional computational complexity, especially when the number of objects per image is high. Embodiments of the deep learning-based techniques disclosed herein include a single-stage object detector that does not require a post-processing stage such as non-maximum suppression, which can improve the performance of estimates for multi-class object detection.

[0030] With reference to the accompanying drawings, embodiments of the present disclosure will now be described. The present disclosure introduces a novel network that can recognize multi-category objects and locate them with a bounding box per object. The proposed technique is a pre-trained object detector that utilizes transfer learning in order to construct a single-stage object detector.

[0031] In order to understand what is in the image, the input image is fed through a convolutional network to build a rich feature representation of the original image. This part of the architecture may be referred to herein as the "backbone" network, which is pre-trained as an image classifier to learn how to extract features from an image. In this approach, it is recognized that image classifications may be easier and cheaper to label than full images because they only require a single label rather than defining bounding box annotations for each image. Training can be done on large labeled datasets (e.g., ImageNet) in order to learn good feature representations.

[0032] An example of a backbone network is Figure 1 , and will now be described according to one or more embodiments. A convolutional neural network 100 (e.g., a VGG network) can be implemented using an architecture configured to receive and process an input image 110 from a training dataset for image classification. The input image 110 is converted to a fixed size and image format 120 and then passed through a plurality of convolutional layers 140, which include a rectified linear activation function, a maximum pooling layer, a softmax output layer 150, and / or other processing steps.

[0033] refer to Figure 2 , after pre-training the backbone architecture 100 as an image classifier, the last few layers of the network are removed so that the backbone network 100 outputs a collection of stacked feature maps 130 that describe the original image at a low spatial resolution (albeit with a high feature (channel) resolution). In the example shown, the 7×7×512 representation of the image observation includes 512 feature maps that describe different characteristics of the original image 110.

[0034] refer to Figure 3 , the 7×7 grid 130 can be related to the original input image 110 in order to understand what each grid cell represents relative to the original image. Based on this data, the system can also roughly determine where the object is located in the coarse (7×7) feature map by observing which grid cell contains the center of the ground truth bounding box annotation. This grid cell can be identified as being "responsible" for detecting that particular object (or representing a particular object that was detected). Figure 4 , for example, a car is identified in a ground truth bounding box 112, the center of the bounding box 112 is identified, and the corresponding grid cell 114 is identified as the cell “responsible” for detecting the car. Figure 5 , feature representations from grid 130 are combined to produce an activation in grid cell 114 that is responsible for detecting a car (e.g., by activating a neuron corresponding to grid cell 114).

[0035] If the input image contains multiple objects, multiple activations can be identified on the grid, representing the objects in each activation region. Figure 6 As shown in the example of , two objects, namely "person" and "motorcycle" are detected in the image. In the first image 600A, the ground truth bounding box 610 defines the detected person and the ground truth bounding box 620 defines the motorcycle. In the next image 600B, the center 610A of the ground truth bounding box 610 and the center 620A of the ground truth bounding box 620 are identified. The corresponding grid cells 610B and 620B are shown in image 600C, respectively. The output of the last layer of the network has two activations 610C and 620C of image 600D related to the two objects.

[0036] In various embodiments disclosed herein, the network learns to find the responsible grid cells to be used for detecting an object. In other words, the network will select all grid cells that are within the ground truth bounding box of the object, such as Figure 7 The grid cells marked with an “X” in FIG. 7 are used as target grid cells that will be used to detect the car in the bounding box 700. The network will then be trained to select one of these target grid cells to activate and use it to detect the object.

[0037] In some embodiments, the last layer generates N*N output probabilities for each category (here assuming N=7 in a 7×7 grid). Assuming the number of categories is C, there will be N*N*C output probabilities (y (1) ,……,y (C) ). For each of the N*N grid cells, the last layer also generates four coordinates corresponding to the four estimated outputs It is related to the x-axis and y-axis positions of the upper left and lower right corners of the rectangular bounding box, such as Figure 8 The output of the network is obtained after applying a sigmoid function that yields a number between 0 and 1. The reference point for each grid cell is the center of the grid cell, as shown by the circle inside grid cell 810. The center of the grid cell corresponds to Move the left corner of the rectangular bounding box to the upper left region of the image, and Move the right corner of the rectangular bounding box to the lower right area of ​​the image. Therefore, when the value changes from 0 to 1, Moving along the horizontal arrow 820 in the image, as the value changes from 0 to 1, Moving along the horizontal arrow 830 in the image, as the value changes from 0 to 1, Moving along vertical arrow 840 in image 800, and as the value changes from 0 to 1, Move along vertical arrow 850 in image 800. By considering that the left corner of the image has (x,y)=(0,0) and the lower right corner of the image has (x,y)=(1,1), using The estimated coordinates will be mapped to the x and y axes of the image as shown in the image below. The estimated mapped coordinates for each grid cell will be named

[0038] The probability that a grid cell contains an object of category j is defined as y (j) , and the number of categories is C. If all y (j) If both are close to zero, it is determined that no object is detected in the image.

[0039] Four bounding box descriptors are used to describe the upper left corner of the bounding box and the lower right corner of the bounding box Considering the reference points at the upper left corner of the image (x=0, y=0) and the lower right corner of the image (x=1, y=1), these values ​​will be mapped to get the corresponding values

[0040] Therefore, the network is configured to learn convolutional filters for each of the above properties such that it produces 4+C output channels to describe a single bounding box at each grid cell location. This means that the network will learn a set of weights to look across all feature maps (in the above example, 512 feature maps) to evaluate the grid cell.

[0041] In some aspects, the size of the neural network model ("model") can be increased by introducing new parameters to learn each category to estimate the bounding box. In other words, there will be 5*C outputs per grid cell, rather than 4+C outputs as shown in the figure below. This will expand the model size at the output layer and can improve the performance of the model for objects with different aspect ratios or shapes. In this embodiment, unless otherwise stated, there are 4+C outputs per grid cell.

[0042] In some aspects, during training of a neural network ("Network"), a rhino loss function may be used to force the Network to detect each object in an image (or each phoneme in a sequence of audio frames) using only one corresponding grid cell activation. Without loss of generality, assume that the number of classes is one (C=1) and that the object of interest is "car". In some embodiments, the Network may generate an estimated "car" confidence score for the nth grid cell: and the estimated bounding box coordinates In each image, each object is shown with a rectangular bounding box around it as its ground truth. All grid cells within the ground truth bounding box of a given object will be considered as target grid cells that will be used to detect the object. For example, Figure 8 Images 800 and Fig.10 The car object in image 1000 has a ground truth bounding box with twelve grid cells as the target grid cell. Slices of the network output corresponding to each object are extracted. When all slices are extracted and no objects are left in the image, the remaining image will belong to background objects. For example, in Fig.10 There is only one object in the image 1000 of FIG. 1 , and a slice 1010 corresponding to the object is extracted from the image 1000 , and the remaining image (or background) is shown in the image 1020 on the right. In some aspects, a ground truth binary mask can be generated and used to generate each corresponding slice. The ground truth binary mask includes a value of 1 for each grid cell in the slice of the object, and a value of 0 for each grid cell elsewhere. For example, Fig.10 As shown, image 1000 includes a ground truth binary mask for a “car” object, where the ground truth binary mask has a value equal to 1 within the ground truth bounding box and has a value equal to 0 elsewhere. Fig.10As shown, the ground truth binary mask may be used to generate a slice 1010 and a residual image 1020 representing a car extracted from image 1000 .

[0043] The Rhino loss function (L rhino (i)) and the total detection loss function for a batch of data of size D Total number of grid cells = N 2 The number of objects or phonemes (or slices) of the jth category of the ith sample of data = S ij Number of categories = C γ ≥ 0 is a hyperparameter that needs to be adjusted during training

[0044] In some aspects, for the i-th sample, the rhino loss function (L rhino (i)) represents the amount of error in the neural network output. That is, the rhino loss represents the difference between the neural network output of the i-th sample and the corresponding ground truth binary mask (e.g., the ground truth binary mask paired with the i-th sample). Similarly, for a batch of D samples, the total detection loss function of Equation 10 is used The calculated total detection loss represents the amount of error in the neural network output. In other words, the total detection loss represents the difference between the neural network output for a batch of D samples and the corresponding ground truth binary masks (where each of the ground truth binary masks is paired with a corresponding sample in the D samples).

[0045] Now refer to Fig.11A , 11B 11C depict an embodiment of a rhino loss function with reallocated overlapping bounding boxes. As shown, bounding boxes of the same category (e.g., bounding boxes 1100A and 1100B) may overlap, and thus the ground truth binary masks corresponding to the respective objects depicted within the bounding boxes (e.g., bounding boxes 1100A and 1100B) may also overlap. For example, in Fig.11A -C, there are three classes "cat", "dog" and "duck", and all bounding boxes of these objects (1100A, 1100B, 1100C and 1100D) have overlapping areas. Therefore, in various embodiments, the ground truth binary masks corresponding to these overlapping objects are modified.

[0046] In one embodiment, the ground truth binary mask (“mask”) of each overlapping object is modified at each update of training. The modified mask is referred to as To this end, for each object Calculate the following rhino soft target scores

[0047] The rhino soft object score will be calculated for each grid cell for all objects. Then, any class of objects with the largest metric value will have the mask for that object set to 1. For example, in Fig. 11B , the two cats have 6 grid cells in the overlapping region (e.g., region 1120 where bounding boxes 1100A and 1100B overlap). After calculating the rhino metric, the system makes a determination to assign the grid cells of the overlapping region to one of the objects. For example, in the illustrated embodiment, the system decides that the first two black grid cells belong to the right object represented by bounding box 1100B, and the other four white grid cells in the overlapping region belong to the left object represented by bounding box 1100A.

[0048] In one embodiment, the system replaces the mask in (1) with a modified mask calculated using the following method to account for overlapping areas of bounding boxes: For each i, calculate for all j, s and n For each n, find the value with the maximum value among all s and j. s * and j * . For other j, s, set and

[0049] If the number of parameters is increased by having one set of coordinates per category, the masks of overlapping regions of objects belonging to different categories do not need to be modified. In this case, the number of outputs will be reduced from (4+C)*N 2 Change to 5*C*N 2 This can increase the number of parameters of a neural network model (e.g., an object detector model), and it can also improve performance when the classes do not have similar shapes or aspect ratios (e.g., people and cars).

[0050] In another embodiment, an alternative method is provided to solve the problem of overlapping bounding boxes when the two bounding boxes belong to different categories. Note that if the overlapping bounding boxes belong to the same category, the above description of Fig.11A and 11BIn this embodiment, the overlapping category of the s-th object of the j-th category of sample i at grid cell n is defined as For example, in Fig. 11C In the image of , which is assumed to be the i-th data sample, there are three categories of "cat" (j=0), "dog" (j=1) and "duck" (j=2). If n is a shared grid cell 1130 that overlaps all three categories as shown in the figure, then This is because the "duck" object, which is the first object of the category "duck" (s=0), overlaps with two categories "cat" and "dog" at grid cell n, so Contains the indices of the two categories, with the indices of the two categories being zero and one.

[0051] Equation (4) can be modified as follows to address the problem of overlapping bounding boxes of different categories:

[0052] As shown in the figure, to solve the problem of overlapping bounding boxes when objects belong to different categories, an additional term is added to the multiplication in (4) As mentioned above, if there is a mixture of within-category and between-category objects, the grid cells of the within-category objects can be reallocated using the method discussed previously, and the rhino loss function for the between-category objects will be modified as shown in (12)-(13).

[0053] The bounding box loss function is designed to estimate the bounding box around the estimated object. The overall bounding box loss is defined as follows:

[0054] in and is the Intersection over Union (IoU) loss between the predicted box B and the target box Bgt for each grid cell n of the s-th object of the j-th category in image i and the penalty term defined in [7]. and Both will use the The output is calculated. Note that in [7], the loss is defined using the height and width of the bounding box and the center point of the bounding box. Therefore, will be converted to height / width with center point, and then the loss will be calculated. In one embodiment, the loss can be calculated as described in “Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression” by Zhaohui Zheng1, Ping Wang1, Wei Liu2, Jinze Li3, Rongguang Ye1, and Dongwei Ren, AAAI 2020, which is incorporated herein by reference.

[0055] The total loss function L total is the bounding box loss and the total detection loss (or rhino loss (L rhino ) is the sum of: β>0 is a hyperparameter that needs to be adjusted to balance the two losses, namely, the total detection loss and the bounding box loss (also called “localization loss”). Phoneme Recognition

[0056] The task of phoneme recognition involves identifying the phonemes of speech (category C) in a sequence of audio data. This is usually the initial step of a speech recognition system. The backbone of phoneme recognition can be a recursive neural network or CNN. Each output is a confidence score of the probability of detecting the jth category and is obtained after applying the sigmoid function. Similar to object detection, a label window is defined for each phoneme to be classified in the sequence. Note that the label window is a 1-D array, unlike the bounding box of an object detector, which is a 2-D array. Therefore, the rhino loss function (L) for the i-th data sample can be obtained as in (9) and (10), respectively. rhino (i)) and the total detection loss for a batch of data of size D

[0057] An embodiment of applying the rhino loss function with overlapping labeled windows in a data sequence with reallocation will now be described. As mentioned above, if two labeled windows have overlapping areas, the system will reallocate the overlapping areas to either of the two classes using a rhino score such as defined in (11). For example, in Fig. 12A, classes A and B have an overlapping area in the middle, which is shown by the shaded area in the audio frame sequence 1200. Unlike object detection, each data frame is not individually assigned based on the rhino score. Instead, the rhino score of the overlapping area of ​​both classes A and B is calculated. Then, the maximum value of the rhino score over the overlapping area is obtained. Depending on whether the maximum value belongs to class A or class B, the left area plus the maximum position or the right area plus the maximum position is reassigned to class A (frame 1210) or class B (frame 1220). Note that this reassignment may affect the labeling window, so it will update the ground truth binary mask at each update of training.

[0058] Now refer to Fig. 12B An embodiment of a rhino loss function with overlapping labeled windows in a data sequence without reallocation is described. Overlapping labeled windows of a data sequence can be solved by modifying the rhino loss function. However, there is at least one difference between the method proposed for object detection and the method that can be used for data sequences. The difference is that each detection of the data sequence is sequential. In other words, the order of detections at each time frame of the sequence is important. For example, if there is a data sequence with label ABC, only the ABC detection sequence is the correct estimate, and all other estimates including BAC or ACB are incorrect. But in object detection, there is no difference in the order of detections, so there are modifications in the method proposed in some embodiments.

[0059] Similar to what is discussed in this article and is defined to include the indices of overlapping categories that appear before and after the sth phoneme. For example, if the sequence ABC has overlapping regions, then the indices of category B will be the index of category A, and category B will be the index of category C. In addition, is defined as the end time frame of the previous category (here category A) in the overlap area, and is defined as the start time frame of the next category (here category C) in the overlap region. Fig. 12B is shown in the example.

[0060] The modified rhino loss function can be written as follows:

[0061] In some aspects, for each time frame n, and If either of these two conditions is not met, then there is no need to compute the multiplications in (18) or (19).

[0062] The techniques described herein provide a general solution to any classification problem and can therefore be applied to many problems, including object detection, keyword localization, acoustic event detection, and speech recognition. The present disclosure may provide opportunities to solve many practical problems where high accuracy at low computational complexity is an important requirement.

[0063] refer to Fig.13 , an example neural network and training process that can be used to train an artificial intelligence model for object detection, speaker recognition, and other classifications based on the rhino loss function disclosed herein will now be described according to one or more embodiments. Neural network 1300 can be implemented as any neural network configured to receive input data samples and generate classifications as taught herein, such as a recurrent neural network, a convolutional neural network (CNN), or other neural network.

[0064] The neural network 1300 is trained using a supervised learning process that compares the network output data to ground truth data (e.g., expected network output data). For a speaker verification system, for example, the training data set 1302 may include sample speech inputs (e.g., audio samples) labeled with corresponding speaker IDs. The input data 1302 may include other labeled data types, such as multiple images labeled with object classification data, audio data labeled for phoneme recognition, etc. In some embodiments, the input data 1302 may include one or more input data frames and one or more ground truth data frames to be paired with the one or more input data frames, respectively. In addition, in some embodiments, the input data 1302 may be provided to a feature extraction process 1304 to produce a batch of features, which the neural network 1300 may process to generate output data. In other words, one or more input data frames (of the input data 1302) may be provided to the feature extraction process 1304 and then mapped to one or more output data frames based on the algorithm of the neural network 1300. In some aspects, a rhino loss function 1340 as disclosed herein (e.g., the rhino loss function of Equation 9) may be used. Or the total detection loss function of Equation 10 ) determines the difference between one or more output data frames and one or more ground truth data frames (of input data 1302), and feeds it back into the neural network 1300 to make corrections to various trainable weights and biases (collectively referred to as "parameters"). The loss can be fed back into the neural network 1300 using a back-propagation technique (e.g., using a stochastic gradient descent algorithm or a similar algorithm). In some examples, the training data combination can be presented to the neural network 1300 multiple times until the total rhino loss value converges to an acceptable level.

[0065] In some examples, each of the input layer 1310, the hidden layer 1320, and / or the output layer 1330 includes one or more neurons, where each neuron applies a combination of its inputs x (e.g., a weighted sum using a trainable weight matrix W), adds an optional trainable bias b, and applies an activation function f to generate an output a, as shown in the equation a=f(Wx+b). In some examples, the activation function f can be a linear activation function, an activation function with an upper and / or lower limit, a log-sigmoid function, a hyperbolic tangent function, a rectified linear unit function, and / or the like. In some examples, each neuron can have the same or different activation functions.

[0066] After training, neural network 1300 can be implemented in a runtime environment of a remote device to receive input data and generate associated classifications. It should be understood that the architecture of neural network 1300 is merely representative, and other architectures are possible, including neural networks with only one hidden layer, neural networks with different numbers of neurons, neural networks without input layers and / or output layers, neural networks with recurrent layers, and / or the like.

[0067] In other embodiments, the training data set 1302 may include captured sensor data associated with one or more types of sensors, such as voice utterances, visible light images, fingerprint data, and / or other types of biometric information. The training data set may include images of a user's face for a facial recognition system, fingerprint images for a fingerprint recognition system, retinal images for a retinal recognition system, and / or a data set for training another type of biometric recognition system.

[0068] Fig.14 An example system 1400 configured to implement a generalized negative log-likelihood loss for speaker verification in accordance with one or more embodiments of the present disclosure is shown. However, not all of the depicted components in the example system 1400 may be required, and one or more embodiments may include additional components not shown in the figure. Variations may be made to the arrangement and type of components, including additional components, different components, and / or fewer components, without departing from the scope of the present disclosure. Although Fig.14 The example system of is configured for speaker verification, but it should be understood that the methods disclosed herein may be implemented through other system configurations.

[0069] System 1400 includes an authentication device 1420, which includes a processing component 1430, an audio input processing component 1440, a user input / output component 1446, a communication component 1448, and a memory 1450. In some embodiments, other sensors / components 1445 may be included to facilitate additional biometric authentication modalities, such as fingerprint recognition, facial recognition, iris recognition, etc. The various components of the authentication device 1420 may interface and communicate via a bus or other electronic communication interface.

[0070] For example, authentication device 1420 may be implemented on a general-purpose computing device such as a system on a chip, an integrated circuit, or other processing system, and may be configured to operate as part of electronic system 1410. In some embodiments, electronic system 1410 may be or may be coupled to a mobile phone, a tablet computer, a laptop computer, a desktop computer, an automobile, a personal digital assistant (PDA), a television, a voice interaction device (e.g., a smart speaker, a conference speaker system, etc.), a network or system access point, and / or other device systems configured to receive user voice input for authentication and / or identification.

[0071] The processing component 1430 may include one or more of a processor, a controller, a logic device, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)), a digital signal processing (DSP) device, an application-specific integrated circuit, or other devices that may be configured by hardwiring, executing software instructions, or a combination of both to perform various operations for audio source enhancement discussed herein. In the illustrated embodiment, the processing component 1430 includes a central processing unit (CPU) 1432, a neural processing unit (NPU) 1434 configured to implement logic for executing a machine learning algorithm, and / or a graphics processing unit (GPU) 1436. The processing component 1430 is configured to execute instructions stored in the memory 1450 and / or other memory components. The processing component 1430 may perform operations of the authentication device 1420 and / or the electronic system 1410, including one or more of the processes and / or calculations disclosed herein.

[0072] Memory 1450 may be implemented as one or more memory devices or components configured to store data, including audio data, user data, trained neural networks, authentication data, and program instructions. Memory 1450 may include one or more types of memory devices, including volatile and non-volatile memory devices, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk drives, and / or other types of memory.

[0073] The audio input processing component 1440 includes circuitry and digital logic components for receiving audio input signals, such as speech from one or more users 1444 sensed by an audio sensor, such as one or more microphones 1442. In various embodiments, the audio input processing component 1440 is configured to process a multi-channel input audio stream received from multiple microphones, such as a microphone array, and generate an enhanced target audio signal that includes speech from the user 1444.

[0074] The communication component 1448 is configured to facilitate communication between the authentication device 1420 and the electronic system 1410 and / or one or more networks and external devices. For example, the communication component 1448 can implement a Wi-Fi (e.g., IEEE 802.11) or Bluetooth connection between the electronic system 1410 and one or more local devices, or implement a connection to a wireless router to provide network access to external computing systems via the network 1480. In various embodiments, the communication component 1448 may include wired and / or other wireless communication components for facilitating direct or indirect communication between the authentication device 1420 and / or other devices and components.

[0075] Depending on the particular embodiment, authentication device 1420 may further include other sensors / components 1445. Other sensors / components 1445 may include other biometric input sensors (e.g., fingerprint sensors, retinal scanners, video or image capture for facial recognition, etc.), and user input / output components 1446 may include I / O components such as a touch screen, a touchpad display, a keypad, one or more buttons, dials, or knobs, a speaker, and / or other components operable to enable a user to interact with electronic system 1410.

[0076] Memory 1450 includes program logic and data configured to facilitate speaker verification according to one or more embodiments disclosed herein, and / or to perform other functions of authentication device 1420 and / or electronic system 1410. Memory 1450 includes program logic for instructing processing component 1430 to perform speech processing 1452 (including speech recognition 1454) on audio input signals received via audio input processing component 1440. In various embodiments, speech processing 1452 logic is configured to identify audio samples including one or more spoken utterances for speaker verification processing.

[0077] The memory 1450 may also include program logic 1462 for implementing user verification controls, which may include security protocols for verifying the user 1444 (e.g., to verify the identity of the user for a secure transaction, to identify access rights to data or programs of the electronic system 1410, etc.). In some embodiments, the user verification control 1462 includes program logic for a registration and / or enrollment process to identify the user and / or obtain user voiceprint information, which may include a unique user identifier and one or more embedding vectors. The memory 1450 may also include program logic for instructing the processing component 1430 to perform a voice authentication process 1464 as described herein, which may include a neural network for speaker verification using a generalized negative log-likelihood loss process, a feature extraction component for extracting features from an input audio sample, a process for identifying an embedding vector and generating a centroid or other vector and a confidence score for speaker identification.

[0078] The memory 1450 may also include other biometric authentication processes 1466, which may include facial recognition, fingerprint recognition, retinal scanning, and / or other biometric processing for a particular implementation. The other biometric authentication processes 1466 may include feature extraction processes, one or more neural networks, statistical analysis modules, and / or other processes. In some embodiments, the user verification control 1462 may process confidence scores or other information from the voice authentication process 1464 and / or one or more other biometric authentication processes 1466 to generate a speaker identification determination. In some embodiments, the other biometric authentication process 1466 includes a neural network trained by a process using a batch of biometric input data and a rhino loss function (as described herein).

[0079] The memory 1450 includes program logic for instructing the processing component 1430 to perform image processing 1456 including object detection 1456 on images received via one or more components (e.g., other sensors / components 1445 such as image capture components, communication components 1448, etc.).

[0080] In various embodiments, the authentication device 1420 may operate in communication with one or more servers across a network 1480. For example, the neural network server 1490 includes processing components and program logic configured to train a neural network (e.g., a neural network training module 1492) for use in speaker verification as described herein. In some embodiments, a database 1494 stores training data 1496, including training data sets and verification data sets for use in training one or more neural network models. The trained neural network 1498 may also be stored in the database 1494 for downloading to one or more runtime environments for use in the voice authentication process 1464. The trained neural network 1498 may also be provided to one or more authentication servers 1482, which provide cloud or other networked speaker recognition services. For example, the verification server 1482 may receive biometric data, such as voice data or other biometric data, from the authentication device 1420 and upload the data to the verification server 1482 for further processing. The uploaded data may include received audio samples, extracted features, embedding vectors, and / or other data. Verification server 1482 may be used to compare samples to known authentication factors and / or user identifiers to determine whether user 1444 has been verified, using a biometric authentication process 1484 including one or more neural networks trained according to the present disclosure (e.g., trained neural network 1488 stored in database 1486) and system and / or user data 1489. In various embodiments, verification server 1482 may be implemented to provide authentication for financial services or transactions, access to a cloud or other online system, cloud or network authentication services for use by electronic system 1410, and the like.

[0081] Fig.15 1 is a flow chart depicting example operations 1500 for training a neural network (e.g., RNN or CNN) according to one or more embodiments. The example operations 1500 may be performed by a machine learning system (e.g., Fig.14 The neural network server 1490) is executed to train the neural network to detect one or more objects in the image (or one or more phonemes in the sequence of audio frames) so that the neural network activates only one neuron for each corresponding object (or phoneme) detected.

[0082] The machine learning system can obtain a frame of first input data and a frame of first ground truth data, wherein the frame of the first ground truth data is associated with a first binary mask having a plurality of first values ​​(e.g., ) and a second binary mask having a plurality of second values ​​(e.g., ) is associated (1510). In some embodiments, the frame of the first input data may represent a frame (or sample) of image data. The frame of the first ground truth data may include an image to be paired with the frame of the first input data, wherein the image depicts one or more objects and one or more bounding boxes associated with the one or more objects, respectively. Each of the one or more objects may represent an object of interest (e.g., an object associated with a particular category). In some other embodiments, the frame of the first input data may represent a frame (or sample) of audio data. The frame of the first ground truth data may include audio data to be paired with the frame of the first input data, wherein the audio data includes a phoneme and at least a portion of a marker window indicating the phoneme. The phoneme may represent a phoneme of interest (e.g., a phoneme associated with a particular category).

[0083] In some embodiments, a first binary mask having a plurality of first values ​​may represent a ground truth binary mask including values ​​of 1 corresponding to grid cells (or portions) of an associated image depicting at least a portion of an object of interest and values ​​of 0 corresponding to the remaining grid cells of the associated image; and a second binary mask having a plurality of second values ​​may represent a ground truth binary mask including values ​​of 0 corresponding to grid cells of an associated image depicting at least a portion of an object of interest and values ​​of 1 corresponding to the remaining grid cells of the associated image. In some other embodiments, a first binary mask having a plurality of first values ​​may represent a ground truth binary mask including values ​​of 1 corresponding to portions (e.g., subframes or frequency bands) of associated audio data including at least a portion of an object of interest and values ​​of 0 corresponding to the remaining portion of the associated audio data; and a second binary mask having a plurality of second values ​​may represent a ground truth binary mask including values ​​of 0 corresponding to portions of associated audio data including at least a portion of an object of interest and values ​​of 1 corresponding to the remaining portion of the associated audio data.

[0084] The machine learning system can also map a frame of first input data to a frame of first output data based on a neural network algorithm, wherein the frame of first output data is associated with a plurality of first probability values ​​(e.g., ) is associated (1520). In some embodiments, each of the plurality of first probability values ​​may indicate a likelihood that a corresponding grid cell (or portion) of a frame of the first input data depicts (or includes) at least a portion of an object of interest (or phoneme). In addition, the neural network algorithm may include a CNN algorithm or a RNN algorithm.

[0085] The machine learning system can generate a prediction based at least in part on a plurality of first probability values ​​(e.g., ) and the corresponding probability values ​​in the plurality of first values ​​(e.g., ) to determine the first detection loss (e.g., )(1530). The machine learning system may also determine the probability of a change based at least in part on the plurality of first probability values ​​(e.g., ) and the corresponding probability values ​​in the plurality of second values ​​(e.g., ) to determine the first non-detection loss (e.g., )(1540). The machine learning system may update the neural network algorithm based at least in part on the first detection loss and the first non-detection loss (1550). In some embodiments, updating the neural network algorithm may include determining a total loss (e.g., a rhino loss, a total detection loss, or other total losses described herein) based at least in part on the first detection loss and the first non-detection loss. Additionally, updating the neural network algorithm may include determining one or more parameters of the neural network algorithm that reduce the total loss.

[0086] In some embodiments, the machine learning system may obtain a frame of second input data and a frame of second ground truth data, wherein the frame of the second ground truth data is associated with a third binary mask having a plurality of third values ​​and a fourth binary mask having a plurality of fourth values. In some embodiments, the frame of the first input data and the frame of the second input data may represent a sequence of audio frames. The machine learning system may map the frame of the second input data to a frame of the second output data based on a neural network algorithm, wherein the frame of the second output data is associated with a plurality of second probability values. The machine learning system may also determine the second detection loss based at least in part on the product (e.g., multiplication) of a corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of third values. The machine learning system may determine the second non-detection loss based at least in part on the product (e.g., multiplication) of a corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of fourth values. In some embodiments, the neural network algorithm may be further updated based on the second detection loss and the second non-detection loss. Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. In addition, where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be combined into a composite component comprising software, hardware, and / or both. Where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware, or both. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0087] Software (such as program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems, networked, and / or otherwise. Where applicable, the ordering of the various steps described herein may be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0088] The foregoing disclosure is not intended to limit the present disclosure to the precise form disclosed or to the specific field of use. Therefore, in accordance with the present disclosure, it is contemplated that various alternative embodiments of the present disclosure and / or modifications to the present disclosure (whether explicitly described or implied herein) are possible. Having thus described the embodiments of the present disclosure, it will be appreciated by those of ordinary skill in the art that changes may be made in form and detail without departing from the scope of the present disclosure. Therefore, the present disclosure is limited only by the claims.

Claims

1. A method comprising: obtaining a frame of first input data and a frame of first ground truth data, wherein the frame of first ground truth data is associated with a first binary mask having a plurality of first values ​​and a second binary mask having a plurality of second values; mapping the first frame of input data to a frame of first output data based on a neural network algorithm, wherein the frame of first output data is associated with a plurality of first probability values; determining a first detection loss based at least in part on a product of a corresponding probability value in the plurality of first probability values ​​and a corresponding value in the plurality of first values; determining a first non-detection loss based at least in part on a product of the corresponding probability value in the plurality of first probability values ​​and a corresponding value in the plurality of second values; and The neural network algorithm is updated based at least in part on the first detection loss and the first non-detection loss.

2. The method of claim 1, wherein the frame of the first input data comprises an image, and each of the plurality of first probability values ​​indicates a likelihood that a corresponding portion of the frame of the first input data depicts at least a portion of an object of interest. 3 . The method of claim 1 , wherein the frame of the first ground truth data comprises an image depicting one or more objects and one or more bounding boxes respectively associated with the one or more objects.

4. The method of claim 1, wherein the frame of the first input data comprises an audio frame, and each probability value of the plurality of first probability values ​​indicates a likelihood that a corresponding portion of the frame of the first input data is associated with a phoneme. 5 . The method of claim 1 , wherein the frame of the first ground truth data comprises audio data, the audio data comprising a phoneme and at least a portion of a marker window indicating the phoneme.

6. The method according to claim 1, further comprising: obtaining a frame of second input data and a frame of second ground truth data, wherein the frame of second ground truth data is associated with a third binary mask having a plurality of third values ​​and a fourth binary mask having a plurality of fourth values; mapping the frame of the second input data to a frame of second output data based on the neural network algorithm, wherein the frame of the second output data is associated with a plurality of second probability values; determining a second detection loss based at least in part on a product of a corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of third values; as well as A second non-detection loss is determined based at least in part on a product of the corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of fourth values, wherein the neural network algorithm is further updated based on the second detection loss and the second non-detection loss.

7. The method of claim 6, wherein the frames of the first input data and the frames of the second input data represent a sequence of audio frames.

8. The method of claim 1, wherein the neural network algorithm comprises a convolutional neural network (CNN) algorithm.

9. The method of claim 1, wherein the neural network algorithm comprises a recurrent neural network (RNN) algorithm.

10. The method of claim 1, wherein updating the neural network algorithm comprises: determining a total loss based at least in part on the first detected loss and the first non-detected loss; as well as One or more parameters of the neural network algorithm that reduce the total loss are determined.

11. A machine learning system comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the machine learning system to: obtaining a frame of first input data and a frame of first ground truth data, wherein the frame of first ground truth data is associated with a first binary mask having a plurality of first values ​​and a second binary mask having a plurality of second values; mapping the first frame of input data to a frame of first output data based on a neural network algorithm, wherein the frame of first output data is associated with a plurality of first probability values; determining a first detection loss based at least in part on a product of a corresponding probability value in the plurality of first probability values ​​and a corresponding value in the plurality of first values; determining a first non-detection loss based at least in part on a product of the corresponding probability value in the plurality of first probability values ​​and a corresponding value in the plurality of second values; and The neural network algorithm is updated based at least in part on the first detection loss and the first non-detection loss.

12. The machine learning system of claim 11, wherein the frame of the first input data comprises an image, and each of the plurality of first probability values ​​indicates a likelihood that a corresponding portion of the frame of the first input data depicts at least a portion of an object of interest.

13. The machine learning system of claim 11, wherein the frame of the first ground truth data comprises an image depicting one or more objects and one or more bounding boxes respectively associated with the one or more objects.

14. The machine learning system of claim 11, wherein the frame of the first input data comprises an audio frame, and each of the plurality of first probability values ​​indicates a likelihood that a corresponding portion of the frame of the first input data is associated with a phoneme.

15. The machine learning system of claim 11, wherein the frame of the first ground truth data comprises audio data, the audio data comprising a phoneme and at least a portion of a marker window indicating the phoneme.

16. The machine learning system of claim 11, wherein execution of the instructions further causes the machine learning system to: obtain a frame of second input data and a frame of second ground truth data, wherein the frame of second ground truth data is associated with a third binary mask having a plurality of third values ​​and a fourth binary mask having a plurality of fourth values; mapping the frame of the second input data to a frame of second output data based on the neural network algorithm, wherein the frame of the second output data is associated with a plurality of second probability values; determining a second detection loss based at least in part on a product of a corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of third values; as well as A second non-detection loss is determined based at least in part on a product of the corresponding probability value in the plurality of second probability values ​​and a corresponding value in the plurality of fourth values, wherein updating the neural network algorithm is further based on the second detection loss and the second non-detection loss.

17. The machine learning system of claim 16, wherein the frames of the first input data and the frames of the second input data represent a sequence of audio frames.

18. The machine learning system of claim 11, wherein the neural network algorithm comprises a convolutional neural network (CNN) algorithm.

19. The machine learning system of claim 11, wherein the neural network algorithm comprises a recurrent neural network (RNN) algorithm.

20. The machine learning system of claim 11, wherein execution of the instructions further causes the machine learning system to: determine a total loss based at least in part on the first detection loss and the first non-detection loss; and One or more parameters of the neural network algorithm that reduce the total loss are determined.

Images