Determining association between object and person using machine learning model

Abstract

The invention discloses determining an association between an object and a person using a machine learning model. In various examples, sensor data, such as sensor data of a mask, may be used as input to a machine learning model to determine a confidence of an association of an object with a person. The masked sensor data may focus the machine learning model on a particular region of an image corresponding to a person, an object, or some combination thereof. In some embodiments, in addition to an area ratio between regions of an image corresponding to a person, an object, or a combination thereof, coordinates corresponding to the person, the object, or the combination thereof may also be used to further assist a machine learning model in focusing on important regions of an image used to determine an association of the object to the person.

Description

[0001] This application is a divisional application of Chinese patent application No. 201980088877.1 filed on November 8, 2019. Background Art

[0002] In order to identify and assign ownership of personal belongings, the monitoring system must effectively associate between items and people in the environment. The determined ownership of personal items can provide insights into security by, for example, identifying whether the personal item is associated with a specific person, whether it is passed from one person to another, and / or whether it is abandoned - and by who and / or where the item was abandoned. Monitoring systems typically rely on detection devices (e.g., cameras) to monitor the environment. However, due to different parameters of the detection devices (e.g., viewing angle, field of view, occlusion, etc.), the detection devices may not be able to capture easily analyzed data for associating people with items. For example, it may be difficult to use a camera to monitor a crowded environment where many people and items are being monitored.

[0003] Some conventional systems for object-to-person association can use still images of a single person and a single object to make one or more determinations. For example, when a child kicks a ball, as represented by the image, a computer vision algorithm can be used to identify what is in the captured image - for example, "child kicking a ball." However, because these still images only represent a single person and a single object, the association between the two may not be determined. For example, when these conventional algorithms are used in a crowded environment, the action of a person relative to an object may be mistakenly identified as not having a relationship (e.g., a false negative detection) or inaccurately positively identified as having a relationship (e.g., a false positive detection), and other conventional systems can perform preliminary associations between people and objects - such as by using a joint area between an object and a person, or representing their boundary shapes. However, these associations are generally inaccurate. For example, in a crowded environment where there are two people and the object (e.g., a backpack) of the first person overlaps with the second person (e.g., a person who does not carry a backpack) from the perspective of the camera, these conventional systems may incorrectly generate an association between the second person and the object. Therefore, the determination made by the system will be inaccurate, and the predictions generated about the object may lead to ineffective safety measures. Summary of the invention

[0004] The present disclosure relates in part to associating objects to people by using one or more machine learning models. Compared to conventional systems (such as those described herein), the system of the present disclosure utilizes a machine learning model (such as a convolutional neural network (CNN)) to generate associations and relationships between people and objects using data corresponding to image regions. For example, the image data and / or coordinates of a mask can be generated using the location of an area of ​​an image corresponding to a person, an object, a combination thereof, or an overlap thereof to be applied to a machine learning model. Using a mask instead of cropping can benefit a neural network because spatial information, such as the dimensions of a person and an object, can be preserved.

[0005] As such, in various embodiments, determining the association between a person and an object may utilize a neural network trained to use the image data and / or coordinates of a mask to calculate the confidence of the association between the person and the object. In some examples, a multi-channel approach may be used, in which multiple versions of an image may be generated using different masks and applied to a single stream of a neural network. In another example, a multi-branch approach may be used, in which multiple versions of an image may be generated and provided as input to corresponding branches or streams of a neural network. The output of each of these streams may then be cascaded and provided to additional layers (e.g., one or more fully connected layers) of the neural network to predict confidence. In a further embodiment, instead of generating three versions of an image, a single image may be used as input to the neural network, and region of interest (ROI) pooling may be used by the neural network to analyze various regions of the image corresponding to objects, people, and their combined regions. For example, an image may be input to a series of layers of a neural network, and in addition to the ROI coordinates, the output of the layer may also be input to an ROI pooling layer to help the neural network learn and predict features corresponding to objects, people, and their combined regions. In such an example, by using a single image with ROI pooling, the runtime of the system can be reduced without loss of accuracy regarding the confidence in predicting associations between objects and people.

[0006] In various examples, a multilayer perceptron (MLP) network may be used in addition to or in place of ROI pooling. For example, in addition to the normalized coordinates corresponding to the boundary shape of the person, the normalized coordinates corresponding to the boundary shape of the object, the ratio of the area of ​​the boundary shape of the person to the joint boundary shape (e.g., the union of the object and the person), and the ratio of the boundary shape of the object to the joint boundary shape, the predicted confidence of the neural network using ROI pooling may be used to generate an updated confidence or final confidence of the association between the object and the person. As such, the confidence of the neural network prediction including ROI pooling may be modified or verified by the MLP network to generate a final result. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present system and method for associating objects to people using machine learning models are described in detail below with reference to the accompanying drawings, wherein:

[0008] Figure 1 is a data flow diagram illustrating an example method for associating people with objects using a machine learning model according to some embodiments of the present disclosure;

[0009] Figure 2A is a diagram of an example machine learning model for associating a person with an object using overlapping regions of the object and the person according to some embodiments of the present disclosure;

[0010] Figure 2B is a data flow diagram of an example method for associating a person with an object based on overlap of the object and the person according to some embodiments of the present disclosure;

[0011] Figure 3A is a diagram of an example machine learning model for associating people with objects using multiple channels according to some embodiments of the present disclosure;

[0012] Figure 3B is a data flow diagram of an example method for associating people with objects using a multi-channel machine learning model according to some embodiments of the present disclosure;

[0013] Figure 4A is a diagram of an example machine learning model for associating people with objects using multiple branches according to some embodiments of the present disclosure;

[0014] Figure 4B is a data flow diagram of an example method for associating people with objects using a multi-branch machine learning model according to some embodiments of the present disclosure;

[0015] Figure 5A is a diagram of an example machine learning model for associating people with objects using region of interest (ROI) pooling according to some embodiments of the present disclosure;

[0016] Figure 5B is a diagram of an example machine learning model for associating people with objects using ROI pooling in conjunction with a multi-layer perceptron (MLP) network according to some embodiments of the present disclosure;

[0017] Figure 5C is a data flow diagram of an example method for associating people with objects using ROI pooling according to some embodiments of the present disclosure; and

[0018] Figure 6 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Disclosed are systems and methods for associating objects to people using machine learning models. For example, in various embodiments, determining the association between a person and an object may utilize a neural network that is trained to calculate the confidence of the association between the person and the object. In such embodiments, one or more masks may be used for a captured image (e.g., a still image and / or a video image) to prepare the image for use by the neural network. Using a mask instead of cropping can benefit the neural network because spatial information, such as the dimensions of the person and the object, can be preserved. In one example, for an image and for a single person and a single object within the image, a mask may be applied to the image to generate a resulting image including an object (or an area of ​​the image within a bounding shape corresponding to the object), a person (or an area of ​​the image with a bounding shape corresponding to the person), and a joint area thereof. This masked image may be fed into a neural network (e.g., a CNN), and the neural network may generate a confidence corresponding to the association between the object and the person. This process may be repeated for each of the object-person associations in the image, and the highest confidence of a particular object with a particular person may be used to determine the association of the object.

[0020] In another example, a multi-channel approach may be used, where multiple versions of an image may be generated using different masks. For example, a first image may be similar to the above description, where a mask may be applied to an image that leaves people, objects, and their joint regions as a result. A second image may include a mask that results in only objects (or portions of an image corresponding to the bounding shape of an object), and a third image may include a mask that results in only people (or portions of an image corresponding to the bounding shape of a person). These three images may be input to a neural network as a stack, and the neural network may generate confidences corresponding to associations between objects and people. This process may be repeated for each of the object-to-person associations in the image, and the highest confidence for a particular object to a particular person may be used to determine the association of the object.

[0021] In yet another embodiment, a multi-branch approach may be used. In such an embodiment, and similar to the multi-channel approach, multiple versions of an image may be generated for each possible association between an object and a person. For example, the same three versions of an image may be generated as described above. In such an example, each version of the image may be provided as input to a corresponding branch or stream of the neural network. The output of each of these streams may then be cascaded and provided to additional layers (e.g., one or more fully connected layers) of the neural network to predict confidence. This process may be repeated for each of the object-person associations in the image, and the highest confidence for a particular object and a particular person may be used to determine the association of the object.

[0022] In some examples, in addition to multiple branches or streams, the neural network can also use additional feature vectors to generate confidence values. For example, a feature vector with normalized detection coordinates can be used, where the coordinates can include coordinates of a person, an object, and / or a ratio of a person to an object area. The feature vector can provide additional information extracted from the image data independently of the branches or streams of the neural network. As such, where a branch or stream of the neural network learns to predict information from an image, the feature vector provides additional information corresponding to the image to increase the amount of data used to determine confidence.

[0023] In a further embodiment, instead of generating three versions of an image, a single image may be used as an input to a neural network, and the neural network may use region of interest (ROI) pooling to analyze various regions of the image corresponding to objects, people, and their combined regions. For example, an image may be input to a series of layers of a neural network, and in addition to the ROI coordinates, the output of the layers may also be input to a ROI pooling layer. The ROI coordinates may include the bounding shape vertices of a person, the bounding shape vertices of an object, and the bounding shape vertices of their combination. Thus, the ROI pooling layer may be used to help the neural network learn and predict features corresponding to objects, people, and their combination - similar to the embodiment of generating three versions of the same image - but without using multiple branches or streams. By using a single image with ROI pooling, the running time of the system can be reduced without a loss of accuracy in the confidence of predicting the association between objects and people.

[0024] In some embodiments, in addition to or in place of ROI pooling, a multilayer perceptron (MLP) network may be used. For example, in addition to the normalized coordinates corresponding to the boundary shape of a person, the normalized coordinates corresponding to the boundary shape of an object, the ratio of the area of ​​the boundary shape of a person to the joint boundary shape (e.g., the union of the object and the person), and the ratio of the boundary shape of the object to the joint boundary shape, the predicted confidence of a neural network using ROI pooling may also be used. This information may be provided as input to an MLP network, which in embodiments may include one or more fully connected layers, to generate an updated confidence or final confidence of the association between the object and the person. As such, the confidence predicted by the neural network including ROI pooling may be modified or verified by the MLP network to generate a final result.

[0025] To increase accuracy over time, for example, for images from a video, temporal smoothing can be used in some embodiments. For example, the trajectories of people and / or objects in an environment can be tracked across images. In addition to the probabilistic outputs of the neural networks described herein, these trajectories can also be provided as input to a recurrent neural network (RNN) (such as a long short-term memory (LSTM) network), and the RNN can generate predictions indicating false positives, false negatives, and / or the like. For example, a sequence of a given length (e.g., 5, 7, 10, etc. time steps) can maintain most of the associations (e.g., false negatives) that are determined but missing in relatively few images throughout the sequence. In this way, the confidence indicating the lack of association for a given image can be updated to include associations, where multiple images sequentially adjacent to a given image indicate associations. Similarly, for false positives, the determination of associations can be updated to not include associations, where multiple images sequentially adjacent to a given image indicate no association. RNNs can be trained to more accurately predict associations across image sequences than simply looking at the results across image sequences, because RNNs can factor in object and / or person trajectory information.

[0026] See now Figure 1 , Figure 1 is a data flow diagram illustrating an example process 100 for associating a person with an object using one or more machine learning models 112 in accordance with some embodiments of the present disclosure. Although the process 100 is primarily described with respect to associating a person with the described objects, this is not intended to be limiting, and the process 100 may similarly be used to associate any object type (e.g., bags, food, items, packaging, equipment, luggage, clothing, etc.) with any other object type (e.g., people, animals, vehicles, robots, etc.) without departing from the scope of the present disclosure.

[0027] The process 100 for associating a person with an object may include generating and / or receiving sensor data 102 from one or more sensors. For example, the sensor data 102 may include image data representing one or more images and / or image data representing one or more videos (e.g., snapshots of videos). In some cases, the image may depict one or more people and one or more objects. Any number of sensors may be used to generate the sensor data 102. For example, in the case where the sensor is a camera, there may be any number of cameras that capture the sensor data 102. In some embodiments, the process 100 may be used with a single camera having a field of view of the environment, while in other embodiments, the process 100 may be used with any number of cameras having a varying field of view. In a non-limiting example, the one or more cameras may include one or more stereo cameras, one or more wide-angle cameras, one or more infrared cameras, one or more surround cameras, one or more long-range and / or mid-range cameras, and / or other camera types. Although primarily described as image data captured by one or more cameras, this is not intended to be limiting, and in some embodiments, other sensor types (e.g., LIDAR sensors, RADAR sensors, infrared sensors, ultrasonic sensors, etc.) may be used in addition to or in place of cameras. In some examples, one or more sensors may be part of a security, surveillance, or monitoring system, such as a system installed in an environment, inside a building, outside a building, at a location, and / or a combination thereof. In one embodiment, one or more sensors may be associated with or disposed on a vehicle (e.g., an autonomous vehicle) or other mobile object.

[0028] In any example, the sensor data 102 may represent any image type and / or format. For example, the sensor data 102 may include a compressed image such as a Joint Photographic Experts Group (JPEG) format, a compressed image as a frame derived from a compressed video format such as H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC), such as a raw image derived from a red-transparent-blue (RCCB), red-transparent-blue (RCCC), or other type of imaging sensor.

[0029] The sensor data preprocessor 104 may use sensor data 102 representing one or more images (or other data representations) and load the sensor data 102 into memory in the form of a multidimensional array / matrix (alternatively referred to as a tensor, or more specifically referred to as an input tensor in some examples). The array size may be calculated and / or represented as WxHxC, where W represents the image width in pixels, H represents the height in pixels, and C represents the number of color channels. Without loss of generality, other types and orderings of input image components are also possible. In addition, when batching is used, the batch size B may be used as a dimension (e.g., an additional fourth dimension). Batching may be used for training and / or reasoning. Thus, the input tensor may represent an array of dimensions WxHxCxB. Any ordering of the dimensions is possible, which may depend on the specific hardware and software used to implement the sensor data preprocessor 104. The ordering may be selected to maximize the training and / or reasoning performance of one or more machine learning models 112.

[0030] The sensor data preprocessor 104 may employ a preprocessing image pipeline to process a raw image acquired by the sensor and included in the sensor data 102 to produce preprocessed sensor data. An example of a suitable preprocessing image pipeline may use a raw RCCB Bayer (e.g., 1 channel) type image from the sensor and convert the image into an RCB (e.g., 3 channel) planar image stored in a fixed precision (e.g., 16 bits per channel) format. The preprocessing image pipeline may include decomposition, noise reduction, demosaicing, white balancing, histogram calculation, and / or adaptive global tone mapping (e.g., in that order or in an alternative order).

[0031] The object detector 106 may receive the sensor data 102 (in embodiments, from the sensor data preprocessor 104). The object detector 106 may use computer vision algorithms, object detection algorithms, and / or one or more machine learning models to detect objects and / or people represented by the sensor data 102 (e.g., depicted in an image represented by the sensor data). For example, the object detector 106 may be used (and accordingly trained or programmed to) generate bounding shapes corresponding to objects (e.g., bags, packages, backpacks, luggage, items, etc.) and people (e.g., people, adults, children, animals, etc.) depicted in an image represented by the sensor data 102 (or other sensor data representations, such as a depth map from a LIDAR sensor). In some examples, the bounding shapes may be calculated by the object detector 106 as pixel locations of vertices of the bounding shape, pixel locations of a centroid of the bounding shape, pixel dimensions (e.g., length and width) of the bounding shape, pixel locations of a boundary of the bounding shape, pixel locations of each pixel of the bounding shape, and / or combinations thereof.

[0032] In some embodiments, the bounding shapes corresponding to the person and the object may be used to generate or determine an object region (e.g., an image region corresponding to the object, such as an image region within the bounding shape corresponding to the object), a person region (e.g., an image region corresponding to the person, such as an image region within the bounding shape corresponding to the person), an overlap region (e.g., an image region corresponding to an overlap between the object and the person, such as an image region where a bounding shape corresponding to the object overlaps a bounding shape corresponding to the person), and / or a joint region (e.g., an image region corresponding to a combination of an image region corresponding to the person and a region corresponding to the object). As such, in some embodiments, the bounding shapes generated by the object detector 106 may be used to determine a person region, an object region, a joint region, and / or an overlap region of an image. As described in more detail herein, one or more of these regions may be used by the mask generator 108 to apply a mask to the sensor data 102 to generate masked sensor data 122 to be applied to one or more machine learning models 112. In some examples, the relationship determiner may use one or more of these regions to determine an association or relationship between an object represented by the sensor data 102 and a person (e.g., within an association region).

[0033] In some embodiments, for each object detected by the object detector, a relationship confidence (e.g., confidence 116) may be determined for one or more persons also detected in the image with the object. For a non-limiting example, the confidence 116 for each object may be determined in view of each person detected from the sensor data 102. However, because some people may be far enough away from the objects in the environment depicted by the sensor data 102, determining the confidence 116 for each person may result in unnecessary computational expense and may increase the running time of the system. As such, in some embodiments, an associated region may be determined by the relationship determiner 110. For example, the relationship determiner 110 may use the output of the object detector 106 to filter out some people who should not have confidence 116 predicted for them (e.g., people who are far away from the object by a threshold distance), and / or focus on people who should have confidence 116 predicted for them (e.g., people who are within a threshold distance from the object). For a given object detected within an instance of the sensor data 102, the pixel coordinates corresponding to the object may be used to determine the associated region. For example, where the object detector 106 outputs a centroid corresponding to the bounding shape of the object (or where the centroid of the bounding shape can be derived from the output of the object detector 106), the centroid can be used to generate an associated region with the centroid at its center. For example, an associated region can be generated to have a radius extending from the centroid of the bounding shape. In a non-limiting embodiment, the radius can be determined as a multiple (e.g., 2x, 3x, etc.) of the radius (or height or width) of the bounding shape corresponding to the object. In other examples, the size and / or position of the bounding shape corresponding to the object, as determined by the object detector 106, can be used to generate an associated region. For example, where the size and / or position of the bounding shape within an instance of the sensor data 102 are known, multiples of these sizes (e.g., 2x, 3x, etc.) can be used to generate an associated region (e.g., the associated region can be a multiple of the size of the bounding shape, centered on the centroid of the bounding shape calculated by the object detector 106). As another example, the height or width of the object or the bounding shape corresponding thereto can be used to determine the radius. For example, the radius may be 2x, 3x, 4x, or another factor of the height or width of the object.

[0034] Once the association region is determined, the relationship determiner 110 may determine the persons (or their corresponding bounding shapes) within the association region corresponding to the given object. For example, for a given object, each person within the association region (e.g., each person at least partially within the association region, each person having a bounding shape that at least partially overlaps the association, and / or each person whose bounding shape is completely within the association region) may be determined to have a potential relationship or association with the object. As such, this set or subset of persons depicted in the instance of the sensor data 102 may be the persons for which the confidence 116 is determined. However, as described above, in some examples, a relationship may be determined for each person without using an association region or regardless of an association region.

[0035] In any example, once the person for whom the confidence 116 is generated is determined, this information can be used to determine the confidence 116 between a given or corresponding object and each of the persons. In some embodiments, an instance of one or more machine learning models 112 can be applied to each object / person pair (e.g., for each person determined to be associated with an object or to have a relationship with an object, data corresponding to the object and the first person can be applied to a first instance or instantiation of one or more machine learning models 112, data corresponding to the object and the second person can be applied to a second instance or instantiation of one or more machine learning models 112, and so on). In addition, this process can be repeated for each object or for each of any number of objects for which a relationship is desired. As such, where it is desired to determine an association or relationship between two or more objects, data corresponding to the first object and the person can be applied to a first instance or instantiation of one or more machine learning models 112, data corresponding to the second object and the person can be applied to a second instance or instantiation of one or more machine learning models 112, and so on.

[0036] Depending on the embodiment, for each person / object pair determined by the relationship determiner 110, the sensor data 102 and / or the masked sensor data 122 may be applied to one or more machine learning models 112 to predict or calculate the confidence 116. In some examples, in addition to the sensor data 102 and / or the masked sensor data 122, the coordinates 120 may be applied to one or more machine learning models 112 to assist one or more machine learning models 112 in predicting or providing more context for the confidence 116. In embodiments using the masked sensor data 122, the mask may include a person mask (e.g., masking out every portion of the image that does not correspond to a person, or a bounding shape thereof), an object mask (e.g., masking out every portion of the image that does not correspond to an object, or a bounding shape thereof), a union mask (e.g., masking out every portion of the image that does not correspond to a union of a person and an object, or a bounding shape thereof), and / or an overlap mask (e.g., masking out every portion of the image that does not correspond to an overlap of a person and an object, or a bounding shape thereof). As such, in a non-limiting example, the mask generator 108 may use the person regions, object regions, union regions, and / or overlap regions determined from the output of the object detector 106 and described in more detail herein to determine one or more masks for a given iteration of the one or more machine learning models 112. In some examples, the mask may include converting or changing the pixel values ​​corresponding to the masked portion of the image to the same value (e.g., 1 or 0, corresponding to black or white). As described herein, a mask may be used instead of cropping an image to enable the one or more machine learning models 112 to consider the relative size or dimensions of the one or more person regions, object regions, union regions, and / or overlap regions relative to the spatial dimensions of the one or more images. For example, where the image is cropped rather than masked (e.g., because the input spatial dimensions of the one or more machine learning models 112 must be the same for each iteration), the one or more machine learning models 112 may not be able to consider the actual size, dimensions, or spatial relationship of the person, object, union thereof, and / or overlap thereof. As such, by masking out portions of the image that are to be ignored, the remainder of the image may be analyzed by one or more machine learning models 112 in the context of the size or dimensions of the remainder.

[0037] In some embodiments, as described in more detail herein, the coordinate determiner 114 may determine coordinates 120 of the object region, the human region, the overlapping region, and / or the joint region—for example, using the output of the object detector 106. As such, in some examples, the coordinate determiner 114 may determine a first coordinate associated with the object region, a second coordinate associated with the human region, a third coordinate associated with the joint region, and / or a fourth coordinate associated with the overlapping region. Additionally, in some embodiments, the coordinate determiner 114 may determine area ratios between and among the human region, the object region, the overlapping region, and / or the joint region. For example, a ratio of the human region to the overlapping region, a ratio of the object region to the overlapping region, and / or other ratios may be determined by the coordinate determiner 114 and represented by the coordinates 120. In any example, the coordinates 120 may include the pixel location of the centroid of the bounding shape and dimensions (e.g., the pixel length and pixel width of the bounding shape), and / or may include pixel locations corresponding to vertices and / or other points along the bounding shape (or a union or overlap thereof).

[0038] In some embodiments, posture features may be used during training and / or during inference to improve the performance of the system. For example, posture features may provide key points of joints of a person's body, which may help determine whether a particular person is carrying or otherwise holding an object. For example, in a case where an arm is bent and a bag is hanging from the bent arm, the bending of the arm may be a posture feature that provides additional indication to one or more machine learning models 112 that the person with the bent arm may be the person holding or otherwise associated with the bag.

[0039] Data representing sensor data 102 (and / or preprocessed sensor data), masked sensor data 122, and / or coordinates 120 may be applied to one or more machine learning models 112—at least with respect to Figure 2A-5CVarious embodiments of the machine learning models are described. One or more machine learning models 112 can be trained to predict confidences 116 corresponding to associations between person / object pairs corresponding to input data at each iteration. For example, the confidence 116 may represent the likelihood that a given object is owned, carried, or otherwise associated with a person for whom the object / person pair is calculated (e.g., for each person determined by the relationship determiner 110). In some aspects, the final determination may be that a person from the person / object pair with the highest confidence 116 is associated with the object. In this way, one or more machine learning models 112 can generate multiple confidences associated with each object / person pair, and the highest confidence 116 can be used, for example, by a post-processor 118 to determine that the object belongs to a person from the corresponding object / person pair. In some examples, as described herein, the confidences 116 calculated by one or more machine learning models 112 can undergo post-processing (such as temporal smoothing) using a post-processor 118.

[0040] Although the present invention relates to the use of neural networks (and in particular, convolutional neural networks (CNNs) and / or multi-layer perceptron (MLP) networks) as machine learning models 112 (e.g., Figure 2A-5C ) describes an example, but this is not intended to be limiting. For example, but not limited to, the machine learning model 112 described herein may include any type of machine learning model, such as a machine learning model using linear regression, logistic regression, decision tree, support vector machine (SVM), naive Bayes, k-nearest neighbor (KNN), K-means clustering, random forest, dimensionality reduction algorithm, gradient boosting algorithm, neural network (e.g., autoencoder, convolution, loop, perceptron, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid machine, etc.), and / or other types of machine learning models.

[0041] As an example, such as in the case where one or more machine learning models 112 include a CNN, one or more machine learning models 112 may include any number of layers. One or more of the layers may include an input layer. The input layer may hold values ​​associated with the sensor data 102 (e.g., before or after post-processing), the masked sensor data 122, and / or the coordinates 120. For example, when the sensor data 102 is an image, the input layer may hold values ​​representing the raw pixel values ​​of the image as a volume (e.g., width, height, and color channels (e.g., RGB), such as 32x32x3).

[0042] One or more layers may include a convolutional layer. A convolutional layer may compute the output of neurons in the input layer connected to a local region, with each neuron computing the dot product between their weights and the small region in the input volume to which they are connected. The result of the convolutional layer may be another volume, where one dimension is based on the number of filters applied (e.g., width, height, and number of filters, such as 32x32x12 if the number of filters is 12).

[0043] One or more layers may include a rectified linear unit (ReLU) layer. The ReLU layer may apply an element-wise activation function, such as max(0, x), such as zero thresholding. The resulting volume of the ReLU layer may be the same as the volume of the input to the ReLU layer.

[0044] One or more of these layers may include a pooling layer. The pooling layer may perform a downsampling operation along the spatial dimensions (e.g., height and width), which may result in a smaller volume than the input to the pooling layer (e.g., 16x 16x 12 from a 32x 32x 12 input volume).

[0045] One or more of the layers may include one or more fully connected layers. Each neuron in a fully connected layer may be connected to each neuron in a previous body. A fully connected layer may calculate a class score, and the resulting body may be a 1x 1x number of classes. In some examples, a CNN may include a fully connected layer, such that the output of one or more layers of the CNN may be provided as input to a fully connected layer of the CNN. In some examples, one or more convolutional streams may be implemented by one or more machine learning models 112, and some or all of the convolutional streams may include one or more fully connected layers, respectively.

[0046] Although input layers, convolutional layers, pooling layers, ReLU layers, and fully connected layers are discussed herein with respect to one or more machine learning models 112, this is not intended to be limiting. For example, additional or alternative layers, such as normalization layers, SoftMax layers, and / or other layer types, may be used in the machine learning model 112.

[0047] In embodiments where the machine learning model 112 includes a CNN, different orders and numbers of layers of the CNN may be used, depending on the embodiment. In other words, the order and number of layers of one or more machine learning models 112 are not limited to any one architecture.

[0048] In addition, some layers may include parameters (e.g., weights), such as convolutional layers and fully connected layers, while other layers may not include parameters, such as ReLU layers and pooling layers. In some examples, the parameters may be learned by one or more machine learning models 112 during training. Further, some layers may include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), such as convolutional layers, fully connected layers, and pooling layers, while other layers may not include, such as ReLU layers. Parameters and hyperparameters are not limited and may vary depending on the embodiment.

[0049] The post-processor 118 may be used to perform a variety of tasks. For example, the post-processor 118 may analyze each of the confidences 116 corresponding to each object / person pair calculated by the machine learning model 112, and determine which confidence 116 (e.g., the highest confidence) indicates the most likely person-object association. In addition, in some embodiments, the post-processor 118 may perform temporal smoothing. For example, temporal filtering of an image sequence may be used to reduce or even eliminate false positive associations (e.g., where an object and a person are incorrectly associated) and false negatives (where a person and an object association are missed or incorrectly disappear due to lighting changes, occlusions, confusion, etc.). In such an example, temporal filtering may be performed by inputting a continuous sequence or fragment of a continuous image into a recurrent neural network (RNN) such as a long short-term memory (LSTM) network. The input to the RNN may include a combination of the probability outputs of one or more machine learning models 112 with the trajectory of the object and its corresponding subject hypothesis. For example, but not limited to, five, seven sequences, or any number of multiple sequences N may be used as input. In some embodiments, associations that are mostly determined but missing or uncertain in relatively few images (eg, false negatives) may still be maintained throughout the sequence. Similarly, singular or otherwise relatively few associations may be determined to be false positives and filtered out or ignored.

[0050] See now Figure 2A , Figure 2A is a diagram of an example machine learning model 112A for associating a person with an object using overlapping areas between the person and the object according to some embodiments of the present disclosure. For example, the machine learning model 112A may be Figure 1 1. In some examples, because machine learning model 112A can be a CNN, machine learning model 112A can be referred to as CNN 112A. In addition, although CNN 112A is shown as having Figure 2A , but this is not intended to be limiting, and CNN 112A may include a different architecture without departing from the scope of the present disclosure.

[0051] CNN 112A may use masked sensor data 202 as input. Figure 1 The masked sensor data 122 may include overlapping areas where the rest of the image is masked out. Figure 2A As shown, the cross-hatched portion of the image may represent a masked portion of the image, and the unmasked portion of the image may include an overlapping area. As described herein, the overlapping area may include an area of ​​the image where object 218 (or its bounding shape) overlaps person 216A (or its bounding shape). Masked sensor data 202 illustrates one example, and in other examples, the comparison may be between person 216B and object 218, and / or may be performed between the person 216B and the object 218, and / or between the person 216A and the object 218 as determined by relationship determiner 110 ( Figure 1 ) between people and objects within the associated area determined by . As such, with respect to masked sensor data 202, masked sensor data 202 may be applied to feature extractor 204 (e.g., including one or more feature extractor layers of CNN 112A, such as described herein with respect to Figure 1 The masked sensor data 202 may represent actual pixel values ​​within the overlap region and mask pixel values ​​(e.g., 1, 0, etc.) corresponding to the mask applied to the remainder of the image. In other examples, instead of applying a mask, cropping may be applied to the sensor data to leave only the remaining overlap region. In such examples, the sensor data may be augmented to match the spatial dimensions of the input layer of the CNN 112A.

[0052] In some non-limiting embodiments, the feature extractor 204 may include a residual network. For example, the feature extractor 204 may include a deep neural network (DNN) that includes a large number of layers (e.g., 30-50 layers), and in order to accelerate convergence and the accuracy of the feature extractor 204, a residual network may be used. The residual network can use shortcut connections between layers by directly connecting the input of the nth layer to a certain (n+x)th layer (e.g., skip connection) to learn a residual function with reference to the layer input instead of learning an unreferenced function. In other examples, the feature extractor 204 may not include a residual network and may include another type of feature extractor 204. The masked sensor data 202 can be received as an input to the feature extractor 204, and the feature extractor 204 can output a feature vector 206 or feature map corresponding to the masked sensor data 202. In some examples, the feature vector 206 output by the feature extractor 204 may have a shape such as Figure 2A112A, which may receive the feature vector as input) may be applied to a fully connected (FC) layer 208. The FC layer 208 may then compute an output, which may be provided as input to another FC layer 210. In some examples, the dimensionality of the FC layer 208 may be greater than the dimensionality of the FC layer 210. Although Figure 2A 1, but this is not intended to be limiting, and any number of FC layers (and / or other layer types) may be used without departing from the scope of the present disclosure. In an embodiment, the output of FC layer 210 may be applied to sigmoid function 212, and the output of sigmoid function 212 may be confidence 116. In this way, CNN 112A may be relative to Figure 2A The example of sensor data 202 with mask in is used to predict or calculate the confidence that object 218 belongs to person 216A.

[0053] See also Figure 2B , Figure 2B is a diagram of an example method 250 for associating objects to people using a machine learning model according to some embodiments of the present disclosure. Each block of method 250 may include a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, different functions may be performed by a processor executing instructions stored in a memory. Method 250 may also be embodied as computer-usable instructions stored on a computer storage medium. To name a few, method 250 may be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product. In addition, by way of example, with respect to Figure 1 and Figure 2A 250. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0054] At block B252, method 250 includes determining from the image one or more persons associated with the object. For example, method 250 may include determining from sensor data 102 and using relationship determiner 110 one or more persons associated with the object. In some embodiments, such as where one or more persons are associated with the object, each of blocks B254, B256, B258, B260, and B262 may be performed simultaneously for each of the one or more persons (e.g., using parallel processing of multiple instances of one or more machine learning models 112) or may be performed sequentially (e.g., one person at a time).

[0055] At block B254 , the method 250 includes determining an overlap region of the image, for example, determining an overlap region of the sensor data 102 corresponding to an overlap of an object region of an object represented by the sensor data 102 and a person region of a person.

[0056] At block B256, method 250 includes applying a mask to portions of the image that are not included in the overlap region to generate a masked image. For example, the mask is applied to portions of sensor data 102 that are not included in the overlap region to generate masked sensor data 122. In some examples, masked sensor data 122 may be generated by mask generator 108.

[0057] At block B258, method 250 includes applying the masked image to a neural network. For example, the masked sensor data (e.g., 122 or 202) can be applied to one or more machine learning models 112 (e.g., CNN 112A).

[0058] At block B260, method 250 includes calculating a confidence level for the association between the object and the person using a neural network. For example, confidence level 116 corresponding to person 216A and object 218 may be calculated using machine learning model 112 (e.g., CNN 112A). In addition, confidence level 116 may be calculated for each person determined to be associated with or having a relationship with an object (e.g., person 216B).

[0059] At block B262, method 250 includes associating the object with the person of the one or more persons with the highest confidence of association. As described herein, one or more machine learning models 112 may generate multiple confidences (e.g., one confidence generated by each instantiation of one or more machine learning models 112 corresponding to each person / object pair), and the object / person pair with the highest confidence output by the one or more machine learning models 112 may be determined as the pairing of the object and the person. For example, object 218 may be determined to be associated with person 216A.

[0060] See now Figure 3A , Figure 3A is a diagram of an example machine learning model 112B for associating a person with an object using multiple channels according to some embodiments of the present disclosure. For example, the machine learning model 112B may be Figure 1 1. In some examples, because machine learning model 112B can be a CNN, machine learning model 112B can be referred to as CNN 112B. In addition, although CNN 112B is shown as having Figure 3A, but this is not intended to be limiting, and CNN 112B may include a different architecture without departing from the scope of the present disclosure.

[0061] CNN 112B may use masked sensor data 302 as input. Masked sensor data 302, similar to Figure 1 The masked sensor data 122 may include first mask sensor data 302A corresponding to a joint region (e.g., a joint image region including a person and an object, or a boundary shape corresponding thereto), wherein the rest of the image is masked out, second mask sensor data 302B corresponding to an object region (e.g., an image portion corresponding to the object, or a boundary shape corresponding thereto), wherein the rest of the image is masked out, and third mask sensor data 302C corresponding to a person region (e.g., an image portion corresponding to a person, or a boundary shape corresponding thereto), wherein the rest of the image is masked out. For example, as Figure 3A As shown, the cross-hatched image portion may represent a masked portion of the image, and the unmasked portion of the image may include a joint region, a person region, and / or an object region. Each of the first mask sensor data 302A, the second mask sensor data 302B, and the third mask sensor data 302C may be generated from the same raw sensor data (e.g., from the same image). The masked sensor data 302 illustrates one example, and in other examples, the comparison may be between the person 216B and the object 218, and / or may be performed between the person 216B and the object 218, and / or ... Figure 1 ) between people and objects within the associated regions determined by the CNN 112B. The masked sensor data 302 may represent actual pixel values ​​within the union region, the object region, and the person region, and the masked pixel values ​​may be used to represent the remainder of the image that is not within the corresponding region. In other examples, instead of applying a mask, cropping may be applied to the sensor data to leave only the remaining union region, the object region, and the person region. In such an example, the sensor data may be augmented to match the spatial dimensions of the input layer of the CNN 112B.

[0062] As such, with respect to masked sensor data 302, masked sensor data 302 may be applied to a feature extractor 304 (e.g., comprising one or more feature extractor layers of CNN 112B, such as described herein with respect to Figure 1In some examples, feature extractor 304 may be similar to feature extractor 204 described herein. In some embodiments, masked sensor data 302 may be applied as an input to CNN 112B as a stack or as multiple channels. Feature extractor 304 may use the combined input of masked sensor data 302 to generate a feature vector 306 and / or feature map corresponding to masked sensor data 302. In some examples, the feature vector output by feature extractor 304 may have a spatial resolution of 3x3, such as Figure 3A , but this is not intended to be limiting. Feature vector 306 (and / or the output of one or more additional layers of CNN 112B) may be applied to FC layer 308. Figure 3A Only a single FC layer 308 is shown in FIG. 1 , but this is not intended to be limiting, and any number of FC layers 308 (and / or other layer types) may be used without departing from the scope of the present disclosure. In an embodiment, the output of the FC layer 308 may be applied to a sigmoid function 310, and the output of the sigmoid function 310 may be the confidence 116. In this way, the CNN 112B may be about Figure 3A The instance of sensor data 302 in the mask is used to predict or calculate a confidence that object 218 belongs to person 216A.

[0063] See also Figure 3B , Figure 3B is a diagram of an example method 350 for associating objects to people using a multi-channel machine learning model according to some embodiments of the present disclosure. Each block of method 350 may include a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, different functions may be performed by a processor executing instructions stored in a memory. Method 350 may also be embodied as computer-usable instructions stored on a computer storage medium. To name a few, method 350 may be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product. In addition, by way of example, Figure 1 and Figure 3A 350. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0064] At block B352, method 350 includes determining one or more persons associated with the object from the image. For example, method 350 may include determining one or more persons associated with the object from sensor data 102. In some examples, relationship determiner 110 may determine an association between a person and an object, for example, by using an association region within the image, as described herein. In the event that one or more persons are associated with the object, each of blocks B354, B356, B358, B360, and B362 may be performed for each of the one or more persons.

[0065] At block B354, the method 350 includes determining a joint region, a person region, and an object region of the image. For example, the output of the object detector 106 may be used to determine a joint region, a person region, and / or an object region of the image.

[0066] At block B356, method 350 includes generating an image of the first mask, an image of the second mask, and an image of the third mask. For example, the joint region may be used to generate the first mask sensor data 302A, the object region may be used to generate the second mask sensor data 302B, and the person region may be used to generate the third mask sensor data 302C.

[0067] At block B358, method 350 includes applying data representing the image of the first mask, the image of the second mask, and the image of the third mask to a neural network. For example, first mask sensor data 302A, second mask sensor data 302B, and third mask sensor data 302C may be applied to a machine learning model 112 (e.g., CNN 112B). Figure 4A As described with respect to machine learning model 112C, masked sensor data 302 may be applied as a stack or channel to a single stream of CNN 112B, rather than applying each of masked sensor data 302A, 302B, and 302C to a corresponding stream (as discussed with respect to CNN 112C).

[0068] At block B360, method 350 includes calculating a confidence level for the association between the object and the person using a neural network. For example, confidence level 116 corresponding to person 216A and object 218 may be calculated using machine learning model 112 (e.g., CNN 112B). In addition, confidence level 116 may be calculated for each person determined to be associated with or having a relationship with an object (e.g., person 216B).

[0069] At block B362, method 350 includes associating the object with the person of the one or more persons with the highest association confidence. As described herein, one or more machine learning models 112 may generate multiple confidences (e.g., one confidence generated by each instantiation of one or more machine learning models 112 corresponding to each person / object pair), and the object / person pair with the highest confidence output by the one or more machine learning models 112 may be determined as the pairing of the object and the person. For example, object 218 may be determined to be associated with person 216A.

[0070] See also Figure 4A , Figure 4A is a diagram of an example machine learning model for associating people with objects using multiple branches or flows according to some embodiments of the present disclosure. For example, machine learning model 112C may be Figure 1 1. In some examples, because the machine learning model 112C can be a CNN, the machine learning model 112C can be referred to as CNN 112C. In addition, although the CNN 112C is shown as having Figure 4A , but this is not intended to be limiting, and CNN 112C may include a different architecture without departing from the scope of the present disclosure.

[0071] The CNN 112C may use the masked sensor data 302 as input, which may be similar to at least the Figure 3A The sensor data 302 of the mask described. However, Figure 3A Unlike the CNN 112B described, Figure 4A The masked sensor data 302 may be applied to multiple convolutional streams or branches of the CNN 112C. For example, the first masked sensor data 302A may be applied to a feature extractor 418 (which may be associated with the feature extractor 418). Figure 2A and / or Figure 3A The second mask sensor data 302B may be applied to a first convolutional stream including a feature extractor 412 (which may be the same as or similar to the feature extractor 204 described above). Figure 2A and / or Figure 3A The third mask sensor data 302C may be applied to a second convolutional stream including a feature extractor 404 (which may be the same as or similar to the feature extractor described with respect to Figure 2A and / or Figure 3AAs such, each corresponding masked sensor data 302 may have respective feature vectors 420, 414, and 406 and / or feature maps generated by feature extractors 418, 412, and 404, respectively. As a result, with respect to masked sensor data 302, masked sensor data 302 may be applied to feature extractors 420, 414, 406 (e.g., each including one or more feature extractor layers of CNN 112C, such as described herein with respect to Figure 1 In some examples, feature extractors 420, 414, and 406 can be similar to feature extractors 204 and / or 304 described herein. However, by using separate convolution streams, each feature extractor can be trained to predict feature vectors corresponding to sensor data 302 of a particular type of mask (e.g., a first stream can be trained on a joint region, a second stream can be trained on an object region, and a third stream can be trained on a person region). In this way, CNN 112C can generate more accurate feature vectors due to the particular focus on updating the parameters (e.g., weights and biases) of the streams for sensor data 302 of the corresponding type of mask during training. In some examples, the feature vectors output by the feature extractors can have a spatial resolution of 3x3 or 8x3x3 (e.g., Figure 4A ), but this is not intended to be limiting. Feature vectors 420, 414, and 406 (and / or the output of one or more additional layers of the stream) may be concatenated at block 408, and the concatenated output may be applied to the FC layer and sigmoid function 422. Although Figure 4A Only a single FC layer is shown in FIG. 1 , but this is not intended to be limiting, and any number of FC layers (and / or other layer types) may be used without departing from the scope of the present disclosure. The output of the FC layer and the sigmoid function may be the confidence score 116. Thus, the CNN 112C may be relative to Figure 4A The example of masked sensor data 302 in is used to predict or calculate the confidence that object 218 belongs to person 216A.

[0072] In some embodiments, Figure 4A As shown in dashed lines, normalized coordinates 424 may be used as an additional input to CNN 112C for determining confidence 116. For example, normalized coordinates 424 may include information about Figure 1Some or all of the coordinates 120 described. In a non-limiting embodiment, the normalized coordinates 424 may include human coordinates (e.g., coordinates corresponding to a human region, such as a boundary shape that depicts a human region), object coordinates (e.g., coordinates corresponding to an object region, such as a boundary shape that defines an object region), and / or a ratio of a human region to an object region. The normalized coordinates 424 may provide additional information to the CNN 112C about the prediction confidence 116. Although described as normalized coordinates, this is not intended to be limiting, and in some examples, the coordinates may not be normalized.

[0073] In an example using normalized coordinates, the normalized coordinates 424 may be applied to a sigmoid function 426, and the output of the sigmoid function 426 may be combined or concatenated with the outputs of the FC layer and the sigmoid function 422. These combined outputs may then be applied to a sigmoid function 428, and the sigmoid function 428 may output the confidence 116. In this way, the CNN 112C may be relative to Figure 4A The instance of sensor data 302 with the mask in and normalized coordinates 424 are used to predict or calculate the confidence that object 218 belongs to person 216A.

[0074] See also Figure 4B , Figure 4B is a diagram of an example method 450 for associating objects to people using a multi-stream machine learning model according to some embodiments of the present disclosure. Each block of method 450 may include a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, different functions may be performed by a processor executing instructions stored in a memory. Method 450 may also be embodied as computer-usable instructions stored on a computer storage medium. To name a few, method 450 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product. In addition, by way of example, Figure 1 and Figure 4A Method 450 is described. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0075] At block B452, method 450 includes determining from the image one or more persons associated with the object. For example, one or more persons associated with the object may be determined. In some examples, relationship determiner 110 may determine an association between a person and an object, for example, by using an association region within the image, as described herein. In the event that one or more persons are associated with the object, each of blocks B454, B456, B458, B460, and B462 may be performed for each of the one or more persons.

[0076] At block B454, the method 450 includes determining a joint region, a person region, and an object region of the image. For example, the output of the object detector 106 may be used to determine a joint region, a person region, and / or an object region of the image.

[0077] At block B456, method 450 includes generating an image of the first mask, an image of the second mask, and an image of the third mask. For example, the joint region can be used to generate sensor data 302A for the first mask, the object region can be used to generate sensor data 302B for the second mask, and the person region can be used to generate sensor data 302C for the third mask.

[0078] At block B458, method 450 includes applying the first data to the first stream of the neural network, applying the second data to the second stream of the neural network, and applying the third data to the third stream of the neural network. For example, the first masked sensor data 302A may be applied to the first stream of CNN 112C, the second masked sensor data 302B may be applied to the second stream of CNN 112C, and the third masked sensor data 302C may be applied to the third stream of CNN 112C.

[0079] At block B460, method 450 includes calculating a confidence of an association between an object and a person using a neural network including the first stream, the second stream, and the third stream. For example, confidence 116 corresponding to person 216A and object 218 may be calculated using machine learning model 112 (e.g., CNN 112C), where the first stream, the second stream, and the third stream are used to make predictions. In addition, confidence 116 may be calculated for each person determined to be associated with or having a relationship with an object (e.g., person 216B).

[0080] At block B462, method 450 includes associating the object with the person of the one or more persons with the highest confidence of association. As described herein, one or more machine learning models 112 may generate multiple confidences (e.g., one confidence generated by each instantiation of one or more machine learning models 112 corresponding to each person / object pair), and the object / person pair with the highest confidence output by the one or more machine learning models 112 may be determined as the pairing of the object and the person. For example, object 218 may be determined to be associated with person 216A.

[0081] See now Figure 5A , Figure 5A is a diagram of an example machine learning model 112D for associating people with objects using region of interest (ROI) pooling. For example, the machine learning model 112D may be Figure 11. In some examples, because machine learning model 112D can be a CNN, machine learning model 112D can be referred to as CNN 112D. In addition, although CNN 112D is shown as having Figure 5A , but this is not intended to be limiting, and CNN112D may include a different architecture without departing from the scope of the present disclosure.

[0082] CNN 112D may use sensor data 502 (e.g., unmasked sensor data) as input, which may be similar to at least the sensor data described herein. Figure 1 Sensor data 502 may be applied to a feature extractor 504 of CNN 112D (e.g., including one or more feature extractor layers of CNN 112D, such as described herein with respect to Figure 1112D). In some examples, the feature extractor 504 may be similar to the feature extractors 204 and / or 304 described herein. The feature vector or feature map output by the feature extractor 504 may be applied to a region of interest (ROI) pooling layer 506. The ROI pooling layer may be used to perform a pooling operation (e.g., a filter) on the feature vector or feature map generated by the feature extractor 504. The pooling layer may reduce the size of the feature vector or feature map by a factor (e.g., a factor of 2, 3, etc.) to account for the possibility of small movements in the location of features in the input image (e.g., sensor data 502). For example, the pooling layer 506 may be used to summarize the average presence of a feature and the most active presence of a feature, respectively. In some examples, in addition to the feature map, the ROI coordinates 524 may also be applied to the ROI pooling layer 506 of the CNN 112D. The ROI coordinates 524 may help the ROI pooling layer 506 predict information specific to the regions (e.g., the person region 518, the object region 520, and the joint region 522) identified by the ROI coordinates 524. For example, the person region 518 of the ROI coordinates 524 may include the coordinates 120 corresponding to the person region 518, the object region 520 of the ROI coordinates 524 may include the coordinates 120 corresponding to the object region 520, and the joint region 522 of the ROI coordinates 524 may include the coordinates 120 corresponding to the joint region 522. By providing the ROI coordinates 524 to the ROI pooling layer 506, the CNN 112D may more accurately account for the portions of the feature map calculated by the feature extractor 504 corresponding to the person region 518, the object region 520, and the joint region 522. For example, when using the ROI coordinates 524, the ROI pooling layer 506 may use accurate or nearly accurate pooled features corresponding to a person, a bag, and / or a joint thereof. As such, rather than having to generate sensor data corresponding to a mask for each of these regions (thereby reducing the pre-processing requirements of the system), the CNN 112D is able to account for these regions using the ROI pooling layer 506 and the ROI coordinates 524, thereby reducing the computational support and runtime of the CNN 112D compared to other approaches. Additionally, when ROI pooling is used, the training time for training the CNN 112D is significantly reduced.

[0083] The output of the ROI pooling layer 506 may include convolutional features 508, and the convolutional features 508 (and / or the output of any additional layers of the CNN 112D) may be provided to a fully connected layer 510. Figure 5AOnly a single FC layer is shown in FIG. 1 , but this is not intended to be limiting, and any number of FC layers (and / or other layer types) may be used without departing from the scope of the present disclosure. The output of the fully connected layer 510 may be applied to a sigmoid function 512 to generate the confidence 116. In this way, the CNN 112D may be relative to Figure 5A The instances of sensor data 502 and ROI coordinates 524 in are used to predict or calculate the confidence that object 218 belongs to person 216A.

[0084] See now Figure 5B , Figure 5B Is used for Figure 5A 1. A diagram of an example machine learning model 112E that associates people with objects using ROI pooling and a multi-layer perceptron (MLP) network. For example, the machine learning model 112E may be Figure 1 1. In some examples, because the machine learning model 112E can be an MLP network, the machine learning model 112E can be referred to as MLP 112E. In addition, although the MLP 112E is shown as having Figure 5B , but this is not intended to be limiting, and MLP 112E may include a different architecture without departing from the scope of the present disclosure.

[0085] MLP 112E may receive as input confidence 116 as calculated by CNN 112D and normalized coordinates 532. For example, normalized coordinates 532 may include information about Figure 1 Some or all of the coordinates 120 described herein. In a non-limiting embodiment, the normalized coordinates 532 may include person coordinates (e.g., coordinates corresponding to a person region, such as a bounding shape that depicts a person region), object coordinates (e.g., coordinates corresponding to an object region, such as a bounding shape that defines an object region), a ratio of a person region to a joint region, and / or a ratio of an object region to a joint region. The normalized coordinates 532 may provide additional information to the MLP 112E regarding the prediction of the final confidence 540. Although described as normalized coordinates, this is not intended to be limiting, and in some examples, the coordinates may not be normalized.

[0086] The normalized coordinates 532 and the confidences 116 may be provided as inputs to one or more FC layers, such as FC layer 534. For example, the normalized coordinates 532 and the confidences 116 may be inputs to FC layer 534, the output of FC layer 534 may be provided as inputs to FC layer 536, the output of FC layer 536 may be provided as inputs to FC layer 538, and the output of FC layer 538 may be the final confidences 540. In some examples, FC layers 534 and 536 may output a larger size than FC layer 538. For non-limiting embodiments, FC layers 534 and 536 may have a size of 64, while FC layer 538 may have a size of 1 (e.g., indicating the final confidences 540). Although Figure 5B Three FC layers are shown in , but this is not intended to be limiting, and any number of FC layers (and / or other layer types) may be used without departing from the scope of the present disclosure.

[0087] See now Figure 5C , Figure 5C 5 is a flowchart of an example method 550 for associating objects to people using ROI pooling according to some embodiments of the present disclosure. Each block of method 550 may include a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, different functions may be performed by a processor executing instructions stored in a memory. Method 550 may also be embodied as computer-usable instructions stored on a computer storage medium. To name a few, method 550 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product. In addition, by way of example, with respect to Figure 1 and Figure 5A 5 to describe method 550. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0088] At block B552, method 550 includes determining from the image one or more persons associated with the object. For example, one or more persons associated with the object may be determined. In some examples, relationship determiner 110 may determine an association between a person and an object, for example, by using an association region within the image, as described herein. In the event that one or more persons are associated with the object, each of blocks B554, B556, B558, B560, B562, B564, and B566 may be performed for each of the one or more persons.

[0089] At block B554, the method 550 includes determining the first coordinate, the second coordinate, and the third coordinate. For example, the ROI coordinates 524 corresponding to the person region 518, the object region 520, and the joint region 522 may be determined.

[0090] At block B556, method 550 includes applying the first data to a convolutional stream of a neural network. For example, sensor data 502 may be applied to a feature extractor 504 of CNN 112D.

[0091] At block B558, method 550 includes using the convolutional flow of the neural network and the first data to calculate second data corresponding to the output of the convolutional flow. For example, feature extractor 504 can be used to calculate a feature map corresponding to sensor data 504.

[0092] At block B560, the method 550 includes applying the second data and the third data to one or more pooling layers of the neural network. For example, data representing a feature map in addition to the ROI coordinates 524 may be applied to the ROI pooling layer 506 of the CNN 112D.

[0093] At block B562, the method 550 includes using one or more pooling layers and calculating fourth data representing a feature map based at least in part on the second data and the third data. For example, the convolutional features 508 may be calculated using the ROI pooling layer 506.

[0094] At block B564, method 550 includes applying the fourth data to an additional layer of the neural network. For example, data corresponding to the feature map calculated by ROI pooling layer 506 can be applied to an additional layer of CNN 112D, such as fully connected layer 510.

[0095] At block B566, method 550 includes using an additional layer of the neural network to calculate a confidence level of the association between the object and the person. For example, confidence level 116 may be calculated by CNN 112D.

[0096] Figure 6 600 is a block diagram of an example computing device 600 suitable for implementing some embodiments of the present disclosure. For example, the computing device 600 can be used to perform some or all of the processing corresponding to any process or method described herein, and / or can be used to at least partially train and / or deploy the system described herein, such as with respect to Figure 2A-5C One or more of the described machine learning models.

[0097] The computing device 600 may include a bus 602 that directly or indirectly couples the following devices: a memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, which may include one or more hardware accelerators dedicated to performing one or more processing tasks (including but not limited to programmable vision accelerators, deep learning accelerators, or codec accelerators), a communication interface 610, an input / output (I / O) port 612, an input / output component 614 (I / O component), a power supply 616, and one or more presentation components 618 (e.g., a display).

[0098] although Figure 6 The various blocks of are shown as being connected via a bus 602 having lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 618 such as a display device may be considered an I / O component 614 (e.g., if the display is a touch screen). As another example, CPU 606 and / or GPU 608 may include memory (e.g., memory 604 may represent a storage device in addition to the memory of GPU 608, CPU 606, and / or other components). In other words, Figure 6 The computing devices described herein are illustrative only. No distinction is made between categories such as "workstations," "servers," "laptops," "desktops," "tablets," "client devices," "mobile devices," "handheld devices," "game consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are considered Figure 6 within the range of computing devices.

[0099] The bus 602 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus 602 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus.

[0100] Memory 604 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 600. Computer-readable media may include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0101] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 600. As used herein, computer storage media does not include the signals themselves.

[0102] Communication media may include computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transport medium. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a way that information is encoded into the signal. By way of example and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer readable media.

[0103] The CPU 606 may be configured to execute computer-readable instructions to control one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. Each of the CPUs 606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. The CPU 606 may include any type of processor, and may include different types of processors, depending on the type of computing device 600 implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 600, the processor may be an ARM processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, the computing device 600 may also include one or more CPUs 606.

[0104] GPU 608 can be used by computing device 600 to render graphics (e.g., 3D graphics). GPU 608 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 608 can generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command from CPU 606 received via a host interface). GPU 608 can include a graphics memory such as a display memory for storing pixel data. Display memory can be included as part of memory 604. GPU 608 can include two or more GPUs operating in parallel (e.g., via a link). When combined together, each GPU 608 can generate pixel data for different parts of an output image or for different output images (e.g., a first GPU for a first image, a second GPU for a second image). Each GPU can include its own memory, or can share memory with other GPUs.

[0105] In examples where computing device 600 does not include GPU 608 , CPU 606 may be used to render graphics.

[0106] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. The communication interface 610 may include components and functionality that enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0107] The I / O ports 612 can enable the computing device 600 to be logically coupled to other devices including I / O components 614, presentation components 618, and / or other components, some of which can be built into (e.g., integrated into) the computing device 600. Illustrative I / O components 614 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The I / O components 614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some examples, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 600 (as described in more detail below). The computing device 600 may include a depth camera such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations of these for gesture detection and recognition. Additionally, computing device 600 may include an accelerometer or gyroscope to enable motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 600 to render immersive augmented reality or virtual reality.

[0108] The power supply 616 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 616 may provide power to the computing device 600 to enable the components of the computing device 600 to operate.

[0109] The presentation component 618 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 618 may receive data from other components (e.g., GPU 608, CPU 606, etc.) and output the data (e.g., as an image, video, sound, etc.).

[0110] The present disclosure may be described in the general context of machine-usable instructions or computer codes executed by a computer or other machine such as a personal digital assistant or other handheld device, including computer-executable instructions such as program modules. Typically, program modules including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network.

[0111] As used herein, the description of "and / or" with respect to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0112] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps different from the steps described herein in conjunction with other current or future technologies or combinations of similar steps. Moreover, although the terms "step" and / or "frame" may be used herein to imply different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A processor, comprising: A processing circuit, the processing circuit being configured to: generating a sub-image of an image depicting a person corresponding to a person region and an object corresponding to an object region, the sub-image depicting a joint region corresponding to a joint between the object region and the person region; applying data representing the sub-image to one or more neural networks; calculating, using the one or more neural networks, one or more confidence values ​​corresponding to an association between the person and the object based at least on the sub-image; as well as A determination is made whether the object is associated with the person based at least on the one or more confidence values.

2. The processor of claim 1 , wherein determining whether the object is associated with the person comprises at least one of determining whether the one or more confidence values ​​are greater than a threshold, or determining that the one or more confidence values ​​are greater than one or more other confidence values ​​calculated using the one or more neural networks and corresponding to another association between another person and the object.

3. The processor of claim 1 , wherein the computing using the one or more neural networks comprises: One or more first layers of the one or more neural networks are used to process first data representing a first sub-image corresponding to the person area, one or more second layers of the one or more neural networks are used to process second data representing a second sub-image corresponding to the object area, and one or more third layers of the one or more neural networks are used to process third data representing the sub-image, wherein the one or more first layers, the one or more second layers and the one or more third layers are different from each other.

4. The processor of claim 1 , wherein the object region is at least partially defined by a first boundary shape corresponding to the object, the person region is at least partially defined by a second boundary shape corresponding to the person, and the joint region is at least partially defined by a combination of the first boundary shape and the second boundary shape.

5. The processor of claim 1 , further comprising a processing circuit configured to: determining a trajectory of the person through a sequence of images, the sequence of images ending with the image; computing a set of confidence values ​​corresponding to the sequence of images using the one or more neural networks, the set of confidence values ​​including the one or more confidence values; and One or more updated confidence values ​​for the one or more confidence values ​​are calculated using another one or more neural networks and based at least on data representing the set of confidence values ​​and the trajectory.

6. The processor of claim 1 , further comprising a processing circuit configured to: determining normalized coordinates corresponding to at least one of the object region, the person region, and the joint region, Wherein calculating the one or more confidence values ​​is further based at least on the normalized coordinates.

7. The processor of claim 1 , wherein generating the sub-image comprises: Pixel values ​​from the image are converted to mask values ​​based at least on the pixel values ​​corresponding to one or more portions of the image not included in the joint region.

8. A processor according to claim 1, wherein the image is the most recent image in a sequence of images, temporal smoothing is used to weight the one or more confidence values ​​based on one or more previous confidence values ​​predicted for an association between the person and the object with respect to other images in the sequence of images, and determining whether the object is associated with the person is based at least on one or more temporally smoothed confidence values.

9. A method comprising: determining, from the image, one or more persons associated with the subject; For a person among the one or more persons, an operation is performed, the operation comprising: generating a sub-image of an image depicting a person corresponding to a person region and an object corresponding to an object region, the sub-image depicting a joint region corresponding to a joint between the object region and the person region; applying the sub-image to one or more layers of one or more neural networks; calculating a confidence level of the association between the object and the person using the one or more neural networks and based at least on outputs of the one or more neural networks; and Based on the confidences of the person among the one or more persons, the object is associated to the person among the one or more persons having the highest association confidence.

10. The method of claim 9, wherein the object region is at least partially defined by a first boundary shape corresponding to the object, the person region is at least partially defined by a second boundary shape corresponding to the person, and the joint region is at least partially defined by a combination of the first boundary shape and the second boundary shape.

11. The method according to claim 9, further comprising: determining a trajectory of the person through a sequence of images, the sequence of images ending with the image; computing a plurality of confidence measures corresponding to the sequence of images using the one or more neural networks, the plurality of confidence measures including the confidence measure; applying data representing the plurality of confidences and the trajectory to one or more additional neural networks; as well as An updated confidence measure for the confidence measure is calculated based at least on the data using the additional one or more neural networks.

12. The method of claim 9, wherein the one or more layers each comprise one or more convolutional layers.

13. The method of claim 9, wherein calculating the confidence level comprises: cascading portions of the output corresponding to the one or more layers to generate a cascaded output; as well as The cascade output is applied to one or more additional layers of the one or more neural networks.

14. The method of claim 9, wherein the operations further comprise: determining normalized coordinates corresponding to at least one of the object region, the person region, and the joint region, Wherein calculating the confidence is further based at least on the normalized coordinates.

15. The method of claim 9, wherein generating the sub-image comprises: A mask is applied to the image.

16. The method of claim 9, wherein the image is the most recent image in a sequence of images, and temporal smoothing is used to weight the confidence according to previous confidences predicted for the association between the person and the object to generate a final confidence, and further wherein, The final confidence is used to associate the object with the person.

17. A system comprising: One or more processing units, the one or more processing units configured to perform operations, the operations comprising: generating a sub-image representing a portion of the image included in a joint region corresponding to a joint between an object region corresponding to an object and a person region corresponding to a person; applying data representing the sub-image to one or more neural networks to generate output data; calculating one or more confidence values ​​corresponding to an association between the person and the object using the output data; and A determination is made whether the object is associated with the person based at least on the one or more confidence values.

18. The system of claim 17, wherein determining whether the object is associated with the person comprises at least one of determining whether the one or more confidence values ​​are greater than a threshold, or determining that the one or more confidence values ​​are greater than one or more other confidence values ​​calculated using the one or more neural networks and corresponding to another association between another person and the object.

19. The system of claim 17, wherein the calculating comprises: One or more first layers of the one or more neural networks are used to process first data representing the object region, one or more second layers of the one or more neural networks are used to process second data representing the person region, and one or more third layers of the one or more neural networks are used to process third data representing the joint region, wherein the one or more first layers, the one or more second layers, and the one or more third layers are different from each other.

20. The system of claim 17, wherein the object region is at least partially defined by a first boundary shape corresponding to the object, the person region is at least partially defined by a second boundary shape corresponding to the person, and the joint region is at least partially defined by a combination of the first boundary shape and the second boundary shape.

Images