Key point detection method and system based on neural network

Abstract

The present invention relates to a computer implementation method for determining key points of multiple objects contained in an image and associating the key points with each corresponding object based on a neural network (2) and a post-processing system (3) coupled to the neural network (2).

Description

Technical Field

[0001] This invention relates to the general field of neural networks, and more particularly to the field of deep neural networks. More specifically, this invention relates to a method and system for detecting key points of multiple objects contained in an image and associating said key points with corresponding objects based on a neural network. The neural network may be, for example, a convolutional neural network. Background Technology

[0002] The goal of keypoint detection is to locate a predefined set of points on an object in an image and group them into individual object instances. This is a challenging task, especially in automotive applications. By detecting keypoints on objects such as pedestrians, cyclists, and vehicles, the pose of the object relative to the camera's line of sight can be determined. Furthermore, skeletal detection can be performed on one or more people in an image provided by the camera.

[0003] Especially in vehicles, computing resources are limited. However, in automotive applications such as autonomous driving or at least partially assisted driving, it is necessary to detect and group key points in real time and associate the grouped key points (e.g., with a delay of 0.1 seconds or less) with objects, because decisions must be made quickly based on the detected object key points or poses.

[0004] In their paper "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields," presented at the International Conference on Computer Vision and Pattern Recognition on November 24, 2016, Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh announced a Gaussian confidence map prediction centered on the locations of keypoints of interest. Associating keypoints with human instances involves two steps: first, introducing directional connections between keypoints to form a complete skeleton; then, attempting to locate these connections by predicting the vector fields aligned with them. After finding all valid connections, the discrete skeleton can be reconstructed. The method involves associating connections that share keypoints.

[0005] The main drawbacks of the pose estimation algorithm proposed by Cao Zhe et al. are the computational complexity of the upscaling algorithm using bicubic pixel interpolation, and the computational complexity of post-processing that involves grouping key points and associating the grouped key points with object instances.

[0006] Xingyi Zhou, Dequan Wang, Philipp Kleinbull The paper "Objects as Points," published on April 16, 2019, at the International Conference on Computer Vision and Pattern Recognition, discloses an algorithm that detects the centroids of objects in an image, associates keypoint predictions with these centroids, and predicts the offset of keypoints relative to the centroids. The proposed method cannot handle cases where keypoints are missing from object instances (e.g., due to missing lower legs, knees, or ankles). In cases of missing keypoints, the method is forced to predict missing keypoints on the image as well, leading to a large number of false positive keypoint detections. Summary of the Invention

[0007] One object of the present invention is to provide a method for detecting keypoints of multiple objects contained in an image and associating the keypoints with corresponding objects, wherein the method requires less computational resources and is less prone to false positive keypoint detections. This task is achieved through the features of the independent claim. Preferred embodiments are given in the dependent claims. Unless otherwise specified, embodiments of the invention can be freely combined with each other.

[0008] According to one perspective, the present invention relates to a method for determining key points of multiple objects contained in an image and associating the key points with corresponding objects. The method is performed based on a neural network and a post-processing system coupled to the neural network. The method includes the following steps:

[0009] First, confidence heatmaps are provided by a neural network. Each confidence heatmap provides information about the approximate location of a keypoint of interest. More specifically, each confidence heatmap can be a Gaussian or Gaussian-like distribution centered on the location of a specific keypoint. Confidence heatmaps can have a lower resolution compared to the input image provided to the neural network. This reduces computational complexity, but the downside is lower accuracy. Due to the reduced resolution or scale of the confidence heatmaps, they can only provide the approximate location of the keypoints.

[0010] Furthermore, a thinning offset vector is provided by a neural network. Each thinning offset vector is associated with a specific keypoint and provides information for thinning the approximate location of the relevant keypoint as shown by the confidence heatmap. The thinning offset vector is a vector, specifically a two-dimensional vector, that provides accurate directional information about the keypoint's approximate location relative to the keypoint's location shown in the confidence heatmap. More specifically, the confidence heatmap can be a probability distribution, providing values ​​that indicate the probability of a keypoint being at a specific location in the image. The confidence heatmap includes a maximum value, which, due to its low resolution, represents the approximate location of the keypoint. The thinning offset vector is then added to the maximum value of the confidence heatmap to obtain the refined location of the keypoint. For example, if the confidence heatmap indicates the approximate location of the keypoint is (x=2, y=2), and the value of the thinning offset vector is (0.1, 0.8), then by adding the value of the thinning offset vector to the value of the approximate location of the keypoint, the refined location of the keypoint (2.1, 2.8) can be obtained.

[0011] Furthermore, the neural network also provides a set of centroid offset vectors or one or more centroid confidence heatmaps, as well as centroid thinning offset vectors. In the first alternative, each centroid offset vector in the set is associated with a specific keypoint and provides information about the distance and orientation between the keypoint and the centroid of the object to which it belongs. More specifically, the centroid position of the object can be obtained by adding the centroid offset vector value to the position of the keypoint. The keypoint position can be an approximate position (i.e., the position of the maximum value of the confidence heatmap without adding the thinning offset vector) or an exact position (i.e., the position after adding the thinning offset vector to the maximum value of the confidence heatmap).

[0012] In the second alternative, each centroid confidence heatmap provides information about the approximate location of the centroid. More specifically, each centroid confidence heatmap can be a Gaussian or Gaussian-like distribution centered at the location of a specific centroid. A centroid confidence heatmap can be a probability distribution, providing probability values ​​indicating the presence of a centroid at a specific location in the image. The centroid confidence heatmap can have a lower resolution compared to the resolution of the input image provided to the neural network. This reduces computational complexity, but at the cost of lower accuracy. Therefore, due to the reduced resolution or scale of the centroid confidence heatmap, it can only provide the approximate location of keypoints.

[0013] Each centroid refinement offset vector is associated with a centroid and provides information that refines the position of the associated centroid. The centroid refinement offset vector is a vector, specifically a two-dimensional vector, that provides directional information about the centroid, providing an accurate position relative to the approximate position indicated by the centroid confidence heatmap. More specifically, the centroid confidence heatmap includes maximum values, which, due to their low resolution, represent the approximate position of the centroid. The refined position of the centroid is obtained by adding the centroid refinement offset vector to the position of the maximum value in the centroid confidence heatmap. For example, if the centroid confidence heatmap indicates an approximate position of the centroid (x = 3, y = 3), and the value of the centroid refinement offset vector is (0.2, 0.7), then adding the value of the centroid refinement offset vector to the value of the approximate centroid position yields the refined position of the centroid (3.2, 3.7).

[0014] In post-processing, the information provided by the neural network is processed.

[0015] To obtain the approximate location of keypoints, the location of the maximum value in each corresponding confidence heatmap is determined. This approximate keypoint location is refined based on information contained in the refinement offset vector. More specifically, the location of the maximum value in the confidence heatmap represents the approximate location of the keypoint. The refined location of the keypoint is obtained by adding the refinement offset vector to the approximate location of the keypoint.

[0016] Furthermore, the object centroid is determined either by applying a centroid offset vector to the keypoint location or by refining the approximate centroid location provided by the centroid confidence heatmap based on a centroid refinement offset vector. More specifically, the location of the maximum value in the centroid confidence heatmap represents the approximate centroid location. The refined centroid location is obtained by adding the centroid refinement offset vector to the approximate centroid location.

[0017] Finally, keypoints are grouped based on the determined object centroids, thereby associating each keypoint group with a corresponding object. This association with the object can be implemented in various ways. For example, information provided by the centroid offset vector can be used to associate a keypoint with a centroid, and thus with the object to which that centroid belongs. Furthermore, object connectivity can be determined based on the grouped keypoints.

[0018] The proposed method is advantageous because using a thinning offset vector to refine the keypoint locations avoids computationally complex magnification operations. Furthermore, by determining the centroid and using it for keypoint grouping and associating keypoint sets with objects, the tasks of keypoint grouping and object association can be handled with lower computational complexity. Moreover, the proposed method is less prone to false positives because, for the purpose of converting keypoint locations to centroid locations, it is unnecessary to include keypoints that are not present in the image.

[0019] Therefore, the proposed method is more effective and has higher reliability in the overall evaluation.

[0020] According to one embodiment, the neural network provides an affinity field vector set, wherein the affinity field vectors provide information about the connections between keypoint pairs. More specifically, the affinity field vector set is a vector field that provides information about existing object connections between a pair of keypoints. The affinity field vector set can be used for keypoint grouping.

[0021] According to one embodiment, affinity field vectors are used to determine key point pairs that are interconnected / interconnected through object connection structures. Therefore, based on the set of affinity field vectors, it can be determined whether a connection exists between a pair of key points. This determination can be a step of verifying the correctness of the detected connection while taking into account information about the centroid. Therefore, the risk of incorrect grouping of key points can be significantly reduced.

[0022] According to one embodiment, in cases where keypoints are ambiguous, affinity field vectors are used to eliminate the ambiguity of the keypoints. Keypoints may include markers indicating a feature or location of the keypoint. The feature might, for example, indicate that the keypoint is "right ear" or "left shoulder." If the above method detects two or more keypoints with the same marker for a specific object, the information contained in the affinity field vectors can be used to determine which keypoint is correctly associated with the object. This significantly reduces the risk of misgrouping of keypoints.

[0023] According to one embodiment, the confidence heatmap and / or centroid confidence heatmap are provided based on a grid containing a lower resolution than the image. This significantly reduces the computational workload of the confidence heatmap and / or centroid confidence heatmap. To mitigate the accuracy loss due to the lower resolution, the keypoint locations can be refined based on a refined offset vector during post-processing.

[0024] According to one embodiment, the neural network provides confidence heatmaps, thinned offset vectors, centroid offset vectors, and / or centroid confidence heatmaps for different types of objects. For example, the first type could be "people," and the second type could be "vehicles." The neural network can provide information about which type of object was detected. Different object detections can be performed in parallel, meaning the method can determine key points of different objects in a single detection process of the neural network. Preferably, only those key points associated with objects actually present in the image are detected. This significantly improves the detection quality.

[0025] According to one implementation, the confidence heatmap is grouped and / or labeled according to specific keypoint types. In other words, keypoints may include an index indicating which type of keypoint exists at a specific location in the image (e.g., left elbow, right shoulder, etc.). This significantly improves the determination of the object's skeleton.

[0026] According to one implementation, the step of grouping a set of keypoints into an object is performed based on searching for the nearest neighboring centroids of one or more keypoints. This significantly reduces the complexity of keypoint grouping.

[0027] According to one embodiment, the step of grouping a set of keypoints to an object is implemented by determining one or more centroid clusters, assigning a label to each centroid cluster, and assigning the label of a centroid cluster to keypoints by considering the centroid offset vector associated with the keypoints. Therefore, this embodiment considers information about which keypoints belong to which centroid to label keypoints associated with the same centroid, grouping the keypoints, and associating the grouped keypoints with objects.

[0028] According to one embodiment, the step of determining the centroid of one or more objects includes using a centroid offset vector for keypoint locations, resulting in multiple temporary centroids, and determining the centroid by applying an interpolation algorithm to the multiple temporary centroids. Therefore, the centroid is determined inwards from the keypoint locations. This reduces the inaccuracy in determining the centroid offset vector.

[0029] According to one embodiment, the method steps are implemented by processing hardware included in a vehicle to process images of the environment surrounding the vehicle and / or images taken from inside the vehicle. Specifically, the proposed method is advantageous in the automotive application field because it offers low runtime and low code complexity, making it highly suitable for automotive processing hardware with typically very limited processing capabilities.

[0030] According to one embodiment, the method steps are implemented by processing hardware included in the vehicle camera device. In other words, at least the neural network task and preferably the post-processing task are performed in the processing hardware within the camera device.

[0031] According to one implementation, grouped key points of an object are correlated to estimate the object's attitude. Information about future motion and / or future attitude can then be derived to coordinate driving controls based on the estimated future motion / attitude.

[0032] According to another perspective, the present invention relates to a computer program product for determining key points of a plurality of objects contained in an image and associating the key points with each object. The computer program product includes a computer-readable storage medium containing program instructions. The program instructions are executable by a processor to cause the processor to perform the method according to any of the embodiments described above.

[0033] According to another perspective, the present invention relates to a system for determining multiple object keypoints contained in an image and associating the keypoints with corresponding objects. The system includes a neural network and a post-processing system coupled to the neural network. The system is configured to perform the following steps:

[0034] - A confidence heatmap is provided by a neural network, wherein the confidence heatmap contains a lower resolution than the image and thereby provides information about the approximate location of key points;

[0035] - A neural network provides thinning offset vectors, where each thinning offset vector is associated with a specific keypoint, and each thinning offset vector provides information about the location of the relevant keypoint.

[0036] -supply:

[0037] o A set of centroid offset vectors is provided by a neural network, wherein each centroid offset vector in the set is associated with a keypoint, and the centroid offset vectors provide information about the distance and orientation between the keypoint and the centroid of the object to which the keypoint belongs; or

[0038] o The neural network provides one or more centroid confidence heatmaps, wherein each centroid confidence heatmap contains a lower resolution than the image and thereby provides information about the approximate location of the centroids, wherein the neural network provides one or more centroid refinement offset vectors, wherein each centroid refinement offset vector is associated with a centroid and provides information about the refinement of the location of the associated centroid.

[0039] - Determine the location of the maximum value in each corresponding confidence heatmap, and use the location of the maximum value in each corresponding confidence heatmap as the approximate location of the key point. Refine the approximate location of the key point by adding a refinement offset vector, thereby obtaining the refined key point location.

[0040] - The centroids of the plurality of objects are determined by the following method.

[0041] o Add the centroid offset vector to the refined keypoint location or approximate keypoint location; or

[0042] o refines the approximate centroid position provided by the centroid confidence heatmap by adding a centroid refinement offset vector to each corresponding approximate centroid position;

[0043] - Associate a set of key points with the object based on the determined object centroid.

[0044] Each of the features described above as an embodiment of the method can also be used as a system feature in the system disclosed in this patent document.

[0045] The term "vehicle" as used in this invention may refer to automobiles, trucks, buses, rail vehicles or any other means of transportation.

[0046] The term "key point" as used in this patent document can refer to the position or point of an object that characterizes the posture of the object. For a person, key points can be "eyes," "ears," "shoulders," "elbows," "wrists," "hips," "knees," "ankles," etc.

[0047] The term "confidence heatmap" as used in this patent document can refer to a region in which a key point of an object is located, and this region is formed by superimposing a probability distribution, such as a Gaussian distribution. The probability distribution indicates the probability at which the key point is located for each corresponding probability value.

[0048] The term "refined offset vector" as used in this patent document can refer to a vector that indicates the approximate location of a key point, by what distance and in which direction it must be moved to reach the actual key point location.

[0049] The term “centroid” as used in this patent document can refer to the center of an object, especially its center of gravity.

[0050] The term "centroid offset vector" as used in this patent document can refer to a vector that indicates the approximate position of the centroid must be moved by what distance and in which direction to reach the actual centroid position.

[0051] The term "centroid confidence heatmap" as used in this patent document refers to the region where the centroid of an object is located, which is a superposition of probability distributions. These probability distributions indicate the position of the centroid according to which probability value it lies.

[0052] As used in this patent document, the term "object connection structure" can refer to a direct connection structure between a pair of key points that constitute part of the skeleton of an object. In the case of a human being, the object connection structure can be, for example, a limb.

[0053] The terms “substantially” or “approximately” as used in this invention refer to deviations from the exact value by + / -10%, preferably + / -5%, and / or deviations that are insignificant to the function and / or to traffic rules in varying forms. Attached Figure Description

[0054] Different aspects of the invention, including its specific features and advantages, will be more readily understood from the following detailed description and accompanying drawings, wherein:

[0055] Figure 1 An exemplary schematic diagram of a system for determining key points of multiple objects contained in an image and associating the key points with each corresponding object is shown;

[0056] Figure 2 The illustration shows the situation based on Figure 1 The steps implemented in the post-processing system of the system;

[0057] Figure 3 An example diagram illustrating the application of the method is shown;

[0058] Figure 4 It shows that according to Figure 3 The key points and centroid of the image;

[0059] Figure 5 The confidence heatmap and refined offset vector are shown, both of which are associated with specific key points.

[0060] Figure 6 This illustration schematically depicts one method for determining the centroid of an object using multiple centroid offset vectors.

[0061] Figure 7 The process of associating key points with the centroid based on the centroid offset vector is illustrated schematically.

[0062] Figure 8 The diagram illustrates multiple affinity field vectors positioned between a pair of key points;

[0063] Figure 9 The diagram illustrates the following: Figure 3 The image shown shows the detected skeleton; and

[0064] Figure 10 A schematic block diagram is shown, illustrating the steps of a method for determining key points of multiple objects contained in an image and associating the key points with the respective objects. Detailed Implementation

[0065] The invention will now be described in more detail with reference to the accompanying drawings, which illustrate exemplary embodiments. The embodiments in the drawings relate to preferred embodiments, and all elements and features described in connection with these embodiments may be used in conjunction with as many other embodiments and features as possible discussed herein, particularly in relation to any other embodiments further discussed above. However, the invention should not be construed as being limited to the embodiments described herein. Throughout the following description, similar reference numerals denote similar elements, parts, items, or features, where applicable.

[0066] The features of the invention disclosed in the description, claims, embodiments and / or drawings can be used individually or in any combination to implement the invention.

[0067] Figure 1The illustration illustrates a system 1 for identifying key points of multiple objects contained in an image and associating those key points with their respective objects. Objects may be, for example, people, vehicles, etc. The image may contain multiple objects of a single object category (e.g., multiple people in the image) or multiple objects of different object categories (e.g., one or more people and one or more vehicles).

[0068] The images may be provided by a camera device, particularly a camera device installed in or contained within a vehicle.

[0069] The image is received by a neural network 2, which may be, for example, a convolutional neural network. The neural network 2 may be a trained neural network, that is, a neural network 2 that has been trained in advance for a specific task in order to provide specific features for determining key point information about multiple objects contained in the image, and to provide specific features for determining the necessary information for associating the key points with the respective objects.

[0070] like Figure 1 As shown, neural network 2 provides different output information. First, neural network 2 provides multiple confidence heatmaps CH. Each confidence heatmap CH defines a specific region on the image and is associated with a specific keypoint of the object (e.g., a person's left elbow). The confidence heatmap CH indicates that the associated keypoint is located within the confidence heatmap CH. Neural network 2 can provide a confidence heatmap CH for each visible keypoint on the image.

[0071] According to the implementation method, the confidence heatmap CH is labeled. Based on the label, key point types such as "right elbow" and "right shoulder" can be determined.

[0072] In addition, the confidence heatmap (CH) provides probability values ​​for the image in at least two dimensions. These probability values ​​indicate the probability that a keypoint is located in a specific region of the image. The probability values ​​of the confidence heatmap (CH) can be set according to a Gaussian distribution or a Gaussian-like distribution.

[0073] To reduce computational complexity, the confidence heatmap CH can have a lower resolution than the image used as input to the neural network 2. For example, the confidence heatmap CH can include only 1 / 4 to 1 / 100, especially 1 / 8, of the image resolution. This reduction in resolution leads to a loss of accuracy.

[0074] To reduce the loss of accuracy, Neural Network 2 provides a thinned offset vector (ROV). Each thinned offset vector (ROV) is associated with a specific confidence heatmap (CH). The thinned offset vector (ROV) provides correction information to correct the location of the maximum value of the associated confidence heatmap (CH) to accurately define the location of keypoints (associated with the confidence heatmap (CH)) on the image.

[0075] More specifically, the refinement offset vector ROV can be a vector that defines the positional variation of the maximum value of the confidence heatmap CH, thereby reducing the loss of accuracy caused by reducing the resolution of the confidence heatmap CH compared to the image resolution.

[0076] It is worth mentioning that the thinning offset vector ROV is provided by neural network 2. That is to say, the loss of accuracy caused by reducing the resolution of the confidence heatmap CH compared with the image resolution is not necessarily caused by the complex thinning algorithm performed in post-processing 3.

[0077] A neural network can be trained based on baseline truth information to estimate the refinement offset vector (ROV). The baseline truth information may include a baseline truth refinement offset vector, which is constructed by computing a vector that starts at the approximate location of a baseline truth keypoint and ends at its exact location. The approximate location of the baseline truth keypoint is determined at a lower resolution of the confidence heatmap, while the exact location can be determined at a higher resolution, i.e., the resolution of the image provided as input to the neural network. Based on this baseline truth information, the neural network can be trained to estimate the refinement offset vector, for example, using a loss function.

[0078] In addition, referencing Xingyi Zhou, Dequan Wang, and Philipp Kleinbull... The example loss function for training a neural network, provided in Section 3 of the paper "Objects as Points" presented at the International Conference on Computer Vision and Pattern Recognition on April 16, 2019, provides a vector to reduce the loss of accuracy due to downsampling. Neural Network 2 can be trained in a similar manner to estimate the refined offset vector ROV.

[0079] Furthermore, neural network 2 can provide centroid information about one or more objects contained in an image. This centroid information can be provided in different ways.

[0080] according to Figure 1 In this embodiment, the neural network 2 provides centroid offset vectors (COVs). Each COV is associated with a specific keypoint of the object. Each COV provides displacement information indicating the direction and distance in which a specific keypoint must move to reach the centroid of the object. In other words, based on the COVs, a specific keypoint of the object can be moved to the centroid position of the object.

[0081] According to another embodiment, the neural network 2 can provide information about the centroids of one or more objects contained in an image based on one or more centroid confidence heatmaps and one or more centroid refinement offset vectors.

[0082] Similar to the confidence heatmap described above, the centroid confidence heatmap defines a specific region where the centroid of an object in an image is located. Furthermore, the centroid confidence heatmap provides signal values ​​in at least two dimensions of the image. These signal values ​​indicate the probability that the object's centroid lies within a specific region of the image. The signal values ​​of the centroid confidence heatmap can be set according to a Gaussian distribution or a Gaussian-like distribution.

[0083] The centroid confidence heatmap may also have a lower resolution than the image used as input to the neural network 2. To reduce the loss of accuracy caused by the reduced resolution, the neural network 2 provides one or more centroid thinning offset vectors. Each centroid thinning offset vector is associated with a specific centroid confidence heatmap. The centroid thinning offset vector provides correction information to correct the location of the maximum value in the associated centroid confidence heatmap to accurately define the position of the centroid on the image.

[0084] More specifically, the centroid thinning offset vector can be a vector that defines the change in the location of the maximum value of the centroid confidence heatmap, thereby reducing the loss of accuracy caused by reducing the resolution of the centroid confidence heatmap compared to the image resolution.

[0085] It is worth mentioning that the centroid thinning offset vector can be provided by neural network 2. That is to say, the loss of accuracy caused by reducing the resolution of the centroid confidence heatmap compared with the image resolution is not necessarily caused by the complex thinning algorithm performed in post-processing 3.

[0086] Neural network 2 can be trained based on baseline truth information to estimate the centroid thinning offset vector. The baseline truth information may include a baseline truth centroid thinning offset vector, which is established by calculating a vector that starts at the approximate location of the baseline truth centroid and ends at its exact location. The approximate location of the baseline truth centroid is determined in a lower-resolution centroid confidence heatmap, while the exact location can be determined at a higher resolution—i.e., the image resolution used as input to neural network 2. Based on the baseline truth information, neural network 2 can be trained, for example, by estimating the centroid thinning offset vector using a loss function.

[0087] In addition, referencing Xingyi Zhou, Dequan Wang, and Philipp Kleinbull... The example loss function for training a neural network, provided in Section 3 of the paper "Objects as Points" presented at the International Conference on Computer Vision and Pattern Recognition on April 16, 2019, provides a vector to reduce the loss of accuracy due to downsampling. Neural Network 2 can be trained in a similar manner to estimate the centroid thinning offset vector.

[0088] Finally, according to one embodiment, the neural network 2 can provide one or more sets of affinity field vectors (AFVs). An affinity field vector set is a vector field representing a specific connection between a pair of keypoints. For example, the affinity field vector set can represent the connection between a first keypoint "right shoulder" and a second keypoint "right elbow," i.e., it can represent, for example, a human limb. The vectors contained in the affinity field vector set are arranged according to existing connections.

[0089] As described in more detail below, affinity field vector sets can be used to associate key points with specific objects and / or eliminate ambiguity.

[0090] The output information of neural network 2 can be provided to post-processing system 3, which is configured to process the received information to determine key points, determine the association between key points and specific objects, and determine existing connections between key points to build the skeleton of the object.

[0091] Figure 2 The process executed by post-processing system 3 is shown in more detail.

[0092] The confidence heatmaps (CH) provided by neural network 2 can be parsed to determine local maxima. In other words, the peak value of each confidence heatmap (CH) is determined to obtain approximate location information of key points associated with each corresponding confidence heatmap. The confidence heatmaps (CH) can be provided on a grid. The local maxima of the confidence heatmaps (CH) can be set at the intersections of grid lines.

[0093] Figure 3 Here is an example image showing two objects: two ballet dancers.

[0094] Figure 4 In the middle, highlighted with a white oval Figure 3 The confidence heatmap of the object shown is displayed, with the centroid of the object highlighted by a white square.

[0095] Figure 5 An example is shown of a specific confidence heatmap (CH) provided on a grid. Different gray levels of the squares represent the probability values ​​at their respective locations. Peak probability values ​​are in the middle and are represented by black squares.

[0096] Back Figure 2After searching for local maxima, keypoints are refined based on information provided by the refined offset vector (ROV). As mentioned above, the refined offset vector (ROV) is associated with a specific confidence heatmap (CH). Due to the low resolution of the confidence heatmap (CH), the determined local maxima can only indicate the approximate location of the keypoints.

[0097] By using the refinement offset vector (ROV) to represent local maxima, the refinement position of keypoints can be determined. In other words, by moving the position of local maxima based on the information provided by the refinement offset vector (ROV), the loss of accuracy can be reduced.

[0098] Figure 5 An example is shown of local maximum movement implemented by refining the offset vector ROV, indicated by a white arrow in the center of a black square. By using the refining offset vector ROV to determine the local maximum of the confidence heatmap CH, the keypoint refining position is determined by the white cross at the tip of the white arrow.

[0099] Back to Figure 2 After refining the key points, the centroids of the objects contained in the image are determined.

[0100] As mentioned above, the centroid can be determined in different ways. Figure 2 The embodiment shown uses the centroid offset vector COV to determine the centroid of the object.

[0101] Each centroid offset vector (COV) is associated with a specific keypoint. By applying the COV to the associated keypoint, the centroid location of the object to which that keypoint belongs can be determined, or at least an estimate of the centroid's location can be determined.

[0102] Figure 6 An example is shown with multiple keypoints of an object represented by black dots, and a set of centroid offset vectors (COVs) represented by white arrows. Each COV is associated with a specific keypoint. If the keypoints are shifted based on the COVs, a centroid estimate of the skier's center, represented by white dots, is obtained. The estimated centroid obtained through this shifting operation may not be exactly at the object's actual centroid. However, by considering all estimated centroids and applying an averaging algorithm to them, the actual centroid can be derived.

[0103] According to another embodiment, as described above, the centroid of an object contained in an image can also be determined based on a centroid confidence heatmap and a centroid refinement offset vector, wherein each centroid refinement offset vector corresponds to one of the centroid confidence heatmaps. The maximum position of the centroid confidence heatmap represents the approximate position of the centroid. The exact position of the centroid is obtained by shifting the approximate position of the centroid based on the information contained in the corresponding centroid refinement offset vector.

[0104] Back to Figure 2 Once the centroid is determined, the keypoint is associated with a specific object. In other words, when there are multiple keypoints in an image, the goal is to determine which keypoint belongs to which object.

[0105] Keypoints provided by a neural network can be labeled. The labeling of a keypoint indicates its location on an object. Thus, a personal keypoint label could represent, for example, "left elbow" or "right shoulder".

[0106] However, in the case of multiple keypoints with the same tag category, it is necessary to determine the association between the keypoints and the objects, that is, which "right shoulder" belongs to which person.

[0107] According to the first embodiment, the association between a keypoint and an object can be determined based on a "nearest neighbor search" algorithm. In other words, a specific keypoint is associated with an object by considering which centroid of the object is closest to the keypoint.

[0108] According to the second embodiment, the information provided by the centroid offset vector (COV) can be used to associate keypoints with objects. As described above, the COV provides information about the direction and distance between a specific keypoint and the centroid of the object to which that keypoint belongs. This information can be used to associate keypoints with objects.

[0109] For example, clustering algorithms can be used to determine clusters whose approximate centroid locations are obtained by using the centroid offset vector (COV) as keypoints. Examples of clustering mechanisms could be, for example, the k-means algorithm or the EM algorithm (EM: expectation maximization). Each cluster can be provided by labels representing a specific object. Using knowledge of the centroid offset vector (COV), the labels provided to the clusters can also be provided to a set of keypoints corresponding to the COV (see [link to relevant documentation]). Figure 6 Therefore, a set of key points can be identified as belonging to a specific cluster or a specific object.

[0110] Figure 7 This illustration demonstrates how a clustering algorithm associates keypoints with objects. Detected keypoints are represented by X, and detected centroids by C. The arrows indicate the centroid offset vector (COV). By considering information about the COV, groupings of keypoints with centroids, or keypoints with objects, can be obtained.

[0111] According to the third embodiment, keypoint grouping can also be obtained by using affinity field vectors (AFVs). As described above, each affinity field vector set provides information about keypoint connections, that is, information about which keypoint is connected to which further keypoint via an object connection structure. Such an object connection structure can be, for example, a human limb. By considering the information contained in the affinity field vectors, keypoint-object grouping can be obtained, because only keypoint pairs connected to each other via affinity field vectors belong to a specific object.

[0112] Figure 8 A set of affinity field vectors is provided. These affinity field vectors may, for example, indicate the connection between two key points, "shoulder" and "elbow". The affinity field vectors are arranged according to object connection structures such as limb orientation.

[0113] Associating keypoints with objects in an image can lead to ambiguity. For example, an object may contain multiple keypoints belonging to the same keypoint tag, such as multiple "left shoulder" tags. To eliminate this ambiguity, affinity field vectors (AFVs) can be used to check which keypoint truly belongs to the object and remove at least one further keypoint association. This ambiguity elimination may be particularly beneficial in implementations that use nearest neighbor search or clustering mechanisms to obtain keypoint-object associations, since in these implementations, the affinity field vector (AFV) is not used to obtain keypoint-object associations.

[0114] Finally, after eliminating ambiguity, the object skeleton is constructed. For example, the skeleton can be constructed by connecting keypoints associated with a specific object based on keypoint markers. Such connections may include, for example, connecting the object to associated keypoints based on a known structure of the object, such as connecting the keypoint "left wrist" to the keypoint "left elbow," and connecting the keypoint "left elbow" to the keypoint "left shoulder."

[0115] Figure 9 The illustration is based on Figure 3 and Figure 4 The image shows the detected skeleton of the object. The skeleton determined based on the detected keypoints is represented by white lines.

[0116] The artificial neural network 2 can be trained using images, which are manually labeled before being used for training. These labels provided to the images are also referred to as baseline truth data. The baseline truth data may, for example, provide a bounding box representing the extent of each object. Based on these bounding boxes, the centroid of the object can be determined, for example, by calculating the center coordinates of the bounding boxes and using these center coordinates as the centroid location of the object.

[0117] Furthermore, baseline truth data can provide information about the location of key points of an object. Additionally, baseline truth data can provide information about key point types such as the left shoulder, right ear, and left knee, or, in the case of a vehicle, information about key point types such as the left side mirror and right headlight.

[0118] By using the aforementioned benchmark truth data, the artificial neural network 2 can be trained to estimate confidence heatmaps, approximate locations of key points, thinning offset vectors, centroid offset vectors, centroid thinning offset vectors, and / or centroid confidence heatmaps.

[0119] Some of the processes performed by the post-processing system 3 have been described previously in sections related to specific implementations. In addition, the post-processing system 3 can also perform other tasks, such as attitude estimation, providing bounding boxes around objects based on detected keypoints (also through a 3D stereo approach extending object depth), providing predictions of relationships between objects (e.g., detecting the relationship between a motorcycle and a person near it, and for this purpose, making a decision about the "motorcycle driver"), performing 3D stereo detection of keypoints through regression analysis or tracking the distance of keypoints, 3D stereo orientation of an object (e.g., by determining the roll, pitch, and yaw angles of the object relative to the camera device), and performing object tracking by regressing the time offset of keypoint coordinates.

[0120] The content disclosed in this patent document can be used for the detection of key points and objects in the environment surrounding a vehicle. However, according to another embodiment, the interior space of the vehicle can also be monitored based on a camera device, and key points of the occupants can be detected to obtain information about the vehicle's passengers, such as whether the driver is ready to take over the vehicle in an autonomous driving situation.

[0121] Figure 10 The diagram illustrates the steps of a method for identifying multiple object key points and associating those key points with objects.

[0122] The first step is to detect multiple objects in the image. Each object contains a centroid and multiple keypoints.

[0123] Subsequently, a confidence heatmap is provided by a neural network (S10). The confidence heatmap provides information about the approximate location of the key points.

[0124] After providing the confidence heatmap, the neural network provides the refinement offset vectors (S11). Each refinement offset vector is associated with a specific keypoint and provides information on the location of the relevant keypoint.

[0125] After providing the confidence heatmap, the neural network provides the centroid offset vector or centroid heatmap (S12).

[0126] As a further step, the maximum value position of each corresponding confidence heatmap that defines the approximate location of the keypoint is determined. The approximate location of the keypoint is then refined based on the refined offset vector (S13).

[0127] In a further step, the centroid of the objects contained in the image is determined (S14).

[0128] Finally, based on the determined centroids, the determined key points are grouped (S15).

[0129] It should be noted that the description and accompanying drawings are merely illustrative of the principles of the invention. Those skilled in the art will be able to implement various arrangements that are not explicitly described or shown herein but embody the principles of the invention.

[0130] List of reference numerals

[0131] 1 System

[0132] 2 Neural Networks

[0133] 3 Post-processing

[0134] AFV Affinity Field Vector

[0135] CH Confidence Heatmap

[0136] COV centroid offset vector

[0137] ROV Refine Offset Vector

Claims

1. A computer implementation method for determining key points of multiple objects contained in an image and associating the key points with corresponding objects based on a neural network (2) and a post-processing system (3) coupled to the neural network (2), wherein, The method includes the following steps: - A confidence heatmap (CH) (S10) is provided by a neural network (2), wherein the confidence heatmap (CH) contains a lower resolution than the image and thereby provides information about the approximate location of key points of interest; - A thinning offset vector (ROV) (S11) is provided by a neural network (2), wherein each thinning offset vector (ROV) is associated with a specific key point, wherein each thinning offset vector (ROV) provides information for thinning the position of the associated key point; -Provided (S12): o A set of centroid offset vectors (COVs) is provided by a neural network (2), wherein each centroid offset vector in the set is associated with a keypoint, and the centroid offset vectors (COVs) provide information about the distance and orientation between the keypoint and the centroid of the object to which the keypoint belongs; or o The neural network (2) provides one or more centroid confidence heatmaps, wherein each centroid confidence heatmap includes a lower resolution than the image and thereby provides information about the approximate location of the centroid, wherein the neural network (2) provides one or more centroid thinning offset vectors, wherein each centroid thinning offset vector is associated with a centroid and provides information for thinning the location of the associated centroid; - Determine the location of the maximum value in each corresponding confidence heatmap (CH), and use the location of the maximum value in each corresponding confidence heatmap (CH) as the approximate location of the key point. Then, refine the approximate location of the key point by adding a refinement offset vector (ROV) to obtain the refined key point location (S13). -The centroids of the plurality of objects are determined by the following method (S14) o Add the centroid offset vector (COV) to the refined keypoint location or approximate keypoint location; or o The approximate position of the centroid provided by the centroid confidence heatmap is refined by adding a centroid refinement offset vector to each corresponding approximate position of the centroid (S14). - Associate key points with objects based on the determined object centroid (S15).

2. The method according to claim 1, wherein, The neural network (2) provides a set of affinity field vectors (AFVs), wherein the affinity field vectors (AFVs) provide information about the connections between key point pairs.

3. The method according to claim 2, wherein, Affinity field vectors (AFVs) are used to identify key pairs of points that are interconnected through object connection structures.

4. The method according to claim 2 or 3, wherein, In cases where key points are unclear, affinity field vectors (AFV) are used to eliminate the ambiguity of key points.

5. The method according to any one of the preceding claims, wherein, The neural network (2) provides confidence heatmaps (CH), refined offset vectors (ROV), centroid offset vectors, and / or centroid confidence heatmaps for different types of objects.

6. The method according to any one of the preceding claims, wherein, Confidence heatmaps (CH) group and / or label key points based on a given key point type.

7. The method according to any one of the preceding claims, wherein, The step of associating keypoints with objects is performed by searching for the centroids that are closest to one or more keypoints.

8. The method according to any one of the preceding claims, wherein, The steps of associating keypoints with objects are performed by identifying one or more centroid clusters, assigning a label to each centroid cluster, and assigning the label of a particular centroid cluster to the keypoint by taking into account the centroid offset vector (COV) associated with the keypoint.

9. The method according to any one of the preceding claims, wherein, The steps for determining the centroid of an object include: applying a centroid offset vector (COV) to the keypoint locations to generate multiple temporary centroids, and applying an interpolation algorithm to the multiple temporary centroids to determine the centroid.

10. The method according to any one of the preceding claims, wherein, Each step of the method is performed by processing hardware included in the vehicle for processing images of the vehicle's surrounding environment and / or processing images taken from inside the vehicle.

11. The method according to any one of the preceding claims, wherein, Each step of the method is performed by processing hardware included in the vehicle camera device.

12. The method according to any one of the preceding claims, wherein, To estimate the pose of an object, connect the key points associated with the object.

13. A computer program product for determining key points of a plurality of objects contained in an image and associating the key points with respective objects, wherein, The computer program product includes a computer-readable storage medium containing program instructions, wherein the program instructions are executable by a processor to cause the processor to perform the method according to any of the preceding claims.

14. A system for determining key points of a plurality of objects contained in an image and associating the key points with respective objects, wherein, The system includes a neural network (2) and a post-processing system (3) coupled to the neural network (2), wherein the system is further configured to perform the following steps: - A confidence heatmap (CH) is provided by a neural network (2), wherein the confidence heatmap (CH) contains a lower resolution than the image and thereby provides information about the approximate location of key points of interest; - A thinning offset vector (ROV) is provided by a neural network (2), wherein each thinning offset vector (ROV) is associated with a specific key point, wherein each thinning offset vector (ROV) provides information for thinning the position of the associated key point; -supply: o A set of centroid offset vectors (COVs) is provided by a neural network (2), wherein each centroid offset vector (COV) in the set is associated with a keypoint, and the centroid offset vector (COV) provides information about the distance and orientation between the keypoint and the centroid of the object to which the keypoint belongs; or o The neural network (2) provides one or more centroid confidence heatmaps, wherein each centroid confidence heatmap includes a lower resolution than the image and thereby provides information about the approximate location of the centroid, wherein the neural network (2) provides one or more centroid thinning offset vectors, wherein each centroid thinning offset vector is associated with a centroid and provides information for thinning the location of the associated centroid; - Determine the location of the maximum value in each corresponding confidence heatmap (CH), use the location of the maximum value in each corresponding confidence heatmap (CH) as the approximate location of the key point, and refine the approximate location of the key point by adding a refinement offset vector (ROV) to obtain the refined key point location; - The centroids of the plurality of objects are determined by the following method. o Add the centroid offset vector (COV) to the refined keypoint location or approximate keypoint location; or o refines the approximate centroid position provided by the centroid confidence heatmap by adding a centroid refinement offset vector to each corresponding approximate centroid position. - Associate a set of key points with the object based on the determined object centroid.

Images