A human pose estimation system and method thereof

Abstract

A human pose estimation system 100 for a motor vehicle 130, the system includes an imaging device 102 and a processor 104, the processor further comprises a base model 110 operable to execute a pose estimation algorithm 112. The pose estimation algorithm is operable to determine a pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120 captured by the imaging device. The processor further comprises a visibility predictor 114 operable to execute a visibility prediction algorithm 116 in parallel with the pose estimation algorithm. The visibility algorithm is operable to determine a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured by the imaging device. Preferably, the processor is operable to generate a value of total loss function of the human pose estimation system and input the total loss function generated to an artificial neural network 106 to train the artificial neural network.

Description

TECHNICAL FIELD

[001] This disclosure relates to a human pose estimation for use in a motor vehicle, and more in particular a human pose estimation for monitoring a cabin of a motor vehicle. BACKGROUND

[002] Occupant safety is an ongoing concern for motor vehicles and increasingly, vehicular application uses sensors to assist with determining actions and behaviours of occupants within passenger compartment of motor vehicles. Traditional onboard monitoring systems uses image processing to determine a state of at least one vehicle occupant.

[003] Human Pose Estimation (HPE) is technique applied to identify and classify poses of human body parts and joints in images or videos, using keypoints. These keypoints may include body joints such as shoulders, elbows, hips and ankles as swell as facial features such as eyes, ears, nose, but not limited thereto. Artificial intelligence such as deep convolution neural networks (CNN) may be deployed to identify and classify pose of human body parts in HPE, to increase accuracy and efficiency. However, conventional HPE networks are unable to handle occlusion of keypoints. This is a main challenge when implementing HPE in motor vehicle, since erroneous predictions of keypoints can pose significant challenges for applications where precise estimation is essential. Henceforth, erroneous predictions of keypoints may be detrimental to safety of a driver and / or passengers.

[004] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure. SUMMARY

[005] A purpose of this disclosure is to ameliorate the problem of occlusion of keypoints in HPE networks for use in a motor vehicle, by providing the subject-matter of the independent claims.

[006] In an aspect of this disclosure, a human pose estimation system for a motor vehicle is provided. The human pose estimation system may comprise an imaging device and a processor. The imaging device may have a field of view of a cabin of a motor vehicle. The imaging device may be operable to capture at least one video image frame containing image information of at least one vehicle occupant within the field of view of a cabin of the motor vehicle. The processor may comprise an artificial neural network and a memory. The processor may be operable to retrieve and execute a set of instructions stored thereon. The processor may be operable to determine a pose of the at least one vehicle occupant in response to the least one video image frame captured. The processor may comprise a base model and a visibility predictor. The base model may be operable to execute a pose estimation algorithm. The pose estimation algorithm may be operable to determine a pose of the at least one vehicle occupant contained in the at least one video image frame captured. The visibility predictor may be operable to execute a visibility prediction algorithm in parallel with the pose estimation algorithm. The visibility prediction algorithm may be operable to determine a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured. Advantageously, the aforesaid system combines a pose estimation algorithm and a visibility prediction algorithm in parallel with the pose estimation algorithm to enhance accuracy of human pose estimation.

[007] In some embodiment, the base model may comprises a backbone. The backbone may include a ResNet-S. An advantage of using ResNet-S as the backbone is that ResNet-S holds a significantly smaller model size compared to original ResNet.

[008] In some embodiment, the base model may include a transformer encoder layer, the transformer encoder layer may further comprises an encoder. The encoder may be operable to perform an encoding task. In some embodiment, the encoding task may include a heatmap prediction.

[009] In some embodiment, the transformer encoder layer may comprise a multi-head attention mechanism.

[0010] In some embodiment, the visibility state of at least one keypoint may be represented by at least one integer label assigned by the visibility predictor. Each of the at least one integer label may be selected from a classification comprising visible, at least partially occulated or occulated. Advantageously, each keypoint identified by in the at least one video image frame captured by the imaging device is assigned an integer label, for example {0, 1, 2} which indicates a level of visibility of the keypoint identified by the pose estimation algorithm of the base model.

[0011] In some embodiment, the processor may be operable to generate a value of total loss function of the human pose estimation system. The processor may be operable to input the value of total loss function generated to the artificial neural network to train the artificial neural network. In some embodiment, the value of total loss function of the human pose estimation system may be generated by a mean square error of a loss of function output of the base model and a loss of function output of the visibility predictor. Advantageously, using a value the total loss function generated to train the artificial neural network improves performance of the encoding task and minimises a value of the total loss function over time to improve accuracy of the human pose system.

[0012] In an aspect of this disclosure, a method of estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle is provided. The method may comprise receiving, by way of an imaging device, at least one video image frame containing image information of at least one vehicle occupant captured within a field of view of a cabin of the motor vehicle and retrieving and executing, by way of a processor comprising an artificial neural network and a memory, a set of instructions stored thereon for determining a pose of the at least one vehicle occupant in response to the least one video image frame captured. The set of instructions may further comprise executing, by way of a base model, a pose estimation algorithm for determining a pose of the at least one vehicle occupant contained in the at least one video image frame captured. The set of instructions may further include in parallel with executing the pose estimation algorithm, executing, by way of a visibility predictor, a visibility prediction algorithm for determining a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured.

[0013] In some embodiment, the method may further comprise labelling, by way of the visibility predictor, at least one integer label for assigning a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured, as determined by the visibility prediction algorithm.

[0014] In some embodiment, the at least one integer label may be assigned by the visibility predictor, the at least one integer label may be selected from a classification comprising visible at least partially occulated or occulated.

[0015] In some embodiment, the method may further comprise generating, by way of the visibility predictor, at least one output video image frame containing visibility state of a maximum number of keypoints allowable by the human pose estimation system in each of the at least one output video image frame.

[0016] In some embodiment, the method may further comprises generating, by way of the processor, a value of total loss function of the human pose estimation system and inputting, by way of the processor, the value of total loss function of the human pose estimation system generated to the artificial neural network, for training the artificial neural network. Consequently, the artificial neural network is continuously trained using a value of the total loss function, to improve accuracy and minimise error margin in identifying a pose and corresponding keypoints of at least one vehicle occupant captured in the at least one video image frame.

[0017] In some embodiment, generating the value of the total loss function of the human pose estimation system may comprise calculating, by way of the processor, a mean square error of a loss of function output of the base model and a loss of function output of the visibility predictor.

[0018] In an aspect of this disclosure, a motor vehicle having a human pose system and a processor adapted to execute the steps of estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle as disclosed herein is provided.

[0019] In an aspect of this disclosure, a computer program product comprising instructions to cause a human pose estimation system as disclosed herein to execute the steps of the method of estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle as disclosed herein is provided.

[0020] In an aspect of this disclosure, a non-transitory computer readable medium having stored thereon a computer program product as disclosed herein is provided.

[0021] In an aspect of this disclosure, a method of training an artificial neural network for estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle is provided. The method may comprise generating, by way of a processor comprising an artificial neural network and a memory, a value of total loss function of a human pose estimation system as disclosed herein and inputting, the value of total loss function of the human pose estimation system to the artificial neural network of the processor. The value of total loss function may be defined by a difference between a loss of function of a module output and a ground truth. The module output may be defined by a visibility predictor of the human pose estimation system and the ground truth may be defined by a base model of the human pose estimation system. The difference between a loss of function of the module output and the ground truth may be determined by a square root error. Advantageously, using a value the total loss function generated to train the artificial neural network improves performance of the encoding task and minimises a value of the total loss function over time to improve accuracy of the human pose system.

[0022] Other objects, features and characteristics, as well as the methods of operation and the functions of the related elements of the structure, the combination of parts and economics of manufacture will become more apparent upon consideration of the following detailed description and appended claims with reference to the accompanying drawings, all of which form a part of this specification. It should be understood that the detailed description and specific examples, while indicating the non-limiting embodiments of the disclosure, are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] Other objects and aspects of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings in which: FIG. 1 shows a human pose system in accordance with an embodiment. FIG. 2 shows a schematic of an artificial neural network in accordance with an embodiment. FIG. 3 shows a flowchart of estimating a human pose in accordance with an embodiment. In various embodiments described by reference to the above figures, like reference signs refer to like components in several perspective views and / or configurations. DETAILED DESCRIPTION OF EMBODIMENTS

[0024] The following detailed description is merely exemplary in nature and is not intended to limit the disclosure or the application and uses of the disclosure. Furthermore, there is no intention to be bound by any theory presented in the preceding background of the disclosure or the following detailed description. It is the intent of this disclosure to present a human pose estimation system for implementation in a motor vehicle which combines a pose estimation algorithm and a visibility prediction algorithm in parallel with the pose estimation algorithm to enhance accuracy of human pose estimation.

[0025] Hereinafter, the term “processor” may also refer to a “computer” and the term “processor” used herein may broadly encompass a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and so forth. Under some circumstances, a "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term "processor" may refer to a combination of processing devices, for example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing. The “processor” may include a memory, for loading a sequence of instruction, causing the “processor” to perform steps of actions. The term "memory" should be interpreted broadly to encompass any electronic component capable of storing electronic information. The term “memory” may refer to various types of processor-readable media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. “Memory” is said to be in electronic communication with a processor if the processor can read information from and / or write information to the memory. Memory that is integral to a processor is in electronic communication with the processor. Henceforth, the term “processor” may also be taken to encompass “system on chip” (SoC) which uses a single integrated circuit (IC) chip that contains multiple resources, computational units, processors and / or cores integrated on a single substrate. A single SOC may contain circuitry for digital, analog, mixed-signal, and radio-frequency functions, as well as any number of general purpose and / or specialized processors (digital signal processors, modem processors, video processors, etc.), memory blocks (e.g., ROM, RAM, Flash, etc.). Unless otherwise specifically stated, the “processor” is not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the processing unit. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this disclosure, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both.

[0026] The term “first”, “second”, “third” and the like used in the context of this disclosure may refer to modification of different elements in accordance to various exemplary embodiments, but not limited thereto. The expressions may be used to distinguish one element from another element, regardless of sequence of importance. By way of an example, “a first layer” and “a second layer” may indicate different layers of a neural network regardless of order or importance. On a similar note, a first capacitor may be referred to as the second capacitor and vice versa without departing from the scope of this disclosure.

[0027] Referring to the accompanying drawings, FIG. 1 shows a human pose system 100 in accordance with an embodiment. The system 100 comprises an imaging device 102 and a processor 104. The imaging device 102 has afield of view of a cabin of a motor vehicle 130, the imaging device is operable to capture and generate at least one video image frame 120 containing information of at least one vehicle occupant 132, 132’ within the field of view of the cabin of the motor vehicle 130.

[0028] The processor 104 comprises an artificial neural network 106 and a memory 108. The processor 104 is operable to retrieve and execute a set of instructions stored in the memory 108, and the processor 104 is operable to determine a pose of the at least one vehicle occupant 132, 132’ in response to the at least one video image frame 120 captured. An example of a suitable artificial neural network may be a deep convolution neural network, but not limited thereto. It shall be understood by a skilled practitioner in other types of artificial neural network operable to perform regression and / or classification task may be applicable.

[0029] In some embodiment, as shown in FIG. 1, the processor 104 may further comprises a base model 110, the base model 110 operable to execute a pose estimation algorithm 112. The pose estimation algorithm 112 may be operable to determine a pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120 captured by the imaging device 102. The processor 104 may further comprise a visibility predictor 114 operable to execute a visibility prediction algorithm 116 in parallel with the pose estimation algorithm 112. The visibility algorithm 116 may be operable to determine a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120 captured by the imaging device 102.

[0030] In some embodiment (not shown in Figures), the base model 110 and the visibility predictor 114 may execute the pose estimation algorithm 112, and a visibility prediction algorithm 116 using separate processors and transmit the information to a single processor to determine a pose of the at least one vehicle occupant 132, 132’ captured in the at least one video frame.

[0031] In some embodiment, the imaging device 102 may be a sub-system within the motor vehicle 130, for example an imaging device of an in-vehicle monitoring system such as a driver monitoring system or an imaging device of a cabin monitoring system. In some embodiments, the transmitting and receiving of the at least one video image frame 120 input data to the processor 104 may be done through a vehicle network 140. In some embodiments, the transmitting and receiving of the at least one video image frame 120 input data to the base model 110 and the visibility predictor 114 is done through a vehicle network 140.

[0032] FIG. 2 shows a schematic of an artificial neural network in accordance with an embodiment. As shown in FIG. 2, the at least one video image frame 120 captured may be transmitted to the base model 110 and the visibility predictor 114 in parallel. As explained above, the base model 110 is operable to execute a pose estimation algorithm 112, to determine a pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120 captured by the imaging device 102. In other words, the base model 110 receives the at least one video image frame 120 transmitted to the base model 110 from the imaging device 102. A suitable example of a base model 110 may be a TransPose. As shown in FIG. 2, the base model 110 includes a transformer encoder layer 202. The transformer encoder layer comprises a backbone 204, a transformer and a head 210. In some embodiment, the backbone 204 of the base model 110 is a ResNet-S, although not limited thereto. ResNet-S is a simplified version of ResNet and it shall be understood by a skilled practitioner, ResNet and other suitable types of backbone suitable for image recognition may be applicable. An advantage of using ResNet-S as the backbone 204 is that ResNet-S holds a significantly smaller model size compared to ResNet.

[0033] In some embodiment, the transformer layer may only utilize an encoder, the encoder may be operable to perform an encoding task, for example a task of heatmap prediction 212, to identify information relating to the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120.

[0034] Also referring to FIG. 2, the transformer encoder layer 202 may further comprise an attention layer 206 and a position-wise feed-forward network (FFN) 208. In some embodiment, the attention layer 206 may be a multi-head attention mechanism, to enable mixing of information and / or compute multiple information in parallel. The multi-head attention score may be computed using an attention score matrix A e R NxN, which may be computed by: A = softmax QKT\ . / where the input sequence X e RLXd projects into sequence Q e RLXd, keys K e rLX d

[0035] The base model 110 may further include a prediction head 210 or multiple prediction head 210, to localize or determine coordinates of a pose of the at least one vehicle occupant 132, 132’ in the at least one video image frame 120 captured by the imaging device 102. The Position-Wise FFN is a feedforward layer for determining position of a sequence, by analysing the at least one video image frame 120 captured.

[0036] In parallel to the base model 110, the visibility predictor 114 is operable to execute a visibility prediction algorithm 116 with the pose estimation algorithm 112. The visibility prediction algorithm 116 operable to determine a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame (120) captured by the imaging device 102. In other words, the visibility predictor 114 receives the at least one video image frame 120 transmitted to the visibility predictor 114 from the imaging device 102. The visibility predictor 114 includes at least a first convolution layer 220 and may include a second convolution layer 222. It shall be understood by a skilled practitioner the convolution layers execute basic computation of input data and may apply certain filters to yield output data for further processing in subsequent layers of the artificial neural network.

[0037] The visibility predictor 114 may further include a pooling layer 224 to reduce parameters of the input image data. A suitable pooling layer may be a max pooling layer, for example a filter to select pixels with maximum value include in the output array. The visibility predictor 114 may further include at least a first fully connected (FC) layer 226 and may further include a second FC layer 228. FC layers perform the task of classification in response to feature extraction. In a next predicted visibility layer 230 of the visibility predictor 114, an output correlating to a predicted visibility of at least one keypoint relative to the pose of the at least one vehicle occupant 132, 132’ contained in the at least one video image frame 120 captured is yield, where a visibility state of the at least one keypoint is represented by at least one integer label. By way of an example, the at least one integer label may include {0,1,2} where ‘0’ indicate the keypoint that is not assigned a label, T indicates the keypoint is labelled as not visible or occulated, and ‘2’ indicates the keypoint is visible. In this manner, a visibility state of each of the at least one keypoint is represented by at least one integer label assigned by the visibility predictor 114 such that each of the at least keypoint is represented by a type of classification corresponding to an integer label representing not visible, visible, at least partially occulated or occulated. The output of the visibility predictor 114 may be an output array or an output vector representing the visibility prediction of keypoints determined in each of the at least one video image frame 120. In some embodiment, the output vector of the visibility predictor 114 may contain a maximum number of keypoints allowable by the human pose system, for example, the maximum number of keypoints allowable by the human pose system may be a 1 X 17 output vector which represents seventeen (17) keypoints in each of the at least one video image frame 120.

[0038] In some embodiment, both branches of the neural network, namely the base model 110 and the visibility predictor 114 generates a loss function. The base model 110 generates a loss of function output of the base model 110 and the visibility predictor 114 generates a loss of function output of the visibility predictor 114. In some embodiment, the value of total loss function of the human pose estimation system 100 combined to yield a value of total loss function and input to the artificial neural network 106 for training. In some embodiment, the value of the total loss function may be generated by a mean square error of the loss of function output of the base model 110 and the loss of function output of the visibility predictor 114.

[0039] It shall be understood by a skilled practitioner the use of mean square error method to the loss function of the base model 110 and the visibility predictor 114 is not limited thereto. Other suitable methods operable to yield a difference between the loss function of a base model a visibility predictor may fulfil the same objective of finding a total loss function of a human pose system, with different advantages.

[0040] In other words, designing a technique to yield a total loss function of a base model 110 and a visibility predictor 114 such as the ones disclosed herein is to calculate a difference between module output and the ground truth annotated in the dataset input to the HPE network. Henceforth, the overall loss of the human pose model or the human pose system 100 may be represented by the following relation: L = (1—oc). Lt+ k . Lv where the parameters: Lt is a loss of function of the base model 110; Lv is a loss function of the visibility predictor 114 oc is used to balance a ratio between the loss functions associated with the tasks, i.e. Lt and Lv. The oc parameter may regulate the relative influence of each task on the training of the feature extraction weights.

[0041] As can be observed from FIG. 2 and the description above, a relatively simple artificial neural network objectively minimizes the number of parameters, memory usage and computation.

[0042] FIG. 3 shows a flowchart illustrating a method 300 of estimating a human pose in accordance with an embodiment. The method 300 may include at step 302, receiving at least one video image frame containing image information of at least one vehicle occupant captured within a field of view of a cabin of the motor vehicle. The at least one video image frame containing image information of at least one vehicle occupant captured within a field of view of a cabin of the motor vehicle may be captured by an independent imaging device, an imaging device of an in-vehicle monitoring system such as a driver monitoring system, a cabin monitoring system or the like.

[0043] At step 304, the method 300 may include retrieving and executing 304, by way of a processor comprising an artificial neural network and a memory, a set of instructions stored thereon for determining a pose of the at least one vehicle occupant in response to the least one video image frame captured.

[0044] The set of instruction retrievable and executable by the processor may further comprises at step 306, executing by way of a base model, a pose estimation algorithm for determining a pose of the at least one vehicle occupant contained in the at least one video image frame captured.

[0045] In parallel to step 306, the set of instruction retrievable and executable by the processor may further include step 308, executing by way of a visibility predictor, a visibility prediction algorithm for determining a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured.

[0046] In some embodiment, the method 300 may further include labelling at least one integer label for assigning a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured determined by the visibility prediction algorithm. The labelling of integer label may be executed by way of the visibility predictor. The at least one integer label assigned by the visibility predictor may be selected from a classification comprising not visible, visible, at least partially occulated or occulated.

[0047] In some embodiment, the method 300 may further include generating at least one output video image frame containing visibility state of a maximum number of keypoints allowable by the human pose estimation system in each of the at least one output video image frame. The generating at least one output video image frame containing visibility state of a maximum number of keypoints may be executed by way of the visibility predictor.

[0048] In some embodiment, the method 300 may further include generating a value of total loss function of the human pose estimation system and inputting the value of total loss function of the human pose estimation system generated to the artificial neural network, for training the artificial neural network. This step may be executed by way of the processor.

[0049] In some embodiment, the method 300 may further include generating the value of total loss function of the human pose estimation system. The generation of the value of total loss function may comprises calculating, a mean square error of a loss of function output of the base model and a loss of function output of the visibility predictor. The aforesaid step may be executed by way of the processor.

[0050] As explained above, the value of the total loss function may be use for training an artificial neural network for estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle.

[0051] Thus, it can be seen that a method and system of a human pose estimation for use in a motor vehicle having combined two neural networks executable in parallel has been provided. Visibility recognition is crucial in in-vehicle monitoring systems to ensure safety of vehicle occupants. Occlusion is a main challenge when implementing existing HPE systems within cabin of motor vehicles caused by high frequency within motor vehicles. Significant occlusions can mislead existing methods, causing them to inaccurately predict keypoints with high confidence. This can present potential issues in applications that require reliable monitoring, such as driver posture understanding in vehicle safety monitoring systems. Advantageously, the disclosure proposes executing a keypoint visibility predictor to existing base model to enhance the accuracy of human pose predictions. While exemplary embodiments have been presented in the foregoing detailed description of the disclosure, it should be appreciated that a vast number of variation exist.

[0052] The foregoing description shall be interpreted as illustrative and not be limited thereto. One of ordinary skill in the art would understand that certain modifications may come within the scope of this disclosure. Although the different non-limiting embodiments are illustrated as having specific components or steps, the embodiments of this disclosure are not limited to those combinations. Some of the components or features from any of the non-limiting embodiments may be used in combination with features or components from any of the other non-limiting embodiments. For these reasons, the appended claims should be studied to determine the true scope and content of this disclosure. 10 List of Reference Signs 100 System 102 Imaging device 104 Processor 106 Artificial neural network 108 Memory 110 Base model 112 Pose estimation algorithm 114 Visibility predictor 116 Visibility prediction algorithm 120 At least one video image frame 130 Motor vehicle 132 At least one vehicle occupant 140 Vehicle network 200 Schematic 202 Transformer encoder layer 204 Backbone 206 Attention layer 208 Position-wise feed-forward network 210 Head 212 Heatmap prediction 220 At least a first convolution layer 222 A second convolution layer 224 Pooling layer 226 At least a first fully-connected layer 228 A second fully-connected layer 230 Predicted visibility 232 loss of pose function 234 Loss of visibility function 236 Total loss function 300 Method 302 Receive at least one video image frame from imaging device 304 Retrieve and execute a set of instructions stored on processor 406 Execute pose estimation algorithm 308 Execute visibility prediction algorithm 310 Determine pose of vehicle occupant captured in video image frame

Claims

1. A human pose estimation system (100) for a motor vehicle (130), the system (100) comprising:an imaging device (102) having a field of view of a cabin of a motor vehicle (130), the imaging device (102) operable to capture at least one video image frame (120) containing image information of at least one vehicle occupant (132, 132’) within the field of view of a cabin of the motor vehicle (130);anda processor (104) comprising an artificial neural network (106) and a memory (108), the processor (104) operable to retrieve and execute a set of instructions stored thereon, and the processor (104) is operable to determine a pose of the at least one vehicle occupant (132, 132’) in response to the least one video image frame (120) captured,characterised in that the processor (104) comprises:a base model (110) operable to execute a pose estimation algorithm (112), the pose estimation algorithm (112) operable to determine a pose of the at least one vehicle occupant (132, 132’) contained in the at least one video image frame (120) captured;anda visibility predictor (114) operable to execute a visibility prediction algorithm (116) in parallel with the pose estimation algorithm (112), the visibility prediction algorithm (116) operable to determine a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant (132, 132’) contained in the at least one video image frame (120) captured.

2. The system (100) according to claim 1, characterised in that the base model (110) comprises a backbone (204), the backbone (204) is a ResNet-S.

3. The system (100) according to claim 1, characterised in that the base model (110) comprises a transformer encoder layer (202), the transformer encoderlayer (202) further comprises an encoder, the encoder operable to execute and perform an encoding task comprising a heatmap prediction (212).

4. The system (100) according to claim 3, characterised in that the transformer encoder layer (202) comprises a multi-head attention mechanism.

5. The system (100) according to claim 1, characterised in that the visibility state of at least one keypoint is represented by at least one integer label assigned by the visibility predictor (114), each of the at least one integer label selected from a classification comprising:• visible;• at least partially occulated; or• occulated6. The system (100) according to claims 1- 5, characterised in that the visibility predictor (114) is operable to generate at least one output video image frame (120) containing visibility state of a maximum number of keypoints allowable by the human pose estimation system in each of the at least one output video image frame (120).

7. The system (100) according to claim 1-6, characterised in thatthe processor (104) is operable to generate a value of total loss function of the human pose estimation system (100), the processor (104) is operable to input the total loss function generated to the artificial neural network (106) to train the artificial neural network (106).

8. The system (100) according to claim 7, characterised in that the value of total loss function of the human pose estimation system (100) is generated by a mean square error of• a loss of function output of the base model;and• a loss of function output of the visibility predictor (114).

9. A method (300) of estimating a human pose of at least one vehicle occupant (132, 132’) within a cabin of a motor vehicle (130), the method comprising:receiving (302), byway of an imaging device, at least one video image frame containing image information of at least one vehicle occupant captured within a field of view of a cabin of the motor vehicle;andretrieving and executing (304), by way of a processor comprising an artificial neural network and a memory, a set of instructions stored thereon for determining a pose of the at least one vehicle occupant in response to the least one video image frame captured,characterised by that the set of instructions further comprises:executing (306), by way of a base model, a pose estimation algorithm for determining a pose of the at least one vehicle occupant contained in the at least one video image frame captured;and in parallel with executing the pose estimation algorithm,executing (308), by way of a visibility predictor, a visibility prediction algorithm for determining a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured.

10. The method (300) according to claim 9, characterised by that the method (300) further comprises:labelling, by way of the visibility predictor, at least one integer label for assigning a visibility state of at least one keypoint relative to the pose of the at least one vehicle occupant contained in the at least one video image frame captured determined by the visibility prediction algorithm.

11. The method (300) according to claims 9-10, characterised by that the at least one integer label assigned by the visibility predictor is selected from a classification comprising:• visible;• at least partially occulated; or• occulated12. The method (300) according to claims 9 - 11, characterised by that themethod (300) further comprises:generating, by way of the visibility predictor, at least one output video image frame containing visibility state of a maximum number of keypoints allowable by the human pose estimation system in each of the at least one output video image frame.

13. The method according to claims 9 - 12, characterised by that the method (300) further comprises:generating, by way of the processor, a value of total loss function of the human pose estimation system;andinputting, by way of the processor, the value of total loss function of the human pose estimation system generated to the artificial neural network, for training the artificial neural network.

14. The method (300) according to claim 13, characterised by thatgenerating the value of total loss function of the human pose estimation system comprisescalculating, by way of the processor, a mean square error of• a loss of function output of the base model;and• a loss of function output of the visibility predictor.

15. A motor vehicle having a human pose estimation system and a processor adapted to execute the steps of claims 9-14.

16. A computer program product comprising instructions to cause a human pose estimation system (100) according to claims 1-8 to execute the steps of a method (300)of estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle defined in claims 9-14.

17. A non-transitory computer readable medium having stored thereon the computer program product of claim 16.

18. A method of training an artificial neural network for estimating a human pose of at least one vehicle occupant within a cabin of a motor vehicle, the method comprising:generating, by way of a processor comprising an artificial neural network and a memory, a value of total loss function of a human pose estimation system as defined according to claims 1 - 8;inputting, the value of total loss function of the human pose estimation system to the artificial neural network of the processorcharacterised by thatthe value of total loss function defined by a difference between a loss of function of a module output and a ground truth, the module output defined by a visibility predictor of the human pose estimation system and the ground truth defined by a base model of the human pose estimation system;wherein the difference between a loss of function of the module output and the ground truth is determined by a square root error.

Images