Human body posture estimation device and method based on images

Abstract

The present disclosure relates to an image-based body posture estimation device and method capable of removing joint information with low reliability. The human body posture estimation device comprises a memory for storing a mixed posture estimation model; and the processor is used for estimating the human body posture in the image based on the mixed posture estimation model. The processor can execute the following operations: acquiring image data, inputting the acquired image data into a pre-trained hybrid attitude estimation model to respectively extract coordinate values of key points corresponding to human joint positions and probability values of visibilities maps in the image data, and calculating the probability values of the visibilities maps according to the coordinate values of the key points and the probability values of the visibilities maps, wherein the probability values of the key points and the probability values of the visibilities maps correspond to human joint positions. And determining whether the probability value of the joint matched with the coordinate value of the key point is below a threshold value, and if the probability value of the joint matched with the coordinate value of the key point is below the threshold value, filtering and removing the key point of the corresponding joint.

Description

Technical Field

[0001] This disclosure relates to an image-based human pose estimation apparatus and method capable of removing unreliable joint information. Background Technology

[0002] Generally speaking, artificial intelligence (AI) is a field of computer engineering and information technology that studies methods that enable computers to think, learn, and improve themselves in a way that resembles human intelligence. It refers to the technology that enables computers to mimic human intelligent behavior.

[0003] Furthermore, artificial intelligence does not exist in isolation, but is directly or indirectly related to various fields of computer science. Especially in modern times, there is a very active effort across multiple areas of information technology to incorporate elements of artificial intelligence to solve problems in those fields.

[0004] Recently, human pose estimation technology has been applied to various fields.

[0005] As an example, human pose estimation technology can be used not only in various fields such as motion posture analysis, motion capture, posture correction, robot control, virtual reality, games, and film production, but also in safety and security-related fields to identify and warn of dangerous situations by detecting abnormal movements.

[0006] Based on their architecture, human pose estimation methods can be broadly categorized into heatmap-free regression methods and heatmap-based estimation methods.

[0007] However, when estimating the coordinates of key points corresponding to joint positions, the regression method without heatmaps may produce unexpected results due to key points that are not present in the scene of the image.

[0008] In addition, although the heatmap-based estimation method can use the predicted heatmap to remove non-existent key points, it introduces quantization errors when obtaining the coordinates of key points through the heatmap, and when key points are densely distributed in the image, it is difficult to distinguish the key points.

[0009] Therefore, there is a need to develop a human pose estimation device that can not only distinguish between densely configured multiple key points, but also improve the accuracy of human pose estimation by filtering out key points that do not exist in the image. Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] The purpose of this disclosure is to resolve the aforementioned problems and other issues.

[0012] The purpose of this disclosure is to provide an image-based human pose estimation device and method that filters out keypoints of joints with low reliability by comparing the coordinate values ​​of keypoints with the probability values ​​of a visibility map, thereby improving the accuracy and speed of human pose estimation.

[0013] Furthermore, the purpose of this disclosure is to provide an image-based human pose estimation device and method that, through learning to align the keypoint distribution map of visualized keypoints with the unrelated visibility map, can clearly distinguish a densely configured plurality of keypoints, thereby improving the reliability of human pose estimation.

[0014] Technical solutions to the problem

[0015] A human pose estimation device according to an embodiment of the present disclosure includes: a memory storing a hybrid pose estimation model; and a processor that estimates human pose in an image based on the hybrid pose estimation model. The processor can perform the following operations: acquiring image data, inputting the acquired image data into a pre-trained hybrid pose estimation model to extract the coordinate values ​​of keypoints and the probability values ​​of the visibility map corresponding to the joint positions of the human body from the image data, and confirming whether the probability value of the joint matching the coordinate value of the keypoint is below a threshold. If the probability value of the joint matching the coordinate value of the keypoint is below the threshold, the keypoint of the corresponding joint is filtered out.

[0016] One embodiment of the human pose estimation method disclosed herein includes: a step of acquiring image data; a step of inputting the acquired image data into a pre-trained hybrid pose estimation model to extract the coordinate values ​​of keypoints and the probability values ​​of the visibility map corresponding to the human joint positions in the image data; a step of confirming whether the probability value of the joint matching the coordinate values ​​of the keypoints is below a threshold; and a step of filtering out the keypoints of the corresponding joints if the probability value of the joint matching the coordinate values ​​of the keypoints is below the threshold.

[0017] Invention Effects

[0018] According to one embodiment of this disclosure, the human posture estimation device filters out key points of joints with low reliability by comparing the coordinate values ​​of keypoints with the probability values ​​of a visibility map, thereby improving the accuracy and speed of human posture estimation.

[0019] Furthermore, this disclosure improves the reliability of human pose estimation by learning to align the keypoint distribution map of visualized keypoints with the unrelated visibility map, thereby clearly distinguishing a large number of densely configured keypoints.

[0020] In addition, this disclosure can readjust the thresholds for each application mode to provide optimal human pose estimation information for the application mode. Attached Figure Description

[0021] Figure 1 An artificial intelligence device according to an embodiment of the present disclosure is shown.

[0022] Figure 2 An artificial intelligence server according to an embodiment of the present disclosure is shown.

[0023] Figure 3 An artificial intelligence system according to an embodiment of this disclosure is shown.

[0024] Figure 4 This is a diagram illustrating the operation of a human posture estimation device according to an embodiment of the present disclosure.

[0025] Figure 5 This is a diagram illustrating a hybrid pose estimation model for a human pose estimation device used to illustrate an embodiment of the present disclosure.

[0026] Figure 6 This is a diagram illustrating the key point filtering process of a human posture estimation device according to an embodiment of the present disclosure.

[0027] Figure 7 This is a diagram illustrating the learning process of a hybrid pose estimation model used to explain an embodiment of the human pose estimation apparatus of this disclosure.

[0028] Figure 8a , Figure 8b , Figure 9a , Figure 9b , Figure 10a and Figure 10b This is a diagram illustrating the key point filtering results of a human posture estimation device according to an embodiment of the present disclosure.

[0029] Figure 11 This is a table used to illustrate the performance of a human posture estimation device according to an embodiment of the present disclosure.

[0030] Figure 12a and Figure 12b This is a diagram of a driver monitoring system applicable to a human posture estimation device according to an embodiment of the present disclosure.

[0031] Figure 13a , Figure 13b , Figure 14a and Figure 14bThis is a graph illustrating the alignment learning results of the probability distributions of key points and visibility maps of a human pose estimation device according to an embodiment of the present disclosure.

[0032] Figure 15 and Figure 16 This is a diagram showing a visibility map corresponding to the image scale of a human pose estimation device according to an embodiment of the present disclosure.

[0033] Figure 17 and Figure 18 This is a diagram illustrating the posture estimation performance of a human posture estimation device according to an embodiment of the present disclosure for each joint.

[0034] Figure 19 This is a diagram illustrating the posture estimation method of a human posture estimation device according to an embodiment of the present disclosure. Detailed Implementation

[0035] The embodiments disclosed in this specification will now be described in detail with reference to the accompanying drawings. Regardless of the reference numerals, the same or similar constituent elements are given the same reference numerals, and repeated descriptions are omitted. The suffixes "module" and "part" used for constituent elements in the following description are assigned or used interchangeably for ease of writing only and do not inherently have a distinguishing meaning or function. Furthermore, when describing the embodiments disclosed in this specification, detailed descriptions of related well-known technologies will be omitted if it is determined that such detailed descriptions might unnecessarily obscure the spirit of the embodiments disclosed in this specification. Additionally, the drawings are only for ease of understanding of the embodiments disclosed in this specification; the technical ideas disclosed in this specification are not limited by the drawings and should be interpreted to include all modifications, equivalents, and substitutions included within the scope of the ideas and techniques of this disclosure.

[0036] Terms containing ordinal numbers, such as "first" and "second," can be used to describe various constituent elements, but these constituent elements should not be limited to these terms. The terms can only be used to distinguish one constituent element from another.

[0037] When referring to any constituent element as being "connected" or "linked" to another constituent element, it should be understood that it can be directly connected or linked to the other constituent element, or that another constituent element may be present in between. Conversely, when referring to any constituent element as being "directly connected" or "directly linked" to another constituent element, it should be understood that no other constituent element is present in between.

[0038] Artificial Intelligence (AI)

[0039] Artificial intelligence (AI) refers to the field of studying human intelligence or the methodologies for creating such intelligence. Machine learning (ML) refers to the field that defines the various problems studied within AI and the methodologies for solving these problems. Machine learning is also defined as algorithms that improve the performance of a given operation through diligent experience.

[0040] Artificial Neural Networks (ANNs), used in machine learning, refer to a model with problem-solving capabilities composed of a complex network of artificial neurons (nodes) connected by synapses. An ANN can be defined by the connection patterns between the complex neurons in different layers, the learning process of upgrading model parameters, and the activation function that generates the output values.

[0041] An artificial neural network can include an input layer, an output layer, and optionally one or more hidden layers. Each layer can include one or more neurons, and the artificial neural network can include synapses connecting neurons. In an artificial neural network, each neuron can output the input signal, weights, and biases of the activation function, which are input through the synapse.

[0042] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, are parameters that need to be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0043] The learning objective of artificial neural networks can be viewed as determining the model parameters that minimize the loss function. The loss function can be used as an indicator to determine the optimal model parameters during the learning process of artificial neural networks.

[0044] Based on the learning method, machine learning can be divided into supervised learning, unsupervised learning, and reinforcement learning.

[0045] Supervised learning refers to the method of enabling an artificial neural network to learn under conditions where labeled learning data is provided. The label can refer to the correct answer (or result value) derived by the artificial neural network after the learning data is input. Unsupervised learning refers to the method of enabling an artificial neural network to learn under conditions where no labeled learning data is provided. Reinforcement learning refers to the learning method of enabling an agent defined in a certain environment to learn to select actions or sequences of actions that maximize cumulative reward in each state.

[0046] In artificial neural networks, machine learning implemented using deep neural networks (DNNs) with multiple hidden layers is also called deep learning, which is a part of machine learning. The machine learning discussed below includes deep learning.

[0047] <Robot>

[0048] A robot can refer to a machine that automatically handles given tasks or actions using its own capabilities. In particular, a robot that has the ability to recognize and independently assess its environment to perform actions can be called an intelligent robot.

[0049] Based on their intended use and application, robots can be categorized into industrial, medical, household, and military applications.

[0050] Robots have a drive unit that includes actuators or motors, enabling them to perform various physical actions such as moving the robot's joints. Furthermore, mobile robots have drive units that include wheels, brakes, propellers, etc., allowing them to travel on the ground or fly in the air.

[0051] Self-Driving

[0052] Autonomous driving refers to the technology of driving automatically. An autonomous vehicle is a vehicle that can drive without the need for user operation or with minimal user operation.

[0053] For example, autonomous driving can include: technologies that keep the vehicle in its lane, technologies that automatically adjust speed like adaptive cruise control, technologies that automatically drive along a predetermined path, and technologies that automatically set a route and drive if a destination is set.

[0054] Vehicles can include: vehicles with only internal combustion engines, hybrid vehicles with both internal combustion engines and electric motors, and electric vehicles with only electric motors. This includes not only cars, but also trains, motorcycles, etc.

[0055] At this point, autonomous vehicles can be viewed as robots with autonomous driving capabilities.

[0056] Extended Reality (XR)

[0057] Extended Reality is a collective term for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR technology uses computer graphics (CG) images to provide real-world objects or backgrounds. AR technology uses CG images created virtually on top of images of actual objects. MR technology combines multiple virtual objects with the real world to provide a more immersive experience.

[0058] MR technology is similar to AR technology in that it presents real-world and virtual objects together. However, in AR technology, virtual objects are used to complement real-world objects, while in MR technology, virtual and real-world objects are used in equal form, which is a key difference.

[0059] XR technology can be applied to head-mounted displays (HMDs), head-up displays (HUDs), mobile phones, tablets, laptops, desktop computers, TVs, digital signage, etc., and devices that use XR technology can be called XR devices (XRDevice).

[0060] Figure 1 An AI device 100 according to an embodiment of the present disclosure is shown.

[0061] AI device 100 can be a fixed or mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, personal digital assistant (PDA), portable multimedia player (PMP), navigator, tablet computer, wearable device, set-top box (STB), digital multimedia broadcast (DMB) receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0062] Reference Figure 1 The AI ​​device 100 may include a communication unit 110, an input unit 120, a learning processor 130, a sensing unit 140, an output unit 150, a memory 170, a processor 180, etc.

[0063] The communication unit 110 can use wired or wireless communication technologies to send and receive data with other external devices such as AI devices 100a-100e and AI server 200. For example, the communication unit 110 can send and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0064] At that time, the communication technologies used by the 110th Research Institute of the Ministry of Communications included: Global System for Mobile Communication (GSM), Code Division MultiAccess (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN, Wi-Fi, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, and Near Field Communication (NFC).

[0065] The input unit 120 can acquire various types of data.

[0066] At this time, the input unit 120 may include: a camera for image signal input, a microphone for receiving audio signals, a user input unit for receiving information input from the user, etc. Here, the camera or microphone can be used as a sensor, and the signal obtained from the camera or microphone is called sensing data or sensor information.

[0067] When the input unit 120 obtains output using learning data and a learning model for model learning, it can acquire the input data to be used. The input unit 120 can also acquire unprocessed input data. In this case, the processor 180 or the learning processor 130 preprocesses the input data to extract input features.

[0068] The learning processor 130 can use learning data to enable a model composed of an artificial neural network to learn. Here, the learned artificial neural network can be referred to as the learning model. The learning model can be used to infer result values ​​from new input data that is not the learning data, and the inferred values ​​can be used as the basis for judgments to perform a certain action.

[0069] At this time, the learning processor 130 can interact with Figure 2 The AI ​​server 200 and the learning processor 240 work together to perform AI processing.

[0070] In this case, the learning processor 130 may include memory integrated or built into the AI ​​device 100. Alternatively, the learning processor 130 may also be implemented using memory 170, external memory directly integrated with the AI ​​device 100, or memory stored in an external device.

[0071] The sensing unit 140 can use various sensors to acquire at least one of the following: internal information of the AI ​​device 100, surrounding environmental information of the AI ​​device 100, and user information.

[0072] At this time, the sensors included in the sensing unit 140 include: proximity sensor, illuminance sensor, accelerometer, gyroscope sensor, inertial sensor, red-green-blue (RGB) sensor, infrared (IR) sensor, fingerprint recognition sensor, ultrasonic sensor, light sensor, microphone, lidar, radar, etc.

[0073] The output unit 150 can generate outputs related to vision, hearing, touch, etc.

[0074] At this time, the output unit 150 may include: a display unit that outputs visual information, a speaker that outputs auditory information, a haptic feedback module that outputs tactile information, etc.

[0075] The memory 170 can store data that supports various functions of the AI ​​device 100. For example, the memory 170 can store input data, learning data, learning models, learning history, etc., obtained from the input unit 120.

[0076] The processor 180 can determine at least one executable action of the AI ​​device 100 based on information determined or generated using data analysis algorithms or machine learning algorithms. Furthermore, the processor 180 can execute the determined action by controlling the constituent elements of the AI ​​device 100.

[0077] To this end, processor 180 may request, retrieve, receive, or apply data from learning processor 130 or memory 170, and may control the components of AI device 100 to perform the predicted action or the action determined to be preferred in the at least one executable action.

[0078] In this case, if it is necessary to link an external device in order to perform a certain action, the processor 180 can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0079] The processor 180 can obtain intent information from user input and determine the user's needs based on the obtained intent information.

[0080] At this time, the processor 180 may utilize at least one of a speech-to-text (STT) engine for converting speech input into a string and a natural language processing (NLP) engine for obtaining intent information corresponding to the user input.

[0081] At this point, at least one of the STT engine and the NLP engine can be composed of at least a portion of an artificial neural network learned according to a machine learning algorithm. Alternatively, at least one of the STT engine and the NLP engine can be an engine learned using the learning processor 130, or an engine learned using the learning processor 240 of the AI ​​server 200, or an engine learned using their distributed processing.

[0082] The processor 180 can collect historical information, including the actions of the AI ​​device 100 or user feedback on the actions, and store it in the memory 170 or the learning processor 130, or transmit it to an external device such as the AI ​​server 200. The collected historical information can be used to upgrade the learning model.

[0083] To drive the application stored in memory 170, processor 180 can control at least a portion of the components of AI device 100. Furthermore, to drive the application, processor 180 can cause two or more components included in the components of AI device 100 to operate in combination.

[0084] Figure 2 An AI server 200 according to an embodiment of the present disclosure is shown.

[0085] Reference Figure 2 AI server 200 can refer to a device that uses machine learning algorithms to enable an artificial neural network to learn or utilize a learned artificial neural network. Here, AI server 200 can be composed of a plurality of servers to perform distributed processing, or it can be defined as a 5G network. In this case, AI server 200 can be included as part of AI device 100, thereby jointly performing at least a portion of the AI ​​processing.

[0086] AI server 200 may include communication unit 210, memory 230, learning processor 240, processor 260, etc.

[0087] The communication unit 210 can send and receive data with external devices such as the AI ​​device 100.

[0088] The memory 230 may include a model storage unit 231. The model storage unit 231 may store a learning or learned model 231a (or artificial neural network) through the learning processor 240.

[0089] The learning processor 240 can use the learning data to enable the artificial neural network 231a to learn. The learning model can be used in the form of an AI server 200 mounted on the artificial neural network, or it can be used on an external device such as an AI device 100.

[0090] The learning model can be implemented by hardware, software, or a combination of hardware and software. When the learning model is partially or entirely implemented by software, one or more instructions constituting the learning model can be stored in memory 230.

[0091] The processor 260 can use the learning model to infer result values ​​from new input data and generate response or control instructions based on the inferred result values.

[0092] Figure 3 An AI system 1 according to an embodiment of the present disclosure is shown.

[0093] Reference Figure 3 In AI system 1, at least one of the following—AI server 200, robot 100a, autonomous vehicle 100b, XR device 100c, smartphone 100d, and home appliance 100e—is connected to cloud network 10. Here, the robot 100a, autonomous vehicle 100b, XR device 100c, smartphone 100d, and home appliance 100e, which utilize AI technology, can be referred to as AI devices 100a to 100e.

[0094] Cloud network 10 can refer to a network that forms part of or exists within cloud computing infrastructure. Here, cloud network 10 can be constructed using 3G networks, 4G or Long Term Evolution (LTE) networks, 5G networks, etc.

[0095] That is, the devices 100a-100e and 200 constituting the AI ​​system 1 can be interconnected through the cloud network 10. In particular, although the devices 100a-100e and 200 can also communicate with each other through a base station, they can also communicate with each other directly without a base station.

[0096] AI server 200 may include servers that perform AI processing and servers that perform big data-related calculations.

[0097] AI server 200 can connect via cloud network 10 to at least one of the following AI devices constituting AI system 1: robot 100a, autonomous vehicle 100b, XR device 100c, smartphone 100d, and home appliance 100e, and can assist at least a portion of the AI ​​processing of the connected plurality of AI devices 100a to 100e.

[0098] At this time, AI server 200 can replace AI devices 100a-100e to enable artificial neural networks to learn according to machine learning algorithms, and the learning model can be directly stored or transmitted to AI devices 100a-100e.

[0099] At this time, the AI ​​server 200 can receive input data from the AI ​​devices 100a to 100e, use the learning model to infer the result value from the received input data, generate a response or control command based on the inferred result value, and transmit it to the AI ​​devices 100a to 100e.

[0100] Alternatively, AI devices 100a-100e can also directly use the learning model to infer the result value from the input data and generate a response or control command based on the inferred result value.

[0101] Hereinafter, various embodiments of the AI ​​devices 100a to 100e to which the above technology is applied will be described. Figure 3 The AI ​​devices 100a-100e shown can be regarded as Figure 1 A specific embodiment of the AI ​​device 100 shown.

[0102] <AI + Robots>

[0103] Robot 100a, through the application of AI technology, can be transformed into a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, and an unmanned flying robot.

[0104] Robot 100a may include a robot control module for controlling actions. The robot control module may refer to a software module or a chip that implements the software module in hardware.

[0105] Robot 100a can use sensor information obtained from various sensors to obtain the status information of robot 100a, or sense (identify) the surrounding environment and objects, or generate map data, or determine the movement path and driving plan, or determine the response to user interaction, or determine the action.

[0106] In order to determine the movement path and driving plan, robot 100a can utilize sensor information obtained from at least one of the following sensors: lidar, radar, and camera.

[0107] Robot 100a can perform the aforementioned actions using a learning model composed of at least one artificial neural network. For example, robot 100a can use the learning model to identify the surrounding environment and objects, and use the identified surrounding environment information or object information to determine the action. Here, the learning model can be a model learned directly in robot 100a or learned in an external device such as AI server 200.

[0108] At this point, although robot 100a can directly use the results generated by the learning model to perform actions, it can also transmit sensor information to external devices such as AI server 200 to receive the results generated therefrom and perform actions.

[0109] Robot 100a can determine a movement path and driving plan using at least one of map data, object information sensed by sensors, and object information obtained from external devices, and control the drive unit to make robot 100a move according to the determined movement path and driving plan.

[0110] The map data can contain object recognition information for various objects configured within the space where the robot 100a moves. For example, the map data can contain object recognition information for fixed objects such as walls and doors, and movable objects such as flower pots and tables. Additionally, the object recognition information can include name, type, distance, and location.

[0111] Furthermore, the robot 100a is driven by a user-controlled / interactive drive unit, enabling it to perform actions or move. In this case, the robot 100a can acquire interactive intent information based on the user's actions or voice utterances, determine the response based on the acquired intent information, and execute the action.

[0112] <AI + Autonomous Driving>

[0113] Autonomous vehicles 100b can be transformed into mobile robots, vehicles, unmanned aircraft, etc. by applying AI technology.

[0114] The autonomous vehicle 100b may include an autonomous driving control module for controlling autonomous driving functions. The autonomous driving control module may refer to a software module or a chip that implements the software module in hardware. The autonomous driving control module may be included inside the autonomous vehicle 100b as part of its structure, or it may be external to the autonomous vehicle 100b and be constructed by additional hardware and connected to the autonomous vehicle 100b.

[0115] The autonomous vehicle 100b can use sensor information obtained from various sensors to obtain state information of the autonomous vehicle 100b, or sense (identify) the surrounding environment and objects, or generate map data, or determine the movement path and driving plan, or determine the action.

[0116] Here, in order to determine the movement path and driving plan, similar to robot 100a, autonomous vehicle 100b can utilize sensor information obtained from at least one of the following sensors: lidar, radar, and camera.

[0117] In particular, the autonomous vehicle 100b can receive sensor information from external devices to identify the environment or objects in areas where the field of vision is obstructed or above a constant distance, or it can receive information from external devices for direct identification.

[0118] The autonomous vehicle 100b can perform the aforementioned actions using a learning model composed of at least one artificial neural network. For example, the autonomous vehicle 100b can use the learning model to identify the surrounding environment and objects, and determine a driving route using the identified surrounding environment information or object information. Here, the learning model can be a model learned directly in the autonomous vehicle 100b or learned in an external device such as an AI server 200.

[0119] At this point, while the autonomous vehicle 100b can directly use the results generated by the learning model to perform actions, it can also transmit sensor information to external devices such as the AI ​​server 200 to receive the results generated therefrom and perform actions accordingly.

[0120] The autonomous vehicle 100b can determine a movement path and driving plan using at least one of map data, object information sensed from sensor information, and object information obtained from external devices, and control the drive unit to make the autonomous vehicle 100b drive along the determined movement path and driving plan.

[0121] Map data can contain object identification information for various objects positioned within the space (e.g., roads) where the autonomous vehicle 100b is traveling. For example, map data can contain object identification information for fixed objects such as streetlights, rocks, and buildings, and movable objects such as vehicles and pedestrians. Additionally, object identification information can include name, type, distance, and location.

[0122] Furthermore, the autonomous vehicle 100b is able to perform actions or drive by controlling the drive unit based on the user's control / interaction. At this time, the autonomous vehicle 100b can acquire interaction intent information based on the user's actions or voice, determine the response based on the acquired intent information, and perform the action.

[0123] <AI+XR>

[0124] The XR device 100c, through the application of AI technology, can be used as a head-mounted display (HMD), a head-up display (HUD) installed in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, a digital signage device, a vehicle, a stationary robot, a mobile robot, etc.

[0125] XR device 100c analyzes 3D point cloud data or image data acquired through various sensors or external devices to generate positional and attribute data for 3D points. It can acquire information about the surrounding space or real-world objects and render and output the XR object to be output. For example, XR device 100c can associate an XR object containing additional information about the identified object with the identified object and output it.

[0126] The XR device 100c can perform the aforementioned actions using a learning model composed of at least one artificial neural network. For example, the XR device 100c can use the learning model to identify real-world objects in 3D point cloud data or image data and provide information corresponding to the identified real-world objects. Here, the learning model can be a model learned directly in the XR device 100c or a model learned in an external device such as the AI ​​server 200.

[0127] At this time, although the XR device 100c can directly use the learning model to generate results to perform actions, it can also transmit sensor information to external devices such as the AI ​​server 200 to receive the results generated therefrom and perform actions.

[0128] <AI + Robotics + Autonomous Driving>

[0129] Robot 100a, through the application of AI and autonomous driving technologies, can be transformed into a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, and an unmanned flying robot.

[0130] The robot 100a that utilizes AI and autonomous driving technologies can refer to the robot itself with autonomous driving capabilities, or the robot 100a that interacts with the autonomous vehicle 100b, etc.

[0131] Robots 100a with autonomous driving capabilities can be collectively referred to as devices that can move independently along a predetermined route or determine their own route without user control.

[0132] To determine one or more of the movement path and driving plan, the robot 100a and the autonomous vehicle 100b with autonomous driving capabilities can use common sensing methods. For example, the robot 100a and the autonomous vehicle 100b with autonomous driving capabilities can use information sensed by lidar, radar, and cameras to determine one or more of the movement path and driving plan.

[0133] The robot 100a, which interacts with the autonomous vehicle 100b, exists separately from the autonomous vehicle 100b and can perform actions related to autonomous driving functions inside the autonomous vehicle 100b or actions related to users riding in the autonomous vehicle 100b.

[0134] At this time, the robot 100a, which interacts with the autonomous vehicle 100b, replaces the autonomous vehicle 100b in acquiring sensor information and providing it to the autonomous vehicle 100b, or acquires sensor information and generates surrounding environment information or object information to provide to the autonomous vehicle 100b, thereby being able to control or assist the autonomous driving function of the autonomous vehicle 100b.

[0135] Alternatively, a robot 100a interacting with the autonomous vehicle 100b can control the functions of the autonomous vehicle 100b by monitoring the user riding in the autonomous vehicle 100b or by interacting with the user. For example, if the robot 100a determines that the driver is drowsy, it can activate the autonomous driving function of the autonomous vehicle 100b or assist in controlling the drive unit of the autonomous vehicle 100b. Here, the functions of the autonomous vehicle 100b controlled by the robot 100a may not only include the autonomous driving function alone, but may also include functions provided by a navigation system or audio system installed inside the autonomous vehicle 100b.

[0136] Alternatively, the robot 100a interacting with the autonomous vehicle 100b can provide information or auxiliary functions to the autonomous vehicle 100b outside the autonomous vehicle 100b. For example, the robot 100a can provide traffic information such as signal information included in intelligent traffic lights to the autonomous vehicle 100b, and can also interact with the autonomous vehicle 100b to automatically connect a charging pile to a charging port, similar to an automatic charging pile for an electric vehicle.

[0137] <AI + Robot + XR>

[0138] By applying AI technology and XR technology, the robot 100a can be implemented as a guiding robot, a handling robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, a drone, etc.

[0139] The robot 100a applied with XR technology can refer to a robot that becomes a control / interaction object in an XR image. In this case, the robot 100a can be distinguished from the XR device 100c and interact with each other.

[0140] If the robot 100a that becomes a control / interaction object in an XR image obtains sensor information from a sensor including a camera, the robot 100a or the XR device 100c generates an XR image based on the sensor information, and the XR device 100c can output the generated XR image. In addition, such a robot 100a can act based on a control signal input through the XR device 100c or the interaction of a user.

[0141] For example, a user can confirm an XR image corresponding to the perspective of the remotely linked robot 100a through an external device such as the XR device 100c, and adjust the autonomous driving path of the robot 100a through interaction, or control the action or driving, or confirm information of surrounding objects.

[0142] <AI + Autonomous Driving + XR>

[0143] By applying AI technology and XR technology, the autonomous vehicle 100b can be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc.

[0144] The autonomous vehicle 100b applied with XR technology can refer to an autonomous vehicle having a means for providing an XR image, or an autonomous vehicle that becomes a control / interaction object in an XR image, etc. In particular, the autonomous vehicle 100b that becomes a control / interaction object in an XR image can be distinguished from the XR device 100c and interact with each other.

[0145] An autonomous vehicle 100b equipped with means for providing XR images can acquire sensor information from sensors including cameras and output generated XR images based on the acquired sensor information. For example, the autonomous vehicle 100b can output XR images by setting up a HUD, thereby providing occupants with XR objects associated with real-world objects or objects in a scene.

[0146] In this case, when an XR object is output to a HUD, at least a portion of the output XR object may overlap with the actual object being viewed by the occupant. Conversely, when an XR object is output to a display installed inside the autonomous vehicle 100b, at least a portion of the output XR object may overlap with objects in the image. For example, the autonomous vehicle 100b may output XR objects associated with lanes, other vehicles, traffic lights, traffic signs, two-wheeled vehicles, pedestrians, buildings, and other objects.

[0147] If the autonomous vehicle 100b, which becomes the control / interaction object in the XR image, acquires sensor information from sensors including cameras, the autonomous vehicle 100b or the XR device 100c can generate an XR image based on the sensor information, and the XR device 100c can output the generated XR image. Furthermore, such an autonomous vehicle 100b can operate based on control signals input from external devices such as the XR device 100c or user interaction.

[0148] Figure 4 This is a diagram illustrating the operation of a human posture estimation device according to an embodiment of the present disclosure.

[0149] like Figure 4 As shown, the human pose estimation device 100 of this disclosure may include: a memory 170 storing a hybrid pose estimation model 300; and a processor 180 that estimates human pose in an image based on the hybrid pose estimation model 300.

[0150] Here, the processor 180 may perform the following operations: acquire image data, input the acquired image data into the pre-trained hybrid pose estimation model 300 to extract the coordinate values ​​of keypoints and the probability values ​​of the visibility map corresponding to the human joint positions in the image data, and confirm whether the probability value of the joint matching the coordinate value of the keypoint is below a threshold. If the probability value of the joint matching the coordinate value of the keypoint is below the threshold, the keypoint of the corresponding joint is filtered out.

[0151] Here, the processor 180 can extract feature information from the image data through the hybrid estimation model 300, and based on the feature information of the image data, simultaneously extract the coordinate values ​​of key points corresponding to human joint positions in the image data and the probability distribution values ​​of the visibility map corresponding to human joint positions in the image data.

[0152] As an example, the hybrid pose estimation model 300 may include: at least one first neural network model that extracts feature information from image data; a second neural network model that extracts the coordinate values ​​of key points in the image data corresponding to the joint positions of the human body based on the feature information from the image data; and a third neural network model that extracts the probability distribution values ​​of the visibility map in the image data corresponding to the joint positions of the human body based on the feature information from the image data.

[0153] Here, the first neural network model can be configured in series with the second and third neural network models respectively, and the feature information of the extracted image data can be output to the second and third neural network models simultaneously.

[0154] In addition, the second and third neural network models can be configured in parallel to each other and simultaneously estimate the position of human joints in the image data based on the feature information of the image data.

[0155] Next, the third neural network model can be pre-trained to generate a keypoint distribution map based on the coordinate values ​​of key points corresponding to the joint positions in the image data, and compare the joint probabilities of the uncorrelated visibility map with the joint probabilities of the keypoint distribution map, so that the joint probabilities of the uncorrelated visibility map are aligned with the joint probabilities of the keypoint distribution map.

[0156] Here, the third neural network model can generate multiple keypoint distribution maps and multiple irrelevant visibility maps for each joint of the human body in the image data, and extract the keypoint distribution map and the irrelevant visibility map corresponding to the same joint to perform learning to align them. If the learning to align the keypoint distribution map with the irrelevant visibility map is completed for all joints of the human body in the image data, the learning can end.

[0157] In addition, the number of the first network models can be increased or decreased proportionally to the size of the input image data and the amount of computation.

[0158] Secondly, the hybrid pose estimation model 300 can extract the coordinate values ​​of key points corresponding to human joint positions in image data through pre-training, generate a keypoint distribution map based on the extracted keypoint coordinate values, and generate an uncorrelated visibility map corresponding to human joint positions in image data. By comparing the correlation between the joint probabilities of the uncorrelated visibility map and the joint probabilities of the keypoint distribution map, the joint probabilities of the uncorrelated visibility map are aligned with the joint probabilities of the keypoint distribution map.

[0159] Here, the hybrid pose estimation model 300 can generate multiple keypoint distribution maps and multiple irrelevant visibility maps for each joint of the human body in the image data, and extract the keypoint distribution map and the irrelevant visibility map corresponding to the same joint to perform learning to align them. If the learning to align the keypoint distribution map with the irrelevant visibility map has been performed on all joints of the human body in the image data, the learning can end.

[0160] As an example, the hybrid pose estimation model 300 can perform learning based on the following mathematical formula 1 when generating the keypoint distribution map.

[0161] [Mathematical Expression 1]

[0162]

[0163] Here, d u，v，a This is a key point distribution map, Ψ is the transformation function, and c i，j，a σ is the coordinates of the key point, σ is the standard deviation, and ρ is the probability of the object's existence.

[0164] In addition, the hybrid pose estimation model 300 can perform learning based on the set of keypoint distribution maps generated by the following mathematical formula 2.

[0165] [Mathematical Expression 2]

[0166]

[0167] Here, in order to filter out low activation of keypoints within the same category, max performs the action in pixels.

[0168] In addition, the hybrid pose estimation model 300 can perform learning by applying the self-correlation loss to the visibility map and the keypoint distribution map based on the following mathematical formula 3.

[0169] [Mathematical Expression 3]

[0170]

[0171] Here, It is the arithmetic mean.

[0172] In addition, when extracting key point coordinates, the processor 180 can estimate key points in the image data corresponding to the joint positions of the human body in batches based on the feature information of the image data, and extract the coordinates of the estimated key points.

[0173] Here, if the processor 180 receives user input to select a specific application mode, it obtains the necessary joint information required for the specific application mode, and based on the necessary joint information, it can selectively extract key points corresponding to the necessary joint positions from all joint positions present in the image data.

[0174] As an example, such as surveillance cameras or driver posture monitoring modules, specific purpose application modes can include all application modes that can be applied to various purposes.

[0175] In addition to human joints, if the processor 180 receives user input selecting a specific part of the human body, it can estimate in batches the key points in the image data that correspond to the positions of human joints and the key points that correspond to the positions of specific parts of the human body.

[0176] Secondly, when extracting the probability values ​​of the visibility map, the processor 180 can generate a visibility map based on the feature information of the image data for each joint position of the human body in the image data, and extract the probability values ​​corresponding to the joint positions of the human body in the visibility map.

[0177] Here, when generating a visibility map, if the processor 180 extracts the coordinate values ​​of key points of a specific joint in the human body, it can generate a visibility map corresponding to the same joint whose key point coordinate values ​​have been extracted.

[0178] In addition, processor 180 can generate the same number of visibility maps as the number of joints from which keypoint coordinates have been extracted.

[0179] Here, the processor 180 can increase or decrease the number of visibility maps proportionally to the number of joints from which keypoint coordinate values ​​are extracted.

[0180] Additionally, the processor 180 can generate visibility maps by using a hybrid pose estimation model that has been pre-trained to align the keypoint distribution maps corresponding to the same joint with unrelated visibility maps when generating visibility maps.

[0181] In addition, when extracting probability values, the processor 180 can extract probability values ​​corresponding to the human joint positions for each visibility map, tabulate the probability values, and store the probability value table.

[0182] Here, the processor 180 can check whether the application mode information of the probability value table exists when storing the probability value table. If the application mode information exists, the probability value table is stored for each application mode.

[0183] In addition, when extracting the probability values ​​of the visibility map, the processor 180 can obtain the application mode information of the image data, and based on the application mode information, determine whether a probability value table with the same application mode is stored in the memory. If a probability value table with the same application mode exists, the processor extracts the probability values ​​of the joints that match the coordinate values ​​of the key points from the probability value table with the same application mode.

[0184] As an example, when generating a visibility graph, processor 180 can generate the visibility graph based on the following mathematical formula 4.

[0185] [Mathematical Expression 4]

[0186]

[0187] Here, It is located in the image (c′) xk , c′ yk The k-th key point of coordinates, v s It is a visibility graph that includes the k-th keypoint.

[0188] Secondly, if the processor 180 extracts the coordinate values ​​of the key points and the probability values ​​of the visibility map, it determines from the probability values ​​of the visibility map whether there is a probability value of a joint that matches the coordinate values ​​of the key points. If there is a probability value of a matching joint, it determines whether the probability value of the matching joint is below a preset threshold.

[0189] Here, if the probability value of the visibility map confirms that there is no probability value of a joint that matches the coordinate value of the keypoint, the processor 180 can maintain the keypoint of the joint corresponding to the keypoint coordinate value.

[0190] In addition, the processor 180 can acquire the shooting environment information of the image data, and if the shooting environment information of the image data is included in the threshold change conditions, the threshold is readjusted.

[0191] Depending on the situation, if the processor 180 receives user input selecting a specific application mode, it checks whether the threshold corresponding to the specific application mode has been pre-stored in memory. If the threshold corresponding to the specific application mode has been pre-stored in memory, it readjusts the currently set threshold to the threshold corresponding to the specific application mode.

[0192] Here, if the specific application mode is a mode aimed at human safety, such as a driver monitoring system, the processor 180 can be readjusted to a higher threshold than the currently set threshold.

[0193] In another scenario, if the processor 180 receives user input requesting a threshold readjustment, it can obtain the threshold to be readjusted from the user input and adjust the currently set threshold to the threshold to be readjusted.

[0194] Here, if the processor 180 receives user input requesting a readjustment of the threshold, it provides a threshold adjustment list window containing a plurality of threshold items. If it receives user input selecting a specified threshold item from the plurality of threshold adjustment list windows, it readjusts the currently set threshold to the threshold corresponding to the selected threshold item.

[0195] As described above, in this disclosure, the human pose estimation device filters out key points of joints with low reliability by comparing the coordinate values ​​of keypoints with the probability values ​​of the visibility map, thereby improving the accuracy and speed of human pose estimation.

[0196] Furthermore, this disclosure, through learning to align the keypoint distribution map of visualized keypoints with the unrelated visibility map, can clearly distinguish densely configured multiple keypoints, thereby improving the reliability of human pose estimation.

[0197] In addition, this disclosure is able to provide optimal human pose estimation information for each application mode by readjusting the thresholds of each application mode.

[0198] Figure 5 This is a diagram illustrating a hybrid pose estimation model for a human pose estimation device used to illustrate an embodiment of the present disclosure.

[0199] like Figure 5 As shown, this disclosure can extract feature information from image data through a hybrid estimation model, and simultaneously extract the coordinate values ​​of key points corresponding to human joint positions in the image data and the probability distribution values ​​of the visibility map corresponding to human joint positions in the image data based on the feature information of the image data.

[0200] The hybrid pose estimation model may include: at least one first neural network model 310, which extracts feature information from image data; a second neural network model 320, which extracts the coordinate values ​​of key points in the image data corresponding to the joint positions of the human body based on the feature information of the image data; and a third neural network model 330, which extracts the probability distribution values ​​of the visibility map in the image data corresponding to the joint positions of the human body based on the feature information of the image data.

[0201] Here, the first neural network model 310 can be connected in series with the second neural network model 320 and the third neural network model 330 respectively, so that the feature information of the extracted image data can be output to the second neural network model 320 and the third neural network model 330 simultaneously.

[0202] In addition, the second neural network model 320 and the third neural network model 330 can be configured in parallel to estimate the position of human joints in the image data based on the feature information of the image data.

[0203] Next, the third neural network model 330 can generate a keypoint distribution map based on the coordinate values ​​of key points corresponding to the joint positions of the human body in the image data after pre-training. By comparing the joint probabilities of the uncorrelated visibility map and the joint probabilities of the keypoint distribution map, the joint probabilities of the uncorrelated visibility map are aligned with the joint probabilities of the keypoint distribution map.

[0204] Here, the third neural network model 330 can generate a plurality of key point distribution maps and a plurality of irrelevant visibility maps for each joint of the human body in the image data, and extract the key point distribution map and the irrelevant visibility map corresponding to the same joint to perform learning to align them. If the learning to align the key point distribution map with the irrelevant visibility map is completed for all joints of the human body in the image data, the learning can end.

[0205] Depending on the circumstances, the number of the first network models 310 can be increased or decreased proportionally to the size of the input image data and the amount of computation.

[0206] Figure 6 This is a diagram illustrating the key point filtering process of a human posture estimation device according to an embodiment of the present disclosure.

[0207] like Figure 6 As shown, this disclosure can acquire image data 410.

[0208] In addition, the present disclosure allows the acquired image data 410 to be input into a pre-trained hybrid pose estimation model 300.

[0209] Next, the pre-trained hybrid pose estimation model 300 can analyze the features of the image data and extract compressed information.

[0210] Here, this disclosure allows the extraction of feature information from image data from a grid block 420 comprising a plurality of grid cells 430.

[0211] At this time, each grid cell 430 can contain object existence information p, frame information t, and key point information c.

[0212] Secondly, this disclosure can extract the coordinate values ​​of keypoints 440 and the probability values ​​of visibility maps 450 corresponding to human joint positions in image data through a pre-trained hybrid pose estimation model 300.

[0213] In addition, this disclosure can confirm whether the probability value of the joint matching the coordinate value of the key point is below a threshold. If the probability value of the joint matching the coordinate value of the key point is below the threshold, the key points of the corresponding joint are filtered out, thereby extracting pose estimation information containing highly reliable key points 470.

[0214] Here, this disclosure can extract feature information from image data using a hybrid estimation model 300, and based on the feature information of the image data, simultaneously extract the coordinate values ​​of key points 440 corresponding to human joint positions in the image data and the probability distribution values ​​of visibility maps 450 corresponding to human joint positions within the image data 410.

[0215] In addition, this disclosure can estimate key points 440 corresponding to human joint positions in image data 410 in batches based on the feature information of image data when extracting key point coordinate values, and extract the coordinate values ​​of the estimated key points.

[0216] Secondly, when extracting the probability value of the visibility map, this disclosure can generate a visibility map 450 based on the feature information of the image data 410 for each joint position of the human body in the image data, and extract the probability value 460 corresponding to the joint position of the human body in the visibility map 450.

[0217] Here, this disclosure allows for the generation of a visibility map 450 if the coordinate values ​​of a key point 440 of a specific joint in the human body are extracted when generating the visibility map 450. In this case, a visibility map 450 corresponding to the same joint as the specific joint whose key point coordinate values ​​have been extracted is generated.

[0218] That is, this disclosure can generate the same number of visibility maps 450 as the number of joints from which keypoint coordinate values ​​are extracted.

[0219] Here, the number of visibility maps 450 can be increased or decreased proportionally to the number of joints from which key point coordinate values ​​are extracted.

[0220] Secondly, if the processor 180 extracts the coordinate values ​​of the key points and the probability values ​​of the visibility map, it determines from the probability values ​​of the visibility map whether there is a probability value of a joint that matches the coordinate values ​​of the key points. If there is a probability value of a matching joint, it can determine whether the probability value of the matching joint is below a preset threshold.

[0221] Here, if this disclosure receives user input requesting a threshold readjustment, it can obtain the threshold to be readjusted from the user input and readjust the currently set threshold to the threshold to be readjusted.

[0222] Figure 7 This is a diagram illustrating the learning process of a hybrid pose estimation model used to explain an embodiment of the human pose estimation apparatus of this disclosure.

[0223] like Figure 7 As shown, this disclosure can extract the coordinate values ​​of key points corresponding to human joint positions in image data through pre-training, generate a keypoint distribution map 520 based on the extracted keypoint coordinate values, and generate an uncorrelated visibility map 510 corresponding to human joint positions in the image data. By comparing the correlation between the joint probabilities of the uncorrelated visibility map 510 and the joint probabilities of the keypoint distribution map 520, the joint probabilities of the uncorrelated visibility map 510 and the joint probabilities of the keypoint distribution map 520 are aligned.

[0224] Here, this disclosure can generate a plurality of keypoint distribution maps 520 and a plurality of irrelevant visibility maps 510 for each joint of the human body in the image data, extract the keypoint distribution map 520 and irrelevant visibility map 510 corresponding to the same joint and perform learning to align them. If the learning to align the keypoint distribution map 520 with the irrelevant visibility map 510 is completed for all joints of the human body in the image data, the pre-training can be terminated.

[0225] As an example, this disclosure can perform learning by generating a keypoint distribution map based on the following mathematical formula when generating the keypoint distribution map.

[0226]

[0227] Here, d u，v，a This is a key point distribution map, Ψ is the transformation function, and c i，j，a σ is the coordinates of the key point, σ is the standard deviation, and ρ is the probability of the object's existence.

[0228] In addition, this disclosure can perform learning by generating a set of complex keypoint distribution maps based on the following mathematical formula.

[0229]

[0230] Here, in order to filter out low activation of keypoints within the same category, max performs the action in pixels.

[0231] In addition, this disclosure can apply the self-correlation loss to the visibility map and keypoint distribution map to perform learning based on the following mathematical formula.

[0232]

[0233] Here, It is the arithmetic mean.

[0234] In addition, this disclosure can extract feature information of image data from a grid block 420 comprising a plurality of grid cells 430.

[0235] At this time, each grid cell 430 may contain object existence information p, frame information t, and key point information c.

[0236] Figure 8a , Figure 8b , Figure 9a , Figure 9b , Figure 10a and Figure 10b This is a diagram illustrating the key point filtering results of a human posture estimation device according to an embodiment of the present disclosure.

[0237] Figure 8a , Figure 9a and Figure 10a This is a graph showing the results of human pose estimation using existing methods. Figure 8b , Figure 9b and Figure 10b This is a diagram of the human pose estimation results using the method disclosed herein.

[0238] exist Figure 8a , Figure 9a and Figure 10a The existing methods of human pose estimation shown have problems such as errors or unexpected results in pose estimation due to the presence of key points of joints that are not visible in the image.

[232]

[0240] On the contrary, in Figure 8b , Figure 9b and Figure 10b In the human pose estimation results of the present disclosure shown, since key points of joints that are not visible in the image are removed by filtering, pose estimation information that is fast and accurate can be provided.

[0241] Figure 11This is a table used to illustrate the performance of a human posture estimation device according to an embodiment of the present disclosure.

[0242] like Figure 11 As shown, in this disclosure, the speed of model inference (MI) and post-processing (PP) corresponding to the input data size is improved compared to existing methods, and the accuracy corresponding to standard average precision (AP) and recall (AR) is improved compared to existing methods.

[0243] Figure 12a and Figure 12b This is a diagram applicable to a driver monitoring system of a human posture estimation device according to an embodiment of the present disclosure.

[0244] like Figure 12a As shown, if the hybrid pose estimation model of this disclosure is not applied, multiple human joint keypoints that are not visible in the image will appear, and therefore unnecessary joint keypoints will lead to unexpected vehicle accidents.

[0245] In comparison, such as Figure 12b As shown, if the hybrid pose estimation model of this disclosure is applied, key points of human joints that are not visible in the image are removed by filtering, so that only the key points of the joints that are actually seen in the image are accurately extracted, thus providing safety information to the driver.

[0246] Figure 13a , Figure 13b , Figure 14a and Figure 14b This is a graph illustrating the alignment learning results of the probability distributions of key points and visibility maps of a human pose estimation device according to an embodiment of the present disclosure.

[0247] like Figure 13a As shown, if the hybrid pose estimation model of this disclosure is not applied, wrist joints are difficult to identify because the probability distribution corresponding to human wrist joints in the image is not accurately shown in the visibility map.

[0248] In comparison, such as Figure 13b As shown, if the hybrid pose estimation model of this disclosure is applied, the wrist joint can be accurately identified because the probability distribution corresponding to the human wrist joint in the image is accurately shown in the visibility map.

[0249] In addition, such as Figure 14a As shown, if the hybrid pose estimation model of this disclosure is not applied, the hip joint may be difficult to identify because the probability distribution corresponding to the human hip joint in the image is not accurately shown in the visibility map.

[0250] In comparison, such as Figure 14bAs shown, if the hybrid pose estimation model of this disclosure is applied, the wrist joint can be accurately identified because the probability distribution corresponding to the human hip joint in the image is accurately shown in the visibility map.

[0251] Figure 15 and Figure 16 This is a diagram showing a visibility map corresponding to the image scale of a human pose estimation device according to an embodiment of the present disclosure.

[0252] like Figure 15 As shown, if the hybrid pose estimation model of this disclosure is applied, the probability distribution of the right elbow joint can be accurately shown in the image even at low image scale, thus enabling accurate identification of the elbow joint.

[0253] In addition, such as Figure 16 As shown, if the hybrid pose estimation model of this disclosure is applied, the probability distribution of the right shoulder joint can be accurately shown in the image even at low image scale, thus enabling accurate identification of the shoulder joint.

[0254] Figure 17 and Figure 18 This is a diagram illustrating the posture estimation performance of a human posture estimation device according to an embodiment of the present disclosure for each joint.

[0255] like Figure 17 As shown, if the hybrid pose estimation model of this disclosure is applied, the distance error between the key points and the visibility map of each joint of the human body in the image is reduced compared with the existing methods when they are aligned according to the probability distribution of the key points and the visibility map.

[0256] That is, compared with existing methods, this disclosed method improves accuracy by minimizing the distance error between the probability distribution of key points and the visibility map at each joint.

[0257] In addition, such as Figure 18 As shown, if the hybrid pose estimation model of this disclosure is applied, the probability value of the visibility map is improved compared with existing methods, based on the individual joints of the human body in the image.

[0258] That is, compared with the existing methods, the accuracy of this disclosure is improved because the probability value of the visibility map increases according to each joint.

[0259] Figure 19 This is a diagram illustrating the posture estimation method of a human posture estimation device according to an embodiment of the present disclosure.

[0260] like Figure 19 As shown, this disclosure can acquire image data (S10).

[0261] In addition, this disclosure can input the acquired image data into a pre-trained hybrid pose estimation model to extract the coordinate values ​​of keypoints corresponding to human joint positions and the probability values ​​of the visibility map from the image data (S20).

[0262] Here, this disclosure can estimate key points in image data corresponding to human joint positions in batches based on feature information of image data, and extract the coordinate values ​​of the estimated key points.

[0263] In addition, this disclosure can generate a visibility map for each joint position of the human body in the image data based on the feature information of the image data, and extract the probability values ​​corresponding to the joint positions of the human body in the visibility map.

[0264] Next, this disclosure can confirm whether the probability value of the joint that matches the coordinate value of the key point is below a threshold (S30).

[0265] Here, if the coordinate values ​​of the key points and the probability values ​​of the visibility map are extracted, the probability values ​​of the visibility map can be used to determine whether there is a probability value of a joint that matches the coordinate values ​​of the key points. If a probability value of a matching joint exists, it can be determined whether the probability value of the matching joint is below a preset threshold.

[0266] In addition, this disclosure can obtain the shooting environment information of the image data, and if the shooting environment information of the image data is included in the threshold change conditions, the threshold can be readjusted.

[0267] Depending on the circumstances, if this disclosure receives user input selecting a specific application mode, it can confirm whether the threshold corresponding to the specific application mode is pre-stored in the memory. If the threshold corresponding to the specific application mode is pre-stored in the memory, the currently set threshold can be readjusted to the threshold corresponding to the specific application mode.

[0268] In other cases, if user input requesting a threshold readjustment is received, this disclosure can obtain the threshold to be readjusted from the user input and readjust the currently set threshold to the threshold to be readjusted.

[0269] Secondly, if the probability value of the joint that matches the coordinate value of the key point is below the threshold, the present disclosure can filter out the key points of the corresponding joint (S40).

[0270] The foregoing disclosure can be implemented in computer-readable code on a medium containing a program. Computer-readable media include all types of recording devices storing data that can be read by a computer system. Examples of computer-readable media include: hard disk drives (HDDs), solid-state drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random access memory (RAM), optical disc read-only memory (CD-ROM), magnetic tape, floppy disks, optical data storage devices, etc. Furthermore, the computer may also include a processor 180 for an artificial intelligence device.

[0271] Industrial applicability

[0272] According to the human posture estimation device disclosed herein, key points of joints with low reliability are filtered by comparing the coordinate values ​​of key points with the probability values ​​of the visibility map, thereby improving the accuracy and speed of human posture estimation, and thus having significant industrial applicability.

Claims

1. A human posture estimation device, characterized in that, include: Memory for storing hybrid attitude estimation models; as well as The processor estimates the human pose in the image based on the hybrid pose estimation model; The processor performs the following operations: Image data is acquired and input into the pre-trained hybrid pose estimation model to extract the coordinate values ​​of key points corresponding to human joint positions and the probability values ​​of the visibility map from the image data. It is then confirmed whether the probability value of the joint matching the coordinate value of the key point is below a threshold. If the probability value of the joint matching the coordinate value of the key point is below the threshold, the key points of the corresponding joint are filtered out.

2. The human posture estimation device according to claim 1, characterized in that, The processor extracts feature information from the image data through the hybrid pose estimation model, and based on the feature information of the image data, simultaneously extracts the coordinate values ​​of key points corresponding to human joint positions in the image data and the probability distribution values ​​of the visibility map corresponding to human joint positions in the image data.

3. The human posture estimation device according to claim 2, characterized in that, The hybrid attitude estimation model includes: At least one first neural network model is used to extract feature information from the image data; The second neural network model, based on the feature information of the image data, extracts the coordinate values ​​of key points in the image data corresponding to the positions of human joints; and The third neural network model extracts the probability distribution values ​​of the visibility map corresponding to the human joint position in the image data based on the feature information of the image data.

4. The human posture estimation device according to claim 1, characterized in that, The hybrid pose estimation model is pre-trained to extract the coordinate values ​​of key points corresponding to human joint positions in the image data. Based on the extracted key point coordinate values, a key point distribution map is generated, and an unrelated visibility map corresponding to human joint positions in the image data is generated. By comparing the correlation between the joint probabilities of the unrelated visibility map and the joint probabilities of the key point distribution map, the joint probabilities of the unrelated visibility map are aligned with the joint probabilities of the key point distribution map.

5. The human posture estimation device according to claim 4, characterized in that, The hybrid pose estimation model generates multiple keypoint distribution maps and multiple irrelevant visibility maps for each joint of the human body in the image data. It extracts the keypoint distribution map and the irrelevant visibility map corresponding to the same joint to perform learning to align them. The learning ends when the learning to align the keypoint distribution map with the irrelevant visibility map has been performed on all joints of the human body in the image data.

6. The human posture estimation device according to claim 1, characterized in that, When extracting the coordinate values ​​of the key points, the processor estimates the key points in the image data that correspond to the positions of human joints in batches based on the feature information of the image data, and extracts the coordinate values ​​of the estimated key points.

7. The human posture estimation device according to claim 6, characterized in that, If the processor receives user input selecting a specific application mode, it obtains the necessary joint information required for the specific application mode, and selectively extracts key points corresponding to the necessary joint positions from all joint positions in the image data based on the necessary joint information.

8. The human posture estimation device according to claim 6, characterized in that, In addition to the human joints, if the processor receives user input to select a specific part of the human body, it batch estimates the key points in the image data that correspond to the positions of the human joints and the key points that correspond to the positions of the specific parts.

9. The human posture estimation device according to claim 1, characterized in that, When extracting the probability values ​​of the visibility map, the processor generates a visibility map for each joint position of the human body in the image data based on the feature information of the image data, and extracts the probability values ​​corresponding to the joint positions of the human body in the visibility map.

10. The human posture estimation device according to claim 9, characterized in that, When generating the visibility map, if the processor extracts the coordinate values ​​of key points of a specific joint in the human body, it generates a visibility map corresponding to the joint with the same coordinate values ​​of the extracted key points.

11. The human posture estimation device according to claim 9, characterized in that, When generating the visibility map, the processor uses a hybrid pose estimation model that has been pre-trained to align the keypoint distribution map corresponding to the same joint with the unrelated visibility map.

12. The human posture estimation device according to claim 1, characterized in that, If the processor extracts the coordinate values ​​of the key point and the probability value of the visibility map, it determines from the probability value of the visibility map whether there is a probability value of a joint that matches the coordinate values ​​of the key point. If there is a probability value of the matching joint, it determines whether the probability value of the matching joint is below a preset threshold.

13. The human posture estimation device according to claim 12, characterized in that, The processor acquires the shooting environment information of the image data, and if the shooting environment information of the image data is included in the threshold change conditions, the threshold is readjusted.

14. The human posture estimation device according to claim 12, characterized in that, If the processor receives user input requesting a threshold readjustment, it obtains the threshold to be readjusted from the user input and readjusts the currently set threshold to the threshold to be readjusted.

15. A method for estimating human pose, characterized in that, include: Steps for acquiring image data; The steps involve inputting the acquired image data into a pre-trained hybrid pose estimation model to extract the coordinate values ​​of key points corresponding to human joint positions and the probability values ​​of the visibility map from the image data, respectively. The step of confirming whether the probability value of the joint that matches the coordinate value of the key point is below the threshold; as well as If the probability value of a joint that matches the coordinate value of the key point is below the threshold, then the key point of the corresponding joint is filtered out.

Images