Method and apparatus for in-vehicle occupant monitoring

Abstract

According to some embodiments of the present disclosure, a method for monitoring an occupant in a vehicle, performed by a computing device, is disclosed. The method may include: identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image, wherein the plurality of body keypoints include a visible area keypoint corresponding to a visible area of ​​the occupant object and an occluded area keypoint corresponding to an occluded area of ​​the occupant object; determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; determining pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for recognizing occupants in a vehicle, and more specifically, to a method and apparatus for recognizing occupants in a vehicle that fuses a 2D camera and a 3D sensor to precisely estimate the position and posture of an occupant and effectively distinguish between an actual occupant and planar false positive objects (photos, drawings, etc.). Background Technology

[0002] With the recent advancements in autonomous driving and connected car technologies, automobiles are evolving beyond mere means of transportation into living spaces for users. Consequently, the importance of Occupant Monitoring Systems (OMS) is emerging to provide safety and convenience for occupants by detecting their presence, location, physique, and posture in real time. Occupant recognition technology serves as the foundation for various services, such as smart airbag deployment control, seatbelt verification, and the provision of personalized climate control and infotainment for each passenger.

[0003] Conventional passenger recognition systems have primarily utilized methods employing RGB cameras, radar, or ultrasonic sensors. The most commonly used RGB camera-based method has the advantage of being able to identify objects through high-resolution images. However, RGB camera-based methods have disadvantages, such as vulnerability to changes in lighting (at night, in tunnels, etc.) and concerns regarding passenger privacy infringement. Furthermore, because RGB camera-based methods rely solely on two-dimensional flat images, they have limitations in misidentifying objects like photographs, dolls, or clothes hanging on hangers as actual people, or in failing to accurately separate adjacent passengers in environments where seats are densely packed.

[0004] To compensate for this, a technology utilizing radar sensors has been proposed. Radar sensors have the advantage of being effective at detecting movements or breathing by utilizing the Doppler effect. However, radar sensors have the disadvantage of low recognition rates due to their low resolution, which makes it difficult to determine the exact shape or posture of the occupant, and the complexity of signal processing.

[0005] In addition, the method using 3D ToF (Time of Flight) sensors has the advantage of enabling three-dimensional detection by acquiring distance information. However, the method using 3D ToF sensors has the disadvantage that it is difficult to clearly separate human bodies between adjacent seats or estimate precise postures due to limitations in depth resolution.

[0006] In particular, with the recent enhancement of in-vehicle entertainment features, the issue of spoofing—where human faces printed on clothing, high-resolution photos, or human-shaped objects are mistakenly identified as passengers—is being pointed out as a serious problem that can cause safety-related malfunctions.

[0007] Therefore, a new occupant recognition technology is required that is robust to lighting environments, accurately maps the seat by determining the occupant's precise 3D position and posture, and simultaneously effectively distinguishes between flat fake objects and actual 3D occupants. Prior art literature

[0008] Republic of Korea Published Patent 10-2022-0104444 Republic of Korea Published Patent 10-2024-0106386 The problem to be solved

[0009] The present disclosure relates to a vehicle occupant monitoring system and aims to provide a method and device for monitoring a vehicle occupant that precisely recognizes an occupant using an object detection algorithm and a pose estimation algorithm based on 2D and 3D sensor data, thereby enabling the control of in-vehicle convenience devices (air conditioning, lighting, sound, etc.) according to the occupant's position. means of solving the problem

[0010] A method for monitoring an in-vehicle occupant, performed by a computing device to solve the problem described above, is disclosed. The method may include: a step of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; a step of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image, wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object; a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; a step of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and a step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats.

[0011] Alternatively, the step of determining the spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image obtained by the 2D camera unit and the externally calibrated 3D sensing unit may include: a step of determining the depth value of the visible area keypoint by obtaining the depth value of a point corresponding to the visible area keypoint from the depth image; and a step of estimating the depth value of the occluded area keypoint using the depth value of the visible area keypoint connected to the occluded area keypoint on a skeletal structure defined by the plurality of body keypoints.

[0012] Alternatively, the step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats may include: a step of determining a coordinate among the plurality of reference seat coordinates that has the shortest distance value from the pelvic center coordinate; and a step of determining the vehicle seat corresponding to the coordinate having the shortest distance value from the pelvic center coordinate as the seat position of the occupant object.

[0013] Alternatively, the method may further include: a step of excluding false positive objects among the occupant objects by determining the stereoscopic structure of the face point cloud region of the occupant object, wherein the stereoscopic structure includes at least one of a depth variance value or a flatness index.

[0014] Alternatively, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include: a step of determining a face polygonal region by connecting face key points among the plurality of body key points; and a step of determining a set of points projected onto the face polygonal region on the depth image as the face point cloud region.

[0015] Alternatively, the step of excluding false positive objects among the occupant objects by determining the stereoscopic structure of the face point cloud region of the occupant object may include: a step of calculating the depth variance value of the face point cloud region; and a step of determining false positive objects by comparing the depth variance value with a predetermined face reference variance value.

[0016] Alternatively, the depth variance value can be calculated by the following Formula 1.

[0017] [Formula 1]

[0018]

[0019] Here, is the depth variance value, and N is the number of points included in the face point cloud region, and is the depth value of the i-th point included in the above-mentioned face point group region, and can be the average depth value of the points included in the face point cloud area.

[0020] Alternatively, the step of excluding false detection objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include: a step of determining the flatness index of the face point cloud region by performing Principal Component Analysis on the planar coordinate components of the face point cloud region orthogonally projected onto the reference plane of the 3D sensing unit; and a step of determining false detection objects by comparing the flatness index of the face point cloud region with a predetermined face elliptical distribution reference value.

[0021] Alternatively, the flatness index can be defined by the following Equation 2.

[0022] [Equation 2]

[0023]

[0024] Here, is the flatness index, is the eigenvalue of the first principal component derived by the above principal component analysis, and may be the eigenvalue of the second principal component derived by the above principal component analysis.

[0025] Alternatively, the step of excluding false positive objects among the occupant objects by determining the stereoscopic structurality of the face point cloud region of the occupant object may include: a step of determining a depth variance normalized value by normalizing the depth variance value using a threshold range of the face reference variance value; a step of determining a flatness normalized value by normalizing the flatness index using a threshold range of the face elliptical distribution reference value; a step of obtaining a weighted stereoscopic structurality value by applying weights to the depth variance normalized value and the flatness normalized value; and a step of determining the false positive objects by comparing the weighted stereoscopic structurality value with a predetermined face reference structurality value.

[0026] Alternatively, the step of obtaining a weighted stereoscopic structural value by applying weights to the depth variance normalized value and the flatness normalized value may include: calculating the weighted stereoscopic structural value by assigning a higher weight to the depth variance normalized value compared to the flatness normalized value in response to the ratio of points corresponding to the visible region among the points included in the face point cloud region being higher than a predetermined reference ratio value compared to the ratio of points corresponding to the occluded region.

[0027] Alternatively, the step of obtaining a weighted three-dimensional structural value by applying weights to the depth variance normalization value and the flatness normalization value may include: a step of determining a face rotation angle based on a face skeletal structure defined by a facial key point among the plurality of body key points; and a step of adjusting the weight for the flatness normalization value by comparing the face rotation angle with the face front angle.

[0028] Alternatively, the method may further include the step of controlling an in-vehicle acoustic module, lighting module, or air conditioning module using the seat position of the occupant object and the posture information of the occupant object estimated by the plurality of body key points.

[0029] A computer program stored on a computer-readable storage medium for solving the problem described above is disclosed. The computer program includes instructions for one or more processors to perform a method for recognizing an occupant in a vehicle, wherein the method comprises: a step of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; a step of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image, wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object; a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; and a step of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints. and the step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats; may be included.

[0030] A computing device is disclosed for performing a method for recognizing an in-vehicle occupant to solve the problem described above. The computing device comprises: a processor; wherein the processor comprises: an operation of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; an operation of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image - wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object -; an operation of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; an operation of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and an operation of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinates among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats. It can perform. Effects of the invention

[0031] The present disclosure relates to a vehicle occupant monitoring system and can provide a method and device for recognizing a vehicle occupant by analyzing the skeletal structure and three-dimensional structure of the facial portion of the occupant based on 2D and 3D sensor data to precisely recognize the occupant, and thereby enable the control of in-vehicle convenience devices (air conditioning, lighting, sound, etc.) according to the occupant's position. Brief explanation of the drawing

[0032] FIG. 1 is a block diagram illustrating a computing device that performs an operation to provide a method for monitoring occupants in a vehicle according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram showing a network function according to some embodiments of the present disclosure. FIG. 3 schematically illustrates an occupant monitoring system according to some embodiments of the present disclosure. FIG. 4 is a drawing for explaining body key points according to some embodiments of the present disclosure. Figure 5 is a graph showing the change in position of each body part of the occupant in pixel units between frames while the vehicle is in motion. FIG. 6 is a drawing for explaining an embodiment of controlling an in-vehicle acoustic module, lighting module, or air conditioning module using seat position and posture information of an occupant object according to some embodiments of the present disclosure. FIG. 7 is a flowchart of a method for monitoring occupants in a vehicle according to some embodiments of the present disclosure. FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Specific details for implementing the invention

[0033] Various embodiments and / or aspects are now disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will be apparent to those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the descriptions are intended to include all such aspects and their equivalents. Specifically, terms such as “exemplary,” “example,” “aspect,” and “example” as used herein may not be interpreted as implying that any described aspect or design is superior or advantageous over other aspects or designs.

[0034] Hereinafter, identical or similar components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art are omitted if it is determined that such detailed descriptions may obscure the essence of the embodiments disclosed in this specification. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings.

[0035] The terms used herein are for describing the embodiments and are not intended to limit the disclosure. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0036] Although terms such as "first," "second," etc. are used to describe various elements or components, it goes without saying that these elements or components are not limited by these terms. These terms are used merely to distinguish one element or component from another. Therefore, it goes without saying that the first element or component mentioned below may be the second element or component within the technical scope of this disclosure.

[0037] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0038] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0039] And, the term “at least one of A or B” should be interpreted to mean “a case including only A,” “a case including only B,” or “a combination of A and B.”

[0040] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0041] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0042] And, terms such as “~ etc.” as in “A, B, etc.” should be interpreted to mean “cases containing only A,” “cases containing only B,” or “cases composed of A and B.”

[0043] When it is stated that one component is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, when it is stated that one component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

[0044] The suffixes “module” and “part” for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles.

[0045] When elements or layers are referred to as being "on" or "on" another element or layer, it includes not only being directly on top of the other element or layer but also cases where another layer or element is interposed in between. On the other hand, when a component is referred to as being "directly on" or "immediately on," it indicates that no other element or layer is interposed in between.

[0046] Spatially relative terms such as "below," "beneath," "lower," "above," and "upper" may be used to easily describe the relationship between one component or other components as illustrated in the drawings. Spatially relative terms should be understood as encompassing different orientations of the element during use or operation, in addition to the directions illustrated in the drawings.

[0047] For example, if a component depicted in a drawing is inverted, a component described as being "below" or "beneath" another component may be placed "above" the other component. Therefore, the exemplary term "below" may encompass both the downward and upward directions. Components may also be oriented in other directions, and accordingly, spatially relative terms may be interpreted according to the orientation.

[0048] The purpose and effects of the present disclosure, and the technical configurations for achieving them, will become clear by referring to the embodiments described in detail below in conjunction with the accompanying drawings. In describing the present disclosure, if it is determined that a detailed description of known functions or configurations might unnecessarily obscure the essence of the present disclosure, such detailed description will be omitted. Furthermore, the terms described below are defined considering their functions in the present disclosure, and these may vary depending on the intentions or practices of the user or operator.

[0049] However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the disclosure, and the present disclosure is defined only by the scope of the claims. Therefore, such definitions should be based on the content throughout this specification.

[0050] FIG. 1 is a block diagram of a computing device for performing a method for monitoring occupants in a vehicle according to some embodiments of the present disclosure.

[0051] As illustrated in FIG. 1, the computing device (100) may include a processor (110), memory (130), and a network unit (150). The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In some embodiments of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0052] The processor (110) may be composed of one or more cores and may include processors for data analysis and processing, deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU). The processor (110) may read a computer program stored in memory (130) and perform data conversion, computation, generation, etc., to perform a method for monitoring occupants in a vehicle according to some embodiments of the present disclosure. For example, the processor (110) may perform steps for performing the method for monitoring occupants in a vehicle described below. To this end, the processor (110) may implement an occupant monitoring system and its components. Additionally, according to some embodiments of the present disclosure, the processor (110) may perform computations for training a neural network using training data to perform the method for monitoring occupants in a vehicle. For example, the processor (110) may implement an object detection model trained to identify passenger objects from 2D images and a pose estimation model trained to generate body keypoints by extracting body feature points of the identified passenger objects. The processor (110) may perform calculations for training a neural network, such as processing input data for training in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process calculations for performing a method for monitoring passengers in a vehicle. For example, the CPU and GPGPU may together process calculations for performing a method for monitoring passengers in a vehicle.In addition, in some embodiments of the present disclosure, processors of a plurality of computing devices may be used together to process data conversion, computation, generation, learning of network functions, and data classification using network functions for performing a method for monitoring occupants in a vehicle. In addition, a computer program executed in a computing device according to some embodiments of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0053] According to some embodiments of the present disclosure, the memory (130) may store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150). For example, the memory (130) may store data generated during the process of performing a vehicle occupant monitoring method by the processor (110). Additionally, the memory (130) may store data received from the outside during the process of performing a vehicle occupant monitoring method by the processor (110). However, not limited thereto, the memory (130) may store various information for performing a vehicle occupant monitoring method according to some embodiments of the present disclosure.

[0054] According to some embodiments of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0055] In some examples, the processor (110) may include memory (130). In some examples, the computing device (100) or the processor (110) may include a System on a Chip (SoC). In some examples, the computing device (100) or the processor (110) may operate based on embedded memory.

[0056] A network unit (150) according to some embodiments of the present disclosure may use any known wired or wireless communication system.

[0057] The network unit (150) can transmit and receive information, user interfaces, etc., processed by the processor (110) through communication with other terminals. For example, the network unit (150) can provide a user interface generated by the processor (110) to a client (e.g., a user terminal). In addition, the network unit (150) can receive external input from a user authorized as a client and transmit it to the processor (110). At this time, the processor (110) can process operations such as outputting, modifying, changing, or adding information provided through the user interface based on the external input from the user received from the network unit (150).

[0058] Specifically, for example, the network unit (150) can transmit and receive various information for performing a vehicle occupant monitoring method according to some embodiments of the present disclosure. For example, the network unit (150) can receive 2D images and / or depth images from an external 2D camera unit and / or 3D sensing unit. In addition, the network unit (150) can transmit some data generated during the process of performing the vehicle occupant monitoring method described below to the outside for storage in a database.

[0059] Meanwhile, a computing device (100) according to some embodiments of the present disclosure may include a server as a computing system that transmits and receives information through communication with a client. In this case, the client may be any type of terminal capable of accessing the server. For example, the computing device (100) which is the server may receive a query from a user terminal and generate a single information processing result corresponding to the query. In this case, the computing device (100) which is the server may provide a user interface including the processing result to the user terminal. In this case, the user terminal may output the user interface received from the computing device (100) which is the server and receive or process information through interaction with the user.

[0060] In an additional embodiment, the computing device (100) may include any type of terminal that receives data resources generated from any server and performs additional information processing.

[0061] In an additional embodiment, the computing device (100) may be an electronic device installed in a vehicle to implement a Driver Monitoring System (DMS) or an Occupant Monitoring System (OMS). For example, the computing device (100) may determine the position and posture of an occupant in real time based on 2D and 3D video / image data acquired from a 2D camera and a 3D sensor installed in the vehicle. The computing device (100) may be implemented in the form of a vehicle infotainment system, an Advanced Driver Assistance System (ADAS) module, or an Electronic Control Unit (ECU) that generates signals to control an acoustic module, a lighting module, or an air conditioning module in the vehicle according to the results determined based on the position and posture of the occupant.

[0062] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to some embodiments of the present disclosure.

[0063] Throughout this specification, artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0064] A neural network can be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting neural networks may be interconnected by one or more links.

[0065] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0066] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight can be variable and can be varied by the user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0067] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0068] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0069] In some embodiments of the present disclosure, a set of neurons or nodes may be defined by the expression a layer.

[0070] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0071] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0072] Artificial intelligence-based models according to some embodiments of the present disclosure may include a deep neural network (DNN). A deep neural network may refer to a neural network that includes a plurality of hidden layers in addition to an input layer and an output layer. Using a deep neural network, latent structures of data can be identified. That is, latent structures of photos, text, video, voice, protein sequence structures, gene sequence structures, peptide sequence structures, music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the voice, etc.), and / or binding affinity between peptides and MHCs can be identified. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), Transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0073] The artificial intelligence-based model of the present disclosure may be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0074] A neural network that can be used in an artificial intelligence-based model of the present disclosure may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of a neural network may be a process of applying knowledge to the neural network to perform a specific operation.

[0075] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training point is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node being updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0076] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed using the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a type of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0077] A computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed. The aforementioned data structure may be stored in a storage unit in the present disclosure, executed by a processor, and transmitted and received by a communication unit.

[0078] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data for solving specific problems (e.g., data analysis, data retrieval, data storage, data modification). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of the computing device's resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0079] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0080] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form a loop in a graph data structure.

[0081] Throughout this specification, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they will be described uniformly as neural network. A data structure may include a neural network. Furthermore, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the configurations described above, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated during the computational process of the neural network, and is not limited to the foregoing. A computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0082] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0083] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0084] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0085] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, R-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0086] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0087] Artificial intelligence-based models according to some embodiments of the present disclosure may include a Large Language Model (LLM). In the present disclosure, a Large Language Model may refer to an artificial intelligence-based model trained using a vast amount of training data to perform Natural Language Processing. A Large Language Model may include a transformer, a transformer encoder family model, and / or a transformer decoder family model. A transformer encoder family model may correspond to an artificial intelligence model that uses a transformer encoder structure. A transformer decoder family model may correspond to an artificial intelligence model that uses a transformer decoder structure.

[0088] In one embodiment, the transformer may be composed of an encoder that encodes input data and a decoder that decodes the encoded data. The transformer may have a structure that takes a series of input data as input, undergoes encoding and decoding steps, and outputs a series of output data. In one embodiment, the series of input data may be processed into a form that the transformer can compute. The process of processing the series of input data into a form that the transformer can compute may include a tokenizing process and an embedding process. The tokenizing process may refer to the process of dividing the series of input data into tokens of a certain unit. For example, the certain unit may include word units. The embedding process may refer to the process of converting at least one token tokenized from the series of input data into an embedding vector.

[0089] In one embodiment, the transformer can obtain an embedding vector to be input to an encoder by combining a token embedding vector that embeds at least one token corresponding to a series of input data, a segment embedding vector that distinguishes sentences containing the tokens for each token, and a position embedding vector that reflects the position of the token. The encoder family model and the decoder family model of the transformer can also obtain embedding vectors by performing the same method.

[0090] In one embodiment, for the transformer to encode and decode a series of input data, the encoder and decoder within the transformer may utilize an attention algorithm. An attention algorithm may refer to an algorithm that, for a given query, calculates similarity by applying a softmax function to an attention score obtained by matrix multiplying the query with a key, and calculates an attention value for the query by matrix multiplying the calculated similarity with a value.

[0091] In one embodiment, the self-attention algorithm may refer to an attention algorithm that uses a query, a key, and a value generated by multiplying the same embedding vector by a query weight, a key weight, and a value weight, respectively. The cross-attention algorithm may refer to an attention algorithm that uses a query generated by multiplying a first embedding vector by a query weight, and a key and a value generated by multiplying a second embedding vector by a key weight and a value weight, respectively. The query weight, the key weight, and the value weight may be trainable parameters that are updated through the training process of a large-scale language model.

[0092] In one embodiment, the encoder of the transformer may include an embedding layer, a self-attention layer that applies a self-attention algorithm to an embedding vector, a normalization layer, and a feed-forward neural network (FFN). Additionally, the encoder may have a form in which N unit structures including a self-attention layer, a normalization layer, and a feed-forward neural network are connected. The decoder of the transformer may include an embedding layer, a masked self-attention layer, a normalization layer, a cross-attention layer that applies a cross-attention algorithm, and a feed-forward neural network. Additionally, the decoder may have a form in which N unit structures including a masked self-attention layer, a normalization layer, a cross-attention layer, and a feed-forward neural network are connected. The masked self-attention layer may correspond to a layer that calculates an attention value for each of the sequences in which words are sequentially included among a plurality of words included in a series of input data.

[0093] A transformer may include additional components such as a linear layer and a softmax layer, in addition to an encoder and a decoder. The encoder family model and the decoder family model of a transformer may also each include the aforementioned additional components in addition to the encoder and decoder. A method for constructing a transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0094] In one embodiment, attention layers such as a self-attention layer, a masked self-attention layer, and a cross-attention layer may correspond to a multi-head attention layer that includes a plurality of attention layers in parallel. The multi-head attention layer may concatenate the attention values ​​output from each of the plurality of attention layers and output an output attention value by matrix multiplying the concatenated matrix by an output weight. The output attention value output from the multi-head attention layer may have the same size as the attention value output from a single attention layer.

[0095] In one embodiment, the transformer may be trained through a Masked Language Model (MLM) process, a Next Sentence Prediction (NSP) process, etc. The MLM process may refer to a training process that predicts a masked word through a series of training data in which some words are masked. The NSP process may refer to a training process that determines whether two sentences are connected sentences in a series of training data containing any two sentences.

[0096] In one embodiment, a large-scale language model can process various data formats, such as natural language text, image data, audio data, and video data. The large-scale language model can embed data to convert data of various data formats into a series of computationally viable data. The large-scale language model can process additional data that represents the relative positional or topological relationships between a series of input data. Alternatively, a series of input data may be embedded by additionally reflecting vectors that represent the relative positional or topological relationships between the input data. In one example, the relative positional relationships between a series of input data may include, but are not limited to, word order within a natural language sentence, the relative positional relationships of each segmented image, and the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or topological relationships between a series of input data may be referred to as positional encoding.

[0097] An example of a large-scale language model for processing image data (Vision Transformer, ViT) is disclosed in Dosovitskiy, et al., AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE, which is incorporated herein by reference.

[0098] An artificial intelligence model according to some embodiments of the present disclosure may include a multimodal large-scale language model. A multimodal large-scale language model may refer to a large-scale language model capable of understanding and processing relationships between different data formats, such as natural language text data, image data, audio data, and video data. A multimodal language model may include a plurality of encoders that encode input data corresponding to each data format. A multimodal language model may be trained to calculate similarity between embedding vectors encoded by encoders of each data format through training data containing data of different data formats, such that similarity between identical pairs is calculated to be higher and similarity between different pairs is calculated to be lower.

[0099] An example of a large-scale multimodal language model that understands and processes the relationship between image data and natural language text data (Contrastive Language-Image Pre-training, CLIP) is disclosed in Alec Radford, et al., *LEARNING TRANSFERABLE VISUAL MODELS FROM NATURAL LANGUAGE SUPERVISION*, which is incorporated herein by reference.

[0100] FIG. 3 schematically illustrates a passenger monitoring system according to some embodiments of the present disclosure. FIG. 4 is a diagram for explaining body keypoints according to some embodiments of the present disclosure. FIG. 5 is a graph measuring the change in position of each body part of a passenger in pixel units between frames while the vehicle is in motion. FIG. 6 is a diagram for explaining an embodiment of controlling an in-vehicle acoustic module, lighting module, or air conditioning module using seat position and posture information of a passenger object according to some embodiments of the present disclosure.

[0101] In some examples, the occupant monitoring system (1000) may include an occupant object detection unit (200), an attitude estimation unit (300), an occupant seat position determination unit (400), a false positive object determination unit (500), and a vehicle device control unit (600). The configuration of the occupant monitoring system (1000) illustrated in FIG. 3 is merely a simplified example. In some embodiments of the present disclosure, the occupant monitoring system (1000) may include other configurations, and only some of the disclosed configurations may constitute the occupant monitoring system (1000).

[0102] According to some embodiments of the present disclosure, a passenger monitoring system (1000) can precisely recognize a passenger object by analyzing the three-dimensional structure of the body points and facial portions of the passenger object based on 2D images and depth images, and thereby control in-vehicle convenience devices (air conditioning, lighting, sound, etc.) according to the passenger's position. Hereinafter, with reference to FIGS. 3 to 6, the operations of a passenger monitoring system (1000) and its components that perform an in-vehicle passenger monitoring method according to some embodiments of the present disclosure are described in detail.

[0103] According to some embodiments of the present disclosure, the occupant object detection unit (200) can identify an occupant object on a 2D image obtained by a 2D camera unit using an object detection algorithm.

[0104] Specifically, the passenger object detection unit (200) can identify a passenger object presumed to be a passenger by analyzing a 2D image obtained by a 2D camera unit using an object detection algorithm.

[0105] In some examples, the passenger object detection unit (200) may be implemented using a deep learning model based on a Convolutional Neural Network (CNN). For example, the passenger object detection unit (200) may extract features by processing a 2D image as input. The passenger object detection unit (200) may identify a passenger object by analyzing the extracted feature map and outputting the coordinates of a bounding box surrounding an area presumed to be a passenger and a confidence score that the object is a person. In some examples, the passenger object detection unit (200) may be implemented using YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), or R-CNN series models, but is not limited thereto.

[0106] In some examples, the 2D image may be an IR image, but is not limited thereto. As described below, the 2D camera unit (not shown) is externally calibrated with the 3D sensing unit (not shown), so that specific pixel coordinates on the 2D image can be precisely converted into the 3D spatial coordinate system of the 3D sensing unit, or conversely, 3D points acquired from the 3D sensing unit can be projected onto the 2D image plane. Through this, as described below, the occupant monitoring system (1000) can accurately match the body keypoints (2D coordinates) of the occupant object detected in the 2D image with the depth values ​​(Z values) acquired from the 3D sensing unit in a 1:1 ratio. Therefore, the occupant monitoring system (1000) can calculate the precise 3D spatial coordinates of the occupant object identified in the 2D image.

[0107] In some examples, a 2D camera unit and / or a 3D sensing unit may be included in the occupant monitoring system (1000). In other examples, the 2D camera unit and / or the 3D sensing unit may be an external device network-connected to the occupant monitoring system (1000) via a wired or wireless connection. In this case, the occupant monitoring system (1000) may receive 2D images and / or depth images through the network unit (150).

[0108] According to some embodiments of the present disclosure, the pose estimation unit (300) can determine a plurality of body keypoints of a occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on a 2D image. Here, the plurality of body keypoints may include a visible area keypoint corresponding to a visible area of ​​the occupant object and an occluded area keypoint corresponding to an occluded area of ​​the occupant object.

[0109] Specifically, the pose estimation algorithm may refer to a technology in the field of computer vision that detects the locations of key points, such as joints or facial landmarks, in the form of coordinates according to a skeletal structure that is predefined based on the human body structure. In some examples, the pose estimation unit (300) may include a pose estimation model that estimates the locations of multiple body keypoints by processing the occupant object region detected by the object detection unit (200) as input. Such a pose estimation model may be implemented using OpenPose, AlphaPose, HRNet (High-Resolution Network), etc., but is not limited thereto.

[0110] In some examples, a body keypoint may be an anatomical reference point that constitutes a predefined human skeleton structure. For example, referring to FIG. 4, an image is shown displaying the body keypoints of a passenger object seated in the rear seat of a vehicle. For example, a plurality of body keypoints may include facial keypoints such as the nose, both eyes, and both ears, and body joint keypoints such as both shoulders, elbows, wrists, pelvis, knees, and ankles. The posture estimation unit (300) can estimate the overall posture or behavior of the passenger object by connecting the plurality of body keypoints detected from the 2D image to the predefined skeleton structure.

[0111] In particular, due to the characteristics of the vehicle interior environment, occlusion situations may frequently occur where parts of a passenger's body are obscured by structures within the vehicle, such as the object dashboard, steering wheel, and front seat, or where the view is blocked by other passengers. Therefore, the body parts of the passenger object identified in the 2D image may be located across the visible area visible in the 2D image and the occlusion area obscured in the 2D image. The pose estimation unit (300) can estimate not only the visible area keypoints located in the visible area of ​​the passenger object but also the occlusion area keypoints located in the occlusion area of ​​the passenger object through a pose estimation algorithm. Specifically, the pose estimation unit (300) utilizes the geometric constraints of a human skeletal structure learned through deep learning (e.g., if there is a shoulder, there is an elbow within a certain distance) and contextual information, so it can estimate the location of body keypoints located in the occlusion area that are not directly visible in the 2D image. As illustrated in FIG. 4, the posture estimation unit (300) can estimate occlusion area keypoints in an occlusion area that is not displayed on a 2D image of a passenger object seated in the rear seat of a vehicle. Through this, the posture estimation unit (300) can estimate the accurate position and posture information of the passenger object by generating a complete skeletal structure including the obscured body parts.

[0112] According to some embodiments of the present disclosure, the occupant seat position determining unit (400) can determine the spatial coordinate values ​​of each of a plurality of body keypoints based on a depth image obtained by a 2D camera unit and an externally calibrated 3D sensing unit.

[0113] Specifically, since the 2D camera unit and the 3D sensing unit are positioned at different physical locations within the vehicle, the viewpoint and coordinate system of the data acquired by the two sensors may differ from each other. In this case, external calibration may refer to a process of setting the coordinate systems of the two sensors to a state where they can be precisely aligned or mutually converted by calculating extrinsic parameters in advance that represent the relative positional relationship (translation) and rotational relationship (rotation) between the two sensors. The passenger seat position determination unit (400) can project specific pixel coordinates detected on the 2D image onto the coordinate system of the 3D sensing unit, or conversely, accurately project coordinates on the depth image of the 3D sensing unit onto the 2D image plane by using the extrinsic parameters obtained through external calibration.

[0114] In some examples, the passenger seat positioning unit (400) can identify accurate points on a depth image corresponding to the location of each of a plurality of body keypoints on a 2D image using external parameters. Through this, the passenger seat positioning unit (400) can determine the depth value of a visible area keypoint by obtaining the depth value of a point corresponding to a visible area keypoint from the depth image. In this case, the passenger seat positioning unit (400) can estimate the depth value of a occluded area keypoint using the depth value of a visible area keypoint connected to an occluded area keypoint on a skeletal structure defined by a plurality of body keypoints.

[0115] To explain specifically with an example, it can be assumed that the shoulder portion of a passenger object seated in the rear seat of a vehicle is clearly identifiable in a 2D image (visible area), but the elbow or wrist portion is obscured by the backrest of the front seat and is not visible (occluded area). In this case, if the passenger seat position determining unit (400) takes the depth value corresponding to the estimated elbow position (2D coordinate) from the depth image of the 3D sensing unit as is, an error may occur in that the value represents the distance to the surface of the front seat rather than the passenger's elbow.

[0116] Accordingly, the passenger seat positioning unit (400) can utilize the connection relationships (i.e., skeleton structure) of body keypoints identified on a 2D image. In the example described above, the passenger seat positioning unit (400) can first obtain the accurate depth value of the shoulder keypoint, which is a visible area keypoint, by using a depth image. Then, the passenger seat positioning unit (400) can use the depth value of the shoulder keypoint as a reference to estimate the depth value of the 'elbow keypoint (occlusion area keypoint)' which is anatomically connected to the shoulder in terms of skeleton structure. For example, the passenger seat positioning unit (400) can calculate a distance vector from the shoulder keypoint to the elbow keypoint using the angle of the arm and predefined human body ratio information according to the skeleton structure defined by a plurality of body keypoints on a 2D image. The passenger seat position determining unit (400) can precisely estimate (or correct) the depth value of the elbow key point (occlusion area key point) by reflecting a distance vector to the 3D coordinates of the shoulder key point (visible area key point) (for example, adding the depth component of the distance vector to the depth value of the shoulder key point). Through this method, the passenger seat position determining unit (400) can precisely restore the 3D posture of the passenger without distortion even when a part of the body is obscured.

[0117] According to some embodiments of the present disclosure, the occupant seat position determining unit (400) can determine the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of a plurality of body key points. Additionally, the occupant seat position determining unit (400) can determine the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinates among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats.

[0118] As described above, the passenger seat position determining unit (400) can estimate the position of a hidden body part using a skeletal structure defined by a plurality of body key points determined using a 2D image. Therefore, even when the pelvic area of ​​a passenger object is located in an obstructed area that is not visible in the 2D image because it is covered by a seat belt, blanket, or vehicle interior structure, the passenger seat position determining unit (400) can accurately calculate the 3D pelvic center coordinates using the relationship between the spatial coordinates of the estimated body key points and the geometric characteristics of the human body structure.

[0119] When the pelvic center coordinates of a passenger object are determined, the passenger seat position determining unit (400) can determine a coordinate that matches the pelvic center coordinates among multiple reference seat coordinates corresponding to multiple vehicle seats. Specifically, the passenger seat position determining unit (400) can determine the seat position of a passenger object with high accuracy by using the pelvic center coordinates among various body parts as a criterion for determining the seat position. FIG. 5 is a graph measuring the amount of position change between frames for each body part (face, body, and pelvis) of a passenger in pixel units while driving a vehicle. Referring to FIG. 5, the face or body may show very large and irregular changes in position between frames depending on movements such as turning the passenger's head or tilting the body. On the other hand, the pelvis is in close contact with the seat and may show a relatively small and stable amount of position change. Therefore, the passenger seat position determining unit (400) can determine the seat position of a passenger object based on the most stable pelvic center coordinates instead of the head or upper body, which have large positional variability.

[0120] For example, the passenger seat position determining unit (400) can calculate the Euclidean distance between the reference seat coordinates and the pelvic center coordinates of each vehicle seat that is pre-set within the vehicle. Then, the passenger seat position determining unit (400) can determine the coordinate having the shortest distance value from the pelvic center coordinates among a plurality of reference seat coordinates. Then, the passenger seat position determining unit (400) can determine the vehicle seat corresponding to the coordinate having the shortest distance value from the pelvic center coordinates as the seat position of the passenger object. Through this, the passenger seat position determining unit (400) can accurately determine the seat position without misrecognition even in dynamic situations where the passenger object leans or moves its body.

[0121] In some cases, the reference seat coordinates may be data that is measured in advance or stored in advance according to the vehicle's design information. For example, the reference seat coordinates may be 3D coordinates corresponding to the geometric center point of the seat cushion of each vehicle seat. As described above, since the pelvic center coordinates and the reference seat coordinates are defined on the same 3D spatial coordinate system through external calibration, precise and rapid distance calculation between the two coordinates may be possible.

[0122] According to some embodiments of the present disclosure, a false detection object determination unit (500) may exclude false detection objects from among passenger objects by determining the three-dimensional structure of the face point cloud region of the passenger object. Here, the three-dimensional structure may include at least one of a depth variance value or a flatness index.

[0123] Specifically, in the interior environment of a vehicle, various objects that can be mistaken for people may exist in addition to actual people. For example, clothing hanging on a hanger, a cushion printed with a human face, or a high-resolution photograph of a person may be mistaken for a person inside the vehicle. Since these false positive objects are difficult to distinguish from actual people using 2D image-based object detection algorithms, they can cause problems of 'false positives' or 'spoofing' where they are incorrectly recognized as passenger objects. In this specification, a passenger object may refer to a passenger who is an actual person. Additionally, a false positive object may refer to various objects that are not actual people but have a shape that can be mistaken for a passenger object.

[0124] These false positive objects may lack a sense of depth as most of them take the form of a two-dimensional plane. On the other hand, since a real human face has a distinct three-dimensional structure due to the curves of the nose, forehead, cheeks, etc., human occupant objects and non-human false positive objects may have different three-dimensional structures.

[0125] Accordingly, the false detection object determination unit (500) can quantitatively determine the three-dimensional structure by utilizing depth data obtained from the 3D sensing unit. And, the false detection object determination unit (500) can effectively filter false detection objects by distinguishing whether the face point cloud region of the identified occupant object has an actual three-dimensional shape or a planar shape using the quantitatively determined three-dimensional structure.

[0126] To this end, the false detection object determination unit (500) can first identify the face point cloud region to be analyzed. In some examples, the false detection object determination unit (500) can determine the face polygonal region on the 2D image by connecting face part keypoints corresponding to the face part among a plurality of body keypoints detected by the pose estimation unit (300). In some examples, the face part keypoints may include keypoints corresponding to both eyes, both ears, and the nose, but are not limited thereto. Subsequently, the false detection object determination unit (500) can determine the set of points projected onto the face polygonal region on the depth image as the face point cloud region.

[0127] Specifically, when a face point cloud area is defined using a simple rectangular bounding box, elements such as hair, accessories being worn, or the headrest behind the occupant object may be included within the face point cloud area in addition to the facial skin. Because these background elements differ significantly in depth from the face surface, they can act as noise that produces high depth variance values, even if they are actually false positive objects such as flat photographs.

[0128] To prevent such problems, the false detection object determination unit (500) can set a face polygon area by connecting major key points (eyes, nose, ears, etc.) of the face. Then, the false detection object determination unit (500) can project the face polygon area onto a depth image using the face polygon area as a mask. That is, the false detection object determination unit (500) can determine the face point cloud area by selectively extracting only the depth values ​​on the depth image that correspond 1:1 to the pixel coordinates inside the face polygon area on the 2D image. Through this, the false detection object determination unit (500) can effectively remove background noise and obtain only pure three-dimensional information of the face (e.g., depth values), thereby significantly improving the accuracy of the three-dimensional structural analysis described later.

[0129] For the face point cloud region extracted in this manner, the false detection object determination unit (500) can determine the three-dimensional structure by determining a 'depth variance value (Z-Variance)' which analyzes the statistical distribution of depth values, or by determining a 'planarity index' which analyzes the geometric shape of the point cloud distribution. Below, an exemplary embodiment is described in which the false detection object determination unit (500) determines the three-dimensional structure using the depth variance value, the planarity index, and a combination thereof.

[0130] According to some embodiments of the present disclosure, the false detection object determination unit (500) can calculate the depth variance value of the face point cloud region. The false detection object determination unit (500) can determine the false detection object by comparing the depth variance value with a predetermined face reference variance value.

[0131] First, an embodiment using depth variance is described. Specifically, the false detection object determination unit (500) can calculate the depth variance of points included in the face point cloud area to quantitatively determine the three-dimensional structure. The depth variance (Z-Variance) may refer to a statistical indicator indicating how widely the depth values ​​of each point constituting the face point cloud area are distributed (deviated) relative to the average depth value of the area.

[0132] In some examples, the depth variance value can be calculated by the following Formula 1.

[0133] [Formula 1]

[0134]

[0135] Here, is the depth variance value, and N is the number of points included in the face point cloud region, and is the depth value of the i-th point included in the face point cloud region, and can be the average depth value of the points included in the face point cloud area.

[0136] In some examples, the false detection object determination unit (500) can determine a false detection object by comparing a depth variance value with a predetermined face reference variance value. Generally, a real human face may have a high depth variance value because it has a relatively large deviation in depth value due to the curvature of the nose, eyes, cheeks, etc. On the other hand, a false detection object such as a photograph or printed material may have a low depth variance value because it has a very small deviation in depth value due to its planar characteristics. For example, the false detection object determination unit (500) determines that the depth variance value is a first threshold value (e.g., If it is greater than ), the object can be determined as a passenger object with an actual three-dimensional face. Conversely, if the depth variance value is greater than the second threshold (e.g., If the value is less than ) the false detection object determination unit (500) may determine the object as a planar false detection object. In this case, the false detection object determination unit (500) may exclude the false detection object from the passenger recognition result.

[0137] According to some embodiments of the present disclosure, the false detection object determination unit (500) can determine the flatness index of the face point cloud region by performing Principal Component Analysis on the planar coordinate components of the face point cloud region orthogonally projected onto the reference plane of the 3D sensing unit. And, the false detection object determination unit (500) can determine the false detection object by comparing the flatness index of the face point cloud region with a predetermined face elliptical distribution reference value.

[0138] Next, an embodiment using a flatness index is described. Specifically, the false detection object determination unit (500) may perform principal component analysis (PCA) to quantitatively analyze the geometric distribution shape of the face point cloud area. In some examples, principal component analysis may be a technique for finding a principal component that indicates the direction in which the data variance is greatest. In some examples, the false detection object determination unit (500) may perform two-dimensional principal component analysis that excludes the depth axis (Z-axis) information of the 3D sensor and analyzes only the point cloud distribution characteristics on the projected plane (XY plane).

[0139] Specifically, for example, the false detection object determination unit (500) calculates the eigenvalues ​​of the first principal component by calculating a covariance matrix representing the correlation between the X-coordinate and the Y-coordinate for the face point cloud region. ) and the eigenvalue of the second principal component ( ) can be calculated. In this case, the eigenvalue of the first principal component ( ) is the eigenvalue of the second principal component ( It can have a large value compared to ).

[0140] In some examples, the flatness index can be defined by the following Formula 2,

[0141] [Equation 2]

[0142]

[0143] Here, is the flatness index, is the eigenvalue of the first principal component derived by the above principal component analysis, and may be the eigenvalue of the second principal component derived by principal component analysis.

[0144] In some examples, the false detection object determination unit (500) can determine the false detection object by comparing the flatness index with a predetermined face elliptical distribution reference value. Generally, since a real human face has a three-dimensional curved surface, the point cloud region of a real human face projected onto a two-dimensional plane may have an elliptical distribution evenly spread along the horizontal and vertical axes. In this case, the first principal component ( ) and the second principal component ( The size ratio of ) is at a certain level (e.g., Since it maintains ), the flatness index of a real human face having an elliptical distribution ( ) can be calculated to be relatively high, with a value of approximately 0.29 to 0.38. On the other hand, the point cloud distribution of the sham plane can have a linear or nearly flat distribution. In this case, the first principal component ( ) and the second principal component ( Since the size ratio of ) appears low (for example, ), flatness index of a fake plane( ) can be calculated as a low value of about 0.09 or less. Accordingly, the false detection object determination unit (500) can determine the object as a false detection object if the calculated flatness index is less than a preset elliptical distribution reference value (e.g., 0.2).

[0145] In the example described above, it was explained that the false detection object determination unit (500) performs a two-dimensional principal component analysis using only planar coordinate components (X, Y) and excluding depth values ​​(Z-axis information) during the principal component analysis process. By using two-dimensional principal component analysis, the false detection object determination unit (500) can increase the reliability of the judgment by performing a shape analysis in the planar direction (i.e., two-dimensional principal component analysis of the planar coordinate components of the depth image) in a manner mutually independent of the depth direction (z-direction) stereoscopic analysis (i.e., depth variance value analysis). While the previously described 'depth variance value' is an indicator for determining the presence or absence of irregularities (curvature) on the surface of the face, the 'flatness index' may be an indicator for determining whether the geometric distribution shape of the point cloud spread in space is normal (elliptical) or abnormal (linear / planar). Therefore, the depth variance value and the flatness index can have a mutually independent relationship that is efficient for quantitatively determining stereoscopic structurality.

[0146] In addition, the false detection object determination unit (500) can increase computational efficiency compared to 3D principal component analysis by performing 2D principal component analysis. Principal component analysis (3D PCA) on the entire 3D data can have a large amount of computation and high complexity. On the other hand, principal component analysis (2D PCA) on projected 2D data can have a relatively low computational load. Since primary stereoscopic verification can be performed using only depth variance values, the false detection object determination unit (500) can secure the fast processing speed required for real-time passenger recognition by avoiding excessive 3D computation in the secondary structural verification stage and utilizing only 2D projected data.

[0147] As described above, the depth variance value and the flatness index can have an independent relationship that is efficient for quantitatively determining three-dimensional structurality. Reflecting this perspective, the false detection object determination unit (500) can determine the false detection object by using a mixed score determination that combines the two indices.

[0148] Specifically, the false detection object determination unit (500) can perform a normalization process to integrally calculate depth variance values ​​and flatness indices having different scales and units. For example, the false detection object determination unit (500) can determine a depth variance normalized value by normalizing the depth variance value using a threshold range of face-based variance values. Specifically, in some examples, the false detection object determination unit (500) can determine a depth variance normalized value that is normalized to a value between 0 and 1 using a pre-set threshold range of face-based variance values ​​for the depth variance value.

[0149] In some examples, the depth variance normalized value can be calculated by the following Formula 3.

[0150] [Formula 3]

[0151]

[0152] Here, is the depth variance normalized value, and is the depth variance value, and is the upper limit of the critical range of the face-based variance value, and can be the lower limit of the threshold range of the face-based variance value.

[0153] In a similar manner, the false detection object determination unit (500) can determine a flatness normalization value by normalizing the flatness index using a threshold range of face elliptical distribution reference values. Specifically, in some examples, the false detection object determination unit (500) can determine a flatness normalization value normalized to a value between 0 and 1 using a threshold range of face elliptical distribution reference values ​​that is preset for the flatness index.

[0154] In some examples, the flatness normalized value can be calculated by the following Formula 4.

[0155]

[0156] Here, is the depth variance normalized value, and is the depth variance value, and is the upper limit of the critical range of the flatness normalization value, and can be the lower limit of the critical range of the flatness normalization value.

[0157] And, the false detection object determination unit (500) can obtain a weighted three-dimensional structural value by applying weights to the depth variance normalized value and the flatness normalized value.

[0158] In some examples, the weighted stereostructural value can be calculated by the following Equation 5.

[0159] [Formula 5]

[0160] ,

[0161] Here is a weighted stereostructural value, and is the depth variance normalized value, and is the flatness normalized value, and is the weight for the depth variance normalization value, and can be a weight for the flatness normalized value. In some examples, the weight for the depth variance normalized value ( ) is 0.6, and the weights for the depth variance normalized values ​​( ) can be 0.4. However, it is not limited to this, and weights can have various values.

[0162] In some examples, the false detection object determination unit (500) can determine the false detection object by comparing the weighted three-dimensional structural value with a predetermined face reference structural value. For example, the false detection object determination unit (500) may determine the weighted three-dimensional structural value ( ) If the face standard structural value (e.g., 0.5) or higher is greater than, the object can be determined to be a real human face. As another example, the false detection object determination unit (500) has a weighted three-dimensional structural value ( If the ) is less than the face-based structural value (e.g., 0.5), the object can be determined as a false positive object.

[0163] Here, the upper and lower limits of the aforementioned threshold range may be predetermined values ​​as optimal boundary values ​​capable of maximizing the accuracy of false positive detection by statistically analyzing the distribution of multiple actual occupant sample data and false positive object (photos, printed materials, etc.) sample data. Additionally, the weights used to determine the weighted three-dimensional structure values ​​may also be predetermined reference values ​​obtained by statistically analyzing the importance or reliability of each indicator (depth variance value, flatness index) in distinguishing between actual faces and false positive objects.

[0164] As described above, in calculating the weighted three-dimensional structurality value, the weights applied to the depth variance normalized value and the flatness normalized value ( , ) may be a value that is not a fixed value but is variably adjusted according to the passenger's state and the recognition environment. Specifically, since the technical reliability of the depth variance value and the flatness index can vary depending on the situation, they may be variably adjusted to increase the accuracy of the determination. For example, in an ideal situation where the passenger object is looking straight ahead and there is no obstruction, both indicators have high reliability, but in specific situations such as when the face is obscured or looking to the side, the judgment ability of one of the indicators may be reduced. Therefore, the false detection object determination unit (500) can dynamically optimize the weights by recognizing such situations.

[0165] First, an embodiment is described in which weights are adjusted according to the relative proportion of points included in the visible region and the occluded region among the points included in the face point cloud region. As described above, an occluded region in which a part of the face area is not visible may occur when the face of a passenger object is obscured by structures inside the vehicle, such as the front seat, or when the face is covered by wearing a mask or by a hand. In this case, the proportion of points corresponding to the visible region and the proportion of points corresponding to the occluded region among the points included in the face point cloud region may differ. Specifically, for the points corresponding to the visible region among the points included in the face point cloud region, the depth value actually measured through the depth image can be directly utilized. Therefore, as the proportion of points corresponding to the visible region among the points included in the face point cloud region increases, the depth variance normalization value using the depth value can have higher reliability. In this regard, when the ratio of points corresponding to the visible area among the points included in the face point group area is higher than the predetermined reference ratio value compared to the ratio of points corresponding to the occluded area, the false detection object determination unit (500) can calculate a weighted three-dimensional structural value by giving a higher weight to the depth variance normalization value compared to the flatness normalization value.

[0166] As another example, an embodiment is described in which weights are adjusted according to the face rotation angle. When a passenger looks to the side rather than the front, even if the face is three-dimensional, the 2D point cloud distribution projected onto the reference plane of the 3D sensor unit may appear as a flattened linear shape rather than an elliptical shape. In this case, there is a possibility that the flatness index is calculated to be low despite the person being real, leading to the possibility of being mistaken for a fake plane (false detection object). To prevent this, the false detection object determination unit (500) can determine the face rotation angle based on the facial skeletal structure defined by the facial keypoint among a plurality of body keypoints. Specifically, the false detection object determination unit (500) can calculate the left-right rotation (Yaw) angle of the face using the difference in depth values ​​between the two ear keypoints. Subsequently, the false detection object determination unit (500) can compare the calculated face rotation angle with the face front angle (0 degrees). The false detection object determination unit (500) can adjust the weight for the flatness normalization value by comparing the face rotation angle with the face front angle. For example, as the rotation angle increases (as the side view increases), the reliability of the flatness index decreases, so the false detection object determination unit (500) can reduce the weight for the flatness normalization value. In this case, the false detection object determination unit (500) can increase or maintain the weight for the depth variance normalization value. Through this, the false detection object determination unit (500) can secure robust false detection filtering performance even with changes in the posture of the occupant object.

[0167] According to some embodiments of the present disclosure, a vehicle device control unit (600) can control an in-vehicle sound module, lighting module, or air conditioning module using posture information of a passenger object estimated by a seat position of a passenger object and a plurality of body key points.

[0168] Specifically, the vehicle device control unit (600) can estimate the current behavior or state of the occupant by analyzing the spatial arrangement of body key points obtained from the posture estimation unit (500) and changes in the skeletal structure. For example, the vehicle device control unit (600) can generate specific posture information, such as whether the occupant is reading, sleeping, or feeling cold, by analyzing the bending angle of the arm joint, the tilt of the head, or the degree of crouching of the upper body.

[0169] First, regarding the control of the lighting module, the vehicle device control unit (600) can detect a reading posture by analyzing changes in the passenger's arm position and upper body angle (bottom image of FIG. 6). If it is determined that the passenger is in a reading posture, the vehicle device control unit (600) can automatically turn on the lighting in the direction of the seat or adjust the illuminance to a reading mode (e.g., 300 lux or more) to ensure visibility. Conversely, if a sleeping posture is detected where the passenger's head is leaning against the seat or the head is drooping (see top image of FIG. 6), the vehicle device control unit (600) can automatically turn off the lighting around the seat or switch to a soft dimming mode to create a comfortable sleeping environment.

[0170] In some examples, regarding the control of the acoustic module, the vehicle device control unit (600) can analyze the activity of the occupant based on the size, speed, frequency, etc. of the movement of the body keypoint. When the occupant's activity level is high (e.g., during conversation, active movement), the audio volume can be increased to a certain level (e.g., +2dB) to help with clear transmission. Additionally, the vehicle device control unit (600) can lower the speaker volume of the corresponding seat area (e.g., -4dB) or stop the output when a sleeping posture is detected or when activity level is low.

[0171] In some examples, regarding the control of the air conditioning module, the vehicle device control unit (600) can estimate thermal comfort through changes in the occupant's posture. For example, if the occupant assumes a posture that is presumed to feel cold, such as crouching the upper body or crossing the arms, the vehicle device control unit (600) can control the air conditioning module to automatically raise the set temperature of the seat (e.g., +1 degree) or increase the volume of warm air. However, not limited thereto, the vehicle device control unit (600) can control the in-vehicle sound module, lighting module, or air conditioning module in various ways using the occupant object's posture information estimated by the occupant object's seat position and multiple body key points.

[0172] FIG. 7 is a flowchart of a method for monitoring occupants in a vehicle according to some embodiments of the present disclosure.

[0173] According to some embodiments of the present disclosure, a method for monitoring occupants in a vehicle may include the step (S100) of identifying an occupant object on a 2D image obtained by a 2D camera unit using an object detection algorithm.

[0174] According to some embodiments of the present disclosure, a method for monitoring an occupant in a vehicle may include a step (S200) of determining a plurality of body keypoints of an occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image. Here, the plurality of body keypoints may include a visible area keypoint corresponding to a visible area of ​​the occupant object and an occluded area keypoint corresponding to an occluded area of ​​the occupant object.

[0175] According to some embodiments of the present disclosure, a method for monitoring occupants in a vehicle may include the step (S300) of determining spatial coordinate values ​​of each of the plurality of body key points based on a depth image obtained by the 2D camera unit and the externally calibrated 3D sensing unit.

[0176] According to some embodiments of the present disclosure, a method for monitoring an occupant in a vehicle may include the step (S400) of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body key points.

[0177] According to some embodiments of the present disclosure, a method for monitoring an occupant in a vehicle may include the step (S500) of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats.

[0178] Alternatively, the step (S300) of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image obtained by the 2D camera unit and an externally calibrated 3D sensing unit may include: a step of determining the depth value of the visible area keypoint by obtaining the depth value of a point corresponding to the visible area keypoint from the depth image; and a step of estimating the depth value of the occluded area keypoint using the depth value of the visible area keypoint connected to the occluded area keypoint on a skeletal structure defined by the plurality of body keypoints.

[0179] Alternatively, the step (S500) of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats may include the step of determining a coordinate having the shortest distance value from the pelvic center coordinate among the plurality of reference seat coordinates, and the step of determining the vehicle seat corresponding to the coordinate having the shortest distance value from the pelvic center coordinate as the seat position of the occupant object.

[0180] Alternatively, the method for monitoring occupants in a vehicle may further include the step of excluding false positive objects among the occupant objects by determining the stereoscopic structure of the face point cloud region of the occupant objects. Here, the stereoscopic structure may include at least one of a depth variance value or a flatness index.

[0181] Alternatively, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include the step of determining a face polygonal region by connecting face key points among the plurality of body key points, and the step of determining a set of points projected onto the face polygonal region on the depth image as the face point cloud region.

[0182] Alternatively, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include the step of calculating the depth variance value of the face point cloud region, and the step of determining false positive objects by comparing the depth variance value with a predetermined face reference variance value.

[0183] Alternatively, the depth variance value can be calculated by the following Formula 1.

[0184] [Formula 1]

[0185]

[0186] Here, is the depth variance value, and N is the number of points included in the face point cloud region, and is the depth value of the i-th point included in the above-mentioned face point group region, and can be the average depth value of the points included in the face point cloud area.

[0187] Alternatively, the step of excluding false detection objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include the step of determining the flatness index of the face point cloud region by performing Principal Component Analysis on the planar coordinate components of the face point cloud region orthogonally projected onto the reference plane of the 3D sensing unit, and the step of determining false detection objects by comparing the flatness index of the face point cloud region with a predetermined face elliptical distribution reference value.

[0188] Alternatively, the flatness index can be defined by the following Equation 2.

[0189] [Equation 2]

[0190]

[0191] Here, is the flatness index, is the eigenvalue of the first principal component derived by the above principal component analysis, and may be the eigenvalue of the second principal component derived by the above principal component analysis.

[0192] Alternatively, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object may include: determining a depth variance normalized value by normalizing the depth variance value using a threshold range of the face reference variance value; determining a flatness normalized value by normalizing the flatness index using a threshold range of the face elliptical distribution reference value; obtaining a weighted three-dimensional structure value by applying weights to the depth variance normalized value and the flatness normalized value; and determining the false positive objects by comparing the weighted three-dimensional structure value with a predetermined face reference structure value.

[0193] Alternatively, the step of obtaining a weighted dimensional structure value by applying weights to the depth variance normalization value and the flatness normalization value may include the step of calculating the weighted dimensional structure value by giving a higher weight to the depth variance normalization value compared to the flatness normalization value in response to the fact that the ratio of points corresponding to the visible region among the points included in the face point group area is higher than a predetermined reference ratio value compared to the ratio of points corresponding to the occluded region.

[0194] Alternatively, the step of obtaining a weighted three-dimensional structural value by applying weights to the depth variance normalization value and the flatness normalization value may include the step of determining a face rotation angle based on a face skeletal structure defined by a face part key point among the plurality of body key points, and the step of adjusting the weight for the flatness normalization value by comparing the face rotation angle with the face front angle.

[0195] Alternatively, the method for monitoring an in-vehicle occupant may further include the step of controlling an in-vehicle acoustic module, lighting module, or air conditioning module using the seat position of the occupant object and posture information of the occupant object estimated by the plurality of body key points.

[0196] The steps of the aforementioned method for monitoring occupants in a vehicle are presented for illustrative purposes only, and some steps may be omitted or additional steps may be added. Additionally, the aforementioned steps may be performed in any order.

[0197] FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0198] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0199] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0200] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0201] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0202] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0203] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1104).

[0204] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0205] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The computer (1102) may also include an external hard disk drive (HDD) (1114) connected to the computer (1102) via ports such as USB, Thunderbolt, and eSATA. A hard disk drive (1114), a magnetic disk drive (1116), and an optical disk drive (1120) can each be connected to a system bus (1108) via a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). An interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0206] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0207] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0208] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0209] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0210] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0211] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0212] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0213] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0214] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0215] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0216] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0217] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0218] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

Claim 1 A method for monitoring an in-vehicle occupant performed by a computing device comprises: a step of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; a step of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image, wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object; a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; a step of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and a step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinates among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats. A method comprising: a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image obtained by the 2D camera unit and the externally calibrated 3D sensing unit, wherein the step comprises: a step of determining the depth value of the visible area keypoint by obtaining the depth value of a point corresponding to the visible area keypoint from the depth image; and a step of estimating the depth value of the occluded area keypoint using the depth value of the visible area keypoint connected to the occluded area keypoint on a skeletal structure defined by the plurality of body keypoints. Claim 2 delete Claim 3 A method according to claim 1, wherein the step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats comprises: a step of determining a coordinate having the shortest distance value from the pelvic center coordinate among the plurality of reference seat coordinates; and a step of determining the vehicle seat corresponding to the coordinate having the shortest distance value from the pelvic center coordinate as the seat position of the occupant object. Claim 4 A method for monitoring an in-vehicle occupant performed by a computing device comprises: a step of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; a step of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image, wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object; a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; a step of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and a step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinates among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats. A method comprising: a step of excluding false positive objects among the occupant objects by determining the stereoscopic structure of the face point cloud region of the occupant object, wherein the stereoscopic structure includes at least one of a depth variance value or a flatness index. Claim 5 In claim 4, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object comprises: a step of determining a face polygonal region by connecting face key points among the plurality of body key points; and a step of determining a set of points projected onto the face polygonal region on the depth image as the face point cloud region; a method. Claim 6 In claim 4, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object comprises: a step of calculating the depth variance value of the face point cloud region; and a step of determining false positive objects by comparing the depth variance value with a predetermined face reference variance value; a method. Claim 7 In claim 6, the depth variance value is calculated by the following Formula 1. [Formula 1] Here, is the depth variance value, and N is the number of points included in the face point cloud region, and is the depth value of the i-th point included in the above-mentioned face point group region, and is the average depth value of the points included in the face point cloud region. Claim 8 In claim 4, the step of excluding false detection objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object comprises: a step of determining the flatness index of the face point cloud region by performing Principal Component Analysis on the planar coordinate components of the face point cloud region orthogonally projected onto the reference plane of the 3D sensing unit; and a step of determining false detection objects by comparing the flatness index of the face point cloud region with a predetermined face elliptical distribution reference value; a method. Claim 9 In claim 8, the method wherein the flatness index is defined by the following Formula 2. [Formula 2] Here, is the flatness index, is the eigenvalue of the first principal component derived by the above principal component analysis, and is the eigenvalue of the second principal component derived by the above principal component analysis. Claim 10 In claim 4, the step of excluding false positive objects among the occupant objects by determining the three-dimensional structure of the face point cloud region of the occupant object comprises: a step of determining a depth variance normalized value by normalizing the depth variance value using a threshold range of the face reference variance value; a step of determining a flatness normalized value by normalizing the flatness index using a threshold range of the face elliptical distribution reference value; a step of obtaining a weighted three-dimensional structure value by applying weights to the depth variance normalized value and the flatness normalized value; and a step of determining the false positive objects by comparing the weighted three-dimensional structure value with a predetermined face reference structure value. Claim 11 In claim 10, the step of obtaining a weighted stereoscopic structural value by applying weights to the depth variance normalized value and the flatness normalized value comprises: a step of calculating the weighted stereoscopic structural value by assigning a higher weight to the depth variance normalized value compared to the flatness normalized value in response to the ratio of points corresponding to the visible region among the points included in the face point group region being higher than a predetermined reference ratio value compared to the ratio of points corresponding to the occluded region. Claim 12 In claim 10, the step of obtaining a weighted three-dimensional structural value by applying weights to the depth variance normalization value and the flatness normalization value comprises: a step of determining a face rotation angle based on a facial skeletal structure defined by a facial key point among the plurality of body key points; and a step of adjusting the weight for the flatness normalization value by comparing the face rotation angle with the face front angle; a method. Claim 13 delete Claim 14 A computer program stored on a computer-readable storage medium, wherein the computer program includes instructions for one or more processors to perform a method for recognizing an occupant in a vehicle, and the method comprises: a step of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; a step of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image - wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object -; a step of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; and a step of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints. A computer program stored on a computer-readable storage medium, comprising: a step of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinate among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats; and a step of determining the spatial coordinate value of each of the plurality of body keypoints based on a depth image obtained by the 2D camera unit and an externally calibrated 3D sensing unit, wherein the step comprises: a step of determining the depth value of the visible area keypoint by obtaining the depth value of a point corresponding to the visible area keypoint from the depth image; and a step of estimating the depth value of the occluded area keypoint using the depth value of the visible area keypoint connected to the occluded area keypoint on a skeletal structure defined by the plurality of body keypoints. Claim 15 A computing device for performing a method for recognizing an occupant in a vehicle, wherein the computing device comprises: a processor; and the processor comprises: an operation of identifying an occupant object on a 2D image acquired by a 2D camera unit using an object detection algorithm; an operation of determining a plurality of body keypoints of the occupant object by performing pose estimation on a visible area of ​​the occupant object displayed on the 2D image - wherein the plurality of body keypoints include a visible area keypoint corresponding to the visible area of ​​the occupant object and an occluded area keypoint corresponding to the occluded area of ​​the occupant object -; an operation of determining spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image acquired by the 2D camera unit and an externally calibrated 3D sensing unit; an operation of determining the pelvic center coordinates of the occupant object based on the spatial coordinate values ​​of each of the plurality of body keypoints; and an operation of determining the seat position of the occupant object by determining a coordinate that matches the pelvic center coordinates among a plurality of reference seat coordinates corresponding to a plurality of vehicle seats. A computing device comprising: an operation to perform, and to determine spatial coordinate values ​​of each of the plurality of body keypoints based on a depth image obtained by the 2D camera unit and an externally calibrated 3D sensing unit, the operation of determining the depth value of the visible area keypoint by obtaining the depth value of a point corresponding to the visible area keypoint from the depth image; and an operation of estimating the depth value of the occluded area keypoint using the depth value of the visible area keypoint connected to the occluded area keypoint on a skeletal structure defined by the plurality of body keypoints.

Images