Information processing device and information processing method

Abstract

This information processing device is provided with: an information acquisition unit (11) that acquires target information pertaining to a control target candidate from a target information acquisition device; a first processing unit (12) that, on the basis of the target information, performs first determination processing in which whether or not the control target candidate is a living body is determined according to whether or not the control target candidate indicates a biological feature; and a second processing unit (13) that, when the first processing unit (12) has determined that the control target candidate is a living body, performs second determination processing in which the control target candidate is designated as a control target and an attribute or situation of the control target is estimated to generate control information.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] Conventionally, various techniques for performing various controls according to the attributes or circumstances of a living body have been known. For example, Patent Document 1 discloses an occupant restraint control device that detects facial features from an image captured by a camera and, based on whether or not the facial features are detected, changes the degree to which an airbag actuator and a seatbelt actuator operate an airbag and a seatbelt pretensioner that restrain the occupant in that seat. Patent Document 1 also describes that a person can be distinguished from a doll based on the trajectory of movement of the facial features, and that if the position of the facial features is extracted within a predetermined range below the image and does not move, it may be determined that a doll or the like has been placed in the seat.

[0003] JP 2013-252863 A

[0004] When performing control based on the attributes or circumstances of a living body, it is necessary to accurately estimate the attributes or circumstances of the living body to prevent the control from being erroneous. To accurately estimate the attributes or circumstances of a living body, it is necessary to accurately distinguish between living and non-living bodies. This is because an incorrect distinction between living and non-living bodies could result in an incorrect estimation of the attributes or circumstances of the living body, and various controls performed based on the incorrectly estimated attributes or circumstances could be erroneous. Conventional technology distinguishes between living and non-living bodies (people and dolls) using information (facial features) related to the attributes or circumstances of the living body for subsequent control (control of an airbag actuator and a seatbelt actuator). While this reduces the burden on the processing involved in distinguishing between living and non-living bodies, the accuracy of distinguishing between living and non-living bodies is low because the distinction is simple and based only on the position and movement of facial features. On the other hand, if conventional technology were to increase the accuracy of distinguishing between living and non-living bodies through more detailed processing, it would result in unnecessary burdens on obtaining information related to the attributes or circumstances of the living body used for subsequent control. As such, conventional technology had the problem of being unable to achieve both improved accuracy in distinguishing between living and non-living organisms, in order to prevent erroneous control based on the attributes or circumstances of the living organism, and preventing an increase in the processing load of estimating the attributes or circumstances of the living organism for such control.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an information processing device that provides control information for performing control according to the attributes or situation of the living body to be controlled, and that is highly accurate and can provide control information that is generated while reducing unnecessary processing load for estimating the attributes or situation of the living body.

[0006] The information processing device of the present disclosure is an information processing device that provides control information for performing control according to the attributes or situation of a living organism to be controlled, and includes an information acquisition unit that acquires target information regarding a control target candidate from a target information acquisition device, a first processing unit that performs a first determination process that determines whether the control target candidate is a living organism based on whether the control target candidate exhibits biological characteristics, based on the target information acquired by the information acquisition unit, and a second processing unit that performs a second determination process that, when the first processing unit determines that the control target candidate is a living organism, sets the control target candidate as a control target and estimates the attributes or situation of the control target to generate control information.

[0007] According to the present disclosure, an information processing device can provide control information that is highly accurate and that is generated while reducing unnecessary processing load for estimating the attributes or status of a living body.

[0008] 6A and 6B are diagrams illustrating an example of the configuration of an information processing device according to embodiment 1. FIG. 6B is a diagram illustrating an example of a test dummy used in a vehicle performance test. FIG. 6C is a flowchart illustrating the operation of the information processing device according to embodiment 1. FIG. 6D is a flowchart illustrating the operation of the first processing unit in step ST2 of FIG. 3 in more detail. FIG. 6E is a flowchart illustrating the operation of the second processing unit in step ST4 of FIG. 3 in more detail. FIGS. 6A and 6B are diagrams illustrating an example of the hardware configuration of an information processing device according to embodiment 1.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. FIG. 1 is a diagram illustrating an example configuration of an information processing device 1 according to embodiment 1. The information processing device 1 is connected to a target information acquisition device 2 and a control device 3. The control device 3 performs control according to the attributes or situation of a living body. Note that in embodiment 1, the "living body" is assumed to be a human. Also, in embodiment 1, the attributes of a living body include, for example, the age, sex, or physique of the living body. Physique includes, for example, body thickness, height, or weight. Also, the situation of a living body includes, for example, the emotion, alertness, posture, or physical condition of the living body. Note that these are merely examples, and the attributes or situations of a living body may be any attributes or situations of a living body that are used to determine the control content performed by the control device 3.

[0010] The target information acquisition device 2 detects objects within a range where a target to be controlled by the control device 3 (hereinafter referred to as a "control target") should be present, and acquires information about the detected object. The objects detected by the target information acquisition device 2 are candidates for the control target (hereinafter referred to as a "control target candidate"). The control target candidate may include living organisms that are the control target and non-living organisms that should not be the control target, i.e., non-living organisms. The target information acquisition device 2 outputs information about the control target candidate (hereinafter referred to as "target information") to the information processing device 1.

[0011] The information processing device 1 acquires target information from the target information acquisition device 2, and based on the acquired target information, determines whether a control target candidate detected by the target information acquisition device 2 is a living organism. If the information processing device 1 determines that the control target candidate is a living organism, the control target candidate determined to be a living organism is designated as a control target and estimates the attributes or status of the control target. The information processing device 1 then generates information (hereinafter referred to as "control information") for the control device 3 to perform control according to the attributes or status of the control target, and outputs the generated control information to the control device 3. If the information processing device 1 determines that a control target candidate detected by the target information acquisition device 2 is not a living organism, i.e., a non-living organism, based on the target information acquired from the target information acquisition device 2, the information processing device 1 does not estimate the attributes or status of the control target candidate determined to be a non-living organism, and does not generate or output the control information to the control device 3. This prevents the information processing device 1 from erroneously determining that a non-living control target candidate is a living organism, resulting in an erroneous estimation of the attributes or status of the living organism, which would result in incorrect control by the control device 3. Details of an example configuration of the information processing device 1 will be described later.

[0012] In the following first embodiment, as an example, it is assumed that the living body to be controlled is a vehicle occupant. It is also assumed that the control device 3 is mounted on the vehicle and performs airbag control in the vehicle according to the age of the vehicle occupant. More specifically, it is assumed that the control device 3 performs airbag control that controls the opening degree or height of the airbag according to the age of the occupant.

[0013] The target information acquisition device 2 is assumed to be an imaging device mounted on a vehicle to monitor the interior of the vehicle, and to acquire captured images of the interior of the vehicle as target information. The imaging device may be, for example, a near-infrared camera or a visible light camera. The imaging device may be, for example, a shared imaging device with a so-called "Driver Monitoring System (DMS)" mounted on a vehicle to monitor the driver's status inside the vehicle. The imaging device is installed so as to be able to capture an image of at least the interior of the vehicle, including the area where the upper bodies of vehicle occupants should be present. The area where the upper bodies of vehicle occupants should be present is, for example, the area corresponding to the seat back and the space in front of the headrest. For example, the imaging device is installed in the center of an overhead console or dashboard inside the vehicle, so as to be able to capture an image of the interior of the vehicle from the center of the overhead console or dashboard. For example, the imaging device may be installed so as to be able to capture an image of only the driver, or so as to be able to capture an image of rear seat occupants.

[0014] In the following first embodiment, as an example, it is assumed that the information processing device 1 determines whether or not an object captured in the captured image, i.e., a control target candidate, is a living body, i.e., an occupant who is a control target, based on the captured image output from the target information acquisition device 2, and estimates the age of the occupant when it is determined that the control target candidate is an occupant. The information processing device 1 generates control information for the control device 3 to control the airbag according to the age of the occupant based on the estimated age of the occupant, and outputs the generated control information to the control device 3.

[0015] For example, in vehicle interior monitoring, there is a possibility that non-living objects may be detected as occupants. For example, because vehicle seats are shaped to fit the shape of a human body, there is a possibility that the seat may be mistakenly detected as an occupant due to its shape. Furthermore, if a doll with a human-like shape and a face or facial features is brought into the vehicle, there is a possibility that the doll may be mistakenly detected as an occupant. For example, if a vehicle is equipped with an airbag control function based on the age of the occupant, such as the control device 3 described above, developers may conduct performance tests using a dummy doll in the vehicle before mass production, etc., to determine whether the airbag control function operates correctly and confirm whether the airbag control is properly performed based on the age of the occupant. If a test dummy doll is mistakenly detected as an occupant and the age of the dummy mistakenly detected as an occupant is incorrectly estimated, the airbag control may not operate correctly or may malfunction. Here, FIG. 2 is a diagram illustrating an example of a test dummy doll used in vehicle performance tests. As shown in Figure 2, the dummy has a shape similar to a human and may have a face or facial features similar to a human, but these are not reproduced to strictly match the shape, face, and facial features of a human. Therefore, if the dummy is mistakenly detected as an occupant, as described above, this may lead to the airbag control not operating correctly or causing abnormal operation.

[0016] Thus, when a vehicle performs control (e.g., airbag control) based on the attributes or situation (e.g., age) of an occupant, if a non-living object that should not be the target of control is mistakenly determined to be a living object (e.g., an occupant), the control performed in the subsequent stage may be erroneous. Therefore, it is necessary to accurately determine whether or not a living object is a living object and to prevent the generation of erroneous control information based on an erroneous determination of whether or not a living object is a living object. In particular, as shown in FIG. 2 , although the above-mentioned dummy doll is not created to match a real person, it may closely resemble a human shape and have facial features similar to a human. Therefore, in order to prevent the dummy doll from being mistakenly determined to be a living object, a detailed determination based on the presence or absence of biological features is required. However, making the determination process of whether or not a living object is a living object more detailed increases the processing load. If an attempt is made to determine whether or not a living object is a living object during the process of detecting information used in the subsequent control stage, the attributes or situation of the non-living object may also be estimated, which may result in unnecessary processing that places a heavy load on the subsequent control stage.

[0017] Taking the above into consideration, the information processing device 1 according to the first embodiment accurately determines whether a control target candidate is a living organism (here, an occupant) and generates control information while suppressing unnecessary processing load for estimating the attributes or status of the living organism (here, age). As a result, the information processing device 1 can provide the control device 3 with control information that is generated with high accuracy and while suppressing unnecessary processing load for estimating the attributes or status of the living organism.

[0018] A description will be given of an example configuration of an information processing device 1 according to embodiment 1. As shown in FIG.

[0019] The information acquisition unit 11 acquires target information from the target information acquisition device 2. Here, the information acquisition unit 11 acquires a captured image of the interior of the vehicle as the target information. The information acquisition unit 11 outputs the acquired captured image to the first processing unit 12.

[0020] The first processing unit 12 performs a process (hereinafter referred to as a "first determination process") of determining whether or not the control target candidate is a living organism depending on whether or not the control target candidate exhibits a biological characteristic, based on the target information acquired by the information acquisition unit 11. That is, here, the first processing unit 12 performs a first determination process of determining whether or not the control target candidate is an occupant depending on whether or not the control target candidate exhibits a biological characteristic, based on the captured image acquired by the information acquisition unit 11.

[0021] The details of the first determination process will be described with an example. The first processing unit 12 determines whether a control target candidate is an occupant based on the captured image in accordance with the living / non-living determination conditions. The living / non-living determination conditions define what characteristics of the control target candidate based on the target information can determine whether the control target candidate exhibits a living / non-living characteristic or whether the control target candidate does not exhibit a living / non-living characteristic. The living / non-living determination conditions are set in advance by a developer or the like. After setting the living / non-living determination conditions, the developer or the like stores information indicating the set living / non-living determination conditions (hereinafter referred to as "living / non-living determination condition information") in a storage unit (not shown). The storage unit (not shown) is provided in a location accessible by the information processing device 1. The living / non-living determination conditions include, for example, the following conditions, <Condition (1)> to <Condition (6)>.

[0022] <Condition (1)> If facial features of a control target candidate are detected and the degree of match between the facial features of the control target candidate and the facial features of a dummy that has been trained in advance is equal to or greater than a first threshold, the control target candidate is determined to not exhibit biological features, and if facial features of the control target candidate are detected and the degree of match between the facial features of the control target candidate and the facial features of a dummy that has been trained in advance is less than a first threshold, the control target candidate is determined to exhibit biological features. If facial features of the control target candidate are not detected, the control target candidate is determined to not exhibit biological features.

[0023] <Condition (2)> If a facial part of a candidate to be controlled is detected and the degree of match between the facial features of the candidate to be controlled and the facial features of a person previously learned is equal to or greater than a second threshold, the candidate to be controlled is determined to exhibit a biometric feature, and if a facial part of the candidate to be controlled is detected and the degree of match between the facial features of the candidate to be controlled and the facial features of a person previously learned is less than the second threshold, the candidate to be controlled is determined to not exhibit a biometric feature. If a facial part of the candidate to be controlled is not detected, the candidate to be controlled is determined to not exhibit a biometric feature.

[0024] <Condition (3)> If facial features of a control target candidate are detected and biological movements are observed in the control target candidate based on the facial features of the control target candidate, the control target candidate is determined to exhibit biological features. If facial features of a control target candidate are detected and biological movements are not observed in the control target candidate based on the facial features of the control target candidate, the control target candidate is determined to not exhibit biological features. Biometric movements based on facial features are movements of the face or facial features seen in humans. More specifically, biological movements based on facial features include head movement (changes in head position or facial direction), eye movement, blinking, or pulse waves. If facial features of a control target candidate are not detected, the control target candidate is determined to not exhibit biological features.

[0025] <Condition (4)> If multiple skeleton points of the control target candidate are detected and the similarity between the positional relationship of each skeleton point and the positional relationship of each skeleton point of a preset person is equal to or greater than a third threshold, the control target candidate is determined to exhibit a biological feature, and if multiple skeleton points of the control target candidate are detected and the similarity between the positional relationship of each skeleton point and the positional relationship of each skeleton point of a preset person is less than the third threshold, the control target candidate is determined to not exhibit a biological feature. If multiple skeleton points of the control target candidate are not detected, the control target candidate is determined to not exhibit a biological feature.

[0026] <Condition (5)> If the skeleton points of the control target candidate are detected and biological movements of the control target candidate are observed from the skeleton points of the control target candidate, the control target candidate is determined to exhibit biological features. If the skeleton points of the control target candidate are detected and biological movements of the control target candidate are not observed from the skeleton points of the control target candidate, the control target candidate is determined to not exhibit biological features. Biological movements based on skeleton points include predetermined skeleton point movements or predetermined periodic skeleton point movements. If the skeleton points of the control target candidate are not detected, the control target candidate is determined to not exhibit biological features.

[0027] <Condition (6)> If the skeleton points of a control target candidate are detected and a feature specific to a dummy doll is detected at the skeleton points, the control target candidate is determined to not exhibit a biological feature, and if the skeleton points of a control target candidate are detected and a feature specific to a dummy doll is not detected at the skeleton points, the control target candidate is determined to exhibit a biological feature. If the skeleton points of a control target candidate are not detected, the control target candidate is determined to not exhibit a biological feature.

[0028] The first, second, and third thresholds are set in advance by developers or the like and stored in a storage unit (not shown). The facial features to be used in <Condition (1)> to <Condition (3)> are predetermined. For example, the predetermined facial features are the outer corners of the eyes, the inner corners of the eyes, the inner corners of the eyebrows, the corners of the mouth, or the tip of the nose. The skeleton points to be used in <Condition (4)> to <Condition (6)> are predetermined. For example, the predetermined skeleton points are the skeleton points representing both shoulders, both elbows, or the neck. The predetermined skeleton point movements or predetermined periodic skeleton point movements in <Condition (5)> are set in advance by developers or the like, and information regarding the movements is stored in a storage unit (not shown). The developers or the like predetermine skeleton point movements or periodic skeleton point movements that are expected to accompany human behaviors, such as natural human body movements or squirming movements, that are not seen in non-living objects such as dummies. Furthermore, the characteristic feature of a dummy in <Condition (6)> is, for example, that it is made of metal parts. A dummy may have exposed metal parts at the joints corresponding to skeletal points such as the neck, shoulders, or elbows (see, for example, Figure 2). In this way, a dummy may have a characteristic feature that a living being cannot have. By detecting this, a living being can be distinguished from a non-living being.

[0029] For example, when determining whether a control target candidate is a living or non-living subject in accordance with the above <Condition (1)>, the first processing unit 12 first detects facial features of the control target candidate based on the captured image. The first processing unit 12 may detect the facial features using, for example, a known image recognition technique that detects multiple facial feature points in the captured image based on the captured image. The detected facial features are represented, for example, by coordinates on the captured image. If the first processing unit 12 detects the facial features of the control target candidate, it calculates facial features of the control target candidate from the facial features and calculates the degree of match with the facial feature of a trained dummy. For example, if the facial feature is the inter-brow distance, the first processing unit 12 can calculate the inter-brow distance from the distance between the points representing both eyebrows on the captured image. The first processing unit 12 calculates the degree of match between the calculated inter-brow distance and the inter-brow distance of the trained dummy. The inter-brow distance of the trained dummy is pre-stored in a storage unit (not shown). Furthermore, for example, if the first processing unit 12 detects facial features of the control target candidate, the first processing unit 12 may input the captured image into a trained model in machine learning (hereinafter referred to as a "machine learning model") and obtain information indicating the degree of match with the facial features of a trained dummy, thereby calculating the degree of match. The machine learning model is a model that receives a captured image of a face and outputs information indicating the degree of match between the facial features of the captured image and the facial features of a dummy, and is generated in advance and stored in a storage unit (not shown). If the calculated degree of match is equal to or greater than a first threshold, the first processing unit 12 determines that the control target candidate does not exhibit biological features and is a non-biological target. On the other hand, if the calculated degree of match is less than the first threshold, the first processing unit 12 determines that the control target candidate exhibits biological features and is a living target, i.e., an occupant. Furthermore, if the first processing unit 12 does not detect facial features of the control target candidate, the first processing unit 12 determines that the control target candidate does not exhibit biological features and is a non-biological target. As mentioned above, the face and facial features of the dummy doll are not designed to be strictly matched to a human face and facial features. By comparing the facial features, it is determined whether the candidate control object exhibits biometric features.

[0030] For example, when determining whether a control target candidate is a living or non-living subject in accordance with the above <Condition (2)>, the first processing unit 12 first detects facial features of the control target candidate based on the captured image. If the first processing unit 12 detects the facial features of the control target candidate, it calculates facial features of the control target candidate from the facial features and calculates the degree of match with the facial features of a trained person. For example, the first processing unit 12 calculates the degree of match between the calculated inter-eyebrow distance and the inter-eyebrow distance of a trained person. Note that the inter-eyebrow distance of the trained person is stored in advance in a storage unit (not shown). Furthermore, for example, if the first processing unit 12 detects the facial features of the control target candidate, it may input the captured image into a machine learning model and obtain information indicating the degree of match with the facial features of a trained person, thereby calculating the degree of match. The machine learning model is a model that receives a captured image of a face as input and outputs information indicating the degree of match between the facial features of the captured image and the facial features of a person. The model is generated in advance and stored in a storage unit (not shown). If the calculated degree of match is equal to or greater than a second threshold, the first processing unit 12 determines that the control target candidate exhibits a biometric feature and is an occupant. On the other hand, if the calculated degree of match is less than the second threshold, the first processing unit 12 determines that the control target candidate does not exhibit a biometric feature and is a non-biological target. Furthermore, if no facial features are detected for the control target candidate, the first processing unit 12 determines that the control target candidate does not exhibit a biometric feature and is a non-biological target.

[0031] For example, when determining whether a control target candidate is a living or non-living subject in accordance with the above-mentioned <Condition (3)>, the first processing unit 12 first detects the facial features of the control target candidate based on the captured image. If the first processing unit 12 detects the facial features of the control target candidate, it determines whether the control target candidate exhibits biological movements (head movement, eye movement, blinking, or pulse wave) based on the facial features of the control target candidate. If the first processing unit 12 determines that the control target candidate exhibits biological movements based on the facial features of the control target candidate, it determines that the control target candidate exhibits biological movements and is an occupant. On the other hand, if the first processing unit 12 does not determine that the control target candidate exhibits biological movements based on the facial features of the control target candidate, it determines that the control target candidate does not exhibit biological movements and is a non-living subject. For example, the first processing unit 12 stores the captured images acquired from the information acquisition unit 11 in a chronological order in an internal buffer or the like. The first processing unit 12 can detect whether or not biological movements are observed in the control target candidate from the facial features of the control target candidate by using a known image recognition technique for the time-series captured images stored in an internal buffer, etc. Furthermore, if no facial features of the control target candidate are detected, the first processing unit 12 determines that the control target candidate does not exhibit biological features and determines that the control target candidate is non-biological.

[0032] For example, when determining whether a control target candidate is a living or non-living object according to the above <Condition (4)>, the first processing unit 12 first detects multiple skeleton points of the control target candidate based on the captured image. The first processing unit 12 may detect multiple skeleton points of the control target candidate using, for example, a known image recognition technique that detects multiple skeleton points on a captured image based on the captured image. The detected multiple skeleton points are represented, for example, by coordinates on the captured image. When the first processing unit 12 detects multiple skeleton points of the control target candidate, it calculates the positional relationship between the skeleton points. The first processing unit 12 then calculates the similarity between the calculated positional relationship between the skeleton points and the positional relationship of a previously set person. Furthermore, when the first processing unit 12 detects multiple skeleton points of the control target candidate, it may input the captured image into a machine learning model and obtain information indicating the similarity with the positional relationship of the skeleton points of a previously set person, thereby calculating the similarity. The machine learning model is a model that takes an image of a person as input, outputs information indicating the similarity between the positional relationship of each skeleton point of the person in the image and the positional relationship of each skeleton point of the person that has been previously set, and is generated in advance and stored in a storage unit (not shown). If the calculated similarity is equal to or greater than a third threshold, the first processing unit 12 determines that the control target candidate exhibits biological characteristics and determines that the control target candidate is an occupant. On the other hand, if the calculated degree of coincidence is less than the third threshold, the first processing unit 12 determines that the control target candidate does not exhibit biological characteristics and determines that the control target candidate is a non-biological object. Furthermore, if multiple skeleton points of the control target candidate are not detected, the first processing unit 12 determines that the control target candidate does not exhibit biological characteristics and determines that the control target candidate is a non-biological object.

[0033] For example, when determining whether a control target candidate is a living or non-living subject in accordance with the above <Condition (5)>, the first processing unit 12 first detects multiple skeleton points of the control target candidate based on the captured image. If the first processing unit 12 detects multiple skeleton points of the control target candidate, it determines whether the control target candidate exhibits biological movement (predetermined skeleton point movement or predetermined periodic skeleton point movement) from the skeleton points of the control target candidate. If the first processing unit 12 detects biological movement from the skeleton points of the control target candidate, it determines that the control target candidate exhibits biological characteristics and is a passenger. On the other hand, if the first processing unit 12 does not detect biological movement from the skeleton points of the control target candidate, it determines that the control target candidate does not exhibit biological characteristics and is a non-living subject. For example, the first processing unit 12 can use a known image recognition technique to detect whether the control target candidate exhibits biological movement from the skeleton points of the control target candidate using a time-series captured image stored in an internal buffer or the like. Furthermore, if multiple skeleton points of the control object candidate are not detected, the first processing unit 12 determines that the control object candidate exhibits biological characteristics and determines that the control object candidate is non-biological.

[0034] For example, when determining whether a control target candidate is a living or non-living object in accordance with the above <Condition (6)>, the first processing unit 12 first detects multiple skeletal points of the control target candidate based on the captured image. If the first processing unit 12 detects multiple skeletal points of the control target candidate, the first processing unit 12 detects characteristics specific to a dummy, such as metal, in a predetermined area of ​​the captured image that includes the skeletal points (hereinafter referred to as the "doll characteristic determination area"). If characteristics specific to a dummy are detected in the doll characteristic determination area, the first processing unit 12 determines that the control target candidate does not exhibit biological characteristics and is a non-living object. For example, the first processing unit 12 may detect metal using a known image recognition processing technique. Alternatively, for example, the first processing unit 12 may detect characteristics specific to a dummy by pattern matching with an image that exhibits characteristics specific to a dummy. For example, if characteristics specific to a dummy are not detected in the doll characteristic determination area, the first processing unit 12 determines that the control target candidate exhibits biological characteristics and is a passenger. Furthermore, if multiple skeleton points of the control object candidate are not detected, the first processing unit 12 determines that the control object candidate does not exhibit biological characteristics, and determines that the control object candidate is non-biological.

[0035] Note that the above-described conditions for determining whether a control target candidate exhibits biometric characteristics, such as <Condition (1)> to <Condition (6)>, are merely examples. Conditions other than <Condition (1)> to <Condition (6)> may be set as the conditions for determining whether a control target candidate exhibits biometric characteristics. Furthermore, the first processing unit 12 may combine multiple conditions set as the conditions for determining whether a control target candidate exhibits biometric characteristics.

[0036] If the first processing unit 12 determines that the control target candidate is a living organism, that is, if the first processing unit 12 determines that the control target candidate is an occupant, the first processing unit 12 designates the control target candidate as a control target, in other words, an occupant, and outputs information about the occupant (hereinafter referred to as "control target information") to the second processing unit 13. The control target information includes information indicating the existence of a control target and information about the feature amounts detected in the first processing. The information about the feature amounts detected in the first processing includes, for example, information indicating facial parts, facial feature amounts, biological movements based on the facial feature amounts, skeletal points, or biological movements of the skeletal points. The control target information may include, for example, a captured image acquired by the information acquisition unit 11.

[0037] When the first processing unit 12 determines that the control target candidate is a living body, the second processing unit 13 performs a process (hereinafter referred to as the "second determination process") in which the control target candidate is the control target, estimates the attribute or situation of the control target based on the control target information, and generates control information. Here, when the first processing unit 12 determines that the control target candidate is an occupant, the second processing unit 13 performs a second determination process in which the control target candidate is the occupant who is the control target, estimates the age of the occupant based on the control target information, and generates control information. For example, when the control target information is output from the first processing unit 12, the second processing unit 13 can determine that the first processing unit 12 has determined that the control target candidate is an occupant.

[0038] The second processing unit 13 may estimate the age of the occupant using a known technique for estimating a person's age. For example, the second processing unit 13 may estimate the age of the occupant using a known technique for estimating a person's age based on facial features. Note that information regarding the facial features is included in the control target information. The second processing unit 13 does not need to perform processing such as detecting facial features from a captured image or calculating facial features. Alternatively, the second processing unit 13 may estimate the facial features of the occupant using a machine learning model that inputs the facial features and outputs information indicating age, and then estimate the age of the occupant from the obtained information indicating age. After estimating the age of the occupant, the second processing unit 13 determines the deployment level or height of the airbag based on the estimated age of the occupant. The second processing unit 13 generates control information for controlling the airbag based on the determined deployment level or height of the airbag. Control conditions that define the airbag control content, such as the airbag opening level or height, depending on the estimated occupant age are generated in advance by an administrator or the like, and information indicating the control conditions (hereinafter referred to as "control condition information") is stored in a storage unit (not shown). The second processing unit 13 determines the airbag opening level or height, etc., according to the estimated occupant age by referring to the control condition information. The second processing unit 13 then outputs the generated control information to the control device 3. The control device 3 controls the airbag based on the control information output from the second processing unit 13.

[0039] The operation of the information processing device 1 according to embodiment 1 will be described. Fig. 3 is a flowchart for describing the operation of the information processing device 1 according to embodiment 1. For example, when the power of the vehicle is turned on and power is supplied to the information processing device 1, the information processing device 1 starts the operation shown in the flowchart of Fig. 3, and repeats the operation shown in the flowchart of Fig. 3 until the power of the vehicle is turned off and power is no longer supplied to the information processing device 1.

[0040] The information acquisition unit 11 acquires target information, in this case, a captured image of the interior of the vehicle, from the target information acquisition device 2 (step ST1). The information acquisition unit 11 outputs the acquired captured image to the first processing unit 12.

[0041] The first processing unit 12 performs a first determination process based on the captured image acquired by the information acquisition unit 11 in step ST1 (step ST2). If the first processing unit 12 determines that the control target candidate is a living body as a result of the first determination process, i.e., if the first processing unit 12 determines that the control target candidate is an occupant (if "YES" in step ST3), the first processing unit 12 sets the control target candidate to the occupant who is the control target, and outputs the control target information to the second processing unit 13. If the first processing unit 12 determines that the control target candidate is a non-living body as a result of the first determination process (if "NO" in step ST3), the first processing unit 12 does not output the control target information to the second processing unit 13. Then, the operation of the information processing device 1 skips the processing of step ST4.

[0042] When the first processing unit 12 determines that the control target candidate is a living body ("YES" in step ST3), the second processing unit 13 determines that the control target candidate is an occupant who is a control target, and performs a second determination process (step ST4). The second processing unit 13 generates control information and outputs the generated control information to the control device 3.

[0043] FIG. 4 is a flowchart for explaining in more detail the operation of first processing unit 12 in step ST2 of FIG.

[0044] The first processing unit 12 determines whether the control target candidate exhibits a biometric characteristic based on the captured image and in accordance with the living / non-living condition (step ST11).

[0045] If it is determined in step ST11 that the control target candidate exhibits a biological characteristic (YES in step ST11), the first processing unit 12 determines that the control target candidate is a living body, in other words, an occupant (step ST12).Then, the first processing unit 12 outputs the control target information to the second processing unit 13 (step ST14).

[0046] On the other hand, if it is determined in step ST11 that the control target candidate does not exhibit a biometric characteristic ("NO" in step ST11), the first processing unit 12 determines that the control target candidate is non-biometric (step ST13). Then, the first processing unit 12 skips step ST14, does not output the control target information to the second processing unit 13, and ends the first determination process.

[0047] FIG. 5 is a flowchart for explaining in more detail the operation of second processing unit 13 in step ST4 of FIG.

[0048] The second processing unit 13 acquires the control object information output from the first processing unit 12 (step ST21). The second processing unit 13 estimates the attribute or situation of the control object, in this case, the age of the occupant, based on the control object information acquired in step ST21 (step ST22), and generates control information (step ST23). The second processing unit 13 then outputs the control information generated in step ST23 to the control device 3 (step ST24).

[0049] In this way, the information processing device 1 performs a process (first determination process) to determine whether the control target candidate is an occupant based on the target information acquired from the target information acquisition device 2. If the information processing device 1 determines that the control target candidate is an occupant, it estimates the age of the occupant and generates control information. If the information processing device 1 determines that the control target candidate is not an occupant, in other words, is a non-living body, it does not perform the process (second determination process) to estimate the occupant's age. This allows the information processing device 1 to provide control information that is generated with high accuracy and reduces unnecessary processing load for estimating the occupant's age. As a result, the information processing device 1 can prevent erroneous airbag control according to the occupant's age.

[0050] As described above, a more detailed determination is required to more accurately determine whether a control target candidate is an occupant or a non-living object such as a dummy. On the other hand, if a control target candidate is a non-living object, if an attempt is made to determine whether the control target candidate is an occupant or a non-living object during the process of estimating the age of the control target candidate to generate control information to be used for subsequent airbag control, age estimation for a non-living object may be performed, which may result in unnecessary, high-load processing that is unnecessary for subsequent airbag control. In response to this, the information processing device 1 separates a first determination process for determining whether the control target candidate is an occupant or a non-living object such as a dummy from a second determination process for estimating the age of the occupant and generating control information. In the first determination process, a more detailed determination of whether the control target candidate is an occupant or a non-living object is performed based on whether the control target candidate exhibits biometric characteristics. Only when the first determination process determines that the control target candidate is an occupant is the second determination process performed, which estimates the age of the occupant and generates control information. In general, estimation processes using machine learning models and biological reaction detection processes require more complex calculations, and are therefore expected to tend to impose a higher processing load than simple image processing.

[0051] Therefore, the information processing device 1 can provide control information that is generated with high accuracy while minimizing unnecessary processing load for estimating the age of the occupant.

[0052] In the first embodiment, in the information processing device 1, the attribute or situation of the occupant estimated by the second processing unit 13 is age, and the control performed by the control device 3 is airbag control. However, this is merely an example. For example, the control performed by the control device 3 may be seat belt control, seat position control, or seat reclining control. Furthermore, for example, the control performed by the control device 3 may be air conditioning or sound control in the vehicle cabin. Furthermore, for example, the attribute or situation of the occupant estimated by the second processing unit 13 may be the occupant's emotion or physique. The second processing unit 13 can estimate the occupant's emotion from a pulse wave, for example, using known technology. Furthermore, the second processing unit 13 can estimate the occupant's physique from skeleton points, for example, using known technology.

[0053] In the first embodiment, the target information acquisition device 2 is an imaging device, but this is merely an example. The target information acquisition device 2 may be, for example, a millimeter-wave sensor. The millimeter-wave sensor emits radio waves toward an area where an occupant should be present, in this case, the vehicle interior, and detects the control target candidate based on the reflected waves from the emitted radio waves reflected by the control target candidate, including the occupant present in the vehicle interior. The millimeter-wave sensor can detect the distance to the control target candidate, the angle, and the speed of the control target candidate. The millimeter-wave sensor outputs information about the detected object (hereinafter referred to as "object detection information") to the information processing device 1 as target information. In the information processing device 1, the information acquisition unit 11 acquires the object detection information, and the first processing unit 12 determines whether the control target candidate is an occupant or a non-living object based on the object detection information and whether the control target candidate exhibits a biometric characteristic. In this case, the living / non-living object determination condition may be set to, for example, the following <Condition (7)>.

[0054] <Condition (7)> ​​If a biological reaction of a control target candidate is detected, the control target candidate is determined to exhibit a biological characteristic, and if a biological reaction of a control target candidate is not detected, the control target candidate is determined to not exhibit a biological characteristic. The biological reaction of a control target candidate includes a heartbeat or respiration.

[0055] Based on the object detection information, the first processing unit 12 determines whether the movement indicated in the object detection information is estimated to be a heartbeat or breathing. The millimeter-wave sensor detects minute movements on the surface of the body. When a person breathes, the chest and abdomen move slightly. The millimeter-wave sensor can detect these minute movements. The first processing unit 12 simply detects the heartbeat or breathing from the minute movements detected by the millimeter-wave sensor based on the object detection information. Since the method of detecting the heartbeat or breathing from the minute movements detected by the millimeter-wave sensor is a well-known method, a detailed description will be omitted. In this case, the second processing unit 13, for example, calculates the occupant's heart rate or breathing rate and estimates the occupant's attributes or situation from the calculated heart rate or breathing rate.

[0056] Furthermore, in the above-described first embodiment, a plurality of target information acquisition devices 2 may be connected to the information processing device 1. For example, an imaging device and a millimeter wave sensor may be connected to the information processing device 1, and the information processing device 1 may perform the first determination process or the second determination process based on the captured image and the object detection information.

[0057] In the first embodiment described above, the information processing device 1 is an in-vehicle device, and the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 are provided in the in-vehicle device. However, the present invention is not limited to this. Some of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 may be provided in the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, and the in-vehicle device and the server may form an information processing system. Furthermore, the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 may all be provided in the server.

[0058] Furthermore, in the above-described first embodiment, the living body to be controlled is a vehicle occupant, but this is merely an example. For example, the living body to be controlled may be an occupant of a moving body other than a vehicle, such as a ship or an airplane, or a person in a factory, a room, or a living space. The control device 3 may be a control device that performs control according to the attributes or situation of a living body in any situation. For example, the control device 3 may be a control device that controls the temperature or direction of an air conditioner in a living space according to the age of a person present in the room. The information processing device 1 may be an information processing device that provides control information for performing various controls according to the attributes or situations of the controlled body performed by the control device 3.

[0059] 6A and 6B are diagrams illustrating an example of the hardware configuration of the information processing device 1 according to the first embodiment. In the first embodiment, the functions of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 are realized by a processing circuit 101. That is, the information processing device 1 includes a processing circuit 101 for performing a first determination process for determining whether a control target candidate is a living or non-living object, and a second determination process only when the first determination process determines that the control target candidate is a living object, thereby generating control information. The processing circuit 101 may be dedicated hardware as shown in FIG. 6A, or a processor 104 that executes a program stored in a memory as shown in FIG. 6B.

[0060] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0061] When the processing circuit is a processor 104, the functions of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. The processor 104 executes the functions of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 by reading and executing the program stored in memory 105. That is, the information processing device 1 includes memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST4 of FIG. 3 described above. It can also be said that the program stored in memory 105 causes a computer to execute the processing procedures or methods of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13. Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0062] The functions of the information acquisition unit 11, the first processing unit 12, and the second processing unit 13 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the information acquisition unit 11 may be implemented by a processing circuit 101 as dedicated hardware, and the first processing unit 12 and the second processing unit 13 may be implemented by a processor 104 reading and executing a program stored in a memory 105. The storage unit (not shown) may be configured, for example, with the memory 105 or a HDD. The information processing device 1 also includes an input interface device 102 and an output interface device 103 that communicate with devices such as the target information acquisition device 2 or the control device 3 via wired or wireless communication.

[0063] As described above, according to the first embodiment, the information processing device 1 provides control information for performing control according to the attributes or circumstances of a living organism to be controlled. The information processing device 1 includes an information acquisition unit 11 that acquires target information about a control target candidate from a target information acquisition device 2, a first processing unit 12 that performs a first determination process to determine whether the control target candidate is a living organism based on whether the control target candidate exhibits biometric characteristics based on the target information acquired by the information acquisition unit 11, and a second processing unit 13 that, if the first processing unit 12 determines that the control target candidate is a living organism, designates the control target candidate as a control target and performs a second determination process to estimate the attributes or circumstances of the control target and generate control information. This allows the information processing device 1 to provide control information that is generated with high accuracy and reduces unnecessary processing load for estimating the attributes or circumstances of the living organism. As a result, the information processing device 1 can prevent erroneous control according to the attributes or circumstances of the living organism.

[0064] In the information processing device 1, if the first processing unit 12 determines that the control target candidate is not a living organism, the second processing unit 13 determines that the control target candidate is not a control target and does not perform the second determination process. This allows the information processing device 1 to provide control information that is generated with high accuracy and reduces unnecessary processing load for estimating the attributes or situation of the living organism. As a result, the information processing device 1 can prevent erroneous control according to the attributes or situation of the living organism.

[0065] Furthermore, the target information acquisition device 2 includes an imaging device, and the target information is an image of the control target candidate captured by the imaging device. In the information processing device 1, the first processing unit 12 detects movements of the control target candidate, including blinking, eye movement, head movement, or pulse wave, as biological characteristics based on the captured image in the first determination process, and determines whether the control target candidate is a living organism. This allows the information processing device 1 to accurately determine whether the control target is a living organism. As a result, the information processing device 1 can prevent erroneous control based on the attributes or situation of the living organism.

[0066] Furthermore, the target information acquisition device 2 includes an imaging device, and the target information is an image captured by the imaging device of the control target. In the information processing device 1, the first processing unit 12, in the first determination process, detects, based on the captured image, movements of the skeleton points of the control target candidate that are expected to accompany actions that a person might perform, or periodic movements of the skeleton points, as biological characteristics, and determines whether the control target candidate is a living organism. This allows the information processing device 1 to accurately determine whether the control target is a living organism. As a result, the information processing device 1 can prevent erroneous control based on the attributes or situation of the living organism.

[0067] Furthermore, the target information acquisition device 2 includes a millimeter wave sensor, and the target information is object detection information obtained by the millimeter wave sensor detecting a control target candidate. In the information processing device 1, the first processing unit 12 detects the heartbeat or breathing of the control target candidate as a biological characteristic based on the object detection information in the first determination process, and determines whether the control target candidate is a living organism. This allows the information processing device 1 to accurately determine whether the control target is a living organism. As a result, the information processing device 1 can prevent erroneous control based on the attributes or situation of the living organism.

[0068] Furthermore, in the information processing device 1, the second processing unit 13 is configured to estimate the age, emotion, or physique of the control target as the attribute or situation of the control target in the second determination process. Only when the control target candidate is determined to be a living organism in the first determination process is the age, emotion, or physique estimated as the attribute or situation of the control target. Therefore, the information processing device 1 can prevent the generation of control information that could result in erroneous control due to an erroneous estimation of the age, emotion, or physique. As a result, the information processing device 1 can prevent erroneous control according to the attribute or situation of the living organism.

[0069] In addition, in the present disclosure, any of the components of the embodiments may be modified or omitted.

[0070] The information processing device of the present disclosure is applied to an information processing device that provides control information for performing control according to the attributes or situation of the living body to be controlled, and can provide control information that is highly accurate and generated while reducing unnecessary processing load for estimating the attributes or situation of the living body.

[0071] REFERENCE SIGNS LIST 1 information processing device, 11 information acquisition unit, 12 first processing unit, 13 second processing unit, 2 target information acquisition device, 3 control device, 101 processing circuit, 102 input interface device, 103 output interface device, 104 processor, 105 memory

Claims

1. An information processing device that provides control information for performing control according to the attributes or situation of a living organism to be controlled, comprising: an information acquisition unit that acquires target information regarding a control target candidate from a target information acquisition device; a first processing unit that performs a first determination process to determine whether the control target candidate is the living organism based on the target information acquired by the information acquisition unit, depending on whether the control target candidate exhibits biological characteristics; and a second processing unit that performs a second determination process, when the first processing unit determines that the control target candidate is the living organism, to designate the control target candidate as the control target, estimate the attributes or situation of the control target, and generate the control information.

2. The information processing device described in claim 1, characterized in that if the first processing unit determines that the control target candidate is not the living body, the second processing unit determines that the control target candidate is not the control target and does not perform the second judgment process.

3. The information processing device according to claim 1 or claim 2, characterized in that the target information acquisition device includes an imaging device, the target information is an image of the control target candidate captured by the imaging device, and the first processing unit, in the first determination process, detects movements of the control target candidate, including blinking, eye movement, head movement, or pulse wave, as the biological characteristics based on the captured image, and determines whether the control target candidate is the living organism.

4. The information processing device according to claim 1 or claim 2, characterized in that the target information acquisition device includes an imaging device, the target information is an image captured by the imaging device of the control target, and the first processing unit, in the first determination process, detects, based on the captured image, movements of the skeleton points of the control target candidate that are expected to accompany actions that a person may perform, or periodic movements of the skeleton points, as the biological characteristics, and determines whether the control target candidate is the living organism.

5. The information processing device according to claim 1 or claim 2, characterized in that the target information acquisition device includes a millimeter wave sensor, the target information is object detection information of the control target candidate detected by the millimeter wave sensor, and the first processing unit, in the first determination process, detects the heartbeat or breathing of the control target candidate as the biological characteristic based on the object detection information, and determines whether the control target candidate is the living organism.

6. An information processing device according to claim 1 or claim 2, characterized in that in the second judgment process, the second processing unit estimates the age, emotion, or physique of the controlled object as the attribute or situation of the controlled object.

7. The information processing device according to claim 1, wherein the controlled object is a vehicle occupant.

8. An information processing method for providing control information for performing control according to the attributes or situation of a living organism to be controlled, comprising: a step in which an information acquisition unit acquires target information regarding a control target candidate from a target information acquisition device; a step in which a first processing unit performs a first determination process in which, based on the target information acquired by the information acquisition unit, the control target candidate is determined to be the living organism depending on whether or not the control target candidate exhibits biological characteristics; and a step in which a second processing unit performs a second determination process in which, when the first processing unit determines that the control target candidate is the living organism, the control target candidate is designated as the control target, and the attribute or situation of the control target is estimated to generate the control information.