Data classification device and data classification program

Abstract

To provide a data classification device capable of preventing deterioration of classification accuracy of an object.SOLUTION: An object classification device includes a data acquisition unit (101) that acquires sensor data from a sensor (20), a data processing unit (104) that generates classification data from the sensor data acquired by the data acquisition unit (101) using representative shape data that is sensor data indicating a feature of a representative shape of an object belonging to a class to be classified, and a classification unit (105) that classifies the object on the basis of the classification data generated by the data processing unit (104).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a data classification device and a data classification program. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for classifying objects based on data obtained from a sensor such as a millimeter wave sensor that can detect the three-dimensional position and shape of the object (hereinafter referred to as "sensor data"). Incidentally, Patent Document 1 discloses an image processing device that distinguishes between adults and children in an image captured by a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-204053 Summary of the Invention [Problem to be solved by the invention]

[0004] Because the radio waves or light emitted by sensors can penetrate objects made of materials other than metal, sensors can detect objects hidden behind shields, etc. However, sensors have characteristics such as low resolution or detection position errors of several centimeters. Therefore, when classifying objects based on sensor data obtained from sensors, there is a problem in that if the sensor data is used as is, the accuracy of object classification may decrease due to the above-mentioned characteristics of the sensors. The technology disclosed in Patent Document 1 as mentioned above is a technology for distinguishing between adults and children from images captured by a camera, and does not take into consideration the effects of the characteristics of the sensors as mentioned above, and therefore cannot solve the above-mentioned problems.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a data classification device that classifies objects based on sensor data acquired from a sensor that can detect the three-dimensional position and shape of the object, and that can prevent a decrease in object classification accuracy. [Means for solving the problem]

[0006] The data classification device disclosed herein is a data classification device that performs classifying of objects based on sensor data acquired from a sensor that can detect the three-dimensional position and shape of the object, and includes a data acquisition unit that acquires the sensor data from the sensor, a data processing unit that generates classification data from the sensor data acquired by the data acquisition unit using representative shape data, which is sensor data that indicates the characteristics of the representative shape of objects that belong to a class to be classified, and a classification unit that classifies the objects based on the classification data generated by the data processing unit. [Effects of the Invention]

[0007] According to the present disclosure, the data classification device is configured as described above, and therefore can prevent a decrease in object classification accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a data classification system including a representative shape data generating device and a data classification device according to a first embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of a representative shape data generating device according to a first embodiment. [Figure 3] 1 is a diagram illustrating an example of a configuration of a data classification device according to a first embodiment. [Figure 4]4A and 4B are diagrams showing an example of how the representative shape data generating unit generates representative shape data based on learning data in the representative shape data generating device of embodiment 1; FIG. 4A is a diagram showing an example of how the representative shape data generating unit generates person representative shape data; FIG. 4B is a diagram showing an example of how the representative shape data generating unit generates adult representative shape data; and FIG. 4C is a diagram showing an example of how the representative shape data generating unit generates infant representative shape data. [Figure 5] 4 is a diagram for explaining an example of a method for generating representative shape data by a representative shape data generating unit in the representative shape data generating device according to the first embodiment. FIG. [Figure 6] FIG. 2 is a diagram illustrating an overview of a complementing process performed by a complementing unit in the data classifying device according to the first embodiment. [Figure 7] FIG. 4 is a diagram for explaining details of a complementing process performed by a complementing unit in the data classifying device according to the first embodiment. [Figure 8] 4 is a diagram for explaining details of a difference acquisition process performed by a difference acquisition unit in the data classifying device according to the first embodiment. FIG. [Figure 9] 4 is a diagram for explaining difference data obtained by a difference obtaining unit performing a difference obtaining process in the first embodiment. FIG. [Figure 10] 4 is a flowchart illustrating the operation of the representative shape data generating device according to the first embodiment. [Figure 11] 4 is a flowchart illustrating an operation of the data classifying device according to the first embodiment. [Figure 12] 12 is a flowchart for explaining an example of detailed operations of the data classification device in steps ST30 to ST40 of FIG. [Figure 13] 13A and 13B are diagrams illustrating an example of the hardware configuration of the data classifying device and the representative shape data generating device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0010] Embodiment 1 The data classification device according to the first embodiment classifies objects into classes based on data acquired from a sensor capable of detecting the three-dimensional position and shape of the object (hereinafter referred to as "sensor data").

[0011] In the following first embodiment, as an example, the sensor is assumed to be a sensor mounted on a vehicle and detecting an object inside the vehicle. Specifically, the sensor is assumed to be a millimeter wave sensor. The sensor is a general millimeter wave sensor, and its outline will be briefly described below. The sensor transmits radio waves (millimeter waves) into the vehicle cabin and receives the waves reflected by objects in the cabin, including moving objects such as people. The sensor extracts the distance, angle, speed, etc., to an object, more specifically, a moving object, based on the received reflected waves, and generates three-dimensional grid data. In the first embodiment, the three-dimensional grid data is data representing a three-dimensional spatial distribution of moving objects present in a target space, in this case, the vehicle interior. In other words, the three-dimensional grid data represents the distribution of areas in the vehicle interior where moving objects exist in three dimensions. More specifically, the three-dimensional grid data represents the minute movements of moving objects, in other words, objects that reflected radio waves transmitted by the sensor, in the vehicle interior, using a plurality of grids that correspond to the reflection points of the radio waves in three-dimensional space. Data such as the intensity or distance at which the radio waves were reflected is recorded in each grid. As a result, the three-dimensional position and shape of the moving object are represented as point cloud data. The sensor can also convert the generated 3D grid data into a 2D image using known methods such as maximum intensity projection or slice display, thereby generating data that represents the 3D position and shape of an object in a 2D image. In the following first embodiment, as an example, it is assumed that the sensor data acquired from the sensor is data in which the three-dimensional position and shape of an object are expressed as a two-dimensional image.

[0012] In addition, in the following embodiment 1, as an example, the data classification device classifies the occupants in the vehicle cabin, more specifically, the physiques of the occupants in the vehicle cabin, and the classes to be classified are "adult" or "child." That is, in the following embodiment 1, the data classification device acquires sensor data from a sensor in which the three-dimensional position and shape of objects in the vehicle cabin are represented as two-dimensional images, and based on the acquired sensor data, performs two-class classification of occupants in the vehicle cabin as either "adult" or "child."

[0013] The radio waves transmitted from the sensor can penetrate non-metallic objects such as seats in the vehicle interior, allowing the sensor to detect, for example, rear seat occupants who are hidden by the front seats. On the other hand, sensors have low resolution, especially when installed in vehicles. Furthermore, as mentioned above, sensors can only detect moving objects. Furthermore, due to frequency bandwidth limitations, antenna characteristics, multipath interference, or the reflection characteristics and surface shape of the object, errors of several centimeters may occur in the position of an object detected by the sensor. Due to these sensor characteristics, the sensor data obtained from the sensor may contain many missing values. Furthermore, the sensor data obtained from the sensor may contain a large proportion of common physical features between each class to be classified, in this case, "adults" and "infants." Therefore, when classifying sensor data obtained from a sensor, if the sensor data is used as is, the classification accuracy, in this case the accuracy of classifying whether the vehicle occupant is an adult or a child, may decrease.

[0014] Taking the above-mentioned problems into consideration, the data classification device according to the first embodiment performs data processing on sensor data obtained from a sensor to generate data (hereinafter referred to as "classification data") from the sensor data that has no missing values ​​and reduces the proportion of physical features common to "adults" and "infants," and performs class classification based on the generated classification data. This prevents the data classifying device from decreasing the accuracy of classifying whether a vehicle occupant is an "adult" or a "child."

[0015] FIG. 1 is a diagram showing an example of the configuration of a data classification system 1 including a representative shape data generating device 30 and a data classification device 10 according to the first embodiment. FIG. 2 is a diagram showing an example of the configuration of the representative shape data generating device 30 according to the first embodiment. FIG. 3 is a diagram illustrating an example of a configuration of data classifying device 10 according to the first embodiment.

[0016] As shown in FIG. 1, a data classification system 1 is configured by a data classification device 10, a sensor 20, a representative shape data generation device 30, and a storage device 40. The data classifying device 10 is connected to the sensor 20 and the storage device 40 via a network. The representative shape data generating device 30 is connected to the storage device 40 via a network. The data classifying device 10 and the representative shape data generating device 30 may be connected to each other. Note that the arrow connecting the data classifying device 10 and the representative shape data generating device 30 is omitted from FIG. 1 . The data classifying device 10, the sensor 20, the representative shape data generating device 30, and the storage device 40 are mounted on, for example, a vehicle.

[0017] Representative shape data generation device 30 generates sensor data (hereinafter referred to as "representative shape data") that indicates the characteristics of the representative shapes of objects (here, vehicle occupants including adults and infants) that belong to a class (here, "adult" or "infant") that is the target of classification performed by data classification device 10. In embodiment 1, the representative shape, more specifically, refers to a typical shape of an object that belongs to a class that is the target of classification performed by data classification device 10. In the first embodiment, "indicating the typical shape characteristics of an object" means that values ​​that should normally be recorded in the data, in this case values ​​that indicate the shape characteristics of the object, are recorded. It is preferable that the representative shape data contains just the right amount of shape characteristics of objects belonging to classes that are the targets of classification by the data classification device 10, i.e., it contains all the values ​​that should normally be included, but this is not essential. The representative shape data may include representative shape data that is sensor data that shows the representative shape characteristics of objects belonging to all classes that are the subject of class classification (hereinafter referred to as "all-class representative shape data"), and representative shape data that is sensor data that shows the representative shape characteristics of objects that belong to each class that is the subject of class classification (hereinafter referred to as "class-specific representative shape data").

[0018] Here, the all-class representative shape data is sensor data that indicates the representative shape characteristics of vehicle occupants, including adults and infants, more specifically, the typical shape characteristics of occupants, and the class-specific representative shape data is sensor data that indicates the representative shape characteristics of adults, more specifically, the typical shape characteristics of adults, and the representative shape characteristics of infants, more specifically, the typical shape characteristics of infants. In the first embodiment, the all-class representative shape data is also referred to as “person representative shape data.” The person representative shape data is representative shape data that can be considered to represent the characteristics of the representative shapes of vehicle occupants (hereinafter referred to as “person representative shapes”) belonging to all classes that are the subject of classification, i.e., adults and children. In addition, in the first embodiment, the class-specific representative shape data representing adults is also referred to as “adult representative shape data.” The adult representative shape data is representative shape data that can be considered to represent the characteristics of the representative shape of an occupant belonging to the adult class, which is the target class for classification (hereinafter referred to as “adult representative shape”). The class-specific representative shape data representing the infant is also referred to as "infant representative shape data." The infant representative shape data is representative shape data that can be considered to represent the characteristics of the representative shape of an occupant belonging to the infant class (hereinafter referred to as "infant representative shape") that is the target of classification.

[0019] The representative shape data generating device 30 stores the generated representative shape data, more specifically, the person representative shape data, adult representative shape data, and child representative shape data, in the storage device 40. A detailed configuration example of the representative shape data generating device 30 and a detailed method of generating representative shape data by the representative shape data generating device 30 will be described later.

[0020] The data classifying device 10 acquires sensor data from the sensor 20, and generates classification data from the acquired sensor data using the representative shape data generated by the representative shape data generating device 30 and stored in the storage device 40. Then, the data classifying device 10 classifies the vehicle occupant as an adult or a child based on the generated classification data. A detailed configuration example of data classifying device 10 and a detailed class classification method performed by data classifying device 10 will be described later.

[0021] An example configuration of the data classifying device 10 and the representative shape data generating device 30 according to the first embodiment will be described in detail. First, a detailed description will be given of an example of the configuration of the representative shape data generating device 30 according to the first embodiment. As shown in FIG. 2, the representative shape data generating device 30 includes a learning data acquiring unit 301, a representative shape data generating unit 302, and a representative shape data output unit 303.

[0022] The learning data acquisition unit 301 acquires learning data from a learning data storage unit (not shown) included in the storage device 40. The storage device 40 is configured with a memory or a HDD, etc., and includes a learning data storage unit, a representative shape data storage unit (not shown), and a model storage unit (not shown). The representative shape data storage unit and the model storage unit will be described later.

[0023] The learning data is data generated based on sensor data previously acquired from the sensor 20, which indicates objects belonging to each class to be classified. Here, the learning data is data generated based on sensor data previously acquired from the sensor 20, which indicates occupants including adults and children. The sensor data previously acquired from the sensor 20 may be, for example, sensor data acquired by test driving a vehicle with an occupant on board, or sensor data acquired from a vehicle that is actually running.

[0024] The learning data is sensor data to which a label indicating whether the sensor data indicates an adult or a child is assigned, and is generated by a developer or the like and stored in the learning data storage unit of the storage device 40. For example, the developer or the like generates learning data by labeling sensor data indicating an occupant acquired from the sensor 20 during a certain period in the past, indicating whether the occupant is an adult or a child. Note that, here, the learning data is data in which a label indicating whether the sensor data indicates an adult or an infant is assigned to the sensor data, but this is merely an example, and the learning data may be data in which a label indicating an age group is assigned to the sensor data. The assigned label may be a label that can be used to identify an object as belonging to each class to be classified, in this case, "adult" or "infant."

[0025] In embodiment 1, the learning data storage unit of storage device 40 stores only learning data that is sensor data to which a label indicating an adult has been assigned (hereinafter referred to as "adult learning data") and learning data that is sensor data to which a label indicating an infant has been assigned (hereinafter referred to as "infant learning data"), and does not store learning data that is sensor data to which a label indicating an object other than an adult or an infant has been assigned. Both adult learning data and infant learning data are stored in the learning data storage unit of storage device 40. The developers etc. generate multiple pieces of learning data for objects that are not to be classified by data classification device 10 and belong to each class, here, “adult” or “infant”, and store them in the learning data storage unit of storage device 40.

[0026] The learning data acquisition unit 301 acquires all learning data stored in the learning data storage unit of the storage device 40 . The learning data acquisition unit 301 outputs the acquired learning data to the representative shape data generation unit 302 .

[0027] The representative shape data generating unit 302 generates representative shape data based on the learning data acquired by the learning data acquiring unit 301 . Specifically, the representative shape data generation unit 302 generates person representative shape data, which is representative shape data for all classes, and adult representative shape data and infant representative shape data, which are representative shape data by class, based on the learning data acquired by the learning data acquisition unit 301.

[0028] Here, the generation of representative shape data by the representative shape data generating unit 302 will be described with reference to the drawings. FIG. 4 is a diagram showing an example in which representative shape data generating unit 302 generates representative shape data based on training data in representative shape data generating device 30 according to the first embodiment. Figure 4A is a diagram showing an example of how the representative shape data generation unit 302 generates person representative shape data, Figure 4B is a diagram showing an example of how the representative shape data generation unit 302 generates adult representative shape data, and Figure 4C is a diagram showing an example of how the representative shape data generation unit 302 generates child representative shape data.

[0029] The representative shape data generation unit 302 generates person representative shape data, which is representative shape data for all classes, based on the learning data generated based on sensor data indicating objects belonging to each class to be classified, which the learning data acquisition unit 301 acquired from the learning data storage unit of the storage device 40, in this case, occupants belonging to the adult or child category. Here, all of the training data acquired by the training data acquisition unit 301 is adult training data or child training data. Therefore, the representative shape data generation unit 302 only needs to generate person representative shape data based on all of the training data acquired by the training data acquisition unit 301 (see FIG. 4A). For example, if the learning data storage unit of the storage device 40 stores learning data generated based on sensor data indicating an occupant belonging to an adult or an infant, as well as learning data generated based on sensor data indicating an object other than an occupant belonging to an adult or an infant, the representative shape data generation unit 302 can identify whether the learning data is adult learning data or infant learning data from the label assigned to the sensor data in the learning data, extract the identified sensor data, and use it to generate person representative shape data.

[0030] The representative shape data generation unit 302 extracts learning data (hereinafter referred to as "class-specific representative shape data generation data") for generating class-specific representative shape data in accordance with extraction conditions from the learning data (here, all learning data) generated based on sensor data indicating objects belonging to each class to be classified, which learning data acquisition unit 301 acquired from the learning data storage unit of storage device 40, and generates class-specific representative shape data, here, adult substitute shape data and infant representative shape data, based on the extracted class-specific representative shape data generation data. The extraction conditions are conditions that define what learning data is used to generate representative shape data for each class from among the learning data acquired from the learning data storage unit of the storage device 40. The extraction conditions are set in advance by a developer or the like. When the developer or the like sets the extraction conditions, data indicating the set extraction conditions (hereinafter referred to as "extraction condition data") is stored in, for example, the storage device 40. The extraction condition data may be updateable as appropriate.

[0031] The extraction conditions are set, for example, as follows: <Extraction conditions (1)> In the learning data, adult learning data in which a label indicating an adult is added to the sensor data is extracted as data for generating representative shape data by class for creating adult representative shape data. <Extraction conditions (2)> In the learning data, infant learning data in which a label indicating an infant is added to the sensor data is extracted as data for generating class-specific representative shape data for generating infant representative shape data.

[0032] The representative shape data generation unit 302 extracts adult learning data in which a label indicating an adult is assigned to the sensor data from the learning data acquired by the learning data acquisition unit 301 in accordance with the extraction conditions described above, and generates adult representative shape data based on the extracted adult learning data (see FIG. 4B). In addition, the representative shape data generation unit 302 extracts infant learning data in which a label indicating an infant is assigned to the sensor data from the learning data acquired by the learning data acquisition unit 301 in accordance with the extraction conditions described above, and generates infant representative shape data based on the extracted infant learning data (see Figure 4C).

[0033] It should be noted that, here, the extraction conditions are set to the above-mentioned <extraction condition (1)> and <extraction condition (2)>, but this is merely an example, and appropriate conditions are set for the extraction conditions depending on the contents of the learning data stored in the storage device 40 or the class-specific representative shape data to be generated. For example, the extraction conditions may be set as follows: "<Extraction condition (3)> In the learning data, sensor data that has been assigned a label indicating that the data is under 6 years old is extracted as data for generating class-specific representative shape data for generating infant representative shape data." Or, the extraction condition may be set as follows: "<Extraction condition (4)> In the learning data, sensor data that has been assigned a label indicating that the data is a child seat is extracted as data for generating class-specific representative shape data for generating infant representative shape data."

[0034] The representative shape data generation method by the representative shape data generation unit 302 will be described below by giving an example with reference to the drawings. FIG. 5 is a diagram for explaining an example of a method for generating representative shape data by the representative shape data generating unit 302 in the representative shape data generating device 30 according to the first embodiment. 5 is a diagram showing an example of a method for generating person representative shape data by the representative shape data generating unit 302. In FIG.

[0035] As shown in FIG. 5, the representative shape data generation unit 302 generates the person representative shape data by, for example, taking the average value of the pixel values ​​of corresponding pixels in a plurality of learning data, in other words, all the learning data acquired by the learning data acquisition unit 301 from the storage device 40, as the pixel value of the corresponding pixel in the person representative shape data.

[0036] The representative shape data generating unit 302 may generate the person representative shape data by, for example, taking the most frequent pixel value of each corresponding pixel in a plurality of learning data, in other words, all learning data acquired by the learning data acquiring unit 301 from the storage device 40, as the pixel value of the corresponding pixel in the person representative shape data.

[0037] FIG. 5 shows an example of a method for generating person representative shape data by the representative shape data generating unit 302, but the representative shape data generating unit 302 may generate adult representative shape data or child representative shape data using a similar method. For example, the representative shape data generation unit 302 may generate adult representative shape data by taking the average value or most frequent value of the pixel values ​​of corresponding pixels in multiple adult learning data as the pixel value of the corresponding pixel in the adult representative shape data. Furthermore, for example, the representative shape data generation unit 302 may generate the infant representative shape data by taking the average value or the most frequent value of the pixel values ​​of corresponding pixels in the multiple infant learning data as the pixel value of the corresponding pixel in the infant representative shape data.

[0038] When the representative shape data generating unit 302 generates the representative shape data, the representative shape data output unit 303 stores the representative shape data generated by the representative shape data generating unit 302 in the representative shape data storage unit of the storage device 40 . The representative shape data output unit 303 assigns data to the representative shape data that can identify the type of the representative shape data (hereinafter referred to as ``type identification data''), more specifically, type identification data that can identify whether the representative shape data is person representative shape data, adult representative shape data, or child representative shape data, and stores the data in the representative shape data storage unit of the storage device 40.

[0039] Note that representative shape data generation unit 302 may have the functions of training data acquisition unit 301 and representative shape data output unit 303. In this case, representative shape data generation device 30 may be configured without training data acquisition unit 301 and representative shape data output unit 303.

[0040] Next, a configuration example of data classifying device 10 according to the first embodiment will be described in detail. As shown in FIG. 3, the data classification device 10 includes a data acquisition unit 101, a representative shape data acquisition unit 102, a process control unit 103, a data processing unit 104, a classification unit 105, and a classification result output unit 106.

[0041] The data acquisition unit 101 acquires sensor data from the sensor 20 . For example, the sensor data output from the sensor 20 may be stored in the storage device 40. In this case, the data acquiring unit 101 may acquire the sensor data output from the sensor 20 via the storage device 40. The data acquisition unit 101 outputs the acquired sensor data to the process control unit 103 .

[0042] The representative shape data acquisition unit 102 acquires representative shape data, more specifically, person representative shape data, adult representative shape data, and child representative shape data, from the representative shape data storage unit of the storage device 40 . The representative shape data acquisition unit 102 outputs the acquired representative shape data to the process control unit 103 .

[0043] The process control unit 103 controls the data processing unit 104 in accordance with the process control conditions. The processing control conditions are conditions that define what kind of processing the data processing unit 104 should perform in what cases. The process control conditions are set in advance by a developer or the like. Once the developer or the like sets the process control conditions, data indicating the set process control conditions (hereinafter referred to as "process control condition data") is stored in the storage device 40. The process control condition data may be updateable as appropriate. Details of the data processing unit 104 will be described later.

[0044] The process control conditions include, for example, the following <process control condition (1)> to <process control condition (3)>. <Processing control conditions (1)> When the data processing unit 104 performs the difference acquisition process, first, the data processing unit 104 performs the difference acquisition process using the adult representative shape data, and generates classification data based on the results of the difference acquisition process. <Processing control conditions (2)> If, as a result of having the data processing unit 104 perform a difference acquisition process in accordance with <Processing control condition (1)>, it is not possible to perform class classification based on the classification data generated by the difference acquisition process, the next time the data processing unit 104 performs a difference acquisition process using infant representative shape data, and generates classification data based on the results of the difference acquisition process. <Processing control conditions (3)> If the data processing unit 104 is caused to perform a difference acquisition process in accordance with <Processing control condition (2)> and class classification cannot be performed based on the classification data generated by the difference acquisition process, the data processing unit 104 is caused to generate classification data without performing the difference acquisition process.

[0045] Here, the processing control conditions are assumed to be set as the above-mentioned <Processing Control Condition (1)> to <Processing Control Condition (3)>, but this is merely an example. The processing control conditions may be conditions that define what processing the data processing unit 104 should perform in what cases.

[0046] The data processing unit 104 generates classification data from the sensor data acquired by the data acquisition unit 101, using the representative shape data acquired by the representative shape data acquisition unit 102 from the storage device 40. The data processing unit 104 may acquire the representative shape data, for example, via the processing control unit 103. The data processing unit 104 may acquire the sensor data, for example, via the processing control unit 103. The data processing unit 104 includes a complementing unit 1041 , a difference acquiring unit 1042 , and a classification data generating unit 1043 .

[0047] The complementing unit 1041 performs a complementing process to complement missing values ​​in the sensor data with the corresponding values ​​of all-class representative shape data, in this case, the person representative shape data. More specifically, the complementing unit 1041 performs a complementing process to complement pixel values ​​of an area in the sensor data where pixel values ​​are missing (hereinafter referred to as the "area to be complemented") with pixel values ​​of the corresponding area of ​​all-class representative shape data, in this case, the person representative shape data.

[0048] FIG. 6 is a diagram illustrating an outline of the complementing process performed by complementing section 1041 in data classifying apparatus 10 according to the first embodiment. As shown in FIG. 6, the complementing unit 1041 identifies a region to be complemented in the sensor data (indicated by "R" in FIG. 6), and complements the pixel values ​​of the identified region to be complemented with the pixel values ​​of a corresponding region in the person representative shape data (hereinafter referred to as a "corresponding region"; indicated by "P" in FIG. 6).

[0049] The method by which the complementing unit 1041 identifies the region to be complemented in the sensor data will be described below with some examples.

[0050] First, the complementing unit 1041 identifies, for example, an area in the sensor data where an object belonging to a class to be classified, in this case occupants including adults and children, should be shown (hereinafter referred to as an "occupant area"). For example, the complementing unit 1041 may identify, as the occupant area, an area in the sensor data corresponding to an area in the person representative shape data where an occupant is shown (hereinafter referred to as a "person representative area"). The person representative area is an area in which a person representative shape is shown in the person representative shape data. For example, in the vehicle interior, the area where an occupant is expected to be present can be estimated in advance as a predetermined area including the seat. Since the installation position and detection range of the sensor 20 are known, the area where an occupant is expected to be detected based on the sensor data, i.e., the predetermined area including the seat, is also known in advance. The complementing unit 1041 may identify the estimated area as the occupant area.

[0051] When the occupant region is identified, the complementing unit 1041, for example, compares the pixel value of each pixel included in the occupant region with the pixel value of each pixel included in the corresponding person representative region on the person representative shape data. As a result of comparing each pixel value, the complementing unit 1041 determines that pixels on the sensor data whose pixel value difference from the person representative shape data is equal to or greater than a preset threshold value (hereinafter referred to as a "first threshold value") are missing pixel values. Then, the complementing unit 1041 identifies an area on the sensor data consisting of pixels determined to be missing pixel values ​​as an area to be complemented. In addition, the complementing unit 1041 may determine that pixels included in the occupant area in the sensor data whose pixel values ​​are less than a predetermined threshold value (hereinafter referred to as the "second threshold value") have missing pixel values, and identify an area on the sensor data consisting of pixels whose pixel values ​​have been determined to be missing as an area to be complemented. The first and second thresholds are set in advance by a developer or the like and stored in the storage device 40. Note that the arrow connecting the data processing unit 104 and the storage device 40 is omitted from Fig. 3.

[0052] When the region to be complemented is identified in the above manner, the complementing unit 1041 complements the pixel values ​​of the region to be complemented with the pixel values ​​of the corresponding region. Then, the completion unit 1041 outputs the sensor data after the completion target region has been completed to the difference acquisition unit 1042. In the first embodiment, the sensor data after the completion target region has been completed by the completion unit 1041 is also referred to as "complemented data."

[0053] The complementing process performed by the complementing unit 1041 will be described in detail below using an example and the accompanying drawings. FIG. 7 is a diagram illustrating details of the complementing process performed by complementing section 1041 in data classifying apparatus 10 according to the first embodiment. In FIG. 7, the left diagram shows the sensor data before complementation, the middle diagram shows the sensor data after the region to be complemented has been complemented, and the right diagram shows the sensor data after the region to be complemented has been complemented and the pixel values ​​around the region to be complemented have been recalculated. In Fig. 7, the area to be complemented in the sensor data before complementation is indicated by "R1," the area to be complemented in the sensor data after the area to be complemented is indicated by "R2," and the area consisting of pixels around the area to be complemented whose pixel values ​​have been recalculated is indicated by "R3." It should be noted that the sensor data shown in Fig. 7 does not match the sensor data shown in Fig. 6. Furthermore, in Fig. 7, the representative shape data is omitted from the illustration.

[0054] In the complementing process, the complementing unit 1041 complements the area to be complemented by setting the pixel value of the pixel included in the area to be complemented to the pixel value obtained by adding the pixel value of the pixel and the pixel value of the pixel included in the corresponding area in the person representative shape data (see "R2" in Figure 7).

[0055] Here, as a result of adding the pixel values ​​of the pixels included in the region to be complemented to the pixel values ​​of the pixels in the corresponding region in the person representative shape data, the pixel values ​​of the pixels included in the region to be complemented and the pixel values ​​of the surrounding pixels may become significantly different. Therefore, in the complementing process, the complementing unit 1041 may recalculate the pixel values ​​of the pixels (hereinafter referred to as "surrounding pixels") surrounding the region to be complemented in the sensor data. The complementing unit 1041 recalculates the pixel values ​​of the surrounding pixels so that, as a result of complementing the region to be complemented, the difference between the pixel values ​​of the pixels included in the region to be complemented and the pixel values ​​of the surrounding pixels becomes smooth.

[0056] The complementation unit 1041 determines, according to a predetermined rule (hereinafter referred to as the "recalculation rule"), how much difference between the pixel values ​​of the surrounding pixels and the pixel values ​​of the pixels included in the area to be complemented needs to be before recalculating the pixel values ​​of the surrounding pixels. The recalculation rules are set in advance by a developer or the like, and data indicating the recalculation rules (hereinafter referred to as “recalculation rule data”) is stored in the storage device 40. The recalculation rule may, for example, state that "if the difference between the pixel value of the surrounding pixel and the pixel value of a pixel included in the area to be complemented that is adjacent or diagonally located is within a preset range, the pixel value of the surrounding pixel is recalculated." Data indicating the extent of the preset range is stored in the storage device 40 together with the recalculation rule data.

[0057] When the complementing unit 1041 determines that there are neighboring pixels whose pixel values ​​need to be recalculated according to the recalculation rules, it recalculates the pixel values ​​of the neighboring pixels. The rules for how to recalculate the pixel values ​​of the surrounding pixels are also set in the recalculation rules. For example, the recalculation rule states that "the pixel value of a pixel whose pixel value is determined to be recalculated (hereinafter referred to as a "pixel to be recalculated") shall be the average value of the pixel value of the pixel to be recalculated and the pixel value of a pixel included in the area to be complemented that is adjacent to the pixel to be recalculated or that is located diagonally from the pixel to be recalculated."

[0058] In this way, the complementing unit 1041 recalculates the pixel values ​​of the surrounding pixels according to the recalculation rule (see "R3" in FIG. 7).

[0059] It should be noted that although the complementing unit 1041 recalculates the pixel values ​​of the surrounding pixels in the complementing process, it is not essential that the complementing unit 1041 recalculates the pixel values ​​of the surrounding pixels in the complementing process.

[0060] The complementing unit 1041 then outputs the sensor data after the region to be complemented has been complemented and the pixel values ​​of the surrounding pixels have been recalculated to the difference acquiring unit 1042 as complemented data.

[0061] The difference acquisition unit 1042 performs a difference acquisition process to acquire difference data indicating the difference between the complemented data output from the complement unit 1041 and the class-specific representative shape data acquired by the representative shape data acquisition unit 102 from the storage device 40, in this case, adult representative shape data or infant representative shape data. More specifically, the difference acquisition unit 1042 performs a difference acquisition process to acquire difference data indicating the difference between the pixel value of each pixel in the complemented data output from the complementation unit 1041 and the pixel value of each pixel in the class-specific representative shape data acquired by the representative shape data acquisition unit 102 from the storage device 40, in this case, the adult representative shape data or the infant representative shape data. The difference acquisition unit 1042 may follow the control of the process control unit 103 to determine whether to acquire difference data indicating the difference between the interpolated data and either the adult representative shape data or the child representative shape data.

[0062] The difference acquisition process performed by the difference acquisition unit 1042 will be described in detail below using an example and the accompanying drawings. FIG. 8 is a diagram for explaining details of the difference acquisition process performed by difference acquisition section 1042 in data classifying device 10 according to the first embodiment. Figure 8 shows details of the difference acquisition process for acquiring difference data indicating the difference between the completed data and the adult representative shape data, as well as details of the difference acquisition process for acquiring difference data indicating the difference between the completed data and the child representative shape data. Here, the differential data is data expressed as a two-dimensional image. In Fig. 8, the differential data is shown as a "difference image."

[0063] 8, in the difference acquisition process, the difference acquisition unit 1042 obtains the difference between the pixel value of each pixel in the interpolated data and the pixel value of each pixel in the corresponding adult representative shape data or the pixel value of each pixel in the corresponding child representative shape data. Then, the difference acquisition unit 1042 obtains the difference data by calculating the pixel value of each pixel as the difference between the corresponding interpolated data and the adult representative shape data or the child representative shape data. Note that the pixels in the adult representative shape data or the child representative shape data that correspond to pixels in the interpolated data specifically refer to pixels that correspond in position on the data, in other words, pixels that are in the same position.

[0064] 9A and 9B are diagrams for explaining difference data obtained by performing difference obtaining processing by the difference obtaining unit 1042 in the first embodiment. In FIGS. 9A and 9B, as an example, the difference acquisition unit 1042 acquires the difference between the interpolated data and the adult representative shape data in the difference acquisition process.

[0065] Let us assume that the interpolated data is data obtained after the interpolation unit 1041 has performed an interpolation process on sensor data in which an adult is detected. In other words, let us assume that the interpolated data is data that indicates the shape characteristics of an adult. In this case, when the difference between the completed data and the adult representative shape data is calculated, it is expected that the features of an adult's shape will be erased in most of the areas of the difference data where they were previously shown (see Figure 9A). On the other hand, suppose that the interpolated data is data obtained after the interpolation unit 1041 has performed an interpolation process on sensor data in which an infant is detected. In other words, suppose that the interpolated data is data that indicates the shape characteristics of an infant. In this case, when the difference between the interpolated data and the adult representative shape data is taken, it is assumed that the shape features common to both infants and adults will be eliminated in the difference data, but the shape features of infants will remain (see Figure 9B).

[0066] 9A and 9B show an example of taking the difference between the interpolated data and the adult representative shape data, but if the difference between the interpolated data and the infant representative shape data is taken, if the interpolated data is data showing the shape characteristics of an adult, it is assumed that the adult shape characteristics will remain in the differential data without being erased. On the other hand, if the interpolated data is data showing the shape characteristics of an infant, it is assumed that the infant shape characteristics will be erased from most of the area of ​​the differential data where the shape characteristics of the infant were shown.

[0067] In other words, if the complemented data is data that shows the shape characteristics of an adult, difference data that highlights the shape characteristics of an adult is obtained by taking the difference from the representative shape data of an infant, and if the complemented data is data that shows the shape characteristics of an infant, difference data that highlights the shape characteristics of an infant is obtained by taking the difference from the representative shape data of an adult. In this way, by performing the difference acquisition process, the difference acquisition unit 1042 can acquire difference data that highlights the shape characteristics of adults or the shape characteristics of infants by combining the complemented data with the representative shape data by class from which the difference is taken.

[0068] When the difference acquisition unit 1042 acquires difference data by performing difference acquisition processing, it outputs the acquired difference data to the classification data generation unit 1043. At this time, the difference acquisition unit 1042 outputs data indicating which class-specific representative shape data was used to obtain the difference between the interpolated data and the difference data to the classification data generation unit 1043, together with the difference data.

[0069] When the process control unit 103 controls the generation of classification data without performing the difference acquisition process, the difference acquisition unit 1042 does not perform the difference acquisition process, and outputs the complemented data output from the complementation unit 1041 as difference data as is to the classification data generation unit 1043. In this case, the difference acquisition unit 1042 outputs difference execution identification data indicating that no difference has been taken with any of the class-specific representative shape data.

[0070] The classification data generating unit 1043 generates classification data by using the difference data output from the difference acquiring unit 1042 as classification data. The classification data generation unit 1043 outputs the classification data to the classification unit 105 together with the differential execution identification data output from the difference acquisition unit 1042 .

[0071] The classification unit 105 classifies the occupants based on the classification data output from the classification data generation unit 1043. That is, here, the classification unit 105 performs two-class classification of the occupants, more specifically, the physique of the occupants, into either "adult" or "child," based on the classification data output from the classification data generation unit 1043.

[0072] The classification unit 105 performs the above two-class classification using, for example, a trained model in machine learning (hereinafter referred to as a "machine learning model.") The machine learning model is, for example, a CNN (Convolutional Neural Network). CNN includes the following first, second, and third machine learning models. The first machine learning model is a CNN that takes as input classification data (hereinafter referred to as "first classification data"), which is differential data (hereinafter referred to as "first differential data") indicating the difference between the completed data and the adult representative shape data, and outputs data indicating whether the subject is an adult or an infant (hereinafter referred to as "classification inference data") and the reliability of the classification inference data. The second machine learning model is a CNN that takes classification data (hereinafter referred to as "second classification data"), which is differential data (hereinafter referred to as "second differential data") that indicates the difference between the completed data and the infant representative shape data, as input, and outputs classification inference data and the reliability of the classification inference data. The third machine learning model is a CNN that takes the imputed data as input and outputs classification inference data and the reliability of the classification inference data. The first machine learning model, the second machine learning model, and the third machine learning model are generated in advance and stored in a model storage unit (not shown) of the storage device 40.

[0073] <Case of differential data between the data with classification data complemented and the adult representative shape data> When the classification data generation unit 1043 outputs, together with the classification data, difference execution identification data indicating that the data is difference data obtained by subtracting the data from the adult representative shape data, i.e., when the classification data is the first classification data, the classification unit 105 infers whether the occupant is classified as a "child" based on the first classification data and the first machine learning model. Specifically, the classification unit 105 inputs the first classification data to the first machine learning model and obtains classification inference data and reliability. The classification unit 105 classifies the occupant as an "infant" when the classification inference data indicates that the occupant is an "infant" and the reliability is equal to or greater than a preset threshold (hereinafter referred to as a "reliability determination threshold"). The reliability determination threshold is set in advance by a developer or the like and stored in the storage device 40. As described above, when the difference from the adult representative shape data is taken, most of the features of the adult shape shown in the interpolated data before the difference acquisition process is performed are erased. In other words, it is difficult to classify an adult from the first classification data, which is the first difference data acquired by taking the difference from the adult representative shape data. Therefore, when the first classification data is based on the first difference data acquired by taking the difference from the adult representative shape data, the classification unit 105 determines whether the occupant can be classified as a "child" based on the first classification data and the first machine learning model. When the occupant is classified as an "infant," the classification unit 105 outputs data indicating that the occupant has been classified as an "infant" (hereinafter referred to as "classification result data") to the classification result output unit .

[0074] If the classification result data indicates an "adult," or if the classification result data indicates an "infant" but the reliability is less than the reliability determination threshold, the classification unit 105 determines that the occupant cannot be classified as either an "adult" or an "infant" based on the first classification data and the first machine learning model. In this case, the classification unit 105 outputs data indicating unclassified data (hereinafter referred to as "unclassified data") to the process control unit 103. The process control unit 103 receives the unknown classification data and determines the next control to be performed on the data processing unit 104 in accordance with the process control conditions.

[0075] <Case of differential data between supplemented data and infant representative shape data for classification> When the classification data generation unit 1043 outputs, together with the classification data, difference execution identification data indicating that the data is difference data obtained by subtracting the data from the infant representative shape data, i.e., when the classification data is second classification data, the classification unit 105 infers whether the occupant is classified as an "infant" based on the second classification data and the second machine learning model. Specifically, the classification unit 105 inputs the second classification data to the second machine learning model and obtains classification inference data and reliability. The classification unit 105 classifies the occupant as an "adult" if the classification inference data indicates that the occupant is an "adult" and the reliability is equal to or greater than the reliability determination threshold. As described above, when the difference from the infant representative shape data is taken, most of the infant shape features shown in the interpolated data before the difference acquisition process are erased. In other words, it is difficult to classify an infant from the difference data obtained by taking the difference from the infant representative shape data. Therefore, when the second classification data is based on the second difference data obtained by taking the difference from the infant representative shape data, the classification unit 105 determines whether or not the occupant can be classified as an "adult" based on the second classification data and the second machine learning model. When the occupant is classified as an "adult," the classification unit 105 outputs, to the classification result output unit 106, classification result data indicating that the occupant has been classified as an "adult."

[0076] If the classification result data indicates that the occupant is an "infant," or if the classification result data indicates that the occupant is an "adult" but the reliability is less than the reliability determination threshold, the classification unit 105 determines that the occupant cannot be classified as either an "adult" or an "infant" based on the classification data and the second machine learning model. In this case, the classification unit 105 outputs the classification unknown data to the process control unit 103. The process control unit 103 receives the unknown classification data and determines the next control to be performed on the data processing unit 104 in accordance with the process control conditions.

[0077] <Case where classification data has been supplemented> When the classification data generation unit 1043 outputs, together with the classification data, difference execution identification data indicating that no difference has been taken with any of the class-specific representative shape data, i.e., when the classification data is interpolated data, the classification unit 105 infers whether the occupant is a child based on the classification data and the third machine learning model. Specifically, the classification unit 105 inputs the classification data to the third machine learning model and obtains classification inference data and reliability. The classification unit 105 classifies the occupant into a category indicated by the classification inference data, for example, "adult" or "child." For example, the classification unit 105 may determine that the occupant is "unclassifiable" if the reliability is less than a reliability determination threshold. The classification unit 105 outputs classification result data indicating whether the occupant has been classified as an "adult," a "child," or "unclassifiable" to the classification result output unit 106.

[0078] When the classification result data is output from the classification unit 105, the classification result output unit 106 outputs the classification result data. For example, the classification result output unit 106 outputs the classification result data to an output device (not shown), and causes the output device to output the two-class classification result of the occupant into "adult" or "child" by the classification unit 105. The output device is, for example, a display device. For example, the classification result output unit 106 may store the classification result data in the storage device 40.

[0079] In the above description of the exemplary configuration of data classifying device 10, data classifying device 10 is assumed to include processing control unit 103, but the functions of processing control unit 103 may be included in data processing unit 104. In this case, data classifying device 10 may be configured without processing control unit 103.

[0080] The operations of the representative shape data generating device 30 and the data classifying device 10 according to the first embodiment will be described. First, the operation of the representative shape data generating device 30 according to the first embodiment will be described. FIG. 10 is a flowchart for explaining the operation of the representative shape data generating device 30 according to the first embodiment. The representative shape data generating device 30 performs the operation shown in the flowchart of FIG. 10, for example, before shipping of a vehicle or when the learning data stored in the storage device 40 is updated.

[0081] The learning data acquisition unit 301 acquires learning data from a learning data storage unit (not shown) of the storage device 40 (step ST1). The learning data acquisition unit 301 outputs the acquired learning data to the representative shape data generation unit 302 .

[0082] The representative shape data generating unit 302 generates representative shape data based on the learning data acquired by the learning data acquiring unit 301 in step ST1 (step ST2). Specifically, the representative shape data generation unit 302 generates person representative shape data, which is representative shape data for all classes, and adult representative shape data and infant representative shape data, which are representative shape data by class, based on the learning data acquired by the learning data acquisition unit 301.

[0083] When the representative shape data generation unit 302 generates the representative shape data in step ST2, the representative shape data output unit 303 stores the representative shape data generated by the representative shape data generation unit 302 in the representative shape data storage unit of the storage device 40 (step ST3). The representative shape data output unit 303 assigns type identification data to the representative shape data, more specifically, type identification data that can identify whether the representative shape data is person representative shape data, adult representative shape data, or child representative shape data, and stores the data in the representative shape data storage unit of the storage device 40.

[0084] The representative shape data generating device 30 may perform the processing of steps ST1 to ST3 shown in the flowchart of FIG. 10 once to generate person representative shape data, adult representative shape data, and child representative shape data, or may perform the processing of steps ST1 to ST3 shown in the flowchart of FIG. 10 three times to generate person representative shape data, adult representative shape data, and child representative shape data, respectively.

[0085] Next, the operation of data classifying device 10 according to the first embodiment will be described. FIG. 11 is a flowchart illustrating the operation of data classifying apparatus 10 according to the first embodiment. Data classifying device 10 performs the operation shown in the flowchart of FIG. 11, for example, when the vehicle engine is turned on, when the vehicle engine is turned off, or at a preset cycle.

[0086] The data acquisition unit 101 acquires sensor data from the sensor 20 (step ST10). The data acquisition unit 101 outputs the acquired sensor data to the process control unit 103 .

[0087] The representative shape data acquisition unit 102 acquires representative shape data, more specifically, person representative shape data, adult representative shape data, and child representative shape data, from the representative shape data storage unit of the storage device 40 (step ST20). The representative shape data acquisition unit 102 outputs the acquired representative shape data to the process control unit 103 .

[0088] The data processing unit 104 performs data processing to generate classification data from the sensor data acquired by the data acquisition unit 101 in step ST10, using the representative shape data acquired by the representative shape data acquisition unit 102 from the storage device 40 in step ST20 (step ST30). The data processing unit 104 outputs the classification data to the classification unit 105 .

[0089] The classification unit 105 performs a classification process to classify the occupants based on the classification data output from the data processing unit 104 in step ST30 (step ST40). That is, here, the classification unit 105 performs two-class classification of the occupants, either "adult" or "child," based on the classification data output from the classification data generation unit 1043.

[0090] 11, the process of step ST10 and the process of step ST20 are performed in parallel, but this is merely an example. For example, the process of step ST20 may be performed after the process of step ST10, or the processes may be performed in the reverse order. It is sufficient that the processing of step ST10 and the processing of step ST20 are performed before the processing of step ST30 is performed.

[0091] FIG. 12 is a flowchart for explaining an example of detailed operations of data classifying device 10 in steps ST30 to ST40 of FIG. Here, as an example, it is assumed that the process control conditions are set to the above-mentioned <process control condition (1)> to <process control condition (3)>.

[0092] The complementing unit 1041 performs a complementing process to complement missing values ​​in the sensor data with values ​​of corresponding all-class representative shape data, here, person representative shape data. More specifically, the complementing unit 1041 performs a complementing process to complement pixel values ​​of a region to be complemented, where pixel values ​​are missing in the sensor data, with pixel values ​​of the corresponding all-class representative shape data, here, person representative shape data (step ST101). The complementing unit 1041 outputs the complemented data to the difference obtaining unit 1042 .

[0093] When the complemented data is first output from the complementing unit 1041 to the difference acquiring unit 1042 after starting operation, the process control unit 103 causes the difference acquiring unit 1042 to first perform difference acquiring processing using the adult representative shape data in accordance with the process control conditions. The process control unit 103 can determine that the complemented data has been first output from the complementing unit 1041 to the difference acquiring unit 1042 after starting operation, for example, by acquiring data indicating that from the complementing unit 1041.

[0094] The difference acquisition unit 1042, under the control of the processing control unit 103, reads the complemented data output from the complementing unit 1041 in step ST101 (step ST102), and performs a difference acquisition process to acquire difference data indicating the difference between the read complemented data and the adult representative shape data acquired by the representative shape data acquisition unit 102 from the storage device 40 in step ST20 of Figure 11 (step ST103). The difference acquisition unit 1042 outputs the acquired difference data to the classification data generation unit 1043. At this time, the difference acquisition unit 1042 outputs difference execution identification data to the classification data generation unit 1043 together with the difference data. The classification data generation unit 1043 generates classification data by using the difference data output from the difference acquisition unit 1042 as classification data. The classification data generation unit 1043 outputs the classification data to the classification unit 105 together with the difference execution identification data output from the difference acquisition unit 1042.

[0095] The classification unit 105 performs inference as to whether the occupant is classified as a "child" based on the classification data (first classification data) and the first machine learning model (step ST104).

[0096] If the classification inference data indicates "infant" and the reliability is equal to or greater than the reliability determination threshold ("YES" in step ST105), the classification unit 105 classifies the occupant as "infant" (step ST106). Then, the classification unit 105 outputs classification result data indicating that the occupant has been classified as a "child" to the classification result output unit 106. When the classification result data is output from the classification unit 105, the classification result output unit 106 outputs the classification result data.

[0097] On the other hand, if the classification inference data indicates an "adult," or if the classification result data indicates a "child" but the reliability is less than the reliability determination threshold ("NO" in step ST105), the classification unit 105 determines that the occupant cannot be classified as either an "adult" or a "child" based on the classification data and the first machine learning model, and outputs unknown classification data to the processing control unit 103.

[0098] The process control unit 103 then causes the difference acquisition unit 1042 to perform difference acquisition processing using the infant representative shape data in accordance with the process control conditions. The difference acquiring unit 1042, under the control of the process control unit 103, reads the complemented data output from the complementing unit 1041 in step ST101 (step ST107), and performs a difference acquiring process of acquiring difference data indicating the difference between the read complemented data and the infant representative shape data acquired from the storage device 40 by the representative shape data acquiring unit 102 in step ST20 of Fig. 11 (step ST108). The difference acquiring unit 1042 may, for example, store the complemented data output from the complementing unit 1041 in step ST101 and read in step ST102 in an internal buffer, and then read the stored complemented data from the internal buffer. The difference acquisition unit 1042 outputs the acquired difference data to the classification data generation unit 1043. At this time, the difference acquisition unit 1042 outputs difference execution identification data to the classification data generation unit 1043 together with the difference data. The classification data generation unit 1043 generates classification data by using the difference data output from the difference acquisition unit 1042 as classification data. The classification data generation unit 1043 outputs the classification data to the classification unit 105 together with the difference execution identification data output from the difference acquisition unit 1042.

[0099] The classification unit 105 performs inference as to whether the occupant is classified as an "adult" based on the classification data (second classification data) and the second machine learning model (step ST109).

[0100] If the classification inference data indicates that the occupant is an "adult" and the reliability is equal to or greater than the reliability determination threshold ("YES" in step ST110), the classification unit 105 classifies the occupant as an "adult" (step ST111). Then, the classification unit 105 outputs classification result data indicating that the occupant has been classified as an "adult" to the classification result output unit 106. When the classification result data is output from the classification unit 105, the classification result output unit 106 outputs the classification result data.

[0101] On the other hand, if the classification inference data indicates a "child," or if the classification result data indicates an "adult" but the reliability is less than the reliability determination threshold ("NO" in step ST110), the classification unit 105 determines that the occupant cannot be classified as either an "adult" or an "child" based on the classification data and the second machine learning model, and outputs unknown classification data to the processing control unit 103.

[0102] The process control unit 103 then causes the difference acquisition unit 1042 to generate classification data in accordance with the process control conditions without performing the difference acquisition process. The difference acquisition unit 1042, under the control of the process control unit 103, reads the complemented data output from the complementing unit 1041 in step ST101 (step ST112) and outputs the read complemented data as it is to the classification data generation unit 1043 as differential data. The difference acquisition unit 1042 outputs difference execution identification data indicating that no difference has been taken from any class-specific representative shape data to the differential data. For example, the difference acquisition unit 1042 may store the complemented data output from the complementing unit 1041 in step ST101 and read in step ST102 in an internal buffer and read the stored complemented data from the internal buffer. After reading the complemented data from the internal buffer in step ST112, the difference acquisition unit 1042 deletes the stored complemented data. The classification data generation unit 1043 generates classification data by using the difference data output from the difference acquisition unit 1042 as classification data. The classification data generation unit 1043 outputs the classification data to the classification unit 105 together with the difference execution identification data output from the difference acquisition unit 1042.

[0103] The classification unit 105 infers whether the occupant is an adult or a child based on the classification data and the third machine learning model (step ST113). The classification unit 105 outputs classification result data indicating whether the occupant has been classified as an "adult," a "child," or "unclassifiable" to the classification result output unit 106. When the classification result data is output from the classification unit 105, the classification result output unit 106 outputs the classification result data.

[0104] In this way, the data classifying device 10 acquires sensor data from the sensor 20 and generates classification data from the acquired sensor data using representative shape data (person representative shape data, adult representative shape data, or child representative shape data).The data classifying device 10 then classifies the occupant into either an "adult" or a "child" based on the generated classification data. This allows the data classifying device 10 to prevent a decrease in the accuracy of classifying the occupant, more specifically, the physique of the occupant.

[0105] In the first embodiment, the classifying unit 105 in the data classifying device 10 classifies occupants using CNN, but this is merely an example. The classifier 105 may, for example, use a large scale multi-modal model to classify the occupants. Furthermore, the classification unit 105 may classify occupants according to, for example, preset conditions (hereinafter referred to as "classification conditions") rather than using a machine learning model. The classification conditions are conditions that define what content of the classification data determines whether an occupant is classified as an "adult" or an "infant" and how to calculate the reliability, and are set in advance by a developer or the like. After setting the classification conditions, the developer or the like stores data indicating the classification conditions (hereinafter referred to as "classification condition data") in the storage device 40. The classification condition data may be updateable as appropriate.

[0106] Furthermore, in the above-described first embodiment, it is assumed that the sensor 20 is a millimeter wave sensor (ToF (Time of Flight) sensor), but this is merely an example. The sensor 20 may be, for example, a LiDAR sensor. When the sensor 20 is a LiDAR sensor, the LiDAR sensor can reflect more objects than a millimeter wave sensor, and therefore, it can acquire three-dimensional data indicating the three-dimensional position and shape of an object with even higher accuracy. The sensor 20 can be any type of sensor capable of detecting the three-dimensional position and shape of an object.

[0107] Furthermore, in the first embodiment described above, data classifying device 10 has the function of performing interpolation processing and the function of performing difference acquisition processing, but this is merely an example, and data classifying device 10 may be configured to not have the function of performing difference acquisition processing, out of the functions of performing interpolation processing and difference acquisition processing. For example, in cases where there are few common shape features among the classes to be classified, data classifying device 10 may be configured to not have the function of performing difference acquisition processing. If the data classifying device 10 does not have the function of performing difference acquisition processing, the data processing unit 104 in the data classifying device 10 may be configured without the difference acquisition unit 1042. In this case, the complementing unit 1041 outputs the complemented data to the classification data generating unit 1043, and the classification data generating unit 1043 uses the complemented data as classification data. The classification data generating unit 1043 outputs, for example, difference execution specifying data indicating that no difference has been taken with any class-specific representative shape data to the classifying unit 105 together with the classification data. In the representative shape data generating device 30, the representative shape data generating unit 302 does not necessarily have to generate the adult representative shape data and the infant representative shape data. In this case, in the operation of data classifying device 10 described using the flowchart of FIG. 12, data classifying device 10 can omit the processes of steps ST103 and ST108. Furthermore, in the operation of the representative shape data generating device 30 described using the flowchart in FIG. 10, the representative shape data generating unit 302 does not necessarily generate the adult representative shape data and the child representative shape data in step ST2.

[0108] Furthermore, in the first embodiment described above, data classification device 10 may be configured to not have the function of performing the interpolation process, out of the functions of performing the interpolation process and the difference acquisition process. For example, if it is estimated that there is a low possibility of missing values ​​occurring in the sensor data due to the characteristics of sensor 20, or if the resolution of the sensor data is high and the obtained data is expected to change little, data classification device 10 may be configured not to have the function of performing the interpolation process. When the data classifying device 10 does not have a function for performing the complementing process, the data processing unit 104 in the data classifying device 10 may be configured without the complementing unit 1041. In this case, the difference acquiring unit 1042 performs a difference acquiring process for acquiring difference data indicating the difference between the sensor data acquired by the data acquiring unit 101 from the sensor 20 and the representative shape data by class. In the representative shape data generating device 30, the representative shape data generating unit 302 does not necessarily generate person representative shape data. In this case, data classifying device 10 can omit the processing of step ST101 in the operation of data classifying device 10 described using the flowchart in Fig. 12. In step ST102, step ST107, or step ST112, difference acquiring unit 1042 reads the sensor data acquired by data acquiring unit 101. Furthermore, in the operation of the representative shape data generating device 30 explained using the flowchart in FIG. 10, the representative shape data generating unit 302 does not necessarily generate person representative shape data in step ST2.

[0109] Furthermore, in the above-described first embodiment, data classification device 10 classifies vehicle occupants into either "adult" or "child," but this is merely an example, and the classes to be classified by data classification device 10 may be classes other than "adult" or "child." Furthermore, in the first embodiment, data classifying device 10 performs two-class classification, but this is merely an example, and data classifying device 10 can also perform three or more classes.

[0110] Furthermore, in the above-described first embodiment, data classification device 10 performs class classification of vehicle occupants, but this is merely an example, and class classification by data classification device 10 can be applied to classifying various objects based on sensor data acquired from sensor 20 capable of detecting the three-dimensional position and shape of the object.

[0111] In the first embodiment, the representative shape data generating device 30 generates adult representative shape data and infant representative shape data as class-specific representative shape data, but this is merely an example. The representative shape data generating device 30 generates an appropriate number and types of class-specific representative shape data according to the classes to be classified by the data classifying device 10.

[0112] Furthermore, in the above embodiment 1, the sensor data acquired from the sensor is data in which the three-dimensional position and shape of an object are represented by a two-dimensional image, but this is merely an example, and the sensor data acquired from the sensor may also be three-dimensional data indicating the three-dimensional position and shape of an object.

[0113] Furthermore, in the first embodiment described above, data classification device 10 is an on-board device mounted on a vehicle, but this is merely an example. For example, each element of the data acquisition unit 101, the representative shape data acquisition unit 102, the processing control unit 103, the complementation unit 1041, the difference acquisition unit 1042, the classification data generation unit 1043, the classification unit 105, or the classification result output unit 106 may be implemented by a separate device. For example, the data acquisition unit 101, the representative shape data acquisition unit 102, the processing control unit 103, the complementation unit 1041, the difference acquisition unit 1042, the classification data generation unit 1043, the classification unit 105, or part of the classification result output unit 106 may be provided in a server (not shown) connected to the in-vehicle device, and the system may be formed by the in-vehicle device and the server. Furthermore, for example, the data acquisition unit 101, the representative shape data acquisition unit 102, the processing control unit 103, the complementation unit 1041, the difference acquisition unit 1042, the classification data generation unit 1043, the classification unit 105, and the classification result output unit 106 may all be provided in the server.

[0114] Furthermore, in the first embodiment, the representative shape data generating device 30 is an on-board device mounted on a vehicle, but this is merely an example. For example, each element of the learning data acquisition unit 301, the representative shape data generation unit 302, or the representative shape data output unit 303 may be implemented by a separate device. For example, a part of the learning data acquisition unit 301, the representative shape data generation unit 302, or the representative shape data output unit 303 may be provided in a server connected to the in-vehicle device, and the in-vehicle device and the server may form a system. Furthermore, for example, all of the learning data acquisition unit 301, the representative shape data generation unit 302, and the representative shape data output unit 303 may be provided in a server.

[0115] In the first embodiment, the data classifying device 10 may include the representative shape data generating device 30.

[0116] 13A and 13B are diagrams illustrating an example of a hardware configuration of data classifying device 10 according to the first embodiment. In the first embodiment, the functions of the data acquiring unit 101, the representative shape data acquiring unit 102, the processing control unit 103, the data processing unit 104, the classification unit 105, and the classification result output unit 106 are realized by the processing circuit 1001. That is, the data classification device 10 includes the processing circuit 1001 for performing data processing using the representative shape data on the sensor data acquired from the sensor 20 to generate classification data, and for controlling the classification of objects based on the classification data. The processing circuit 1001 may be dedicated hardware as shown in FIG. 13A, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 13B.

[0117] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0118] When the processing circuit is a processor 1004, the functions of the data acquisition unit 101, representative shape data acquisition unit 102, process control unit 103, data processing unit 104, classification unit 105, and classification result output unit 106 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 reads and executes the program stored in memory 1005 to execute the functions of the data acquisition unit 101, representative shape data acquisition unit 102, process control unit 103, data processing unit 104, classification unit 105, and classification result output unit 106. In other words, the data classification device 10 includes memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST10 to ST40 in FIG. 11 described above. It can also be said that the program stored in the memory 1005 causes the computer to execute the processing procedures or methods of the data acquisition unit 101, the representative shape data acquisition unit 102, the processing control unit 103, the data processing unit 104, the classification unit 105, and the classification result output unit 106. Here, the memory 1005 corresponds to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0119] It is also possible to realize some of the functions of the data acquisition unit 101, the representative shape data acquisition unit 102, the processing control unit 103, the data processing unit 104, the classification unit 105, and the classification result output unit 106 with dedicated hardware and some with software or firmware. For example, the functions of the data acquisition unit 101 and the representative shape data acquisition unit 102 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the processing control unit 103, the data processing unit 104, the classification unit 105, and the classification result output unit 106 can be realized by the processor 1004 reading and executing a program stored in the memory 1005. The data classification device 10 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the sensor 20 or the storage device 40 .

[0120] An example of the hardware configuration of the representative shape data generating device 30 is also the example of the configuration shown in FIGS. 13A and 13B. In representative shape data generation device 30, the functions of training data acquisition unit 301, representative shape data generation unit 302, and representative shape data output unit 303 are realized by processing circuit 1001. In other words, representative shape data generation device 30 includes processing circuit 1001 for controlling the generation of representative shape data used by data classification device 10 when performing class classification. The processing circuit 1001 may be dedicated hardware as shown in FIG. 13A, or may be a processor 1004 that executes a program stored in memory as shown in FIG. 13B.

[0121] Processing circuit 1001 reads and executes a program stored in memory 1005, thereby performing the functions of learning data acquisition unit 301, representative shape data generation unit 302, and representative shape data output unit 303. That is, representative shape data generation device 30 includes memory 1005 for storing a program that, when executed by processing circuit 1001, results in the execution of steps ST1 to ST3 in Fig. 10 described above. It can also be said that the program stored in memory 1005 causes a computer to execute the processing procedures or methods of learning data acquisition unit 301, representative shape data generation unit 302, and representative shape data output unit 303. The representative shape data generating device 30 also includes devices such as the storage device 40, an input interface device 1002 for performing wired or wireless communication, and an output interface device 1003.

[0122] As described above, according to the first embodiment, the data classification device 10 is a data classification device 10 that performs class classification of objects based on sensor data acquired from a sensor 20 that can detect the three-dimensional position and shape of the object, and is configured to include a data acquisition unit 101 that acquires the sensor data from the sensor 20, a data processing unit 104 that generates classification data from the sensor data acquired by the data acquisition unit 101 using representative shape data, which is sensor data that indicates the characteristics of the representative shapes of objects that belong to a class to be classified, and a classification unit that classifies the objects based on the classification data generated by the data processing unit 104. Therefore, data classifying device 10 can prevent a decrease in the accuracy of object classification.

[0123] According to embodiment 1, the representative shape data includes all-class representative shape data, which is sensor data that indicates the characteristics of the representative shapes of objects belonging to all classes that are the subject of class classification, and in data classification device 10, data processing unit 104 is configured to have: completion unit 1041 that performs an interpolation process to interpolate missing values ​​in the sensor data with the corresponding values ​​of the all-class representative shape data; and classification data generation unit 1043 that generates classification data by using the sensor data after the interpolation process by completion unit 1041 as classification data. This allows data classifying device 10 to prevent a decrease in the accuracy of object classification.

[0124] Furthermore, according to embodiment 1, the representative shape data includes all-class representative shape data, which is sensor data that indicates the characteristics of the representative shapes of objects belonging to all classes that are the subject of class classification, and the sensor data is data in which the three-dimensional position and shape of an object are represented by a two-dimensional image. In data classification device 10, data processing unit 104 is configured to have: completion unit 1041 that performs an interpolation process to interpolate pixel values ​​of regions to be interpolated that are missing pixel values ​​in the sensor data, with pixel values ​​of corresponding regions in the all-class representative shape data; and classification data generation unit 1043 that generates classification data by using the sensor data after completion process by completion unit 1041 as classification data. This allows data classifying device 10 to prevent a decrease in the accuracy of object classification.

[0125] Furthermore, according to the first embodiment, the complementing unit 1041 is configured to recalculate the pixel values ​​of the pixels surrounding the region to be complemented in the sensor data in the complementing process. This allows data classifying device 10 to suppress extreme changes in the data values ​​in the sensor data after completion unit 1041 has performed the completion process.

[0126] Furthermore, according to embodiment 1, the representative shape data includes class-specific representative shape data, which is sensor data indicating the characteristics of the representative shapes of objects belonging to each class to be classified, and in data classification device 10, data processing unit 104 is configured to have a difference acquisition unit 1042 that performs a difference acquisition process to acquire difference data indicating the difference between the sensor data and the class-specific representative shape data, and a classification data generation unit 1043 that generates classification data by using the difference data acquired by difference acquisition unit 1042 in the difference acquisition process as classification data. This allows data classifying device 10 to extract features specific to objects belonging to a class to be classified, thereby preventing a decrease in the accuracy of object classification.

[0127] Furthermore, according to embodiment 1, the representative shape data includes class-specific representative shape data, which is sensor data indicating the characteristics of the representative shapes of objects belonging to each class to be classified, and the sensor data is data in which the three-dimensional position and shape of the object are represented by a two-dimensional image. In data classification device 10, data processing unit 104 is configured to have a difference acquisition unit 1042 that performs a difference acquisition process to acquire difference data indicating the difference between the pixel value of each pixel in the sensor data and the pixel value of each pixel in the class-specific representative shape data, and a classification data generation unit 1043 that generates classification data by using the difference data acquired by difference acquisition unit 1042 in the difference acquisition process as classification data. This allows data classifying device 10 to extract image features specific to objects belonging to a class that is the target of classification, thereby preventing a decrease in the accuracy of object classification.

[0128] Furthermore, according to the first embodiment, the representative shape data includes all-class representative shape data, which is sensor data indicating the characteristics of the representative shapes of objects belonging to all classes to be subjected to class classification, and class-specific representative shape data, which is sensor data for each class to be subjected to class classification, indicating the characteristics of the representative shapes of objects belonging to that class. In the data classification device 10, the data processing unit 104 is configured to have a completion unit 1041 that performs an completion process to complete missing values ​​in the sensor data with the values ​​of the corresponding all-class representative shape data, a difference acquisition unit 1042 that performs a difference acquisition process to acquire difference data indicating the difference between the completed data, which is the sensor data after the completion process by the completion unit 1041, and the class-specific representative shape data, and a classification data generation unit 1043 that generates classification data by using the difference data acquired by the difference acquisition process by the difference acquisition unit 1042 as classification data. This allows data classifying device 10 to prevent a decrease in the accuracy of object classification.

[0129] Furthermore, according to the first embodiment, the representative shape data includes all-class representative shape data, which is sensor data indicating the characteristics of the representative shapes of objects belonging to all classes to be subjected to class classification, and class-specific representative shape data, which is sensor data for each class to be subjected to class classification, indicating the characteristics of the representative shapes of objects belonging to that class. The sensor data is data in which the three-dimensional position and shape of an object are represented as a two-dimensional image. In the data classification device 10, the data processing unit 104 includes a completion unit 1041 that performs an completion process to complete pixel values ​​of regions to be completed, where pixel values ​​are missing in the sensor data, with pixel values ​​of the corresponding all-class representative shape data; a difference acquisition unit 1042 that performs a difference acquisition process to acquire difference data indicating the difference between the pixel values ​​of each pixel in the completed data, which is the sensor data after the completion process by the completion unit 1041, and the pixel values ​​of each pixel in the class-specific representative shape data; and a classification data generation unit 1043 that generates classification data by using the difference data acquired by the difference acquisition process by the difference acquisition unit 1042 as classification data. This allows data classifying device 10 to prevent a decrease in the accuracy of object classification.

[0130] Any of the components of the embodiments may be modified or omitted.

[0131] Various aspects of the present disclosure are summarized below as appendices.

[0132] (Appendix 1) A data classification device that classifies an object based on sensor data acquired from a sensor capable of detecting a three-dimensional position and shape of the object, a data acquisition unit that acquires the sensor data from the sensor; a data processing unit that generates classification data from the sensor data acquired by the data acquisition unit, using representative shape data that is the sensor data indicating features of the representative shape of the object that belongs to the class that is the target of the classification; a classification unit that classifies the object based on the classification data generated by the data processing unit; A data classification device comprising: (Appendix 2) the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; The data processing unit a complementing unit that performs a complementing process to complement missing values ​​in the sensor data with corresponding values ​​of the all-class representative shape data; a classification data generation unit that generates classification data by using the sensor data after the completion process by the completion unit as classification data; 2. The data classification device according to claim 1. (Appendix 3) the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The data processing unit a complementing unit that performs a complementing process to complement pixel values ​​of a complement target region in which pixel values ​​are missing in the sensor data with pixel values ​​of a corresponding region in the all-class representative shape data; a classification data generation unit that generates classification data by using the sensor data after the completion process by the completion unit as classification data; 2. The data classification device according to claim 1. (Appendix 4) The complementing unit In the complementation process, the pixel values ​​of the pixels surrounding the region to be complemented in the sensor data are recalculated. 4. The data classification device according to claim 3. (Appendix 5) The all-class representative shape data is generated based on learning data that is generated based on sensor data that indicates the objects that belong to all of the classes that are the target of the classification and that have been previously acquired from the sensor. 5. The data classification device according to claim 2, wherein: (Appendix 6) the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; The data processing unit a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the sensor data and the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1. (Appendix 7) the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The data processing unit a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between a pixel value of each pixel in the sensor data and a pixel value of each pixel in the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1. (Appendix 8) the class to be classified includes infants; The difference acquisition unit performing a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the sensor data and the pixel value of each pixel in infant representative shape data, which is the class-specific representative shape data indicating the infant; 8. The data classification device according to claim 7. (Appendix 9) the class targeted for classification includes adults; The difference acquisition unit performing a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the sensor data and the pixel value of each pixel in adult representative shape data, which is the class-specific representative shape data indicating the adult; 8. The data classification device according to claim 7. (Appendix 10) The representative shape data by class is The sensor data is generated based on learning data in which a label indicating which class of the class classification target the sensor data corresponds to is assigned to the sensor data previously acquired from the sensor. 8. The data classification device according to claim 6 or 7. (Appendix 11) The label includes a label indicating a date. 11. The data classification device according to claim 10. (Appendix 12) The label includes the label indicating a child. 11. The data classification device according to claim 10. (Appendix 13) The label includes the label indicating an adult 11. The data classification device according to claim 10. (Appendix 14) the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all of the classes that are the targets of the class classification, and class-specific representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to each class that is the target of the class classification, The data processing unit a complementing unit that performs a complementing process to complement missing values ​​in the sensor data with corresponding values ​​of the all-class representative shape data; a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the complemented data, which is the sensor data after the complementing unit has performed the complementing process, and the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1. (Appendix 15) the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all of the classes that are the targets of the class classification, and class-specific representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to each class that is the target of the class classification, the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The data processing unit a complementing unit that performs a complementing process to complement pixel values ​​of a complement target region in which pixel values ​​are missing in the sensor data with corresponding pixel values ​​of the all-class representative shape data; a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the interpolated data, which is the sensor data after the interpolation process has been performed by the interpolation unit, and the pixel value of each pixel in the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1. (Appendix 16) The sensor is a millimeter wave sensor 16. The data classification device of any one of appendices 1 to 15. (Appendix 17) The sensor is a LiDAR 16. The data classification device of any one of appendices 1 to 15. (Appendix 18) The classification unit performs two-class classification based on the classification data. 2. The data classification device according to claim 1. (Appendix 19) The classification unit classifies the object according to classification conditions based on the classification data. 2. The data classification device according to claim 1. (Appendix 20) The classification unit classifies the object using a CNN based on the classification data. 2. The data classification device according to claim 1. (Appendix 21) The classification unit classifies the object using a large-scale multimodal model based on the classification data. 2. The data classification device according to claim 1. (Appendix 22) a representative shape data generating unit for generating the representative shape data; 22. The data classifier of any one of Supplementary Notes 1 to 21, comprising: (Appendix 23) the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; The representative shape data generation unit generates the all-class representative shape data based on learning data generated based on the sensor data that indicates the objects that belong to all the classes that are the target of the classification and that have been previously acquired from the sensor. 23. The data classification device according to claim 22. (Appendix 24) the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; The representative shape data generation unit Extracting data for generating representative shape data by class according to extraction conditions from learning data generated based on the sensor data indicative of the objects belonging to all of the classes to be classified in the classification, which data was previously acquired from the sensor, and generating the representative shape data by class based on the extracted data for generating representative shape data by class. 23. The data classification device according to claim 22. (Appendix 25) the learning data is data in which a label indicating to which class the sensor data corresponds to the class to be classified is assigned to the sensor data, The extraction condition includes a condition for extracting the learning data including the sensor data to which the label is assigned as the data for generating the representative shape data by class. 25. The data classification device according to claim 24. (Appendix 26) The label includes a label indicating a year; The extraction condition includes a condition as to which age group the learning data to which the label indicating the age group is assigned should be extracted as data for generating the class-specific representative shape data. 26. The data classification device according to claim 25. (Appendix 27) the label includes the label indicating an adult; The extraction condition is set to extract the learning data to which the label indicating an adult is assigned as data for generating representative shape data by class. 26. The data classification device according to claim 25. (Appendix 28) the label includes the label indicating an infant; The extraction condition is set to extract the learning data to which the label indicating the infant is assigned as the data for generating the class-specific representative shape data. 26. The data classification device according to claim 25. (Appendix 29) the learning data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The representative shape data generation unit The representative shape data is generated by setting the average value of the pixel values ​​of the corresponding pixels in the plurality of learning data as the pixel value of the corresponding pixel in the representative shape data. 25. The data classification device according to claim 24. (Appendix 30) the learning data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The representative shape data generation unit The representative shape data is generated by taking the most frequent pixel value of each corresponding pixel in the plurality of learning data as the pixel value of the corresponding pixel in the representative shape data. 25. The data classification device according to claim 24. (Appendix 31) A data classification program for classifying an object based on sensor data acquired from a sensor capable of detecting a three-dimensional position and shape of the object, Computer, a data acquisition unit that acquires the sensor data from the sensor; a data processing unit that generates classification data from the sensor data acquired by the data acquisition unit, using representative shape data that is the sensor data indicating features of the representative shape of the object that belongs to the class that is the target of the classification; a classification unit that classifies the object based on the classification data generated by the data processing unit; A data classification program to function as a [Explanation of symbols]

[0133] 1 Data classification system, 10 Data classification device, 101 Data acquisition unit, 102 Representative shape data acquisition unit, 103 Processing control unit, 104 Data processing unit, 1041 Complement unit, 1042 Difference acquisition unit, 1043 Classification data generation unit, 105 Classification unit, 106 Classification result output unit, 20 Sensor, 30 Representative shape data generation device, 301 Learning data acquisition unit, 302 Representative shape data generation unit, 303 Representative shape data output unit, 40 Storage device, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. A data classification device that classifies an object based on sensor data acquired from a sensor capable of detecting a three-dimensional position and shape of the object, a data acquisition unit that acquires the sensor data from the sensor; a data processing unit that generates classification data from the sensor data acquired by the data acquisition unit, using representative shape data that is the sensor data indicating features of the representative shape of the object that belongs to the class that is the target of the classification; a classification unit that classifies the object based on the classification data generated by the data processing unit; A data classification device comprising:

2. the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; The data processing unit a complementing unit that performs a complementing process to complement missing values ​​in the sensor data with corresponding values ​​of the all-class representative shape data; a classification data generation unit that generates classification data by using the sensor data after the completion process by the completion unit as classification data; 2. The data classification device according to claim 1.

3. the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image; The data processing unit a complementing unit that performs a complementing process to complement pixel values ​​of a complement target region in which pixel values ​​are missing in the sensor data with pixel values ​​of a corresponding region in the all-class representative shape data; a classification data generation unit that generates classification data by using the sensor data after the completion process by the completion unit as classification data; 2. The data classification device according to claim 1.

4. The complementing unit In the complementation process, the pixel values ​​of the pixels surrounding the region to be complemented in the sensor data are recalculated.

4. The data classification device according to claim 3.

5. The all-class representative shape data is generated based on learning data that is generated based on sensor data that indicates the objects that belong to all of the classes that are the target of the classification and that have been previously acquired from the sensor.

5. The data classification device according to claim 2, wherein the data classification device is a data classification device.

6. the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; The data processing unit a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the sensor data and the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1.

7. the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image; The data processing unit a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between a pixel value of each pixel in the sensor data and a pixel value of each pixel in the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1.

8. the class to be classified includes infants; The difference acquisition unit performing a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the sensor data and the pixel value of each pixel in infant representative shape data, which is the class-specific representative shape data indicating the infant; 8. The data classification device according to claim 7.

9. the class targeted for classification includes adults; The difference acquisition unit performing a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the sensor data and the pixel value of each pixel in adult representative shape data, which is the class-specific representative shape data indicating the adult; 8. The data classification device according to claim 7.

10. The representative shape data by class is The sensor data is generated based on learning data in which a label indicating which class of the class classification target the sensor data corresponds to is assigned to the sensor data previously acquired from the sensor.

8. The data classification device according to claim 6 or 7.

11. The label includes a label indicating a date.

11. The data classification device according to claim 10.

12. The label includes the label indicating a child.

11. The data classification device according to claim 10.

13. The label includes the label indicating an adult 11. The data classification device according to claim 10.

14. the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all of the classes that are the targets of the class classification, and class-specific representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to each class that is the target of the class classification, The data processing unit a complementing unit that performs a complementing process to complement missing values ​​in the sensor data with corresponding values ​​of the all-class representative shape data; a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the complemented data, which is the sensor data after the complementing unit has performed the complementing process, and the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1.

15. the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all of the classes that are the targets of the class classification, and class-specific representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to each class that is the target of the class classification, the sensor data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image; The data processing unit a complementing unit that performs a complementing process to complement pixel values ​​of a complement target region in which pixel values ​​are missing in the sensor data with corresponding pixel values ​​of the all-class representative shape data; a difference acquisition unit that performs a difference acquisition process to acquire difference data indicating a difference between the pixel value of each pixel in the interpolated data, which is the sensor data after the interpolation process has been performed by the interpolation unit, and the pixel value of each pixel in the class-specific representative shape data; a classification data generation unit that generates classification data by using the difference data acquired by the difference acquisition unit in the difference acquisition process as classification data; 2. The data classification device according to claim 1.

16. The sensor is a millimeter wave sensor 2. The data classification device according to claim 1.

17. The sensor is a LiDAR 2. The data classification device according to claim 1.

18. The classification unit performs two-class classification based on the classification data.

2. The data classification device according to claim 1.

19. The classification unit classifies the object according to classification conditions based on the classification data.

2. The data classification device according to claim 1.

20. The classification unit classifies the object using a CNN based on the classification data.

2. The data classification device according to claim 1.

21. The classification unit classifies the object using a large-scale multimodal model based on the classification data.

2. The data classification device according to claim 1.

22. a representative shape data generating unit for generating the representative shape data; 2. The data classification device according to claim 1, comprising:

23. the representative shape data includes all-class representative shape data, which is the sensor data indicating the characteristics of the representative shapes of the objects belonging to all the classes that are the targets of the class classification; The representative shape data generation unit generates the all-class representative shape data based on learning data generated based on the sensor data that indicates the objects that belong to all the classes that are the target of the classification and that have been previously acquired from the sensor.

23. The data classification device according to claim 22.

24. the representative shape data includes, for each of the classes to be classified, class-specific representative shape data that is the sensor data indicating features of the representative shapes of the objects that belong to the class; The representative shape data generation unit Extracting data for generating representative shape data by class according to extraction conditions from learning data generated based on the sensor data indicative of the objects belonging to all of the classes to be classified in the classification, which data was previously acquired from the sensor, and generating the representative shape data by class based on the extracted data for generating representative shape data by class.

23. The data classification device according to claim 22.

25. the learning data is data in which a label indicating to which class the sensor data corresponds to the class to be classified is assigned to the sensor data, The extraction condition includes a condition for extracting the learning data including the sensor data to which the label is assigned as the data for generating the representative shape data by class.

25. The data classification device according to claim 24.

26. The label includes a label indicating a year; The extraction condition includes a condition as to which age group the learning data to which the label indicating the age group is assigned should be extracted as data for generating the class-specific representative shape data.

26. The data classification device according to claim 25.

27. the label includes the label indicating an adult; The extraction condition is set to extract the learning data to which the label indicating an adult is assigned as data for generating representative shape data by class.

26. The data classification device according to claim 25.

28. the label includes the label indicating an infant; The extraction condition is set to extract the learning data to which the label indicating the infant is assigned as the data for generating the class-specific representative shape data.

26. The data classification device according to claim 25.

29. the learning data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The representative shape data generation unit The representative shape data is generated by setting the average value of the pixel values ​​of the corresponding pixels in the plurality of learning data as the pixel value of the corresponding pixel in the representative shape data.

25. The data classification device according to claim 24.

30. the learning data is data in which the three-dimensional position and the shape of the object are represented by a two-dimensional image, The representative shape data generation unit The representative shape data is generated by taking the most frequent pixel value of each corresponding pixel in the plurality of learning data as the pixel value of the corresponding pixel in the representative shape data.

25. The data classification device according to claim 24.

31. A data classification program for classifying an object based on sensor data acquired from a sensor capable of detecting a three-dimensional position and shape of the object, Computer, a data acquisition unit that acquires the sensor data from the sensor; a data processing unit that generates classification data from the sensor data acquired by the data acquisition unit, using representative shape data that is the sensor data indicating features of the representative shape of the object that belongs to the class that is the target of the classification; a classification unit that classifies the object based on the classification data generated by the data processing unit; A data classification program to function as a

Images