Information processing device and method of generating data for learning model

Abstract

To enable stable state estimation.SOLUTION: An information processing device 10 provided herein comprises a millimeter wave sensor 11 and a control unit 12. The control unit 12 extracts a portion of output data from among multiple sets of output data output form the millimeter wave sensor 11 using a learning model. The learning model estimates output data suitable for estimation of the state of a target object ob.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a method for generating data for a learning model. [Background technology]

[0002] There is a demand for monitoring objects such as people while protecting their privacy. To meet such demands, the use of millimeter wave sensors has been proposed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-154902 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when monitoring an object using a millimeter wave sensor, state estimation was unstable depending on the amount of movement of the object.

[0005] Therefore, an object of the present disclosure, made in consideration of the above-described problems of the conventional technology, is to stabilize state estimation based on millimeter-wave radar. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an information processing device according to a first aspect comprises: A millimeter wave sensor, The device is equipped with a control unit that extracts a portion of output data from the multiple output data outputted from the millimeter wave sensor using a learning model that estimates output data suitable for estimating the state of the target.

[0007] A method for generating data for a learning model according to a second aspect includes: an acquisition step of acquiring a plurality of output data output from the millimeter wave sensor and a true value of a state of an object detected by the millimeter wave sensor; an extraction step of randomly extracting some output data from the plurality of output data; an estimation step of estimating a state of an object detected by the millimeter wave sensor using the extracted output data; an evaluation step of performing a comparison evaluation between the estimated state of the object and the true value; a repeating step of repeating the extracting step, the estimating step, and the evaluating step until a maximum value of the comparative evaluation is obtained; a selection step of selecting the extracted output data and the plurality of output data from which the maximum value is obtained. [Effects of the Invention]

[0008] According to the information processing device and learning model data generation method configured as described above, state estimation based on millimeter-wave radar is stabilized. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional block diagram illustrating a schematic configuration of an information processing apparatus according to an embodiment. [Figure 2] This is a skeleton model generated by the information processing device of FIG. [Figure 3] 2 is a flowchart for explaining a state estimation process performed by the control unit of FIG. 1. [Figure 4] 10 is a flowchart illustrating a learning data generation process performed by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of an information processing device to which the present disclosure is applied will be described with reference to the drawings.

[0011] 1, an information processing device 10 according to an embodiment of the present disclosure may include a millimeter wave sensor 11 and a control unit 12. The information processing device 10 may further include a communication unit 13.

[0012] The millimeter-wave sensor 11 may emit millimeter waves into space. Millimeter waves are electromagnetic waves with a frequency band of 30 GHz to 300 GHz. The millimeter-wave sensor 11 may receive millimeter waves reflected from multiple object points in space. The millimeter-wave sensor 11 generates output data based on the millimeter waves reflected from one object point. The output data may include the three-dimensional coordinates of any one point on the surface of the object ob and information about noise. The information about noise is, for example, SNR (Signal Noise Ratio). Therefore, the millimeter-wave sensor 11 generates multiple output data corresponding to multiple object points.

[0013] The millimeter wave sensor 11 may include, for example, a Tx antenna, multiple Rx antennas, a mixer, and a DSP. The Tx antenna may transmit a modulated wave into a space to be measured. Each of the multiple Rx antennas may receive a reflected wave of the modulated wave from the object. The mixer may mix the transmitted wave from the Tx antenna with the received wave at the Rx antenna to generate a mixed signal. The DSP may perform signal processing on the mixed signal to calculate the three-dimensional coordinates, velocity, and SNR of each object point.

[0014] The control unit 12 includes one or more processors and a memory. The processor may include at least one of a general-purpose processor that loads a specific program to execute a specific function and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 12 may include at least one of a system-on-a-chip (SoC) and a system in a package (SiP) in which one or more processors work together.

[0015] The control unit 12 may further include a storage unit. The storage unit may include any storage device, such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage unit may store various programs that cause the control unit 12 to function and various information used by the control unit 12.

[0016] For example, the storage unit may store an extraction model that extracts output data suitable for estimating the state of the target ob from among the multiple output data. The extraction model is a learning model trained by machine learning, which will be described later. The extraction model may be trained using information on noise, three-dimensional coordinates, and speed, and may extract some of the output data based on the information on noise, three-dimensional coordinates, and speed. The storage unit may also store a state estimation model that estimates the state of the target ob based on the multiple output data. The state estimation model is a learning model trained by machine learning.

[0017] The control unit 12 extracts a portion of output data from the multiple pieces of output data output from the millimeter wave sensor 11. The control unit 12 may use an extraction model to extract the portion of output data. Alternatively, the control unit 12 may preferentially extract output data with a larger SNR until the number of pieces of output data to be extracted exceeds a certain value. The certain value may be determined so as to maintain the estimation accuracy of the state estimation using the state estimation model at or above a required level. Alternatively, the control unit 12 may extract the output data when the number of pieces of output data with an SNR equal to or greater than a first reference value is equal to or greater than a threshold, or when the number is less than the threshold, extract output data with an SNR equal to or greater than a second reference value that is less than the first reference value.

[0018] The control unit 12 may estimate the state of the object ob using a state estimation model based on the extracted output data. A human body may be defined as the object ob whose state is to be estimated. The estimated state may be the posture of the human body.

[0019] The control unit 12 creates a skeletal model sm that indicates the posture of the object ob based on the multiple pieces of output data od extracted by the extraction model, as shown in Fig. 2, for example. The skeletal model sm may be a model that does not include the surface shape of the object ob, and in which joints such as knees are connected by straight lines. The control unit 12 may generate the skeletal model sm as information. The control unit 12 may transmit the skeletal model sm to an information processing device having a display via the communication unit 13.

[0020] Next, the state estimation process executed by the control unit 12 in this embodiment will be described with reference to the flowchart in Fig. 3. The extraction process starts, for example, after the millimeter wave sensor 11 receives a reflected wave in response to a modulated wave emitted into space.

[0021] In step S100, the control unit 12 extracts a part of the output data od from the plurality of output data od using an extraction model. After the extraction, the process proceeds to step S101.

[0022] In step S101, the control unit 12 generates a skeleton model sm using a state estimation model based on the output data od extracted in step S100. After generation, the process proceeds to step S102.

[0023] In step S102, the control unit 12 transmits the skeleton model sm generated in step S101 to the information processing device. After transmission, the state estimation process ends.

[0024] Next, a data generation method for training the extraction model will be described below. Any information processing device may be used to generate data used for training the extraction model. The information processing device acquires multiple pieces of output data od output from the above-mentioned millimeter wave sensor 11. The information processing device also acquires a true value of the state of the object ob that caused the millimeter wave sensor 11 to output the multiple pieces of output data od. Specifically, the true value of the state of the object ob is the posture of the object ob. The true value of the state of the object ob is generated, for example, by image analysis or motion capture of a camera that photographs the object ob.

[0025] The information processing device randomly extracts a portion of output data od from the acquired plurality of output data od. The information processing device estimates the state of the object ob measured by the millimeter wave sensor 11 based on the extracted portion of output data od. The information processing device may use the same state estimation model as that used in the information processing device 10 described above to estimate the state of the object ob. Alternatively, the information processing device may use a state estimation model different from the state estimation model used in the information processing device 10.

[0026] The information processing device performs a comparative evaluation by comparing the estimated state of the target ob with the true value of the acquired state. The information processing device may quantify the comparative evaluation result. A higher quantified comparative evaluation may indicate a closer relationship between the estimated state and the true value.

[0027] The information processing device repeats the process of randomly extracting output data od, estimating a state based on the extracted output data od, and comparing the estimated state with a true value a predetermined number of times. The information processing device stores the output data od that estimates the state of the object closest to the true value and the multiple output data od output from the millimeter wave sensor 11 as a set of learning data.

[0028] The information processing device applies the above-described data generation method to a variety of people in a variety of postures to generate a plurality of combinations of training data for training an extraction model. Each set of training data may be composed of output data od that obtains the above-described maximum value for an arbitrary person in an arbitrary posture and a plurality of output data od output from the millimeter wave sensor 11. The information processing device may train the extraction model using the plurality of combinations of training data.

[0029] Next, the learning data generation process executed by the information processing device in this embodiment will be described with reference to the flowchart in Fig. 4. The learning data generation process starts, for example, when an operation input for starting the generation process is detected in a state where a combination of a plurality of output data od and the corresponding true values ​​of the postures has already been acquired.

[0030] In step S200, the information processing device adds 1 to the number of comparisons and evaluations. The number of comparisons and evaluations means the number of times steps S201 to S203, which will be described later, have been executed. After the addition, the process proceeds to step S201.

[0031] In step S201, the information processing device randomly extracts some output data od from a plurality of output data od. After the extraction, the process proceeds to step S202.

[0032] In step S202, the information processing device estimates the state of the object ob using the state estimation model based on the output data od extracted in step S201. After the estimation, the process proceeds to step S203.

[0033] In step S203, the information processing device calculates a numerical value by comparing and evaluating the state estimated in step S202 with the true value. After the comparison and evaluation, the process proceeds to step S204.

[0034] In step S204, the information processing device determines whether the number of comparison evaluations has reached a predetermined number. If not, the process returns to step S200. If yes, the process proceeds to step S205.

[0035] In step S205, the information processing device stores the output data od that has estimated the state of the object closest to the true value together with the plurality of output data od as a set of learning data. After storing, the learning data generation process ends.

[0036] The information processing device 10 of this embodiment, configured as described above, includes a millimeter-wave sensor 11 and a control unit 12 that extracts a portion of output data od from the multiple output data od output by the millimeter-wave sensor 11 using a learning model that estimates output data od suitable for estimating the state of the target ob. When the target ob is moving, the millimeter-wave sensor 11 outputs more object points with a relatively high SNR. On the other hand, when the target ob is stationary, the millimeter-wave sensor 11 outputs fewer object points with a relatively high SNR, while output data od with a relatively low SNR increases. In such a case, estimating the state of the target ob based on output data od with an SNR above a threshold may not provide sufficient output data od for estimation when the target ob is stationary. This can result in unstable state estimation when the target ob is stationary. On the other hand, in a configuration that uses a certain number or more of output data od, output data od with a low SNR may be used, which can reduce estimation accuracy. Therefore, the information processing device 10 configured as described above extracts output data od suitable for state estimation from the multiple output data od, thereby enabling stable state estimation.

[0037] Furthermore, in the information processing device 10, the output data od includes information about the coordinates of a point on the surface of the object ob, the speed, and noise. With this configuration, the information processing device 10 can use information about noise to extract the output data od, and therefore the detection accuracy of the output data od can be referenced.

[0038] The learning model data generation method of the present embodiment also includes an acquisition step of acquiring multiple pieces of output data od output from the millimeter wave sensor 11 and true values ​​of the state of the target ob detected by the millimeter wave sensor 11, an extraction step of randomly extracting a portion of output data od from the multiple pieces of output data od, an estimation step of estimating the state of the target ob detected by the millimeter wave sensor 11 using the extracted output data od, an evaluation step of comparing and evaluating the estimated state of the target ob with the true value, a repetition step of repeating the extraction step, estimation step, and evaluation step a predetermined number of times, and a selection step of selecting output data od and multiple pieces of output data od that have estimated the state of the target closest to the true value in the repetition step. With this configuration, the learning model data generation method can extract learning model data suitable for estimating the state of the target ob from the multiple pieces of output data od.

[0039] The above has described an embodiment of the information processing device 10, but the embodiment of the present disclosure can also be implemented as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0040] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0041] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0042] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0043] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0044] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein. [Explanation of symbols]

[0045] 10. Information processing equipment 11 Millimeter wave sensor 12 Control Unit 13 Communications Department ob target od output data sm skeletal model

Claims

1. A millimeter wave sensor, and a control unit that extracts a portion of output data from the plurality of output data output from the millimeter wave sensor using a learning model that estimates output data suitable for estimating a state of an object. Information processing device.

2. 2. The information processing device according to claim 1, the object is a human body, The state is the posture of the human body. Information processing device.

3. 3. The information processing device according to claim 1, The output data includes information about the coordinates, velocity, and noise of a point on the surface of the object. Information processing device.

4. 4. The information processing device according to claim 3, The learning model extracts the portion of output data based on information about the noise, the coordinates, and the velocity. Information processing device.

5. an acquisition step of acquiring a plurality of output data output from the millimeter wave sensor and a true value of a state of an object detected by the millimeter wave sensor; an extraction step of randomly extracting some output data from the plurality of output data; an estimation step of estimating a state of an object detected by the millimeter wave sensor using the extracted output data; an evaluation step of performing a comparison evaluation between the estimated state of the object and the true value; a repeating step of repeating the extracting step, the estimating step, and the evaluating step a predetermined number of times; a selection step of selecting output data that has been estimated to be the state of the object closest to the true value in the repeating step and the plurality of output data. A method for generating data for a learning model.

Images