Electronic device, method for controlling an electronic device, and program

Abstract

To provide an electronic apparatus, a method for controlling an electronic apparatus, and a program that are useful to monitor a monitoring target.SOLUTION: The electronic apparatus mechanically learns a predetermined site of an object by using information on a point group obtained in such a manner that a sensor detects an object that reflects a transmission wave and information of an image obtained in such a manner that an imaging unit takes an image of an object on the basis of a transmission signal sent as a transmission signal and a reception signal received as a reflection signal as the transmission signal reflected.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an electronic device, a method for controlling an electronic device, and a program.

Background Art

[0002] For example, at a site such as a nursing facility, a device for monitoring the behavior of a person to be monitored, such as a person requiring nursing care or a person requiring care, has been proposed. For example, Patent Document 1 discloses a monitoring system that detects a predetermined behavior of a person to be monitored based on an image obtained by an imaging device. Further, Patent Document 2 discloses a sensor that can estimate the position and posture of a living body using a radio signal. Further, Patent Document 3 discloses a processing device that detects that a target has changed its posture from a sleeping posture to a posture in which the upper body is raised based on a change in the reception level of a radio wave.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, the applicant has conducted repeated research on an electronic device that monitors a target to be monitored using an image captured by an imaging unit such as a digital video camera, and has already filed a plurality of patent applications (for example, Japanese Patent Application No. 2020-145012, etc.). Therefore, for example, if it is possible to estimate a body part such as the position of the head or the position of the leg of the target to be monitored based on the result of detecting an object based on the reflected wave of a transmitted wave like a radar, it is beneficial for monitoring the target to be monitored.

[0005] An object of the present disclosure is to provide an electronic device, a control method for the electronic device, and a program that are useful for monitoring an object to be monitored.

Means for Solving the Problems

[0006] An electronic device according to an embodiment Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, information on a point cloud obtained by a sensor detecting an object that reflects the transmission wave, and Using information on an image obtained by an imaging unit imaging the object, Machine learning is performed on a predetermined part of the object. Obtained by the sensor detecting After recognizing a predetermined part of the object The information on the point cloud When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X'max, the coordinates before normalization (X) are converted into information converted into the coordinates after normalization (X') using the following formula (1). X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

[0007] Also, an electronic device according to an embodiment Using information on a point cloud obtained by detecting an object that reflects the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, machine learning is performed on a predetermined part of the object, and Using information on a point cloud obtained by a sensor detecting a monitoring target that reflects the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, A predetermined part of the monitoring target is estimated. Obtained by the sensor detecting After recognizing a predetermined part of the object The information on the point cloud Let the maximum value of the coordinates of the extracted point cloud be Xmax, the minimum value of the extracted coordinates be Xmin, and the maximum value of the normalized coordinates be X’max. When using the following formula (1), the coordinates (X) before normalization are the information converted into the coordinates (X’) after normalization. X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

[0008] A control method for an electronic device according to an embodiment includes: acquiring point cloud information by detecting, with a sensor, an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; acquiring image information by imaging the object with an imaging unit; performing machine learning on a predetermined part of the object; and. The point cloud information obtained by the sensor detecting After recognizing a predetermined part of the object is Let the maximum value of the coordinates of the extracted point cloud be Xmax, the minimum value of the extracted coordinates be Xmin, and the maximum value of the normalized coordinates be X’max. When using the following formula (1), the coordinates (X) before normalization are the information converted into the coordinates (X’) after normalization. X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

[0009] Also, a control method for an electronic device according to an embodiment includes: performing machine learning on a predetermined part of the object using the point cloud information obtained by detecting an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; estimating a predetermined part of the monitored object using the point cloud information obtained by the sensor detecting a monitored object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; and. Obtained by the sensor detecting After recognizing a predetermined part of the object The point cloud information is When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X’max, the coordinates before normalization (X) are converted into the information of the coordinates after normalization (X’) using the following formula (1). X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

[0010] A program according to an embodiment causes an electronic device to acquire point cloud information by detecting, with a sensor, an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; acquire image information by imaging the object with an imaging unit; perform machine learning on a predetermined part of the object; and execute these steps. Obtained by the sensor detecting After recognizing a predetermined part of the object The point cloud information is When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X’max, the coordinates before normalization (X) are converted into the information of the coordinates after normalization (X’) using the following formula (1). X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

[0011] Also, a program according to an embodiment causes an electronic device to perform machine learning on a predetermined part of the object using the point cloud information obtained by detecting an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; Based on the transmission signal transmitted as a transmission wave and the reception signal received as a reflected wave obtained by reflecting the transmission wave, using the information of the point cloud obtained by the sensor detecting the object to be monitored that reflects the transmission wave, a step of estimating a predetermined part in the object to be monitored; Execute it. Obtained by the sensor detecting After recognizing a predetermined part of the object The information of the point cloud is When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X’max, the following formula (1) is used to convert the coordinates (X) before normalization into the information converted into the coordinates (X’) after normalization. X’=((X-Xmin) / (Xmax-Xmin))·X’max (1)

Effect of the Invention

[0012] According to one embodiment, it is possible to provide an electronic device, a control method for the electronic device, and a program that are useful for monitoring an object to be monitored.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0014] In the present disclosure, an "electronic device" may be a device driven by electric power. Also, a "system" may include a device driven by electric power. Further, a "user" may be a person (typically a human) who uses a system and / or an electronic device according to one embodiment. The user may include a person who monitors a subject to be monitored by using a system and / or an electronic device according to one embodiment. Also, a "subject to be monitored" may be a person (e.g., a human or an animal) who is the target to be monitored by a system and / or an electronic device according to one embodiment. Furthermore, the user may include the subject to be monitored.

[0015] Scenarios where an electronic device according to one embodiment is assumed to be used may be, for example, specific facilities used by persons engaging in social activities, such as companies, hospitals, nursing homes, schools, sports gyms, and care facilities. For example, in a company, it is extremely important to grasp and / or manage the health status of employees and the like. Similarly, in a hospital, it is extremely important to grasp and / or manage the health status of patients and medical staff, etc., and in a nursing home, it is extremely important to grasp and / or manage the health status of residents and staff, etc. The scenarios where an electronic device according to one embodiment is used are not limited to the above-mentioned facilities such as companies, hospitals, and nursing homes, but may be any facility where it is desired to grasp and / or manage the health status of the subject to be monitored. Any facility may include, for example, a non-commercial facility such as the user's home. Also, the scenarios where an electronic device according to one embodiment is used may be, for example, inside a moving body such as a train, a bus, and an airplane, as well as stations and boarding areas.

[0016] An electronic device according to one embodiment may be used, for example, in a care facility or the like for the purpose of monitoring the behavior of a subject to be monitored such as a person in need of care or a person requiring nursing care. An electronic device according to one embodiment can monitor, for example, the action of a subject to be monitored such as a person in need of care or a person requiring nursing care slipping off the chair on which they are seated. Here, the action of a subject to be monitored slipping off the chair on which they are seated may be, for example, an action of falling off the chair by sliding their body partially on the seat surface when the subject to be monitored is seated on the seat surface of the chair.

[0017] In particular, for the electronic device according to one embodiment, when a monitored person, such as a person requiring nursing care or a person requiring care, is seated on a chair, for example, before the action of slipping off the chair ends, for example, before falling onto the floor surface or the like, a predetermined warning can be issued. Therefore, according to the electronic device according to one embodiment, for example, the staff in a nursing facility can recognize that the monitored person, such as a person requiring nursing care or a person requiring care, is about to slip off the chair before the monitored person actually slips off the chair.

[0018] Also, the electronic device according to one embodiment can also monitor actions such as a monitored object, such as a person requiring nursing care or a person requiring care, falling while standing. In particular, for the electronic device according to one embodiment, when a monitored person, such as a person requiring nursing care or a person requiring care, is standing, for example, before the action of falling ends, for example, before collapsing onto the floor surface or the like, a predetermined warning can be issued. Therefore, according to the electronic device according to one embodiment, for example, the staff in a nursing facility can recognize that the monitored person, such as a person requiring nursing care or a person requiring care, is about to fall before the monitored person actually falls.

[0019] Also, the electronic device according to one embodiment can also monitor a monitored object, such as a person requiring nursing care or a person requiring care, from falling into various dangerous states. In particular, for the electronic device according to one embodiment, a predetermined warning can be issued before the monitored person, such as a person requiring nursing care or a person requiring care, actually falls into a dangerous state. According to the electronic device according to one embodiment, for example, the staff in a nursing facility can recognize various dangers that can occur to the monitored object, such as a person requiring nursing care or a person requiring care, at an earlier stage.

[0020] The electronic device according to an embodiment described below generates information on a point group that can be processed two-dimensionally, like an image, from information on a point group in a three-dimensional space detected by a sensor based on a technology such as a millimeter-wave radar. The electronic device according to an embodiment can recognize, for example, a body part of a subject to be monitored based on the information on the point group that can be processed two-dimensionally by adopting a technology such as image recognition. By monitoring, for example, at any time, the movement of the body part of the subject to be monitored recognized in this way, the electronic device according to an embodiment can determine whether the subject to be monitored falls into various dangerous states. Hereinafter, the electronic device according to an embodiment will be described in detail with reference to the drawings.

[0021] FIG. 1 is a functional block diagram schematically showing the configuration of an electronic device 1 according to an embodiment. As shown in FIG. 1, the electronic device 1 according to an embodiment includes a controller 10. In one embodiment, the electronic device 1 may appropriately include at least any one of, for example, a storage unit 20, a communication unit 30, a display unit 40, and a notification unit 50. The above-described controller 10, storage unit 20, communication unit 30, display unit 40, notification unit 50, etc. may be arranged or incorporated at any location in the electronic device 1. Further, at least any one of the above-described controller 10, storage unit 20, communication unit 30, display unit 40, notification unit 50, etc. may be arranged outside the electronic device 1. The electronic device 1 according to an embodiment may omit at least a part of the functional units shown in FIG. 1, or may appropriately include other functional units other than the functional units shown in FIG. 1.

[0022] The electronic device 1 according to one embodiment may be various devices. For example, the electronic device according to one embodiment may be any device such as a dedicatedly designed terminal, a general-purpose smartphone, tablet, phablet, notebook personal computer (notebook PC), computer, or server. Further, the electronic device according to one embodiment may have a function of communicating with other electronic devices, such as a mobile phone or a smartphone. Here, the above-mentioned "other electronic devices" may be electronic devices such as a mobile phone or a smartphone, or may be any device such as a base station, server, dedicated terminal, or computer. The "other electronic devices" may be, for example, the sensor 100 and / or the imaging unit 300 described later. Also, the "other electronic devices" in the present disclosure may be devices or apparatuses driven by electric power. When the electronic device according to one embodiment communicates with other electronic devices, it may perform wired and / or wireless communication.

[0023] As shown in FIG. 1, the electronic device 1 according to one embodiment may be connected to the sensor 100 by wire and / or wirelessly. With such a connection, the electronic device 1 according to one embodiment can acquire information on the result detected by the sensor 100. Further, the electronic device 1 according to one embodiment may be connected to the imaging unit 300 by wire and / or wirelessly. With such a connection, the electronic device 1 according to one embodiment can acquire information on the image captured by the imaging unit 300. The sensor 100 and the imaging unit 300 will be described in more detail later.

[0024] The controller 10 controls and / or manages the entire electronic device 1, including each functional unit that constitutes the electronic device 1. To provide control and processing capabilities for executing various functions, the controller 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The controller 10 may be implemented by a single processor, by several processors, or by individual processors respectively. The processor may be implemented as a single integrated circuit. The integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0025] In one embodiment, the controller 10 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or the DSP. Programs executed in the controller 10, and results of processes executed in the controller 10, etc. may be stored, for example, in the storage unit 20. The controller 10 may appropriately include memory necessary for the operation of the controller 10.

[0026] In the electronic device 1 according to one embodiment, the controller 10 can perform various processes on information output as a result of detection by, for example, the sensor 100. For this reason, in the electronic device 1, the controller 10 may be connected to the sensor 100 by wire and / or wirelessly. Also, in the electronic device 1 according to one embodiment, the controller 10 can perform various processes on information (image) output as a result of imaging by, for example, the imaging unit 300. For this reason, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and / or wirelessly. The operation of the controller 10 of the electronic device 1 according to one embodiment will be further described later.

[0027] The storage unit 20 may have a function as a memory for storing various types of information. The storage unit 20 may store, for example, a program executed in the controller 10, and the result of the process executed in the controller 10. The storage unit 20 may also store or accumulate the detection result by the sensor 100 and / or the image captured by the imaging unit 300. Further, the storage unit 20 may function as a working memory of the controller 10. The storage unit 20 can be configured by, for example, a semiconductor memory or the like, but is not limited thereto, and can be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to one embodiment. Further, the storage unit 20 may be configured to include, for example, a hard disk drive (HDD) and / or a solid state drive (SSD). Further, the storage unit 20 may be an internal memory of the CPU used as the controller 10 described later, or may be connected to the controller 10 as a separate unit.

[0028] The storage unit 20 may store, for example, machine learning data. Here, the machine learning data may be data generated by machine learning. Further, machine learning may be based on the technology of AI (Artificial Intelligence) that enables a specific task to be executed by training. More specifically, machine learning may be a technology in which an information processing device such as a computer learns a large amount of data and automatically constructs an algorithm or model for performing tasks such as classification and / or prediction. In this specification, machine learning may be included as a part of AI (Artificial Intelligence).

[0029] In this specification, machine learning may include supervised learning that learns the features or rules of input data based on correct data. Also, machine learning may include unsupervised learning that learns the features or rules of input data without correct data. Furthermore, machine learning may include reinforcement learning that gives rewards or penalties to learn the features or rules of input data. Also, in this specification, machine learning may be an arbitrary combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of the machine learning data of this embodiment may include an algorithm that outputs a predetermined inference (estimation) result using the algorithm learned for the input data. This embodiment may use, for example, linear regression that predicts the relationship between a dependent variable and an independent variable, a neural network (NN) that mathematically models the neurons of the human nervous system, the least squares method that calculates the error by squaring, a decision tree that structures problem solving in a tree structure, and regularization that transforms data in a predetermined method, and other appropriate algorithms. This embodiment may utilize deep learning, which is a type of neural network. Deep learning is a type of neural network, and a neural network with deep network layers is called deep learning.

[0030] In the technology of the present disclosure, it may be generally assumed that there is a certain relationship between the body movement a of the object to be monitored and the movement result A of the object to be monitored generated from this movement a. Here, the movement result may include the movement of the object to be monitored, the start time of the movement of the object to be monitored, accidents, events, and other incidents generated from the movement of the object to be monitored. For example, assume that the body movement a of the object to be monitored is performed and the movement result A of the object to be monitored is generated from this movement a. Also, assume that the body movement b of the object to be monitored is performed and the movement result B of the object to be monitored is generated from this movement b. The technology of the present disclosure accumulates the relationships between the above movement a and movement result A, movement b and movement result B, and other movements and movement results as machine learning data (for example, in the storage unit 20). Then, when the movement x is extracted, the technology of the present disclosure may estimate the movement result X related to the movement x using the above machine learning data.

[0031] In particular, in one embodiment, the machine learning data may be data obtained by machine learning the movement of feature points when the object to be monitored performs a predetermined movement. The machine learning data may also be data obtained by machine learning the movement of feature points when a specific person (for example, a specific person requiring care) performs a predetermined movement as the object to be monitored.

[0032] The communication unit 30 has the function of an interface for communicating by wire or wirelessly. The communication method performed by the communication unit 30 in one embodiment may be a wireless communication standard. For example, the wireless communication standard includes cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (registered trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, the wireless communication standard includes WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, WiFi, Bluetooth (registered trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include, for example, a modem whose communication method is standardized in ITU-T (International Telecommunication Union Telecommunication Standardization Sector). The communication unit 30 can support one or more of the above communication standards.

[0033] The communication unit 30 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. The communication unit 30 may perform wireless communication with, for example, the communication unit of another electronic device via the antenna. Further, the communication unit 30 may be configured as an interface such as a connector for wired connection to the outside. Since the communication unit 30 can be configured by known techniques for performing wireless communication, a more detailed description of hardware and the like is omitted.

[0034] The various types of information received by the communication unit 30 may be supplied to, for example, the storage unit 20 and / or the controller 10. The various types of information received by the communication unit 30 may be stored in, for example, the memory built into the controller 10. Further, the communication unit 30 may transmit, for example, the processing results by the controller 10 and / or the information stored in the storage unit 20 to the outside.

[0035] The display unit 40 may be, for example, any display device such as a liquid crystal display (LCD), an organic electro-luminescence panel, or an inorganic electro-luminescence panel. The display unit 40 may display various types of information such as characters, graphics, or symbols. Further, the display unit 40 may display, for example, objects and icon images that constitute various GUIs in order to prompt the user to operate the electronic device 1. The various types of data necessary for performing display on the display unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. Further, the display unit 40 may be configured to include, for example, a backlight as appropriate when including an LCD or the like. In one embodiment, the display unit 40 may display information based on, for example, the detection result by the sensor 100. Further, in one embodiment, the display unit 40 may display information based on, for example, the imaging result by the imaging unit 300.

[0036] The notification unit 50 may notify a predetermined warning for prompting attention to the user of the electronic device 1 or the like based on a predetermined signal output from the controller 10. The notification unit 50 may be any functional unit that stimulates at least one of the user's hearing, vision, and touch as a predetermined warning, such as sound, voice, light, text, video, and vibration. Specifically, the notification unit 50 may be at least one of, for example, an audio output unit such as a buzzer or a speaker, a light emitting unit such as an LED, a display unit such as an LCD, and a tactile presentation unit such as a vibrator. Thus, the notification unit 50 may notify a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the notification unit 50 may notify a predetermined alarm as information acting on at least one of hearing, vision, and touch.

[0037] In one embodiment, the notification unit 50 may notify a warning that there is a risk of the monitored object slipping off the chair before starting the action of slipping off the chair on which the monitored object was seated. Also, in one embodiment, the notification unit 50 may notify a warning that there is a risk of the monitored object slipping off the chair before ending the action of slipping off the chair on which the monitored object was seated. For example, in one embodiment, the notification unit 50 that outputs visual information may warn the user of that fact by emitting light or a predetermined display when it is detected that there is a risk of the monitored object slipping off the chair. Also, in one embodiment, the notification unit 50 that outputs auditory information may warn the user of that fact by a predetermined sound or voice when it is detected that there is a risk of the monitored object slipping off the chair. In this embodiment, the above warning may combine light emission or a predetermined display, and a predetermined sound or voice.

[0038] The electronic device 1 shown in FIG. 1 incorporates the notification unit 50. However, in one embodiment, the notification unit 50 may be provided outside the electronic device 1. In this case, the notification unit 50 and the electronic device 1 may be connected by wire, wirelessly, or a combination of wire and wireless.

[0039] At least a part of each functional unit constituting the electronic device 1 according to one embodiment, as shown in FIG. 1, may be configured by specific means in which software and hardware resources cooperate with each other.

[0040] The sensor 100 shown in FIG. 1 is configured to detect, for example, a human body such as an object to be monitored as information on a point cloud in a three-dimensional space. Hereinafter, the sensor 100 according to one embodiment will be described in more detail.

[0041] FIG. 2 is a functional block diagram schematically showing the configuration of the sensor 100 according to one embodiment. The sensor 100 shown in FIG. 2 is, for example, based on the technology of a millimeter-wave radar (RADAR (Radio Detecting and Ranging)) (millimeter-wave radar sensor). However, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor. The sensor 100 according to one embodiment may be, for example, a quasi-millimeter-wave radar sensor. Further, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor or a quasi-millimeter-wave radar sensor, and may be various radar sensors that transmit and receive radio waves. Further, the sensor 100 according to one embodiment may be, for example, a sensor based on a technology such as a microwave sensor, an ultrasonic sensor, or LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0042] When measuring distance or the like using a millimeter-wave radar, a frequency-modulated continuous-wave radar (hereinafter referred to as an FMCW radar (Frequency Modulated Continuous Wave radar)) may be used. The FMCW radar generates a transmission signal by sweeping the frequency of the transmitted radio wave. Therefore, for example, in a millimeter-wave FMCW radar using radio waves in the 79 GHz frequency band, the frequency of the radio waves used has a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. The radar in the 79 GHz frequency band has a characteristic that the available frequency bandwidth is wider than that of other millimeter-wave / sub-millimeter-wave radars, such as those in the 24 GHz, 60 GHz, and 76 GHz frequency bands. Hereinafter, as an example, the case of adopting such an FMCW radar will be described. The FMCW radar system used in the present disclosure may include an FCM method (Fast-Chirp Modulation) that transmits a chirp signal in a shorter period than usual. The signal generated by the sensor 100 is not limited to an FMCW signal. The signal generated by the sensor 100 may be a signal of various methods other than the FMCW method. The transmission signal sequence stored as the transmitted signal may be different for these various methods. For example, in the case of the above-described FMCW radar signal, signals that increase and decrease in frequency for each time sample may be used. Since the above-described various methods can appropriately apply known techniques, more detailed descriptions will be omitted as appropriate.

[0043] As shown in FIG. 2, the sensor 100 according to an embodiment may be configured to include a radar control unit 110, a transmission unit 120, and a reception unit 130. The above-described radar control unit 110, transmission unit 120, and reception unit 130 may be arranged or incorporated at any location in the sensor 100. Further, at least any one of the above-described radar control unit 110, transmission unit 120, and reception unit 130 may be arranged outside the sensor 100. The sensor 100 according to an embodiment may omit at least a part of the functional units shown in FIG. 2, or may appropriately include other functional units other than the functional units shown in FIG. 2.

[0044] The radar control unit 110 controls and / or manages the entire sensor 100, including each functional unit that constitutes the sensor 100. The radar control unit 110 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for executing various functions. The radar control unit 110 may be implemented by a single processor, by several processors, or by individual processors respectively. The processor may be implemented as a single integrated circuit. An integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0045] In one embodiment, the radar control unit 110 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or the DSP. Programs executed in the radar control unit 110, and results of processes executed in the radar control unit 110, etc., may be stored, for example, in any storage unit built into the radar control unit 110. The radar control unit 110 may appropriately include memory necessary for the operation of the radar control unit 110.

[0046] In the sensor 100 according to one embodiment, the radar control unit 110 may appropriately perform various processes such as distance FFT (Fast Fourier Transform) processing, velocity FFT processing, arrival angle estimation processing, and clustering processing. Since each of these processes performed by the radar control unit 110 is known as general radar technology, a more detailed description is omitted.

[0047] As shown in FIG. 2, the transmission unit 120 may include a signal generation unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmission antenna 125. The sensor 100 according to one embodiment may include a plurality of transmission antennas 125. In this case, the sensor 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to each of the plurality of transmission antennas 125.

[0048] As shown in FIG. 2, the reception unit 130 may include a reception antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. The sensor 100 according to one embodiment may include a plurality of reception units 130 corresponding to each of the plurality of transmission antennas 125.

[0049] In the sensor 100 according to one embodiment, the radar control unit 110 can control at least one of the transmission unit 120 and the reception unit 130. In this case, the radar control unit 110 may control at least one of the transmission unit 120 and the reception unit 130 based on various information stored in an arbitrary storage unit. For example, an arbitrary storage unit built into the radar control unit 110 may store various parameters for setting a range for detecting an object by the transmitted wave transmitted from the transmission antenna 125 and the reflected wave received from the reception antenna 131. Also, in the sensor 100 according to one embodiment, the radar control unit 110 may instruct the signal generation unit 121 to generate a signal or control the signal generation unit 121 to generate a signal.

[0050] The signal generation unit 121 generates a signal (transmission signal) transmitted as a transmission wave from the transmission antenna 125 under the control of the radar control unit 110. When generating the transmission signal, the signal generation unit 121 may assign the frequency of the transmission signal, for example, based on the control by the radar control unit 110. Specifically, the signal generation unit 121 may assign the frequency of the transmission signal according to the parameters set by the radar control unit 110, for example. For example, the signal generation unit 121 receives frequency information from the radar control unit 110 or an arbitrary storage unit, and generates a signal with a predetermined frequency in a frequency band such as 77 to 81 GHz. The signal generation unit 121 may be configured to include a functional unit such as a voltage controlled oscillator (VCO).

[0051] The signal generation unit 121 may be configured as hardware having the function, or may be configured by, for example, a microcomputer, or may be configured as a processor such as a CPU or a DSP and a program executed by the processor. Each functional unit described below may also be configured as hardware having the function, or may be configured by, for example, a microcomputer if possible, or may be configured as a processor such as a CPU or a DSP and a program executed by the processor.

[0052] In the sensor 100 according to an embodiment, the signal generation unit 121 may generate a transmission signal (transmission chirp signal) such as a chirp signal. In particular, the signal generation unit 121 may generate a signal whose frequency linearly changes periodically (linear chirp signal). For example, the signal generation unit 121 may be a chirp signal whose frequency periodically and linearly increases from 77 GHz to 81 GHz as time elapses. Also, for example, the signal generation unit 121 may generate a signal whose frequency periodically repeats linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz as time elapses. The signal generated by the signal generation unit 121 may be preset, for example, in the radar control unit 110. Also, the signal generated by the signal generation unit 121 may be pre-stored, for example, in an arbitrary storage unit or the like. Since chirp signals used in technical fields such as radar are known, more detailed descriptions will be appropriately simplified or omitted. The signal generated by the signal generation unit 121 is supplied to the synthesizer 122.

[0053] FIG. 3 is a diagram for explaining an example of the chirp signal generated by the signal generation unit 121.

[0054] In FIG. 3, the horizontal axis represents the elapsed time, and the vertical axis represents the frequency. In the example shown in FIG. 3, the signal generation unit 121 generates a linear chirp signal whose frequency linearly changes periodically. In FIG. 3, each chirp signal is shown as c1, c2, …, c8. As shown in FIG. 3, in each chirp signal, the frequency linearly increases as time elapses.

[0055] In the example shown in FIG. 3, including eight chirp signals such as c1, c2, …, c8, it is regarded as one subframe. That is, subframe 1 and subframe 2 shown in FIG. 3, etc., are each composed of including eight chirp signals such as c1, c2, …, c8. Also, in the example shown in FIG. 3, including 16 subframes such as subframe 1 to subframe 16, it is regarded as one frame. That is, frame 1 and frame 2 shown in FIG. 3, etc., are each composed of including 16 subframes. Also, as shown in FIG. 3, between frames, a frame interval of a predetermined length may be included. One frame shown in FIG. 3 may have a length of about 30 milliseconds to 50 milliseconds, for example.

[0056] In FIG. 3, the subsequent frames such as frame 2 may have the same configuration. Also, in FIG. 3, the subsequent frames such as frame 3 may have the same configuration. In the sensor 100 according to an embodiment, the signal generation unit 121 may generate a transmission signal as any number of frames. Also, in FIG. 3, some of the chirp signals are shown omitted. In this way, the relationship between the time and frequency of the transmission signal generated by the signal generation unit 121 may be stored in, for example, any storage unit.

[0057] In this way, the sensor 100 according to an embodiment may transmit a transmission signal composed of subframes including a plurality of chirp signals. Also, the sensor 100 according to an embodiment may transmit a transmission signal composed of frames including a predetermined number of subframes.

[0058] Hereinafter, the sensor 100 will be described as transmitting a transmission signal having a frame structure as shown in FIG. 3. However, the frame structure as shown in FIG. 3 is an example, and for example, the number of chirp signals included in one sub-frame is not limited to eight. In one embodiment, the signal generation unit 121 may generate a sub-frame including an arbitrary number (for example, an arbitrary plurality) of chirp signals. Also, the sub-frame structure as shown in FIG. 3 is an example, and for example, the number of sub-frames included in one frame is not limited to 16. In one embodiment, the signal generation unit 121 may generate a frame including an arbitrary number (for example, an arbitrary plurality) of sub-frames. The signal generation unit 121 may generate signals having different frequencies. The signal generation unit 121 may generate a plurality of discrete signals having different bandwidths with different frequencies f.

[0059] Returning to FIG. 2, the synthesizer 122 raises the frequency of the signal generated by the signal generation unit 121 to the frequency in a predetermined frequency band. The synthesizer 122 may raise the frequency of the signal generated by the signal generation unit 121 to the frequency selected as the frequency of the transmission wave transmitted from the transmission antenna 125. The frequency selected as the frequency of the transmission wave transmitted from the transmission antenna 125 may be set by, for example, the radar control unit 110. Also, the frequency selected as the frequency of the transmission wave transmitted from the transmission antenna 125 may be stored in, for example, an arbitrary storage unit. The signal whose frequency has been raised by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. When there are a plurality of phase control units 123, the signal whose frequency has been raised by the synthesizer 122 may be supplied to each of the plurality of phase control units 123. Also, when there are a plurality of reception units 130, the signal whose frequency has been raised by the synthesizer 122 may be supplied to each mixer 133 in the plurality of reception units 130.

[0060] The phase control unit 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control unit 123 may adjust the phase of the transmission signal by advancing or delaying the phase of the signal supplied from the synthesizer 122 as appropriate based on the control by, for example, the radar control unit 110. In this case, the phase control unit 123 may adjust the phase of each transmission signal based on the path difference of each transmission wave transmitted from the plurality of transmission antennas 125. By appropriately adjusting the phase of each transmission signal by the phase control unit 123, the transmission waves transmitted from the plurality of transmission antennas 125 reinforce each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the direction of beamforming and the phase amount to be controlled of the transmission signals to be transmitted by the plurality of transmission antennas 125 may be stored, for example, in an arbitrary storage unit. The transmission signal whose phase is controlled by the phase control unit 123 is supplied to the amplifier 124.

[0061] The amplifier 124 amplifies the power (electric power) of the transmission signal supplied from the phase control unit 123 based on the control by, for example, the radar control unit 110. When the sensor 100 includes a plurality of transmission antennas 125, the plurality of amplifiers 124 may respectively amplify the power (electric power) of the transmission signals supplied from the corresponding ones of the plurality of phase control units 123 based on the control by, for example, the radar control unit 110. Since the technology of amplifying the power of the transmission signal itself is already known, a more detailed description is omitted. The amplifier 124 is connected to the transmission antenna 125.

[0062] The transmission antenna 125 outputs (transmits) the transmission signal amplified by the amplifier 124 as a transmission wave. When the sensor 100 includes a plurality of transmission antennas 125, the plurality of transmission antennas 125 may respectively output (transmit) the transmission signals amplified by the corresponding ones of the plurality of amplifiers 124 as transmission waves. Since the transmission antenna 125 can be configured in the same manner as the transmission antenna used in known radar technology, a more detailed description is omitted.

[0063] In this way, the sensor 100 according to one embodiment includes a transmission antenna 125 and can transmit a transmission signal (e.g., a transmission chirp signal) as a transmission wave from the transmission antenna 125. Here, at least one of the functional units constituting the sensor 100 may be housed in one housing. Also, in this case, the one housing may have a structure that cannot be easily opened. For example, it is preferable that the transmission antenna 125, the reception antenna 131, and the amplifier 124 are housed in one housing and this housing has a structure that cannot be easily opened. Furthermore,

[0064] The sensor 100 shown in FIG. 2 shows an example of including one transmission antenna 125. However, in one embodiment, the sensor 100 may include any number of transmission antennas 125. On the other hand, in one embodiment, when the transmission wave transmitted from the transmission antenna 125 forms a beam in a predetermined direction, the sensor 100 may include a plurality of transmission antennas 125. In one embodiment, the sensor 100 may include any plurality of transmission antennas 125. In this case, the sensor 100 may also include a plurality of phase control units 123 and a plurality of amplifiers 124 corresponding to the plurality of transmission antennas 125, respectively. Then, the plurality of phase control units 123 may control the phases of the plurality of transmission waves supplied from the synthesizer 122 and transmitted from the plurality of transmission antennas 125, respectively. Also, the plurality of amplifiers 124 may amplify the powers of the plurality of transmission signals transmitted from the plurality of transmission antennas 125, respectively. Also, in this case, the sensor 100 may be configured to include a plurality of transmission antennas. In this way, when the sensor 100 includes a plurality of transmission antennas 125, the functional units necessary for transmitting transmission waves from the plurality of transmission antennas 125 may also be configured to include a plurality of each.

[0065] The receiving antenna 131 receives the reflected wave. The reflected wave may be the one obtained by reflecting the transmitted wave from a predetermined object 200. The receiving antenna 131 may be configured to include a plurality of antennas. Since the receiving antenna 131 can be configured in the same manner as the receiving antenna used in known radar technology, a more detailed description is omitted. The receiving antenna 131 is connected to the LNA 132. The received signal based on the reflected wave received by the receiving antenna 131 is supplied to the LNA 132.

[0066] The sensor 100 according to one embodiment can receive the reflected wave obtained by reflecting the transmitted wave (transmission chirp signal) transmitted as a transmission signal, such as a chirp signal, from a plurality of receiving antennas 131 by a predetermined object 200. Thus, when transmitting a transmission chirp signal as the transmitted wave, the received signal based on the received reflected wave is referred to as a received chirp signal. That is, the sensor 100 receives the received signal (for example, received chirp signal) as the reflected wave from the receiving antenna 131.

[0067] The LNA 132 amplifies the received signal based on the reflected wave received by the receiving antenna 131 with low noise. The LNA 132 is preferably a low noise amplifier, and amplifies the received signal supplied from the receiving antenna 131 with low noise. The received signal amplified by the LNA 132 is supplied to the mixer 133.

[0068] The mixer 133 generates a beat signal by mixing (multiplying) the received signal of RF frequency supplied from the LNA 132 with the transmission signal supplied from the synthesizer 122. The beat signal mixed by the mixer 133 is supplied to the IF unit 134.

[0069] The IF unit 134 reduces the frequency of the beat signal to an intermediate frequency (IF (Intermediate Frequency) frequency) by performing frequency conversion on the beat signal supplied from the mixer 133. The beat signal whose frequency has been reduced by the IF unit 134 is supplied to the AD conversion unit 135.

[0070] The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be configured by an arbitrary analog-to-digital conversion circuit (Analog to Digital Converter (ADC)). The beat signal digitized by the AD conversion unit 135 is supplied to the radar control unit 110. When there are a plurality of receiving units 130, each beat signal digitized by the plurality of AD conversion units 135 may be supplied to the radar control unit 110.

[0071] The radar control unit 110 may perform FFT processing (hereinafter, appropriately referred to as "range FFT processing") on the beat signal digitized by the AD conversion unit 135. For example, the radar control unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be represented as a temporal change in signal intensity (power). By performing FFT processing on such a beat signal, the radar control unit 110 can represent it as the signal intensity (power) corresponding to each frequency. By performing range FFT processing in the radar control unit 110, a complex signal corresponding to the range can be obtained based on the beat signal digitized by the AD conversion unit 135.

[0072] When the peak in the result obtained by the range FFT processing is equal to or greater than a predetermined threshold, the radar control unit 110 may determine that there is a predetermined object 200 at the range corresponding to the peak. For example, there is known a method of determining that there is an object (reflecting object) that reflects the transmitted wave when a peak value equal to or greater than the threshold is detected from the average power or amplitude of the disturbance signal, such as detection processing based on a constant false alarm rate (CFAR).

[0073] As described above, the sensor 100 according to one embodiment can detect an object 200 that reflects the transmitted wave, targeting the transmitted signal transmitted as the transmitted wave and the received signal received as the reflected wave.

[0074] The radar control unit 110 can estimate the distance to a predetermined object based on one chirp signal (e.g., c1 shown in FIG. 3). That is, the sensor 100 can measure (estimate) the distance between the sensor 100 and the predetermined object 200 by performing distance FFT processing. Since the technology of measuring (estimating) the distance to a predetermined object by performing FFT processing on the beat signal is already known, a more detailed description will be appropriately simplified or omitted as needed.

[0075] Further, the radar control unit 110 may perform further FFT processing (hereinafter, appropriately referred to as "velocity FFT processing") on the beat signal on which the distance FFT processing has been performed. For example, the radar control unit 110 may perform FFT processing on the complex signal obtained by the distance FFT processing. The radar control unit 110 can estimate the relative velocity to a predetermined object based on a sub-frame of the chirp signal (e.g., sub-frame 1 shown in FIG. 3). By performing velocity FFT processing on a plurality of chirp signals in the radar control unit 110, a complex signal corresponding to the relative velocity can be obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

[0076] When distance FFT processing is performed on the beat signal as described above, a plurality of vectors can be generated. By obtaining the phase of the peak in the result of performing velocity FFT processing on these plurality of vectors, the relative velocity to a predetermined object can be estimated. That is, the electronic device 1 can measure (estimate) the relative velocity between the sensor 100 and the predetermined object 200 by performing velocity FFT processing. Since the technology of measuring (estimating) the relative velocity to a predetermined object by performing velocity FFT processing on the result of performing distance FFT processing is already known, a more detailed description will be appropriately simplified or omitted as needed.

[0077] In general FMCW radar technology, it is possible to determine whether a target exists based on the result of performing fast Fourier transform processing on the extracted beat frequency from the received signal. Here, the result of extracting the beat frequency from the received signal and performing fast Fourier transform processing includes components of noise (such as clutter (unwanted reflection components)). Therefore, it may be possible to execute a process for removing the noise component from the result of processing the received signal and extracting only the target signal.

[0078] Also, the radar control unit 110 may estimate the direction (arrival angle) from which the reflected wave arrives from a predetermined object 200 based on the determination of whether a target exists. The radar control unit 110 may perform the estimation of the arrival angle for the point where it is determined that a target exists. The sensor 100 can estimate the direction from which the reflected wave arrives by receiving the reflected wave from a plurality of receiving antennas 131. For example, it is assumed that the plurality of receiving antennas 131 are arranged at a predetermined interval. In this case, the transmitted wave transmitted from the transmitting antenna 125 is reflected by the predetermined object 200 to become a reflected wave, and the plurality of receiving antennas 131 arranged at a predetermined interval each receive the reflected wave R. Then, the radar control unit 110 can estimate the direction in which the reflected wave arrives at the receiving antenna 131 based on the phase of the reflected wave received by each of the plurality of receiving antennas 131 and the path difference of each reflected wave. That is, the sensor 100 can measure (estimate) the arrival angle θ indicating the direction in which the reflected wave reflected by the target arrives based on the result of performing the velocity FFT processing.

[0079] Various techniques have been proposed for estimating the direction from which the reflected wave R arrives based on the result of performing velocity FFT processing. For example, as known algorithms for estimating the direction of arrival, MUSIC (MUltiple SIgnal Classification), ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique), etc. are known. Therefore, a more detailed explanation of the known techniques will be appropriately simplified or omitted as needed.

[0080] The radar control unit 110 detects an object existing in the range where the transmission wave was transmitted based on at least any one of distance FFT processing, velocity FFT processing, and arrival angle estimation. The radar control unit 110 may perform object detection, for example, by performing clustering processing based on the supplied distance information, velocity information, and angle information. As an algorithm used for clustering data, for example, DBSCAN (Density-based spatial clustering of applications with noise) etc. are known. In the clustering processing, for example, the average power of the points constituting the detected object may be calculated.

[0081] As described above, the sensor 100 can detect an object that reflects the transmission wave as point cloud information in a three-dimensional space. That is, in one embodiment, based on the detection result output from the sensor 100, it is possible to determine (detect) whether or not there is an object that reflects the transmission wave at a certain coordinate in the three-dimensional space. Also, in one embodiment, the sensor 100 can detect the signal intensity and velocity at each point in the three-dimensional space. As described above, the sensor 100 according to one embodiment may detect an object that reflects the transmission wave as point cloud information in a three-dimensional space based on the transmission signal transmitted as the transmission wave and the reception signal received as the reflected wave reflected by the transmission wave.

[0082] Further, as shown in FIG. 1, the electronic device 1 according to one embodiment may be configured to include an imaging unit 300. The electronic device 1 and the imaging unit 300 may be connected by wire, wirelessly, or a combination of wire and wireless.

[0083] The imaging unit 300 may be configured to include an image sensor that electronically captures an image, such as a digital camera. The imaging unit 300 may be configured to include an imaging element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture an image of, for example, an object to be monitored. Here, the object to be monitored may be, for example, a human. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the extraction unit 11, the storage unit 20, and / or the controller 10 of the electronic device 1. The imaging unit 300 is not limited to an imaging device such as a digital camera as long as it captures an object to be monitored, and may be any device.

[0084] In one embodiment, the imaging unit 300 may capture an image of an object to be monitored as a still image at predetermined intervals (for example, 15 frames per second). Also, in one embodiment, the imaging unit 300 may capture an image of an object to be monitored as a continuous moving image.

[0085] Next, the operation of the electronic device 1 according to one embodiment will be described.

[0086] The operation of the electronic device 1 according to one embodiment can typically be divided into a "learning phase" and an "estimation phase". In the learning phase, for example, when a human such as a person to be monitored performs a predetermined action, an operation of performing machine learning on the relationship between the positions (coordinates) of each part of the body in the action and the timing of the action may be performed. Further, in the estimation phase, based on the result of machine learning in the learning phase, an operation of estimating the start of the action may be performed from the positions (coordinates) of each part of the body when the person to be monitored performs a predetermined action. Furthermore, in the estimation phase, for example, when it is determined that there is a risk of a predetermined level or more, such as when the above-described action starts, a predetermined alarm or the like may be output.

[0087] Also, the operation of the electronic device 1 according to one embodiment may determine the presence or absence of a risk according to a predetermined algorithm without dividing it into a "learning phase" and an "estimation phase". For example, in the electronic device 1 according to one embodiment, factors of risks that can be applied to the person to be monitored may be defined in advance as an algorithm. In this case, when it is determined that the risk exceeds a predetermined level according to a predetermined algorithm, the electronic device 1 according to one embodiment may output a predetermined alarm or the like.

[0088] FIG. 4 is a flowchart showing an example of the operation performed by the electronic device 1 according to one embodiment. The operation shown in FIG. 4 may start, for example, when the electronic device 1 according to one embodiment performs an operation of monitoring a person to be monitored. Hereinafter, a case where the person to be monitored by the electronic device 1 is a human will be assumed for explanation. However, the person to be monitored by the electronic device 1 is not limited to a human, and may be an animal other than a human or the like. Further, the person to be monitored is not limited to a human or an animal, and may be a machine such as a work robot, for example. The operation shown in FIG. 4 may be based on the processing executed by the controller 10 of the electronic device 1. That is, in the following description, the operation and / or processing performed by the "electronic device 1" may be, for example, the operation and / or processing executed by the controller 10 of the electronic device 1.

[0089] When the operation shown in FIG. 4 starts, the electronic device 1 acquires the point group information output by the sensor 100 (step S1). Here, the point group information output by the sensor 100 may be the information detected by the sensor 100. That is, the information detected by the sensor 100 may be the information detected as the point group information of the object that reflects the transmission wave in the three-dimensional space based on the transmission signal transmitted as the transmission wave and the reception signal received as the reflected wave reflected by the transmission wave.

[0090] When the point group information is acquired in step S1, the electronic device 1 executes preprocessing of the acquired point group information (step S2). The preprocessing of the point group information performed in step S2 may be to generate point group information that can be processed two-dimensionally from the point group information in the three-dimensional space. That is, when the preprocessing of the point group information is performed in step S2, the point group information becomes information that can be processed two-dimensionally and can be processed in the same way as image data, for example, by performing image recognition. The preprocessing of the point group information performed in step S2 will be further described later.

[0091] When the preprocessing of the point cloud information is executed in step S2, the electronic device 1 recognizes, for example, the body part of the object to be monitored based on the point cloud information that can be processed two-dimensionally (step S3). In step S3, the electronic device 1 may recognize the body part of the object to be monitored based on the point cloud information that can be processed two-dimensionally by adopting a technique such as image recognition. Also, in step S3, the electronic device 1 may extract the coordinates of a predetermined part obtained by image recognition, for example. As described above, by performing the processing in step S2, the point cloud information in the three-dimensional space becomes the point cloud information that can be processed two-dimensionally. Therefore, in step S3, the electronic device 1 can process the point cloud information as if handling an image. In step S3, the electronic device 1 may associate each point of the point cloud that can be processed two-dimensionally with a pixel in the image. In step S3, the electronic device 1 may determine each part such as the head, neck, shoulders, arms, hands, torso, or legs of the object to be monitored by image recognition based on the point cloud information that can be processed two-dimensionally.

[0092] When the body part is recognized in step S3, the electronic device 1 normalizes the coordinates of the part (step S4). By normalizing the coordinates of the part in step S4, the electronic device 1 can process the point cloud information that can be processed two-dimensionally regardless of differences such as viewing angles, for example. By normalizing the coordinates of the part in step S4, data for predicting the behavior of the object to be monitored can be generated (step S5).

[0093] Here, the normalization of the coordinates of the part performed in step S4 described above will be explained. The coordinates of the predetermined part extracted in step S3 are assumed to vary due to, for example, the size of the body of the object to be monitored. Also, the coordinates of the predetermined part extracted in step S3 are assumed to vary due to, for example, the distance between the sensor 100 and the object to be monitored, and the direction in which the sensor 100 faces the object to be monitored. Therefore, in one embodiment, by normalizing the X - direction component and the Y - direction component of the coordinates extracted in step S3 respectively, the extracted coordinates can be generally used for machine learning.

[0094] In this case, for example, based on the maximum and minimum values of the X and Y coordinates extracted in 15 frames per second, the extracted X and Y coordinates may be normalized. Here, let the maximum value of the X coordinates extracted in step S3 be Xmax, and the minimum value of the X coordinates extracted in step S3 be Xmin. Also, let the maximum value of the X coordinates after normalization be X’max. In this case, using the following formula (1), the X coordinate (X) before normalization can be converted into the X coordinate (X’) after normalization.

[0095] X’ = ((X - Xmin) / (Xmax - Xmin))·X’max (1)

[0096] Similarly, let the maximum value of the Y coordinates extracted in step S3 be Ymax, and the minimum value of the Y coordinates extracted in step S3 be Ymin. Also, let the maximum value of the Y coordinates after normalization be Y’max. In this case, using the following formula (2), the Y coordinate (Y) before normalization can be converted into the Y coordinate (Y’) after normalization.

[0097] Y’ = ((Y - Ymin) / (Ymax - Ymin))·Y’max (2)

[0098] By normalizing the X - direction component and Y - direction component of the extracted coordinates according to the above formulas (1) and (2), it is expected to reduce the individual differences of the object to be monitored and the influence of the environment in which the sensor 100 images the object to be monitored on machine learning.

[0099] Thus, in one embodiment, the controller 10 may normalize each direction component of the coordinates of a predetermined number of joint points in the body of the object to be monitored extracted two - dimensionally based on the maximum value and minimum value of each direction component.

[0100] By normalizing the coordinates in step S4, each coordinate (X, Y) shown in FIG. 5 is normalized to the coordinate (X’, Y’), respectively.

[0101] Next, the data generation in step S5 will be further described. As described above, the data generation in step S5 may be to generate data for predicting the behavior of the object to be monitored. When the sensor 100 images each image of a predetermined number of frames per second, the controller 10 may extract a predetermined part of the body of the object to be monitored in a predetermined number of frames per second. Also, when the controller 10 acquires images of a predetermined number of frames per second, the controller 10 may extract a predetermined part of the body of the object to be monitored in a predetermined number of frames per second. As an example, the controller 10 may extract a predetermined part of the body of the object to be monitored in 15 frames per second.

[0102] FIG. 5 is a diagram showing coordinates of a predetermined part extracted from the body of a subject to be monitored, for example, in 15 frames per second. As shown in FIG. 5, in step S5, the controller 10 may arrange the coordinates of the predetermined part extracted from the body of the subject to be monitored for each frame. As shown in FIG. 5, the controller 10 may extract the coordinates of the predetermined part two-dimensionally (as X, Y coordinates) from the body of the subject to be monitored for each frame. In the table shown in FIG. 5, each row schematically shows how the predetermined part of the body of the subject to be monitored is extracted as X, Y coordinates in each frame. Also, in the table shown in FIG. 5, the rows indicating each frame are shown from top to bottom according to the passage of time. The coordinates of the 15 frames shown in FIG. 5 may be, for example, those obtained by tracking the coordinates for 1 second in an image (or video). Also, after the 15 frames shown in FIG. 5, the coordinates of the predetermined part may be sequentially extracted from the body of the subject to be monitored.

[0103] As described above, in one embodiment, the controller 10 may two-dimensionally extract the coordinates of a predetermined number of joint points in the body of the subject to be monitored for each image of a predetermined number of frames per second captured by the sensor 100.

[0104] When data is generated in step S5, the electronic device 1 predicts the behavior of the subject to be monitored (step S6). The method of predicting the behavior of the subject to be monitored in step S6 can be variously assumed, such as, for example, by machine learning or simply by an algorithm without machine learning. In the present disclosure, since the method of predicting the behavior of the subject to be monitored is not a main feature, a more detailed description is omitted.

[0105] If the behavior of the object to be monitored is predicted in step S5, the electronic device 1 determines whether there is a risk of a predetermined level or more with respect to the behavior of the object to be monitored (step S7). When performing the process of step S7, the electronic device 1 may set criteria for risks for various behaviors of the object to be monitored. Then, the electronic device 1 may determine whether the behavior of the object to be monitored exceeds the criteria for the risk (that is, whether there is a risk of a predetermined level or more).

[0106] If there is no risk of a predetermined level or more in step S7, the electronic device 1 may determine that risk notification is not necessary and end the operation shown in FIG. 4. On the other hand, if there is a risk of a predetermined level or more in step S7, the electronic device 1 issues a predetermined warning from the notification unit 50 (step S8).

[0107] In this way, the electronic device 1 according to one embodiment can monitor that the object to be monitored falls into various dangerous states. In particular, the electronic device 1 according to one embodiment can issue a predetermined warning before the person to be monitored actually falls into a dangerous state. According to the electronic device 1 according to one embodiment, various dangers that can be imposed on the object to be monitored can be recognized at an earlier stage.

[0108] Next, the preprocessing of the point group information executed in step S2 of FIG. 4 will be further described.

[0109] FIG. 6 is a flowchart showing in more detail the preprocessing of the point group information performed by the electronic device 1 in step S2 of FIG. 4. The operation shown in FIG. 6 may be based on the process executed by the controller 10 of the electronic device 1. That is, in the following description, the operations and / or processes performed by the "electronic device 1" may be, for example, the operations and / or processes executed by the controller 10 of the electronic device 1.

[0110] When the operation shown in FIG. 6 starts, the electronic device 1 reduces the dimensionality of the point cloud information (step S11). More specifically, in step S11, the electronic device 1 generates point cloud information that can be processed two-dimensionally from the point cloud information in the three-dimensional space output by the sensor 100.

[0111] FIG. 7 is a diagram for explaining the point cloud information in the three-dimensional space output by the sensor 100. FIG. 7 shows an example of a situation where a monitored object Tm standing at a certain location is three-dimensionally detected by the sensor 100 installed at the origin O.

[0112] As shown in FIG. 7, with the position where the sensor 100 is installed as a reference (origin O), the direction approaching the monitored object Tm is defined as the positive X-axis direction, and the direction moving away from the monitored object Tm is defined as the negative X-axis direction. Also, as shown in FIG. 7, with the position where the sensor 100 is installed as a reference (origin O), the right side of the sensor 100 is defined as the positive Y-axis direction, and the left side of the sensor 100 is defined as the negative Y-axis direction. Further, as shown in FIG. 7, with the position where the sensor 100 is installed as a reference (origin O), the upper side of the sensor 100 is defined as the positive Z-axis direction, and the lower side of the sensor 100 is defined as the negative Z-axis direction. That is, in FIG. 7, lx indicates the distance in the depth direction, ly indicates the distance in the horizontal direction, and lz indicates the distance in the vertical (height) direction. In the situation shown in FIG. 7, the output of the sensor 100 may be, at a certain moment, data of four channels of data elements (signal intensity and speed) of the position (X, Y, Z) in the three-dimensional space. In step S11 shown in FIG. 7, the electronic device 1 converts the information detected in such a three-dimensional space into two-dimensional information.

[0113] FIG. 8 is a diagram for explaining an example of generating point cloud information that can be processed two-dimensionally from the point cloud information in the three-dimensional space output by the sensor 100. FIG. 8 shows an example of converting what has been three-dimensionally (spatially) detected by the sensor 100 installed at the origin O for a monitored object Tm standing at a certain location into two-dimensionally (planarly) in the electronic device 1.

[0114] As shown in FIG. 8, with the position of the origin O as a reference, the right side of the origin O is defined as the positive X-axis direction, and the lower side of the origin O is defined as the positive Y-axis direction. That is, in FIG. 8, px indicates the horizontal coordinate, and py indicates the vertical coordinate. In the situation shown in FIG. 8, the output of the sensor 100 is converted, at a certain moment, from the data of the above-described four channels into the data of three channels of the data elements (signal intensity and velocity) of the position (X, Y) in the two-dimensional plane. Thus, in step S11 shown in FIG. 7, the electronic device 1 converts the information detected in the three-dimensional space into the information in the two-dimensional plane.

[0115] When converting the information detected in the three-dimensional space into the information in the two-dimensional plane as described above, the coordinates in the two-dimensional plane may be calculated based on, for example, the following formulas (3) and (4).

[0116]

Equation

[0117]

Equation

[0118] In the above formulas (3) and (4), lx i , ly i , lz i represents the output based on the detection result by the sensor 100, that is, the information of the point cloud in the three-dimensional space. In particular, lx i represents the distance in the x direction of the i-th information detected by the sensor 100. Also, ly i represents the distance in the y direction of the i-th information detected by the sensor 100. Also, lz i represents the distance in the z direction of the i-th information detected by the sensor 100.

[0119] Also, in the above formulas (3) and (4), px i , py i represents the coordinates of the point cloud converted into the information in the two-dimensional plane by the electronic device 1. In particular, pxi indicates the x - coordinate of the i - th information detected by the sensor 100. Also, py i indicates the y - coordinate of the i - th information detected by the sensor 100.

[0120] Furthermore, in the above formula (3), M indicates the number of pixels in the horizontal direction when assuming an image on a two - dimensional plane, and αx indicates the horizontal viewing angle when assuming an image on a two - dimensional plane. Also, in the above formula (4), N indicates the number of pixels in the vertical direction when assuming an image on a two - dimensional plane, and αy indicates the vertical viewing angle when assuming an image on a two - dimensional plane.

[0121] In the above formulas (3) and (4), px i , py i may be rounded to the nearest integer to function as a coordinate value. Also, in the above formulas (3) and (4), px i , py i should be within the size of the image after conversion to a two - dimensional plane. For example, data that does not satisfy 0≦px i ≦M or 0≦py i ≦N may be discarded.

[0122] Thus, the electronic device 1 according to one embodiment generates information on a point group that can be processed two - dimensionally from the output of the sensor 100. In particular, the electronic device 1 according to one embodiment may convert the information on the point group detected by the sensor 100 into information on a point group that can be processed two - dimensionally based on at least one of a predetermined number of pixels and a predetermined viewing angle in a two - dimensional image. The electronic device 1 according to one embodiment may be based on a conversion formula other than the above formulas (3) and (4) when generating information on a point group that can be processed two - dimensionally from the output of the sensor 100.

[0123] According to the electronic device 1 according to one embodiment, in order to perform dimensionality reduction of the point - group information in step S11, the amount of calculation can be significantly reduced. Therefore, according to the electronic device 1 according to one embodiment, for example, the processing load on the controller 10 can be reduced.

[0124] When the dimensionality reduction of the point cloud information is performed in step S11, the electronic device 1 superimposes the frame information (step S12). In step S12, the electronic device 1 may execute a process of superimposing a plurality of frames as shown in FIG. 3, for example.

[0125] FIG. 9 is a diagram for explaining the superimposition of the frame information performed in step S12. As shown in FIG. 9, the electronic device 1 may superimpose the point cloud data generated in, for example, 50 frames. Here, the electronic device 1 may simply add the values of the point cloud data generated in, for example, 50 frames. On the other hand, the electronic device 1 may average the values of the point cloud data generated in, for example, 50 frames according to the number of occurrences in the 50 frames. For example, when the information at a certain point is detected in 5 frames out of 50 frames, the electronic device 1 may average the values detected in the 5 frames.

[0126] In this way, the electronic device 1 according to one embodiment may collect the outputs of the sensor 100 for each of a plurality of frames and generate point cloud information that can be processed two-dimensionally. Also, the electronic device 1 according to one embodiment may average the point cloud information detected by the sensor 100 according to the number of detections for each of a plurality of frames to generate point cloud information that can be processed two-dimensionally.

[0127] According to the electronic device 1 according to one embodiment, since the frame information is superimposed in step S12, for example, the point cloud information is likely to become dense two-dimensionally. Therefore, according to the electronic device 1 according to one embodiment, for example, the recognition rate of the body part performed in step S3 of FIG. 4 can be improved.

[0128] When the frame information is superimposed in step 12, the electronic device 1 enlarges the points in the point cloud information (step S13). In step S13, the electronic device 1 may enlarge each point of the frame superimposed in step 12. Further, when enlarging each point in step S13, the electronic device 1 may enlarge each point in an arbitrary size in a predetermined direction.

[0129] FIG. 10 is a diagram for explaining the enlargement of the points in the point cloud information performed in step S13. As shown on the left side of FIG. 10, for example, after steps S11 and S12, it is assumed that one point indicated by a filled dot is detected in the point cloud information that can be processed two-dimensionally. In this case, as shown on the right side of FIG. 10, the electronic device 1 may execute a process of enlarging the points, for example, by one dot (one pixel) in the left-right direction and by two dots (two pixels) in the up-down direction. Here, the electronic device 1 may also execute a process of enlarging the points, for example, by two dots (two pixels) in the left-right direction and by one dot (one pixel) in the up-down direction. In this way, the electronic device 1 may execute a process of enlarging the points by an arbitrary number of dots (pixels) in the left-right direction and / or the up-down direction.

[0130] In this way, the electronic device 1 according to one embodiment may enlarge the size of each point in a predetermined direction of the points constituting the point cloud in the point cloud information that can be processed two-dimensionally.

[0131] According to the electronic device 1 according to one embodiment, in order to enlarge the points in the point cloud information, for example, the point cloud information becomes likely to be dense two-dimensionally. Therefore, according to the electronic device 1 according to one embodiment, for example, the recognition rate of the body part performed in step S3 of FIG. 4 can be improved.

[0132] In the above-described embodiment, the electronic device 1 has been described as expanding each point with respect to the information of the point group dimensionality-reduced in step S11. However, in one embodiment, the order of such processing is not limited to this. For example, in one embodiment, the electronic device 1 may expand each point with respect to the information of the point group before being dimensionality-reduced in step S11, and then perform dimensionality reduction of the information of the point group in step S11. In this way, the electronic device 1 according to one embodiment may expand the size of each point in a predetermined direction of the point group in the information of the point group detected by the sensor 100 to generate information of a point group that can be processed two-dimensionally.

[0133] If each point in the information of the point group is expanded in step S13, the electronic device 1 weights each point in the information of the point group (step S14). In step S14, the electronic device 1 may, for example, adopt information with a predetermined frequency or higher or exclude information with a predetermined frequency or lower according to the detection frequency in a predetermined number of frames.

[0134] FIG. 11 is a diagram for explaining the weighting of each point in the information of the point group performed in step S14. In FIG. 11, Dr shown on the left side indicates the distance information detected as a point group, and Di indicates the intensity information detected as a point group. In this way, the left side of FIG. 11 shows the information of two channels of the detected distance and intensity. Also, in FIG. 11, HM shown in the upper row may be information indicating the detection frequency of the detected point group for each pixel (hereinafter, also referred to as "detection frequency information") like a heat map, for example. In FIG. 11, Dr' shown on the right side indicates the result of integrating the detection frequency information HM with the distance information Dr shown on the left side. Also, in FIG. 11, Di' shown on the right side indicates the result of integrating the detection frequency information HM with the intensity information Di shown on the left side. In this way, the right side of FIG. 11 shows the result of integrating the detection frequency information HM with the information of two channels of the detected distance and intensity.

[0135] As shown on the right side of FIG. 11, based on the integrated detection frequency information HM in the detected distance and intensity information, the electronic device 1 may adopt information with a predetermined frequency or higher, or exclude information with a predetermined frequency or lower. For example, the electronic device 1 may be configured to adopt only the distance and intensity information detected with a frequency of 10 frames or more in 50 frames. Further, for example, the electronic device 1 may exclude the distance and intensity information detected with a frequency of 5 frames or less in 50 frames.

[0136] In this way, in the information of the point cloud that can be processed two-dimensionally, the electronic device 1 according to an embodiment may weight each point constituting the point cloud according to the detection frequency for each of a plurality of frames. In particular, in the information of the point cloud that can be processed two-dimensionally, the electronic device 1 according to an embodiment may exclude points with a detection frequency of a predetermined value or less for each of a plurality of frames among the points constituting the point cloud.

[0137] When an object is detected by a sensor 100 such as a millimeter-wave radar, the detection result includes noise. FIG. 12 is a diagram showing an example of the point cloud information that has undergone the processing from step S21 to step S23 in FIG. 6. FIG. 12 shows an image obtained by superimposing an image captured by the imaging unit 300 on the point cloud information based on the output of the sensor 100. As shown in FIG. 12, most of the point cloud information based on the output of the sensor 100 overlaps the monitored target Tm in the image captured by the imaging unit 300. On the other hand, among the point cloud information based on the output of the sensor 100, there is also information that does not overlap the monitored target Tm in the image captured by the imaging unit 300. Thus, when the sensor 100 is, for example, a millimeter-wave radar, when an object is detected by the sensor 100, the detection result includes noise.

[0138] FIG. 13 is a diagram showing an example of the point group information after the process of step S24 in FIG. 6. As shown in FIG. 13, the point group information based on the output of the sensor 100 comes to substantially overlap the monitored object Tm in the image captured by the imaging unit 300 through the process of step S24. That is, through the process of step S24, the influence of noise is considerably reduced in the point group information based on the output of the sensor 100.

[0139] As described above, according to the electronic device 1 according to one embodiment, the result of detecting an object based on the reflected wave of a transmission wave, such as a millimeter-wave radar, can be processed (pre-processed) like an image captured by the imaging unit. Therefore, according to the electronic device 1 according to one embodiment, it can be beneficial for monitoring the object to be monitored.

[0140] Through the processes from step S21 to step 24 shown in FIG. 6, the process of step S2 shown in FIG. 4, that is, the pre-processing of the point cloud data can be performed. Through the pre-processing of the point cloud data shown in step S2 of FIG. 4, point group information that can be processed two-dimensionally is generated. Therefore, thereafter, the electronic device 1 may perform processes such as image recognition on the point group information that can be processed two-dimensionally as the processes after step S3 shown in FIG. 4. That is, the electronic device 1 may recognize the monitored object Tm as a two-dimensional object in the point group information that can be processed two-dimensionally. For this reason, the electronic device 1 may generate point group information that can be recognized as a two-dimensional object through the pre-processing of the point cloud data in step S2. That is, the electronic device 1 according to one embodiment may generate point group information that can be recognized as a two-dimensional object from the output of the sensor 100.

[0141] As described above, in the process of step S3 shown in FIG. 4, the body part of the object to be monitored Tm is recognized from the information of the point group that can be processed two-dimensionally. Therefore, the electronic device 1 may generate information of a point group that can recognize the body part of the object to be monitored Tm by preprocessing the point group data in step S2. Thus, the electronic device 1 according to one embodiment may generate information of a point group that can recognize the body part of the object to be monitored as an object from the output of the sensor 100.

[0142] Also, as shown in FIG. 13, the electronic device 1 may output a superposition of the information of the point group that can be processed two-dimensionally and the information of the two-dimensional image of the object captured by the imaging unit 300. The information of the point group superposed on the information of the two-dimensional image of the object captured by the imaging unit 300 as shown in FIG. 13 is not essential when monitoring the behavior of the object to be monitored Tm. However, for example, when recognizing the body part of the object to be monitored Tm by image recognition or by a human, it is convenient to have the information of the point group superposed on the information of the two-dimensional image of the object captured by the imaging unit 300 as shown in FIG. 13. Therefore, the electronic device 1 according to one embodiment may superpose the information of the point group that can be processed two-dimensionally on the information of the two-dimensional image of the object captured by the imaging unit 300.

[0143] Next, in the electronic device 1 according to one embodiment, an aspect of recognizing (or estimating) the body part of the object to be monitored Tm shown in step S3 of FIG. 4 by machine learning will be further described. The electronic device 1 according to one embodiment can estimate the body part of the object to be monitored Tm, for example, as shown in FIG. 5, from the output of the sensor 100 by recognizing the body part of the object to be monitored Tm by machine learning.

[0144] First, in the electronic device 1 according to an embodiment, a detection device for recognizing (estimating) a body part of the object to be monitored Tm by machine learning will be described. FIGS. 14 and 15 are diagrams showing an example of the configuration of a detection device that supplies information for machine learning to the electronic device 1 according to an embodiment. FIG. 14 is a front view showing an example of the detection device according to an embodiment viewed from the front. Further, FIG. 15 is a side view showing an example of the detection device according to an embodiment viewed from the side (left side).

[0145] As shown in FIGS. 14 and 15, the detection device 3 according to an embodiment includes a sensor 100 and an imaging unit 300. Further, as shown in FIGS. 14 and 15, the detection device 3 according to an embodiment may appropriately include at least one of a stand portion 5 and a grounding portion 7, etc.

[0146] The sensor 100 shown in FIGS. 14 and 15 may be the sensor 100 described in FIGS. 1 and / or 2. As shown in FIGS. 14 and 15, the sensor 100 may include a radio wave input unit 101 that receives a reflected wave obtained by reflecting a transmission wave by an object. As shown in FIG. 15, the radio wave input unit 101 may face the optical axis Ra of the sensor 100. Here, the optical axis of the sensor 100 may be, for example, a direction perpendicular to the surface on which at least one of the transmission antenna 125 and the reception antenna 131 of the sensor 100 is installed. Further, when the optical axis of the sensor 100 includes a plurality of at least one of the transmission antenna 125 and the reception antenna 131, the optical axis of the sensor 100 may be a direction perpendicular to the surface on which at least any one of the plurality of antennas is installed. With such a configuration, the sensor 100 can transmit a transmission wave and / or receive a reflected wave around the optical axis Ra. That is, the sensor 100 can detect an object as a point cloud within a range centered on the optical axis Ra.

[0147] Further, the imaging unit 300 shown in FIGS. 14 and 15 may be the imaging unit 300 described in FIG. 1. As shown in FIGS. 14 and 15, the imaging unit 300 may include a light input unit 301 that receives light reflected by an object. As shown in FIG. 15, the light input unit 301 may face the optical axis La of the imaging unit 300. Further, the light input unit 301 may be the position where a lens is disposed in the imaging unit 300. Here, the optical axis of the imaging unit 300 may be, for example, a direction perpendicular to the surface on which a light receiving element (or an imaging element) used for imaging is installed in the imaging unit 300. With such a configuration, the imaging unit 300 can capture an image centered on the optical axis La.

[0148] As shown in FIGS. 14 and 15, the stand unit 5 maintains the sensor 100 and the imaging unit 300 at a predetermined height from the ground point in the detection device 3. The stand unit 5 may maintain the sensor 100 at a height at which the sensor 100 can easily detect an object such as a monitored object. Further, the stand unit 5 may maintain the sensor 100 at a height at which the imaging unit 300 can easily capture an image of an object such as a monitored object. The stand unit 5 may include a mechanism that can adjust the sensor 100 and the imaging unit 300 in, for example, the height direction in the detection device 3.

[0149] As shown in FIGS. 14 and 15, the grounding unit 7 fixes the sensor 100 and the imaging unit 300 to the ground surface in the detection device 3. The grounding unit 7 can assume various configurations, such as having a shape like a pedestal, in order to stabilize the detection device 3 including the sensor 100 and the imaging unit 300.

[0150] As shown in FIGS. 14 and 15, in the detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other in the vicinity of each other. In the example shown in FIGS. 14 and 15, the sensor 100 and the imaging unit 300 are arranged adjacent to each other in the vertical direction. In the detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other, for example, in the left - right direction or the diagonal direction.

[0151] Also, as shown in FIG. 15, in the detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be arranged such that their respective optical axes Ra and La are parallel. That is, in the electronic device 1 according to an embodiment, the information of the point cloud by the sensor 100 and the information of the image by the imaging unit 300 may be used in a state where the optical axis La of the imaging unit 300 is installed parallel to the optical axis Ra of the sensor 100.

[0152] Also, as shown in FIG. 15, in the detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be arranged such that the distance between their respective optical axes Ra and La is maintained at a distance G. By arranging in this way, the information of the point cloud by the sensor 100 and the information of the image by the imaging unit 300 are information shifted from each other by the distance G. For example, in the arrangement configuration shown in FIG. 15, the information of the image by the imaging unit 300 is information shifted upward by the distance G from the information of the point cloud by the sensor 100. Also, in the arrangement configuration shown in FIG. 15, the information of the point cloud by the sensor 100 is information shifted downward by the distance G from the information of the image by the imaging unit 300.

[0153] Therefore, in the arrangement configuration shown in FIG. 15, for example, by correcting the information of the point cloud by the sensor 100 to be shifted upward by the distance G, the position of the information of the point cloud by the sensor 100 can be made to correspond to the position of the information of the image by the imaging unit 300. Also, in the arrangement configuration shown in FIG. 15, for example, by correcting the information of the image by the imaging unit 300 to be shifted downward by the distance G, the position of the information of the image by the imaging unit 300 can be made to correspond to the position of the information of the point cloud by the sensor 100. In this way, the electronic device 1 may make the information of the point cloud by the sensor 100 and the information of the image by the imaging unit 300 correspond to each other in position by correcting at least one of the information of the point cloud by the sensor 100 and the information of the image by the imaging unit 300.

[0154] That is, in the electronic device 1 according to one embodiment, at least one of the point cloud information obtained by the sensor 100 detecting an object (object to be monitored) and the image information obtained by the imaging unit 300 imaging the object may be corrected. The electronic device 1 according to one embodiment may use information in which the point cloud information from the sensor 100 and the image information from the imaging unit 300 are made consistent by correction.

[0155] Also, it is assumed that the detection range (angle) of the point cloud by the sensor 100 and the imaging range (angle or field of view) of the image by the imaging unit 300 may not be the same. In such a case, the electronic device 1 may adjust the wider range (angle) of the two to the narrower range (angle) so that the point cloud information from the sensor 100 and the image information from the imaging unit 300 correspond to each other positionally. That is, the electronic device 1 may use only the information within the overlapping range of the imaging range of the imaging unit 300 and the detection range of the sensor 100, and may delete or ignore the information in the non-overlapping range.

[0156] As described above, in the electronic device 1 according to one embodiment, as the image information obtained by the imaging unit 300 imaging an object (object to be monitored), information that corresponds positionally to the point cloud information obtained by the sensor 100 detecting the object (object to be monitored) may be used.

[0157] Hereinafter, in the electronic device 1 according to one embodiment, the operation when recognizing (or estimating) the body part of the object to be monitored Tm shown in step S3 of FIG. 4 by machine learning using the detection device 3 will be further described.

[0158] An electronic device 1 according to an embodiment performs machine learning on a predetermined part (for example, a body part) of an object such as a monitored object using the point cloud information output from the sensor 100. Here, the electronic device 1 according to an embodiment may perform machine learning on a predetermined part of an object using the image information obtained by imaging the object (monitored object) by the imaging unit 300. Hereinafter, the aspect of machine learning by the electronic device 1 according to such an embodiment may be simply referred to as the "learning phase". After undergoing machine learning as described above, the electronic device 1 according to an embodiment can estimate a predetermined part (for example, a body part) of an object such as a monitored object using the point cloud information output from the sensor 100. Hereinafter, the aspect of estimating a part by the electronic device 1 according to such an embodiment may be simply referred to as the "estimation phase". When a predetermined part (for example, a body part) of an object such as a monitored object is thus estimated (recognized), the electronic device 1 according to an embodiment can perform the processing after step S4 shown in FIG. 4 based on the information of the estimated (recognized) part.

[0159] (Learning Phase) First, the learning phase will be described. The electronic device 1 according to an embodiment performs machine learning on a predetermined part (for example, a body part) of an object (monitored object) using the output of the sensor 100.

[0160] FIG. 16 is a diagram for explaining an example of machine learning by the electronic device 1 according to an embodiment. The electronic device 1 according to an embodiment performs machine learning on a predetermined part (for example, a body part) of an object (for example, a monitored object) detected by the sensor 100 using point cloud data based on the output of the sensor 100 as shown on the left side of FIG. 16. At this time, the electronic device 1 may perform machine learning by using the coordinates of the part of the monitored object based on the image captured by the imaging unit 300 as shown on the upper side of FIG. 16. FIG. 16 conceptually shows the mode in which the electronic device 1 according to an embodiment performs machine learning based on the point cloud data based on the output of the sensor 100 and the coordinates of the part based on the image captured by the imaging unit 300.

[0161] When performing machine learning as shown in FIG. 16, the electronic device 1 according to an embodiment may use data suggesting correct answers (hereinafter sometimes referred to as "correct answer data") as information indicating the coordinates of each part based on the image captured by the imaging unit 300. That is, the coordinates of the part of the object to be monitored based on the image captured by the imaging unit 300 as shown in the upper part of FIG. 16 may be used as correct answer data indicating the coordinates of each part of the body of the object to be monitored. Such correct answer data may be specified in advance by a person, or may be recognized by image recognition using an information processing device such as the electronic device 1.

[0162] FIG. 17 is a diagram for explaining an example of specifying a predetermined part and image recognition by the electronic device 1 according to an embodiment. As shown on the left side of FIG. 17, for example, for the image of the object to be monitored captured by the imaging unit 300, by performing designation or image recognition by a person, as shown on the right side of FIG. 17, the coordinates of each part of the object to be monitored can be obtained.

[0163] For example, the electronic device 1 may be configured such that the positions corresponding to each part in the image of the object to be monitored captured by the imaging unit 300 are designated manually by a predetermined person or the like. For example, the electronic device 1 according to an embodiment may request a person such as a user of the electronic device 1 to designate the "neck" part in the image of the object to be monitored captured by the imaging unit 300. In this case, a person such as a user of the electronic device 1 may perform an operation of designating the neck part in the image of the object to be monitored captured by the imaging unit 300 in response to the request by the electronic device 1. For example, a person such as a user of the electronic device 1 may use a pointer to designate the neck part in the image of the object to be monitored displayed on a display unit or the like as shown on the left side of FIG. 17. Also, for example, a person such as a user of the electronic device 1 may touch and designate the neck part in the image of the object to be monitored displayed on a display unit or the like as shown on the left side of FIG. 17 using a finger or a stylus or the like.

[0164] When the neck part is pointed out in this way, as shown on the right side of FIG. 17, the electronic device 1 may specify the coordinates of the position corresponding to the part in the image of the object to be monitored captured by the imaging unit 300. Similarly hereinafter, the electronic device 1 may request a person such as a user of the electronic device 1 to specify parts such as the "right shoulder" part and then the "left shoulder" part in the image of the object to be monitored captured by the imaging unit 300.

[0165] Thus, in the electronic device 1 according to one embodiment, as information of an image obtained by the imaging unit 300 imaging an object (object to be monitored), information of an image in which a predetermined part in the object is specified in advance (for example, by a person's hand) may be used.

[0166] Further, for example, in the electronic device 1, in the image of the object to be monitored captured by the imaging unit 300, the positions corresponding to the respective parts may be image-recognized (for example, without depending on the hands of a predetermined person or the like). For example, the electronic device 1 according to one embodiment may specify the coordinates of the part by image-recognizing the "neck" part in the image of the object to be monitored captured by the imaging unit 300. In this case, the electronic device 1 may use information learned by machine learning using the correct data of each part specified in advance by the appearance of a predetermined person or the like as described above. Also, in this case, the electronic device 1 can improve the accuracy of image recognition of each part by machine-learning the positions corresponding to the respective parts in a large number of images of the object to be monitored captured by the imaging unit 300.

[0167] When the neck part is image-recognized in this way, as shown on the right side of FIG. 17, the electronic device 1 may specify the coordinates of the position corresponding to the part in the image of the object to be monitored captured by the imaging unit 300. Similarly hereinafter, the electronic device 1 may also image-recognize parts such as the "right shoulder" part and then the "left shoulder" part in the image of the object to be monitored captured by the imaging unit 300.

[0168] Thus, in the electronic device 1 according to one embodiment, as information on an image obtained by the imaging unit 300 imaging an object (object to be monitored), information in which a predetermined part of the object is recognized by image recognition may be used.

[0169] The right side of FIG. 17 represents that, as a predetermined part specified in an object such as an object to be monitored, the position of the predetermined part, that is, the coordinates of the position of the predetermined part are shown.

[0170] As described above, the electronic device 1 according to one embodiment may perform machine learning on a predetermined part (body part) of an object (object to be monitored) using the output of the sensor 100 and the output of the imaging unit 300. Here, the output of the sensor 100 may be point cloud information obtained by the sensor 100 detecting an object (object to be monitored) that reflects the transmission wave based on the transmission signal transmitted as the transmission wave and the reception signal received as the reflected wave reflected by the transmission wave. Further, the output of the imaging unit 300 may be image information obtained by the imaging unit 300 imaging an object (object to be monitored). Further, the electronic device 1 according to one embodiment may perform machine learning on the position of a predetermined part of an object (object to be monitored). Furthermore, the electronic device 1 according to one embodiment may perform machine learning on the coordinates of the position of a predetermined part of an object (object to be monitored).

[0171] (Estimation phase) Next, the estimation phase will be described. When the electronic device 1 according to one embodiment performs machine learning as described above, it can estimate a predetermined part (for example, a body part) of an object such as an object to be monitored using the point cloud information output from the sensor 100.

[0172] FIG. 18 is a diagram for explaining an example of estimating a predetermined part by the electronic device 1 according to an embodiment. The electronic device 1 according to an embodiment uses the point cloud data based on the output of the sensor 100 shown on the left side of FIG. 18, and via the machine learning shown in the center of FIG. 18, estimates a predetermined part (for example, a body part) in an object such as a subject to be monitored as shown on the right side of FIG. 18. The right side of FIG. 18 represents that the position of the predetermined part, that is, the coordinates of the position of the predetermined part, are shown as the predetermined part in an object such as a subject to be monitored.

[0173] As described above, the electronic device 1 according to an embodiment may perform machine learning on a predetermined part in the object using the point cloud information obtained by detecting the object by, for example, the sensor 100 or the like. By performing such machine learning, the electronic device 1 according to an embodiment can estimate a predetermined part in the object to be monitored using the point cloud information obtained by detecting the object to be monitored by the sensor 100 or the like.

[0174] According to the electronic device 1 according to an embodiment, each part of a predetermined object such as a subject to be monitored can be recognized based on the point cloud data output from a sensor 300 such as a millimeter-wave radar sensor. Therefore, according to the electronic device 1 according to an embodiment, even if a predetermined object such as a subject to be monitored is performing a complex operation, each part of the object can be recognized. Further, according to the electronic device 1 according to an embodiment, after predetermined machine learning, each part of a predetermined object such as a subject to be monitored can be recognized based on the point cloud data output from a sensor 300 such as a millimeter-wave radar sensor. Therefore, according to the electronic device 1 according to an embodiment, each part can be recognized without using, for example, an image of the object to be monitored. For this reason, a use case considering the privacy of the object to be monitored can be assumed.

[0175] The embodiments according to the present disclosure have been described based on the drawings and examples. However, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically contradictory, and a plurality of components or steps can be combined into one or divided. Although the embodiments according to the present disclosure have been described mainly centered on the apparatus, the embodiments according to the present disclosure can also be realized as a method including the steps executed by each component of the apparatus. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium recording the program executed by a processor included in the apparatus. It should be understood that these are also included in the scope of the present disclosure.

[0176] The above-described embodiments are not limited to being implemented only as the electronic device 1. For example, the above-described embodiments may be implemented as the electronic device 1 included in the electronic device 1. Further, the above-described embodiments may be implemented, for example, as a monitoring method by a device such as the electronic device 1. Furthermore, the above-described embodiments may be implemented, for example, as a program executed by a device such as the electronic device 1 or an information processing apparatus (for example, a computer).

Description of Reference Numerals

[0177] 1 Electronic device 3 Detection device 5 Stand portion 7 Grounding portion 10 Controller 20 Storage portion 30 Communication portion 40 Display portion 50 Notification portion 100 Sensor 101 Radio wave input portion 110 Radar control portion 120 Transmission portion 121 Signal generation portion 122 Synthesizer 123 Phase control portion 124 Amplifier 125 Transmission Antenna 130 Receiver 131 Reception Antenna 132 LNA 133 Mixer 134 IF Section 135 AD Conversion Section 300 Imaging Section 301 Optical Input Section

Claims

1. Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, the information of a point cloud obtained by a sensor detecting an object that reflects the transmission wave, and using the information of an image obtained by an imaging unit imaging the object, performing machine learning on a predetermined part of the object, After recognizing a predetermined part of the object obtained by detection by the sensor, the information of the point cloud is as follows: An electronic device that, when the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X'max, uses the following formula (1) to convert the coordinates (X) before normalization into information converted into the coordinates (X') after normalization. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

2. The electronic device according to claim 1, which performs machine learning on the position of a predetermined part of the object.

3. The electronic device according to claim 2, which performs machine learning on the coordinates of the position of a predetermined part of the object.

4. The electronic device according to any one of claims 1 to 3, which performs machine learning on a predetermined part of the object by using information that is positionally corresponding to the information of the point cloud obtained by the sensor detecting the object as the information of the image obtained by the imaging unit imaging the object.

5. In a state where the optical axis of the imaging unit is installed parallel to the optical axis of the sensor, using the information of the point cloud obtained by the sensor detecting the object and the information of the image obtained by the imaging unit imaging the object, the electronic device according to claim 4, which performs machine learning on a predetermined part of the object.

6. The electronic device according to any one of claims 1 to 5, which performs machine learning on a predetermined part of the object by using information obtained by correcting at least one of the information of the point cloud obtained by the sensor detecting the object and the information of the image obtained by the imaging unit imaging the object so as to match both.

7. The electronic device according to any one of claims 1 to 6, which performs machine learning on a predetermined part of the object by using the information of an image in which a predetermined part of the object is specified in advance as the information of the image obtained by the imaging unit imaging the object.

8. Using information indicating that a predetermined part of the object has been recognized by image recognition as information of an image obtained by the imaging unit imaging the object, machine learning is performed on the predetermined part of the object, the electronic device according to any one of claims 1 to 6.

9. Using point cloud information obtained by detecting an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave, machine learning is performed on a predetermined part of the object, Using point cloud information obtained by a sensor detecting a monitoring target that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave, estimating a predetermined part of the monitoring target, The point cloud information after recognizing a predetermined part of the object, obtained by the sensor detecting, An electronic device, which is information obtained by converting a coordinate (X) before normalization into a coordinate (X') after normalization using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

10. A step of acquiring point cloud information by detecting, by a sensor, an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave obtained by reflecting the transmitted wave; A step of acquiring image information by imaging the object with an imaging unit; A step of performing machine learning on a predetermined part of the object; including The point cloud information after recognizing a predetermined part of the object, obtained by the sensor detecting, An electronic device control method, which is information obtained by converting a coordinate (X) before normalization into a coordinate (X') after normalization using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

11. Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, using information on a point cloud obtained by detecting an object that reflects the transmission wave, a step of performing machine learning on a predetermined part of the object; Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, using information on a point cloud obtained by a sensor detecting a monitoring target that reflects the transmission wave, a step of estimating a predetermined part of the monitoring target; including; The information on the point cloud after recognizing a predetermined part of the object, obtained by the sensor detecting, When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X'max, using the following formula (1), the coordinates before normalization (X) are converted into information converted into the coordinates after normalization (X'), a control method for an electronic device. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

12. In an electronic device, a step of obtaining information on a point cloud by detecting, by a sensor, an object that reflects a transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave; a step of obtaining information on an image by imaging the object by an imaging unit; a step of performing machine learning on a predetermined part of the object; causing it to execute, The information on the point cloud after recognizing a predetermined part of the object, obtained by the sensor detecting, When the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X'max, using the following formula (1), the coordinates before normalization (X) are converted into information converted into the coordinates after normalization (X'), a program. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

13. In an electronic device, Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, using information on a point cloud obtained by detecting an object that reflects the transmission wave, a step of performing machine learning on a predetermined part of the object; Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave obtained by reflecting the transmission wave, using the point cloud information obtained by a sensor detecting a monitoring target that reflects the transmission wave, a step of estimating a predetermined part in the monitoring target; to cause to execute; After recognizing a predetermined part in the object, the point cloud information obtained by the sensor detecting is A program that, when the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the normalized coordinates is X'max, uses the following formula (1) to convert the coordinates (X) before normalization into information converted into the coordinates (X') after normalization. X' = ((X - Xmin) / (Xmax - Xmin)) · X'max (1)

Images