Management device and management method

Abstract

Sensing the surroundings. [Solution] A first device (ay311) that senses a second device (ay312) by radio waves includes a communication unit (ay315) that receives frames transmitted by the second device (ay312) by radio waves and senses the second device (ay312) using the received frames, and a control unit (ay316) that selects the frequency of the radio waves transmitted by the second device (ay312) from a predetermined number of frequencies, notifies the second device (ay312) of the selected frequency, and controls the communication unit (ay315) to perform sensing using the frequency.

Description

[Technical Field]

[0001] The present disclosure relates to a communication device and a communication method. [Background technology]

[0002] Methods for sensing the surrounding conditions include, for example, detecting ambient light using a camera, or irradiating the surroundings with light, infrared rays, ultrasonic waves, etc. and detecting reflected light, infrared rays, ultrasonic waves, etc. In recent years, a method for sensing the surrounding conditions using radio waves has also been proposed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Schuster, S. Scheiblhofer, R. Feger, and A. Stelzer, “Signal model and statistical analysis for the sequential sampling pulse radar technique,” ​​in Proc. IEEE Radar Conf, 2008, pp. 1-6, 2008 [Non-patent document 2] D. Cao, T. Li, P. Kang, H. Liu, S. Zhou, H. Su, “Single-Pulse Multi-Beams Operation of Phased Array Radar”, 2016 CIE International Conference on Radar (RADAR), pp. 1-4, 2016 [Non-patent document 3] A. Bourdoux, K. Parashar, and M. Bauduin, “Phenomenology of mutual interference of FMCW and PMCW automotive radars,” in 2017 IEEE Radar Conference (Radar Conf.), pp. 1709-1714, 2017 [Non-patent document 4] J. Fink, FK Jondral, “Comparison of OFDM radar and chirp sequence radar,” in 2015 16th International Radar Symposium (IRS), pp. 315-320, 2015 Summary of the Invention [Problem to be solved by the invention]

[0004] The purposes, uses, and environments in which spatial sensing is used are becoming more diverse.

[0005] Therefore, one aspect of the present disclosure provides a communication transmission device and a communication transmission method capable of sensing surroundings using radio waves. Another aspect of the present disclosure provides a communication device and a communication method that operate according to a communication protocol for controlling the time and frequency of surrounding sensing using radio waves. Another aspect of the present disclosure provides a device, a system, and a method that present information to a user based on the results of sensing the surroundings using radio waves, light, infrared rays, ultrasound, etc. Another aspect of the present disclosure provides a device, a system, and a method that control the operation of a device based on the results of sensing the surroundings using any one or a combination of two or more of radio waves, light, infrared rays, ultrasound, etc. Another aspect of the present disclosure provides a device, a system, and a method that generate data based on the results of sensing the surroundings using any one or a combination of two or more of radio waves, light, infrared rays, ultrasound, etc. Another aspect of the present disclosure provides a device, a system, and a method that transmit the results of sensing the surroundings using any one or a combination of two or more of radio waves, light, infrared rays, ultrasound, etc., or data generated based on the sensing results, to another device, a server, etc. Another aspect of the present disclosure provides an apparatus, system, and method for obtaining data generated based on sensing results of surroundings or sensing results using any one or a combination of two or more of radio waves, light, infrared rays, ultrasound, etc. from another device or server in order to perform processing such as controlling a device. [Means for solving the problem]

[0006] A management device that is one aspect of the present disclosure is a management device that includes a communication unit that communicates with a plurality of sensing devices placed within a building, and a processor, wherein the processor receives capability information from the plurality of sensing devices that indicates the sensing capabilities of the sensing devices, selects a sensing device to be used to measure a specified area within the building, and estimates the presence or position of a target in the specified area based on the sensing results received from the sensing devices.

[0007] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0008] Some of the aspects included in the present disclosure can promote the realization and widespread use of surrounding sensing using radio waves.

[0009] The present disclosure also includes aspects that can promote the realization and spread of new services that utilize ambient sensing using any one or a combination of two or more of radio waves, light, infrared rays, ultrasound, etc. This makes it possible to provide, for example, any one or more of the following: control of device operation, control of information presented to users, and data generation, based on the state, shape, and movement of people and objects in real space. As a result, it is expected to contribute in any one or more aspects, for example, improved user convenience, simplification of user input operations, automation of processing, provision of new services, and detection of events occurring in real space that have been difficult to detect until now. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a detection device according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the detection device according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the detection device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a modified example of the configuration of the detection device according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing a modified example of the configuration of the detection device according to the first embodiment. [Figure 6]FIG. 6 is a diagram showing a modified example of the configuration of the detection device according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a separation configuration of the detection device according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a separation configuration of the detection device according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a separation configuration of the detection device according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a separation configuration of the detection device according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a state of the device according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a frame configuration according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a frame configuration according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of the configuration of the device according to the third embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of a configuration of a device according to the third embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a configuration of a device according to the third embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a system configuration according to the fourth embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of the configuration of the device according to the fourth embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 20] FIG. 20 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 21] FIG. 21 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 22] FIG. 22 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 23]FIG. 23 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 24] FIG. 24 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 25] FIG. 25 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 26] FIG. 26 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 27] FIG. 27 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 28] FIG. 28 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 29] FIG. 29 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 30] FIG. 30 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 31] FIG. 31 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 32] FIG. 32 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 33] FIG. 33 is a diagram showing an example of a frame configuration according to the fourth embodiment. [Figure 34] FIG. 34 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 35] FIG. 35 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 36] FIG. 36 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 37] FIG. 37 is a diagram showing an example of a frame structure according to the fourth embodiment. [Figure 38] FIG. 38 is a diagram illustrating an example of the configuration of a control system according to the fifth embodiment. [Figure 39] FIG. 39 is a diagram illustrating an example of the configuration of a control system according to the fifth embodiment. [Figure 40] FIG. 40 is a diagram showing an example of the appearance of the control system according to the fifth embodiment. [Figure 41] FIG. 41 is a diagram illustrating an example of processing performed by the control system according to the fifth embodiment. [Figure 42] FIG. 42 is a diagram illustrating an example of a configuration of a transmitting device according to the sixth embodiment. [Figure 43] FIG. 43 is a diagram illustrating an example of processing performed by the transmitting device according to the sixth embodiment. [Figure 44] FIG. 44 is a diagram showing an example of the configuration of the device according to the seventh embodiment. [Figure 45] FIG. 45 is a diagram showing an example of a relationship between a device and a person according to the seventh embodiment. [Figure 46] FIG. 46 is a diagram illustrating an example of the states of a person, a terminal, and a device according to the seventh embodiment. [Figure 47] FIG. 47 is a diagram showing an example of the states of a person, a terminal, and a device according to the seventh embodiment. [Figure 48] FIG. 48 is a diagram illustrating an example of the relationship between a device, a terminal, and a person according to the seventh embodiment. [Figure 49] FIG. 49 is a diagram illustrating an example of the relationship between a device, a terminal, and a person according to the seventh embodiment. [Figure 50] FIG. 50 is a diagram showing an example of the state of each device in the eighth embodiment. [Figure 51] FIG. 51 is a flowchart of an example of operation when an AP is installed in a home according to the eighth embodiment. [Figure 52] FIG. 52 is a flowchart of an example of the operation of the AP according to the eighth embodiment. [Figure 53] FIG. 53 is a flowchart of an example of the operation of the system according to the eighth embodiment. [Figure 54] FIG. 54 is a flowchart illustrating an example of the operation of the AP and the cloud server according to the eighth embodiment. [Figure 55] FIG. 55 is a diagram showing an example of the state of each device in the ninth embodiment. [Figure 56]FIG. 56 is a flowchart of an operation example when an AP is installed in a home according to the ninth embodiment. [Figure 57] FIG. 57 is a flowchart of an example of the operation of the AP according to the ninth embodiment. [Figure 58] FIG. 58 is a flowchart of an example of the operation of the system according to the ninth embodiment. [Figure 59] FIG. 59 is a flowchart illustrating an example of the operation of the AP and the cloud server according to the ninth embodiment. [Figure 60] FIG. 60 is a diagram illustrating an example of a system configuration according to the tenth embodiment. [Figure 61] FIG. 61 is a diagram illustrating an example of a system configuration according to the tenth embodiment. [Figure 62] FIG. 62 is a diagram illustrating an example of a system configuration according to the tenth embodiment. [Figure 63] FIG. 63 is a diagram illustrating an example of a system configuration according to the tenth embodiment. [Figure 64] FIG. 64 is a diagram showing a sensing method in each embodiment. [Figure 65] FIG. 65 is a diagram showing a sensing method in each embodiment. [Figure 66] FIG. 66 is a diagram illustrating an example of a system configuration according to the eleventh embodiment. [Figure 67] FIG. 67 is a diagram illustrating an example of a system configuration according to the eleventh embodiment. [Figure 68] FIG. 68 illustrates an example of a configuration of the first device according to the eleventh embodiment. In FIG. [Figure 69] FIG. 69 illustrates an example of a configuration of the first device according to the eleventh embodiment. In FIG. [Figure 70] FIG. 70 is a flowchart showing an example of the operation of the first device in the eleventh embodiment. [Figure 71] FIG. 71 is a flowchart showing an example of the operation of the first device in the eleventh embodiment. [Figure 72]FIG. 72 is a flowchart showing an example of the operation of the first device in the eleventh embodiment. [Figure 73] FIG. 73 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 74] FIG. 74 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 75] FIG. 75 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 76] FIG. 76 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 77] FIG. 77 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 78] FIG. 78 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 79] FIG. 79 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 80] FIG. 80 shows an example of a display on the monitor unit in the eleventh embodiment. [Figure 81] FIG. 81 is a flowchart showing an example of the operation of the first device in the eleventh embodiment. [Figure 82] FIG. 82 is a flowchart showing an example of the operation of the first device in the eleventh embodiment. [Figure 83] FIG. 83 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 84] FIG. 84 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 85] FIG. 85 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 86] FIG. 86 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 87] FIG. 87 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 88] FIG. 88 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 89] FIG. 89 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 90] FIG. 90 shows an example of a display on the monitor unit in the eleventh embodiment. [Figure 91] FIG. 91 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 92] FIG. 92 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 93] FIG. 93 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 94] FIG. 94 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 95] FIG. 95 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 96] FIG. 96 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 97] FIG. 97 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 98] FIG. 98 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 99] FIG. 99 is a diagram showing a display example of the monitor unit in the eleventh embodiment. [Figure 100] FIG. 100 shows an example of a display on the monitor unit in the eleventh embodiment. [Figure 101] FIG. 101 is a diagram showing an example of a display on the monitor unit in the eleventh embodiment. [Figure 102] FIG. 102 is a diagram showing a device that communicates with the second device in the twelfth embodiment. [Figure 103] FIG. 103 is a diagram showing an example of a display on the monitor unit in the twelfth embodiment. [Figure 104] FIG. 104 is a diagram showing an example of a display on the monitor unit in the twelfth embodiment. [Figure 105] FIG. 105 is a diagram showing a map according to the thirteenth embodiment. [Figure 106]FIG. 106 is a diagram showing a map according to the thirteenth embodiment. [Figure 107] FIG. 107 is a diagram showing an example of how the display is switched in the thirteenth embodiment. [Figure 108] FIG. 108 is a diagram showing an example of how the display is switched in the thirteenth embodiment. [Figure 109] FIG. 109 is a diagram showing an example of how the display is switched in the thirteenth embodiment. [Figure 110] FIG. 110 is a diagram showing an example of a system configuration according to the fourteenth embodiment. [Figure 111] FIG. 111 is a diagram showing an example of a transmission status in the fourteenth embodiment. [Figure 112] FIG. 112 is a diagram showing an example of a transmission status in the fourteenth embodiment. [Figure 113] FIG. 113 is a diagram showing an example of a frame structure in the fourteenth embodiment. [Figure 114] FIG. 114 is a diagram showing an example of a frame structure in the fourteenth embodiment. [Figure 115] FIG. 115 is a diagram showing an example of a frame structure in the fourteenth embodiment. [Figure 116] FIG. 116 is a diagram showing an example of the state of the device in the fifteenth embodiment. [Figure 117] FIG. 117 is a diagram showing an example of the state of the device in the seventeenth embodiment. [Figure 118] FIG. 118 is a diagram showing an example of the state of the device in the eighteenth embodiment. [Figure 119] FIG. 119 is a diagram showing an example of the state of the device in the nineteenth embodiment. [Figure 120] FIG. 120 is a diagram illustrating an example of a sensing system or a sensing and communication system according to the twentieth embodiment. [Figure 121] FIG. 121 is a diagram showing an example of the structure of information regarding sensing capabilities in the twentieth embodiment. [Figure 122]FIG. 122 is a diagram showing an example of the operation of the device and the target according to the twentieth embodiment. [Figure 123] FIG. 123 is a diagram showing an example of the operation of the device and the target according to the twentieth embodiment. [Figure 124A] FIG. 124A is a diagram showing an example of the operation of the device and the target in the twentieth embodiment. [Figure 124B] FIG. 124B is a diagram showing an example of the operation of the device and the target in the twentieth embodiment. [Figure 125A] FIG. 125A is a diagram showing an example of a state when the device according to the twentieth embodiment is performing a sensing operation. [Figure 125B] FIG. 125B is a diagram showing an example of a state when the device according to the twentieth embodiment is performing a sensing operation. [Figure 126] FIG. 126 is a diagram showing an example of the state of the sensing and communication system in the twenty-first embodiment. [Figure 127] FIG. 127 is a diagram showing an example of the configuration of a signal transmitted by a device in the twenty-first embodiment. [Figure 128A] FIG. 128A is a diagram showing an example of a configuration of a control information symbol in the twenty-first embodiment. [Figure 128B] FIG. 128B is a diagram showing an example of the configuration of a sensing symbol in the twenty-first embodiment. [Figure 129] FIG. 129 is a diagram showing an example of the configuration of a signal transmitted by a device in the twenty-first embodiment. [Figure 130] FIG. 130 is a diagram showing an example of the configuration of a signal transmitted by a device in the twenty-first embodiment. [Figure 131] FIG. 131 is a diagram showing an example of the configuration of the device and the target according to the twenty-first embodiment. [Figure 132] FIG. 132 is a diagram illustrating an example of a configuration related to a transmitting antenna in the twenty-first embodiment. [Figure 133]FIG. 133 is a diagram showing a specific example of a frame of a sensing signal transmitted by the device according to the twenty-first embodiment. [Figure 134] FIG. 134 is a diagram showing an example of the configuration of a sensing signal transmitted using an antenna in the twenty-first embodiment. [Figure 135] FIG. 135 is a diagram showing an example of the configuration of a sensing signal transmitted using an antenna and parameters in the twenty-first embodiment. [Figure 136] FIG. 136 is a diagram showing an example of a system configuration according to the twenty-second embodiment. [Figure 137A] FIG. 137A is a diagram showing an example of an operation when charging a car in the twenty-second embodiment. [Figure 137B] FIG. 137B is a diagram showing an example of an operation when charging a car in the twenty-second embodiment. [Figure 137C] FIG. 137C is a diagram showing an example of an operation when charging a car in the twenty-second embodiment. [Figure 137D] FIG. 137D is a diagram showing an example of the operation when charging a car in the twenty-second embodiment. [Figure 138A] FIG. 138A is a diagram showing an example of an operation for ending charging of a car in the twenty-second embodiment. [Figure 138B] FIG. 138B is a diagram showing an example of an operation for ending charging of a car in the twenty-second embodiment. [Figure 139] FIG. 139 is a diagram showing an example of a system configuration according to the twenty-second embodiment. [Figure 140] FIG. 140 is a diagram showing an example of a system configuration according to the twenty-third embodiment. [Figure 141A] FIG. 141A is a diagram showing an example of an operation for unlocking a car door in the twenty-third embodiment. [Figure 141B] FIG. 141B is a diagram showing an example of an operation for unlocking a car door in the twenty-third embodiment. [Figure 142] FIG. 142 is a diagram showing an example of a system configuration according to the twenty-third embodiment. [Figure 143A] FIG. 143A is a diagram showing an example of operating the vehicle drive system in the twenty-third embodiment. [Figure 143B] FIG. 143B is a diagram showing an example of operating the vehicle drive system in the twenty-third embodiment. [Figure 144] FIG. 144 is a diagram showing an example of a system configuration in the twenty-fourth embodiment. [Figure 145A] FIG. 145A is a diagram showing an example of the operation of the device in the twenty-fifth embodiment. [Figure 145B] FIG. 145B is a diagram showing an example of the operation of the device in the twenty-fifth embodiment. [Figure 146] FIG. 146 is a diagram showing an example of the operation of the device in the twenty-fifth embodiment. [Figure 147] FIG. 147 is a diagram showing an example of the configuration of sensing related information in the twenty-fifth embodiment. [Figure 148A] FIG. 148A is a diagram showing an example of a frame in the twenty-fifth embodiment. [Figure 148B] FIG. 148B is a diagram showing an example of a frame in the twenty-fifth embodiment. [Figure 149] FIG. 149 is a diagram showing an example of a system configuration according to the twenty-sixth embodiment. [Figure 150] FIG. 150 is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 151A] FIG. 151A is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 151B] FIG. 151B is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 151C] FIG. 151C is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 151D] FIG. 151D is a diagram showing an example of the configuration of the device in the twenty-seventh embodiment. [Figure 152A] FIG. 152A is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 152B]FIG. 152B is a diagram showing an example of the configuration of the device according to the twenty-seventh embodiment. [Figure 153A] FIG. 153A is a diagram showing an example of a display in the twenty-seventh embodiment. [Figure 153B] FIG. 153B is a diagram showing an example of a display in the twenty-seventh embodiment. [Figure 154A] FIG. 154A is a diagram showing an example of a display in the twenty-seventh embodiment. [Figure 154B] FIG. 154B is a diagram showing an example of a display in the twenty-seventh embodiment. [Figure 155] FIG. 155 is a diagram showing an example of a system configuration according to the twenty-eighth embodiment. [Figure 156A] FIG. 156A is a diagram showing an example of a target region in the twenty-eighth embodiment. [Figure 156B] FIG. 156B is a diagram showing an example of how a sensing signal is transmitted in the twenty-eighth embodiment. [Figure 156C] FIG. 156C is a diagram showing an example of how a sensing signal is received in the twenty-eighth embodiment. [Figure 156D] FIG. 156D is a diagram showing an example of how a sensing signal is received in the twenty-eighth embodiment. [Figure 157A] FIG. 157A is a diagram showing an example of distances for each region in the twenty-eighth embodiment. [Figure 157B] FIG. 157B is a diagram showing an example of the distance for each region in the twenty-eighth embodiment. [Figure 157C] FIG. 157C is a diagram showing an example of distances for each region in the twenty-eighth embodiment. [Figure 157D] FIG. 157D is a diagram showing an example of the distance for each region in the twenty-eighth embodiment. [Figure 158A] FIG. 158A is a diagram showing an example of a target area in the twenty-eighth embodiment. [Figure 158B] FIG. 158B is a diagram showing an example of how a sensing signal is transmitted in the twenty-eighth embodiment. [Figure 158C]FIG. 158C is a diagram showing an example of how a sensing signal is transmitted in the twenty-eighth embodiment. [Figure 159] Figure 159 is a diagram showing an example of a signal transmitted by a device in embodiment 28. [Figure 160A] FIG. 160A is a diagram showing an example of the configuration of the device according to the twenty-ninth embodiment. [Figure 160B] FIG. 160B is a diagram showing an example of the configuration of the device according to the twenty-ninth embodiment. [Figure 161] FIG. 161 is a diagram showing an example of speaker placement in the 30th embodiment. [Figure 162] FIG. 162 is a diagram showing an example of the structure of information regarding sensing capabilities in each embodiment. [Figure 163] FIG. 163 is an explanatory diagram of position measurement or estimation using triangulation in each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A management device according to one aspect of the present invention includes a communication unit that communicates with a plurality of sensing devices arranged within a building, and a processor, wherein the processor receives capability information from the plurality of sensing devices indicating the sensing capabilities of the sensing devices, selects a sensing device to be used to measure a specified area within the building, and estimates the presence or position of a target in the specified area based on the sensing results received from the sensing devices.

[0012] A management method according to one aspect of the present invention is a management method implemented by a management device that communicates with a plurality of sensing devices arranged within a building, the management method receiving capability information from the plurality of sensing devices indicating the sensing capabilities of the sensing devices, selecting a sensing device to be used to measure a specified area within the building, and estimating the presence or position of a target in the specified area based on the sensing results received from the sensing devices.

[0013] A communication device according to one embodiment of the present invention is a communication device that is a first device that senses a second device using radio waves, and includes a communication unit that receives frames transmitted by the second device using radio waves and senses the second device using the received frames, and a control unit that selects a frequency of the radio waves transmitted by the second device from a predetermined number of frequencies, notifies the second device of the selected frequency, and controls the communication unit to perform the sensing using the frequency.

[0014] According to the above aspect, a communication device (i.e., a first device) notifies a second device, which is a target of sensing, of the frequency of radio waves transmitted by the second device, and senses the second device using the notified frequency. Generally, the frequency of radio waves appropriate for sensing depends on the distance between the first device and the second device, or the radio wave environment surrounding one or both of the first device and the second device. Therefore, by configuring the frequency used for sensing to be selectable from among multiple frequencies, the first device can appropriately sense the second device. In this way, the first device can sense its surroundings.

[0015] For example, when selecting the frequency, the control unit may select the frequency independently of a frequency used by the communication unit for communication.

[0016] According to the above aspect, the communication device (i.e., the first device) selects a frequency independent of the frequency used for communication. Therefore, regardless of the frequency used for communication, the first device can select a frequency suitable for sensing by the second device. Therefore, the first device can more appropriately sense its surroundings.

[0017] For example, the sensing may include at least a process of detecting the position of an object, a process of detecting the presence or absence of an object, or a process of detecting the outer shape of an object by analyzing the radio waves received by the communication unit.

[0018] According to the above aspect, the communication device (i.e., the first device) can more easily obtain sensing results of the surroundings of the transmitting device by processing to detect the position of an object, processing to detect the presence or absence of an object, or processing to detect the outer shape of an object.

[0019] A communication device according to one embodiment of the present invention may be a second device that is sensed by a first device using radio waves, and may include a communication unit that transmits frames for the sensing using radio waves, and a control unit that receives a frequency notification from the first device and controls the communication unit to transmit the frames using radio waves using the frequency indicated by the notification.

[0020] According to the above aspect, a communication device (i.e., a second device) receives notification of the frequency of radio waves transmitted by a first device that targets the second device for sensing, and is sensed by the first device by transmitting radio waves of the frequency indicated in the notification. Generally, the radio wave frequency appropriate for sensing depends on the distance between the first device and the second device or the radio wave environment surrounding one or both of the first device and the second device. Therefore, the second device can be a target of sensing by the first device by being configured to be able to transmit the frequency notified by the first device. In this way, the second device can be sensed by the first device.

[0021] For example, the communication unit may transmit, as the frame, a frame that includes a preamble but does not include a data field.

[0022] According to the above aspect, the communication device (that is, the second device) does not include a data field in the frame that is transmitted to be sensed by the first device, and therefore it is possible to shorten the radio wave transmission time.

[0023] A communication method according to one embodiment of the present invention is a communication method executed by a communication device that is a first device that senses a second device using radio waves, and includes a communication step of receiving a frame transmitted by the second device using radio waves and sensing the second device using the received frame, and a control step of selecting a frequency of the radio waves to be transmitted by the second device from a predetermined number of frequencies, notifying the second device of the selected frequency, and controlling the second device to perform the sensing using the frequency in the communication step.

[0024] According to the above aspect, the same effects as those of the above communication device are achieved.

[0025] A communication method according to one embodiment of the present invention is a communication method executed by a communication device that is a second device sensed by a first device using radio waves, and includes a communication step of transmitting a frame for the sensing using radio waves, and a control step of receiving a frequency notification from the first device and controlling the communication step to transmit the frame using radio waves using the frequency indicated by the notification.

[0026] According to the above aspect, the same effects as those of the above communication device are achieved.

[0027] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0028] The transmitting device according to the present disclosure will be described in detail below with reference to the drawings.

[0029] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0030] (Embodiment 1) In this embodiment, a configuration of a detection device that can detect the position of a surrounding object, the positional relationship between a plurality of objects, the distance to an object, etc. The detection device is also called a transmission device.

[0031] FIG. 1 shows an example of the configuration of a detection device according to this embodiment.

[0032] The transmitting device Y101 receives a control signal Y100 as input. At this time, the control signal Y100 includes information for controlling the operation of one or more components included in the detecting device, such as information for "starting object detection operation," information for "ending object detection operation," information for "starting storage of still images or moving images," information for "ending storage of still images or moving images," etc.

[0033] When the control signal Y100 includes information of "start of object detection operation," the transmitting device Y101 generates, for example, M modulated signals used for estimating the direction of arrival of radio waves in the receiving device Y106, and outputs M transmission signals. That is, the transmitting device Y101 outputs transmission signals Y102_1 to Y102_M. Note that M is an integer equal to or greater than 1. The transmission signal Y102_i is then output as a radio wave from the antenna Y103_i. Note that i is an integer equal to or greater than 1 and equal to or less than M. The antenna Y103_i may be configured with one antenna or multiple antennas. The antenna Y103_i configured with multiple antennas may have a function of performing directivity control.

[0034] The modulated signal transmitted from the transmitting device Y101 is reflected by an object in the direction or range of radiation. The receiving device Y106 receives this reflected wave. Therefore, the receiving device Y106 receives as input "a group of received signals Y105_1 received by the antenna Y104_1" to "a group of received signals Y105_N received by the antenna Y104_N." N is an integer equal to or greater than 1.

[0035] In the following, a case where the antenna Y104_i is configured with multiple antennas will be described. Therefore, in the following description, signals received by the antenna Y104_i will be referred to as a "received signal group Y105_i." For example, if the antenna Y104_i is configured with 16 antennas, the received signal group Y105_i will be configured with 16 received signals.

[0036] The receiving device Y106, for example, performs direction-of-arrival estimation for the received signal group Y105_1 and estimates the distance to the object using the time when the transmitting device Y101 transmitted the modulated signal and the time when the received signal group Y105_1 was obtained. Therefore, the receiving device Y106 outputs object estimation information Y107_1. Note that the distance in the term "object distance" in the above description refers to, for example, the distance between the object and the detection device. Here, possible values ​​calculated as the distance include the distance between the object and the antenna, the distance between the object and the center position of multiple antennas, and the distance between the object and a sensor unit (described later). The "object distance" may also be, for example, the distance between the detection device and a point or region on the object where the modulated signal is reflected. Furthermore, for example, when multiple modulated signals are transmitted simultaneously, the "object distance" may be measured simultaneously at each of multiple points or regions on a single object.

[0037] Similarly, the receiving device Y106 estimates the direction of arrival for the received signal group Y105_i, and estimates the distance to the object using the time when the transmitting device Y101 transmitted the modulated signal and the time when the received signal group Y105_i was obtained. Therefore, the receiving device Y106 outputs object estimation information Y107_i, where i is an integer between 1 and N.

[0038] The first processing unit Y108 receives the object estimation information Y107_1 to Y107_N as input. The first processing unit Y108 performs detailed object estimation using the object estimation information Y107_1 to Y107_N, for example, and outputs an object estimation signal Y109.

[0039] The display unit Y113 receives as input image information Y124 and area information Y111 for limiting an area in which an object is recognized, associates the image with the area in which the object is recognized, and outputs an area signal Y114. Note that associating the image with the area in which an object is recognized means, for example, "specifying an area in the image obtained on the display unit Y113 where object recognition is performed." Alternatively, associating the image with the area in which an object is recognized may also mean, for example, "specifying an area in which the first processing unit Y108 performs object recognition according to the area identified in the image obtained on the display unit Y113."

[0040] The selection unit Y112 receives as input area information Y111 and an area signal Y114 for limiting the area in which an object is to be recognized. The selection unit Y112 then determines the area in which the object is to be detected based on the area information Y111 and the area signal Y114, and outputs a selected area signal Y115. Note that the selection unit Y112 does not have to limit the area in which the object is to be detected, in which case the selection unit Y112 does not have to output the selected area signal Y115, or the selected area signal Y115 may include information that "the area in which the object is to be detected is not limited."

[0041] Here, it is stated that "the display unit Y113 exists" and "the display unit Y113 outputs the area signal Y114," but this configuration is not essential. Also, the display unit Y113 may limit the object detection area according to a user's designation by using a touch panel function (a device that combines a display device such as a panel such as an LCD with a position input device such as a touchpad) on a screen such as an LCD.

[0042] The second processing unit Y116 receives the object estimation signal Y109, the selected area signal Y115, and the image information Y124 as input. At this time, the second processing unit Y116 performs a first processing method or a second processing method. The second processing unit Y116 may implement either the first processing method or the second processing method, or may switch between the first processing method and the second processing method depending on the situation. The second processing unit Y116 may also generate auxiliary information for storing distance information of multiple positions in the object estimation signal Y109. The auxiliary information may be, for example, multiple pieces of "position information corresponding to objects that are candidates for shooting," and the second processing unit Y116 may select "position information corresponding to an object to be shot" from the multiple "position information corresponding to objects that are candidates for shooting."

[0043] First processing method: The second processing unit Y116 recognizes objects from the image information Y124. Then, the second processing unit Y116 estimates the distance between each recognized object and the detection device based on this object recognition information and the object estimation signal Y109, and outputs estimated distance information Y117. Note that the second processing unit Y116 may limit the area in which objects are recognized by a selected area signal Y115. Also, the second processing unit Y116 may limit the objects for which distances are to be estimated by the selected area signal Y115.

[0044] Second processing method: The second processing unit Y116 estimates the distance between each object and the detection device from the object estimation signal Y109 and image information Y124, and outputs estimated distance information Y117. The second processing unit Y116 may also limit the area in which objects are recognized by a selected area signal Y115. The second processing unit Y116 may also limit the objects for which distances are to be estimated by the selected area signal Y115.

[0045] The lens control unit Y110 receives the object estimation signal Y109 and the estimated distance information Y117 as input. Then, the lens control unit Y110 determines control of lens-related operations, such as "control of the focal length for the target object," "control of the lens focus for the target object," and "control of the direction in which to photograph the target object," "using the object estimation signal Y109 and the estimated distance information Y117," or "using either the object estimation signal Y109 or the estimated distance information Y117," and outputs an operation control signal Y118.

[0046] The lens unit Y119 receives an operation control signal Y118 as input, and controls lens-related operations based on the operation control signal Y118, such as "control of the focal length relative to the target object," "control of the lens focus relative to the target object," and "control of the direction in which the target object is photographed," and outputs an object signal Y120. The object signal Y120 is an optical signal.

[0047] The shutter section Y121 receives the control signal Y100 and the object signal Y120 as input, controls the operation of the shutter based on the control signal Y100, and outputs a controlled object signal Y122.

[0048] The sensor unit Y123 receives the controlled object signal Y122 as input, converts, for example, an optical / electrical signal, and outputs image information Y124. The sensor unit Y123 may be, for example, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or an organic CMOS image sensor.

[0049] The storage unit Y125 receives the control signal Y100 and image information Y124 as input, and stores image information, such as moving images and still images, based on the control signal Y100. The storage unit Y125 may store the image information acquired by the sensor unit Y123 as is, or may store encoded data encoded using an image encoding technique.

[0050] The storage unit Y125 may store analysis data obtained as a result of signal processing of an image in addition to or instead of image information. The analysis data may be, for example, information about a predetermined detection target, such as a person, animal, vehicle, or drone, as to whether the detection target is being photographed, i.e., whether the detection target is staying within the photographed area. The analysis data may also include information about the attributes of the detection target, such as its color or size, as well as information about the detection target's behavior, such as its orientation, movement path, speed, and stay time, or what it is doing or looking at. For example, the attribute information may be estimated gender or age for a person, or the vehicle model, number of occupants, and amount of luggage for a vehicle.

[0051] As described above, the detection device of this embodiment can estimate the distance to an object using radio waves. Furthermore, the detection device of this embodiment can control the lens used when photographing an object based on the estimated distance to the object, thereby enabling lens control according to the purpose, such as capturing a clear image of the object being photographed. Furthermore, the detection device of this embodiment may be able to estimate the distance to an object even in dark surroundings, and improve the reliability of the estimation of the distance to the object. Furthermore, by performing distance estimation to an object using radio waves and distance estimation to an object based on an optical signal (image), it may be possible to obtain the effect of performing distance estimation with higher accuracy or reliability.

[0052] Next, a description will be given of the configuration of a detection device that can detect an object different from that shown in FIG. 1 with high accuracy.

[0053] Fig. 2 shows an example of the configuration of a detection device that is different from that of Fig. 1. In Fig. 2, components that operate in the same manner as in Fig. 1 are given the same numbers, and descriptions thereof will be omitted.

[0054] The sensor unit Y200 with shutter function receives a control signal Y100 and an object signal Y120. The sensor unit Y200 with shutter function receives the control signal Y100, controls the operation of the shutter based on the control signal Y100, and generates and outputs image information Y124, for example, by converting optical and electrical signals. The shutter of the sensor unit Y200 with shutter function may be, for example, an electronic shutter or a global shutter.

[0055] In FIG. 2, the operations of the parts other than the sensor unit Y200 with shutter function are the same as those explained using FIG.

[0056] The detection device having the above configuration can estimate the distance to an object using radio waves. Furthermore, the detection device having the above configuration can control the lens used when photographing the object based on the estimated distance to the object, thereby enabling lens control according to the purpose, such as capturing a clear image of the object being photographed. The detection device having the above configuration can potentially estimate the distance to an object even in dark surroundings, and improve the reliability of the distance estimation. Furthermore, by estimating the distance to an object using radio waves and estimating the distance to an object based on an optical signal (image), it may be possible to achieve an effect of more accurate or reliable distance estimation.

[0057] Next, a description will be given of the configuration of a detection device that can detect an object different from that shown in FIGS. 1 and 2 with high accuracy.

[0058] Figure 3 shows an example of the configuration of a detection device that is different from Figures 1 and 2. In Figure 3, components that operate in the same way as in Figures 1 and 2 are given the same numbers, and descriptions thereof will be omitted.

[0059] In FIG. 3, a distinctive feature is that the detection device is equipped with both a shutter unit Y121 and a sensor unit Y200 with a shutter function.

[0060] For example, the shutter unit Y121 is assumed to be configured with a mechanical shutter such as a focal plane shutter, and the shutter provided in the sensor unit Y200 with shutter function is assumed to be an electronic shutter or a global shutter.

[0061] The sensor unit Y200 with shutter function receives the control signal Y100, and when the operation information included in the control signal Y100 indicates the video mode, the shutter function of the sensor unit Y200 with shutter function operates. On the other hand, the shutter unit Y121 receives the control signal Y100, and when the operation information included in the control signal Y100 indicates the video mode, the shutter unit Y121 does not operate the shutter, i.e., the shutter remains open.

[0062] The sensor unit Y200 with shutter function receives the control signal Y100 as input, and when the operation information included in the control signal Y100 indicates a still image mode, the control signal Y100 includes, for example, shutter speed information. In the still image mode, the sensor unit Y200 with shutter function operates its own shutter function in accordance with the shutter speed information.

[0063] Furthermore, the shutter unit Y121 receives the control signal Y100 as input, and when the operation information included in the control signal Y100 indicates a still image mode, the control signal Y100 includes, for example, shutter speed information. In the still image mode, the shutter unit Y121 operates the shutter function in accordance with the shutter speed information.

[0064] In still image mode, when the shutter function of the shutter-equipped sensor unit Y200 is operating, the shutter function of the shutter unit Y121 is not operating, and conversely, when the shutter function of the shutter unit Y121 is operating, the shutter function of the shutter-equipped sensor unit Y200 is not operating.

[0065] In FIG. 3, the operations of the parts other than those described above are the same as those described using FIG.

[0066] The detection device having the above configuration can estimate the distance to an object using radio waves. Furthermore, the detection device having the above configuration can control the lens used when photographing the object based on the estimated distance to the object, thereby enabling lens control according to the purpose, such as capturing a clear image of the object being photographed. The detection device having the above configuration can potentially estimate the distance to an object even in dark surroundings, and improve the reliability of the distance estimation. Furthermore, by estimating the distance to an object using radio waves and estimating the distance to an object based on an optical signal (image), it may be possible to achieve an effect of more accurate or reliable distance estimation.

[0067] FIG. 4 is a diagram showing a modification of FIG.

[0068] In FIG. 4, components that operate in the same manner as in FIG. 1 are given the same numbers, and the description thereof will be omitted.

[0069] The second processing unit Y116 receives the object estimation signal Y109, the selected area signal Y115, the image information Y124, and the data group Y403 as input. Based on the data group Y403, the second processing unit Y116 recognizes objects from the image information Y124. Based on the object recognition information and the object estimation signal Y109, the second processing unit Y116 estimates the distance between each recognized object and the detection device, and outputs estimated distance information Y117. The second processing unit Y116 may limit the objects for which distances are to be estimated using the selected area signal Y115.

[0070] Examples of signal processing used when recognizing an object using the image information Y124 include, for example, processing to detect a person or a part of a body such as a person's face, processing to identify a person, processing to detect an object such as a car or drone, processing to identify an object such as a car or drone, processing to detect the motion or movement of a detected person or object, processing to track a detected person or object, etc. Here, the image information Y124 may be one or more still images, or may be a moving image composed of multiple frames acquired continuously at predetermined timing.

[0071] These signal processes are performed, for example, by extracting one or more feature quantities obtained by a predetermined calculation process determined according to the purpose of the signal process from the image information Y124, and comparing the extracted feature quantities with known feature quantities corresponding to the object or its movement to be recognized. These signal processes may also be performed based on the results of determining whether the extracted feature quantities exceed a predetermined threshold. These signal processes may also be performed based on signal processes other than those exemplified above, such as a model created by machine learning using a multi-layered neural network. When using a model created by machine learning using a multi-layered neural network, preprocessing may be performed on the captured video data, and the preprocessed data may then be input to the model created by machine learning using the multi-layered neural network.

[0072] 4, the second processing unit Y116 may output data to the query data unit Y402. For example, based on this data, the query data unit Y402 may reduce the amount of output data of the data group Y403.

[0073] In the above explanation, an example has been described in which the second processing unit Y116 recognizes an object using the image information Y124, but the object may also be recognized using the object estimation signal Y109 in addition to the image information Y124. In this case, the object estimation signal Y109 does not need to be composed of distance information only, and may also include information such as reflectance obtained by analyzing the received signal group Y105_i, for example.

[0074] The second processing unit Y116 may also output object recognition information Y404.

[0075] The communication device Y400 receives object recognition information Y404 as input, generates a modulated signal including this data, and transmits it to a communication device of a communication partner. At this time, the communication device of the communication partner is connected to, for example, a server, and the server obtains the object recognition information Y404 from the modulated signal transmitted by the communication device Y400 and creates an object recognition database. The server generates a modulated signal including this database via the communication device and transmits it to the communication device Y400.

[0076] The communication device Y400 receives this modulated signal, acquires the object recognition database Y401, and outputs it to the query data unit Y402. The query data unit Y402 then receives the object recognition database Y401 as input and updates the data group Y403 for object recognition performed by the second processing unit Y116.

[0077] The detection device having the above configuration can estimate the distance to an object using radio waves. Furthermore, the detection device having the above configuration can control the lens used when photographing the object based on the estimated distance to the object, thereby enabling lens control according to the purpose, such as capturing a clear image of the object being photographed. The detection device having the above configuration can potentially estimate the distance to an object even in dark surroundings, and improve the reliability of the distance estimation. Furthermore, by estimating the distance to an object using radio waves and estimating the distance to an object based on an optical signal (image), it may be possible to achieve an effect of more accurate or reliable distance estimation.

[0078] Furthermore, with the detection device having the above configuration, the mechanism for updating the database used for object recognition can improve the accuracy and reliability of object recognition, which in turn can contribute to improving the accuracy and reliability of distance estimation. Furthermore, when object recognition is performed using information obtained using radio waves in addition to image information, the accuracy and reliability of object recognition may be improved.

[0079] Figure 5 is a modified example of Figure 2, and the same numbers are used to denote components that operate in the same manner as in Figures 1, 2, and 4, and their explanations will be omitted. The operation of each unit in Figure 5 has already been explained, so explanations will be omitted.

[0080] Figure 6 is a modified example of Figure 3, and the same numbers are used to denote components that operate in the same manner as in Figures 1, 2, and 4, and their explanations will be omitted. The operation of each component in Figure 6 has already been explained, so the explanation will be omitted.

[0081] The detection device having the above configuration can estimate the distance to an object using radio waves. Furthermore, the detection device having the above configuration can control the lens used when photographing the object based on the estimated distance to the object, thereby enabling lens control according to the purpose, such as capturing a clear image of the object being photographed. The detection device having the above configuration can potentially estimate the distance to an object even in dark surroundings, and improve the reliability of the distance estimation. Furthermore, by performing distance estimation based on radio waves and optical signals (images), it may be possible to achieve the effect of more accurate or reliable distance estimation.

[0082] Furthermore, with the detection device having the above configuration, the mechanism for updating the database used for object recognition can improve the accuracy and reliability of object recognition, which in turn can contribute to improving the accuracy and reliability of distance estimation. Furthermore, when object recognition is performed using information obtained using radio waves in addition to image information, the accuracy and reliability of object recognition may be improved.

[0083] In this embodiment, the device shown in Figures 1 to 6 as an example of the configuration has been described as a "detection device," but the name is not limited to this. For example, since the device in this embodiment includes a memory unit, it may be called a "memory device," or since it includes a sensor unit, it may be called a "camera," "movie," "surveillance camera," "security camera," "recording device," or "still image capturing device." It may also be simply called an "device." However, the name is not limited to this.

[0084] 1, 2, 3, 4, 5, and 6 may be realized by a system that is a combination of multiple separate devices. An example of realizing the configurations as multiple separate devices will be described below.

[0085] Figure 7 shows the first "device separation example."

[0086] The second device Y702 is a device that includes the lens unit Y119 in the devices of Figures 1, 2, 3, 4, 5, and 6. The first device Y701 is a device that includes parts other than the lens unit Y119 in the devices of Figures 1, 2, 3, 4, 5, and 6.

[0087] The first device Y701 has a first interface Y703 for connecting to the second device Y702, and the second device Y702 has a second interface Y704 for connecting to the first device Y701.

[0088] Therefore, the first interface Y703 receives the operation control signal Y118 as an input. The second interface Y704 outputs a signal to the lens unit Y119. Therefore, by connecting the first interface Y703 and the second interface Y704, the lens unit Y119 can obtain a signal equivalent to the operation control signal Y118. The first interface Y703 and the second interface Y704 may be connected in any manner. For example, the first interface Y703 and the second interface Y704 may be connected directly, or the first interface Y703 and the second interface Y704 may be connected via a connection cable or the like. The connection method is not limited to these examples.

[0089] Figure 8 shows the second "device separation example."

[0090] The third device Y802 is a device that includes the transmitting device Y101, antennas Y103_1 to Y103_M, antennas Y104_1 to Y104_N, receiving device Y106, and first processing unit Y108 in the devices of Figures 1, 2, 3, 4, 5, and 6. The first' device Y801 is a device that includes all parts other than the "transmitting device Y101, antennas Y103_1 to Y103_M, antennas Y104_1 to Y104_N, receiving device Y106, and first processing unit Y108" in the devices of Figures 1, 2, 3, 4, 5, and 6.

[0091] The first' device Y801 has a third interface Y803 for connection with the third device Y802, and the third device Y802 has a fourth interface Y804 for connection with the first' device Y801.

[0092] Therefore, the third interface Y803 receives the control signal Y100 as an input, and the fourth interface Y804 outputs a signal to the transmitting device Y101. Therefore, by connecting the third interface Y803 and the fourth interface Y804, the transmitting device Y101 can obtain a signal equivalent to the control signal Y100.

[0093] Furthermore, the fourth interface Y804 receives the object estimation signal Y109 as input. The third interface Y803 outputs a signal. Therefore, by connecting the third interface Y803 and the fourth interface Y804, the third interface Y803 outputs a signal equivalent to the object estimation signal Y109.

[0094] The third interface Y803 and the fourth interface Y804 may be connected in any manner. For example, the third interface Y803 and the fourth interface Y804 may be connected directly, or the third interface Y803 and the fourth interface Y804 may be connected via a connection cable or the like. The connection method is not limited to these examples.

[0095] Fig. 9 shows a third "example of device separation." In Fig. 9, the same reference numerals are used to denote components that operate in the same manner as in Fig. 7 and Fig. 8, and the description thereof will be omitted.

[0096] The first'' device Y901 is a device that includes all parts of the devices in Figures 1, 2, 3, 4, 5, and 6 except for the ``transmitting device Y101, antennas Y103_1 to Y103_M, antennas Y104_1 to Y104_N, receiving device Y106, first processing unit Y108, and lens unit Y119.'' The first'' device Y901 also includes a first interface Y703 and a third interface Y803.

[0097] In FIG. 9, the operations of the first interface Y703, the second interface Y704, the third interface Y803, and the fourth interface Y804 are as already explained.

[0098] Fig. 9 shows a third "example of device separation." In Fig. 9, the same reference numerals are used to denote components that operate in the same manner as in Fig. 7 and Fig. 8, and the description thereof will be omitted.

[0099] The first'' device Y901 is a device that includes all parts of the devices in Figures 1, 2, 3, 4, 5, and 6 except for the ``transmitting device Y101, antennas Y103_1 to Y103_M, antennas Y104_1 to Y104_N, receiving device Y106, first processing unit Y108, and lens unit Y119.'' The first'' device Y901 also includes a first interface Y703 and a third interface Y803.

[0100] In FIG. 9, the operations of the first interface Y703, the second interface Y704, the third interface Y803, and the fourth interface Y804 are as already explained.

[0101] Fig. 10 shows the fourth "example of device separation." In Fig. 10, the same reference numerals are used to designate components that operate in the same manner as in Figs. 7, 8, and 9, and the description thereof will be omitted.

[0102] The fourth device Y1002 includes the "transmitting device Y101, antennas Y103_1 to Y103_M, antennas Y104_1 to Y104_N, receiving device Y106, first processing unit Y108, and lens unit Y119" in the devices of Figures 1, 2, 3, 4, 5, and 6, as well as the second interface Y704 and the fourth interface Y804. Note that the operation of each unit in Figure 10 has already been explained, so explanation will be omitted.

[0103] Although an example of the operation of this embodiment has been described above using FIGS. 1 to 10, the configurations of FIGS. 1 to 10 are merely examples, and the present invention is not limited to these configurations.

[0104] The transmitting device Y101 and receiving device Y106 in Figures 1 to 10 handle radio waves, and the operation in this case has been described. Alternatively, it is possible to implement a configuration in which the transmitting device Y101 in Figures 1 to 10 generates an optical modulated signal, such as visible light, and the receiving device Y106 receives the optical modulated signal. In this case, LEDs (Light Emitting Diodes) or organic EL (Electro-Luminescence) would be used instead of the antennas Y103_1 to Y103_M, and photodiodes, image sensors, etc. would be used instead of the antennas Y104_1 to Y104_N.

[0105] Note that object recognition may be performed using information acquired using radio waves in devices such as those shown in Figures 1 to 10. The distinctive features of Figures 1 to 10 are that they perform "object recognition or distance estimation to an object" and record (store) "still images or videos," but the configuration of devices having this characteristic is not limited to the configurations shown in Figures 1 to 10.

[0106] To extend the sensing distance, it is necessary to increase the transmission power. For example, this can be achieved by narrowing the transmission band. Sensing using polarized waves is also possible.

[0107] (Embodiment 2) In this embodiment, a method for constructing a modulated signal transmitted by a device that performs distance estimation using radio waves or object recognition using radio waves, such as those shown in Figures 1 to 10 described in embodiment 1, will be described.

[0108] 11 shows an example of the states of a device Y1101 that performs "distance estimation using radio waves or object recognition using radio waves" described in embodiment 1, an object Y1102 that is the target of "distance estimation or object recognition," and other devices. Note that, hereinafter, the device Y1101 that performs "distance estimation using radio waves or object recognition using radio waves" will be simply referred to as "device Y1101."

[0109] In FIG. 11, in addition to a device Y1101 and an object Y1102, there are a terminal Y1103 and an AP (access point) Y1104.

[0110] The APs of terminals Y1103 and Y1104 use the first frequency band when communicating with each other, and the device Y1101 uses the first frequency band when estimating the distance to or recognizing an object Y1102.

[0111] In this case, if "communication between APs of terminals Y1103 and Y1104" and "distance estimation or object recognition of object Y1102 by device Y1101" are performed simultaneously, there is a risk of interference between the radio waves of each other, which may result in a decrease in communication quality and a decrease in the accuracy of distance estimation and object recognition.

[0112] In this embodiment, a method for improving these degradations will be described.

[0113] FIG. 12 shows an example of a frame configuration when the terminal Y 1103 and the AP (Access Point) Y 1104 in FIG. 11 transmit modulated signals for communication, and the horizontal axis represents time.

[0114] The communication reference symbol Y1201 is a symbol for the communication partner to perform signal detection, time synchronization, frequency synchronization, channel estimation, etc. The communication reference symbol Y1201 may also be a MAC (Media Access Control) frame, which is control information required for the procedure of data communication.

[0115] The communication control information symbol Y1202 is assumed to be, for example, a symbol for notifying, for example, a communication method of the communication data symbol Y1203. Therefore, the communication control information symbol Y1202 is assumed to include, for example, information on the error correction coding method (such as coding rate, block length (code length), and error correction code information), information on the modulation method, information on the transmission method (such as information on whether single-stream transmission or multiple-stream transmission is used) (MCS (Modulation and Coding Scheme) information), data length, etc.

[0116] The communication data symbol Y1203 is a symbol for transmitting data to a communication partner.

[0117] Note that the frame configuration in Fig. 12 is merely an example, and other symbols may be included. Furthermore, when transmitting the frame in Fig. 12, a single-carrier transmission method may be used, or a multi-carrier transmission method such as OFDM (Orthogonal Frequency Division Multiplexing) may be used.

[0118] The communication reference symbol Y1201 may include a data transmission area, which may include data for identifying whether the transmission signal is "transmitting a communication signal" or "transmitting a signal for object detection."

[0119] Also, although it is called the communication reference symbol Y1201, the name is not limited to this, and the communication reference symbol Y1201 may also be called a pilot symbol, training symbol, training field, preamble, control information symbol, midamble, etc.

[0120] Fig. 13 shows an example of the frame structure of a modulated signal transmitted by device Y 1101. In Fig. 13, the same numbers are used for components that operate in the same way as in Fig. 12, and since they have already been explained, their explanation will be omitted.

[0121] As explained in the first embodiment, based on the information contained in the control signal Y100 such as "start of object detection operation" or "start of recording of still images or videos," the device Y1101 will first transmit a communication reference symbol Y1201.

[0122] In addition, "start of object detection operation" or "start of recording of still images or videos" may be realized by "user pressing a button," or "user touching a touch panel," or "user pressing a shutter button," or "user pressing a recording start button," which are provided in device Y1101.

[0123] Then, the device Y1101 transmits a "modulated signal for object detection or a modulated signal Y1302 for estimating the distance to the object."

[0124] By transmitting a modulated signal as shown in Figure 13, the APs of terminals Y1103 and Y1104 in Figure 11 can detect the communication reference symbol Y1201 in Figure 13 and thereby determine that a modulated signal is present in the first frequency band. Therefore, upon detecting the communication reference symbol Y1201, the APs of terminals Y1103 and Y1104 temporarily suspend transmission of the modulated signal. Then, once the communication reference symbol Y1201 is no longer detected, the APs of terminals Y1103 and Y1104 can begin transmitting the modulated signal.

[0125] By doing the above, it is possible to suppress the occurrence of situations in which the "modulated signal transmitted by device Y1101" and the "modulated signal transmitted by the APs of terminals Y1103 and Y1104" interfere with each other, thereby achieving the effects of "improving the reception quality of communication data between the APs of terminals Y1103 and Y1104" and "improving the accuracy of distance estimation and object recognition by device Y1101."

[0126] It should be noted that the frame configuration in FIG. 13 is merely an example, and other modulated signals may be included.

[0127] The following describes several examples of sensing methods using radio waves. Note that a device that implements a sensing method using radio waves, which will be described in the following embodiments, may use the sensing method in combination with processing that uses sensing data obtained by a sensor with an imaging function, such as a camera, as described in the first and second embodiments. Also, a device that implements a sensing method using radio waves, which will be described in the following embodiments, may use the sensing method without combining it with processing that uses sensing data obtained by a sensor with an imaging function, such as a camera.

[0128] Here, the implementation of a sensing method using radio waves without combining it with processing using sensing data obtained by a sensor with an imaging function, such as a camera, does not only mean that a device without a camera performs sensing using radio waves. For example, a device with a camera, such as a smartphone, may independently perform sensing, such as image capture and distance measurement, using one or more cameras, and sensing using radio waves, such as a wireless communication unit or wireless radar. Furthermore, a device with a camera, such as a smartphone, may not use the sensing results using radio waves, such as a wireless communication unit or wireless radar, to control the sensing, such as image capture and distance measurement, using one or more cameras, and may not use the sensing results using radio waves, such as a wireless communication unit or wireless radar, to control the sensing, such as image capture and distance measurement, using one or more cameras, but may combine the results of both sensing methods in a single application.

[0129] (Embodiment 3) In this embodiment, for example, the system is configured with one or more of a device that performs communication, a device that performs sensing, and a device that performs communication and sensing. First, the configuration of the device that performs sensing and the device that performs communication and sensing will be described.

[0130] FIG. 14 is a diagram showing an example of the configuration of a device X100 that transmits a sensing signal, receives the sensing signal that has been reflected off a surrounding object and returned, and performs sensing.

[0131] The transmitting device X101 generates transmission signals X102_1 to X102_M as sensing signals and transmits the transmission signals X102_1 to X102_M from antennas X103_1 to X103_M, respectively. Here, the number of antennas used for transmission is M, where M is an integer equal to or greater than 1 or an integer equal to or greater than 2.

[0132] The transmitting device X101 may, for example, multiply the same sensing signal by a coefficient determined for each antenna to generate transmission signals X102_1 to X102_M and transmit the transmission signals from antennas X103_1 to X103_M, thereby controlling the directivity of the sensing signal. The transmitting device X101 may also, for example, multiply each of a plurality of sensing signals by a coefficient determined for each sensing signal and each antenna, combine the signals, and generate transmission signals X102_1 to X102_M, which are then transmitted from antennas X103_1 to X103_M. This allows directivity control for each sensing signal. The coefficient determined for each antenna or the coefficient determined for each sensing signal and each antenna is expressed as a complex number or a real number. The amplitude and / or phase of the sensing signal transmitted from each antenna is changed according to the value of this coefficient. However, the coefficient may be 1, in which case the sensing signal generated by the transmitting device X101 is transmitted as is from the antenna with a coefficient value of 1.

[0133] The transmitting device X101 may transmit a transmission signal without performing directivity control. For example, the transmitting device X101 may output each of the multiple sensing signals as a transmission signal for the corresponding antenna and transmit them from antennas X103_1 to X103_M. Although the above description has been given of a case where there are multiple sensing signals and multiple antennas, the number of sensing signals generated by the transmitting device X101 and the number of antennas that transmit the sensing signals may be one.

[0134] The sensing signals transmitted from antennas X103_1 to X103_M are reflected by object #1 X110_1 and object #2 X110_2, and the reflected sensing signals are received by antennas X104_1 to X104_N provided in device X100. Here, the number of antennas that receive the sensing signals is N, where N is an integer greater than or equal to 1 or an integer greater than or equal to 2. The number M of antennas used for transmission may be the same as or different from the number N of antennas used for reception.

[0135] Received signals X105_1 to X105_N received by antennas X104_1 to X104_N are input to the receiving device X106. The receiving device X106 performs, for example, filtering to extract only components of the frequency band in which the sensing signal is transmitted or of a channel within the frequency band, frequency conversion to convert from a radio frequency band to an intermediate frequency band (IF) or the frequency band of a baseband signal, and weighting and combining of the N received signals, and outputs an estimated signal X107.

[0136] The coefficients used in the weighting and combining process for the N received signals can be set for each of the received signals X105_1 to X105_N, and the reception directionality can be controlled by changing the coefficient values. The coefficients may be estimated in advance, or the received signals X105_1 to X105_N may be used to estimate a coefficient that will result in a larger amplitude or signal-to-noise ratio (SNR) of the sensing signal component after weighting and combining than when other coefficients are used, or that exceeds a predetermined threshold. Furthermore, the receiving device X106 may simultaneously acquire directional signals corresponding to each set of coefficients by using multiple sets of N coefficients corresponding to the received signals X105_1 to X105_N. However, weighting and combining can also be performed without using the coefficients.

[0137] The estimation unit X108 performs sensing, that is, estimation processing related to the surrounding environment, using the estimation signal X107. Details of the estimation processing performed by the estimation unit X108 will be described later.

[0138] The control signal X109 is a control signal input to the transmitting device X101, the receiving device X106, and the estimation unit X108, and instructs the transmitting device X101, the receiving device X106, and the estimation unit X108 to perform sensing, controls the sensing range and sensing timing, etc.

[0139] The above is a description of an example of the configuration of device X100 in this embodiment.

[0140] Note that, in FIG. 14, an example has been described in which signals generated by device X100 are transmitted by M antennas, and signals received by N antennas are subjected to signal processing by receiving device X106, but the configuration of the device that implements the sensing method described in this embodiment is not limited to this.

[0141] For example, a transmitting antenna unit for transmitting a signal may be composed of multiple antenna units, each including multiple antennas. Here, the multiple antenna units may have the same directivity and directivity control function, or the range of directivity control may differ between the antenna units. In this case, one transmitting device X101 may be configured to select an antenna unit to be used for transmitting a sensing signal from the multiple antenna units, or the same sensing signal may be transmitted simultaneously from the multiple antenna units. Furthermore, the transmitting device X101 may be able to switch between transmitting one sensing signal from one antenna unit or simultaneously transmitting it from multiple antenna units. Furthermore, the device X100 may include multiple transmitting devices X101, or may include a transmitting device X101 for each antenna unit.

[0142] Received signals X105_1 to X105_N received by antennas X104_1 to X104_N are input to the receiving device X106. The receiving device X106 performs, for example, filtering to extract only components of the frequency band in which the sensing signal is transmitted or of a channel within the frequency band, frequency conversion to convert from a radio frequency band to an intermediate frequency band (IF) or the frequency band of a baseband signal, and weighting and combining of the N received signals, and outputs an estimated signal X107.

[0143] The coefficients used in the weighting and combining process for the N received signals can be set for each of the received signals X105_1 to X105_N, and the reception directionality can be controlled by changing the coefficient values. The coefficients may be estimated in advance, or the received signals X105_1 to X105_N may be used to estimate a coefficient that will result in a greater amplitude or signal-to-noise ratio (SNR) of the sensing signal component after weighting and combining than when other coefficients are used, or that exceeds a predetermined threshold. Furthermore, the receiving device X106 may simultaneously acquire directional signals corresponding to each set of coefficients by using multiple sets of N coefficients corresponding to the received signals X105_1 to X105_N. However, it is also possible to perform this process without performing weighting and combining.

[0144] The estimation unit X108 performs sensing, that is, estimation processing related to the surrounding environment, using the estimation signal X107. Details of the estimation processing performed by the estimation unit X108 will be described later.

[0145] The control signal X109 is a control signal input to the transmitting device X101, the receiving device X106, and the estimation unit X108, and instructs the transmitting device X101, the receiving device X106, and the estimation unit X108 to perform sensing, controls the sensing range and sensing timing, etc.

[0146] The above is a description of an example of the configuration of device X100 in this embodiment.

[0147] Note that, in FIG. 14, an example has been described in which signals generated by device X100 are transmitted by M antennas, and signals received by N antennas are subjected to signal processing by receiving device X106, but the configuration of the device that implements the sensing method described in this embodiment is not limited to this.

[0148] For example, a transmitting antenna unit for transmitting a signal may be composed of multiple antenna units, each including multiple antennas. Here, the multiple antenna units may have the same directivity and directivity control function, or the range of directivity control may differ between the antenna units. In this case, one transmitting device X101 may be configured to select an antenna unit to be used for transmitting a sensing signal from the multiple antenna units, or the same sensing signal may be transmitted simultaneously from the multiple antenna units. Furthermore, the transmitting device X101 may be able to switch between transmitting one sensing signal from one antenna unit or simultaneously transmitting it from multiple antenna units. Furthermore, the device X100 may include multiple transmitting devices X101, or may include a transmitting device X101 for each antenna unit.

[0149] Similarly, the receiving antenna unit for transmitting signals may be composed of multiple antenna units, each including multiple antennas. Here, the multiple antenna units may have the same directivity control capabilities, such as the directivity control range and directivity control accuracy, or the directivity control capabilities may differ between the antenna units. Furthermore, the multiple antenna units may have the same directivity control capabilities, such as the directivity control range and directivity control accuracy, but may be installed so that the spatial regions in which the directivity can be controlled are different. In this case, one receiving device X106 may be configured to select and use an antenna unit from among the multiple antenna units to acquire a received signal, or may be configured to simultaneously process signals received from the multiple antenna units. Furthermore, the receiving device X106 may be able to switch between processing only a received signal received from one antenna unit and simultaneously processing received signals received by multiple antenna units. Furthermore, the device X100 may include multiple receiving devices X106, or a receiving device X106 may be provided for each antenna unit.

[0150] Furthermore, device X100 may be provided with multiple antennas that can be used for both transmitting and receiving signals, rather than having multiple antennas for transmitting and receiving separately. In this case, device X100 may be able to select and switch between using each antenna for transmission or reception, or may switch over time between using the multiple antennas for transmission and reception.

[0151] Furthermore, device X100 may include a transmitting / receiving antenna unit that can be used in common for transmitting and receiving signals. Here, the transmitting / receiving antenna unit includes multiple antenna units, and each antenna unit can be switched between being used for transmission and being used for reception. Device X100 may include a selection unit that selects and switches between the antenna unit used for transmitting signals generated by transmitting device X101 and the antenna unit used for receiving signals that are signal-processed by receiving device X106.

[0152] When multiple antenna units are used to simultaneously transmit sensing signals, the directivity of the signals transmitted from each antenna unit may be the same or different. If device X100 transmits sensing signals from multiple antenna units with the same directivity, it may be possible to increase the reach of the sensing signals or the distance to the reflection position where the reflected sensing signal can be received.

[0153] The number of antennas constituting the antenna units described above does not need to be the same between the antenna units, and the number of antennas may differ between the antenna units.

[0154] Next, the estimation process performed by the estimation unit X108 will be described using an example.

[0155] The estimation unit X108 estimates, for example, the distance between the device itself and the object that reflected the sensing signal. The distance between the device itself and the object that reflected the sensing signal can be estimated by, for example, detecting the delay time between the transmission time of the sensing signal and the reception time, and multiplying the delay time by the propagation speed of the electromagnetic wave. The estimation unit X108 may also estimate the direction of arrival of the received signal, i.e., the direction of the object that reflected the sensing signal, using a direction of arrival estimation method such as the MUSIC (Multiple Signal Classification) method. The estimation unit X108 can estimate the position of the object that reflected the transmitted signal by estimating the direction in addition to the distance between the device itself and the object. The estimation unit X108 can estimate the position of the object by triangulation using, for example, direction of arrival estimation such as the MUSIC method, the position of the transmitting antenna, the position of the receiving antenna, and information on the direction of transmission directivity control. The estimation unit X108 may also use the received signal to detect the object, its movement, the material of the object, etc.

[0156] Here, the position of the object may be represented in a polar coordinate system or a three-dimensional Cartesian coordinate system. The origin of the coordinate system may be any position within the device X100, for example, and the coordinate axes of the coordinate system may be in any direction. Note that if the equipment including the device X100 includes multiple wireless sensors or other distance sensors in addition to the device X100, each with a similar or different configuration to the device X100, the origin and coordinate axes of the coordinate system for the data acquired by each sensor may be common to the sensors or may be unique to each sensor. The estimation unit X108 may output the position information expressed in the unique coordinate system as is, or may convert it into a coordinate system common within the device and output it. The converted coordinate system may be a coordinate system unique to the device, or it may be a coordinate system common to other devices, such as the same coordinate system as the 3D map data used by the device.

[0157] The estimation unit X108 may also estimate the distance to the object that reflected the signal in each of multiple directions and acquire the three-dimensional coordinates of the estimated multiple reflection positions as a point cloud. Note that the format of the data of the multiple distance measurement results acquired by the estimation unit X108 does not have to be a point cloud format having three-dimensional coordinate values, and may be, for example, a range image or other format. When using a range image format, the position (coordinates) in a two-dimensional plane of the range image corresponds to the arrival direction of the received signal as seen from the device, and the distance to the object in the direction corresponding to the pixel position of each image is stored as a pixel sample value.

[0158] The estimation unit X108 may further perform recognition processing such as estimating the shape of an object using the point cloud data or range image data. For example, it may extract "one or more points located close to each other within a predetermined distance" or multiple points or image regions as the same object, and estimate the shape of the object based on the positional relationship of the one or more points or the shape of the image region. The estimation unit X108 may perform recognition processing such as identifying the sensed object using the results of the object shape estimation. In this case, the estimation unit X108 may, for example, identify whether the object in the sensing range is a person or another animal, or identify the type of object. Note that the recognition processing performed by the estimation unit X108 may be other than object identification. For example, the estimation unit X108 may detect the number of people or vehicles within the sensing range, or estimate the position and orientation of the detected person's face, as part of the recognition processing. The estimation unit X108 may perform a different recognition process from the above-described recognition process, such as face recognition, to determine whether the shape of the detected person's face matches that of a person registered in advance and to determine which person it is.

[0159] Furthermore, the estimation unit X108 may measure the distance between the device itself and an object multiple times at different times to acquire a change over time in the distance between the device itself and the object or the position of a detected point. In this case, the estimation unit X108 may estimate the speed or acceleration of a moving object as a recognition process using the change over time in the distance between the device itself and the object or the position of a point. For example, the estimation unit X108 may estimate the speed or direction of movement of a vehicle traveling within the sensing range. Note that the recognition process performed by the estimation unit X108 using the change over time in the distance or the position of a point may be other than the estimation of the speed or acceleration of an object. For example, the estimation unit X108 may detect whether a person has performed a specific action from a change in the detected person's posture, thereby allowing the device X100 to be used as a gesture input device for electronic devices such as smartphones, tablets, and personal computers.

[0160] In addition, the speed of the moving object described above may be estimated by comparing the frequency of the transmitted sensing signal with the frequency of the received reflected signal and estimating the change in frequency due to the Doppler effect experienced by the reflected signal.

[0161] Next, an example of the sensing signal used in the transmitting device X101 and the receiving device X106 will be described.

[0162] Device X100 may transmit, as a sensing signal, a pulse signal disclosed in Non-Patent Document 1 or Non-Patent Document 2. Device X100 transmits a pulse signal in a frequency band used for sensing, and measures the distance to the object that reflected the sensing signal based on the delay time between the transmission time of the pulse signal and the reception time of the reflected signal.

[0163] As another example of a sensing signal, the device X100 may use a frequency modulated continuous wave (FMCW) or phase modulated continuous wave (PMCW) signal described in Non-Patent Document 3. The FMCW signal is a signal obtained by converting a chirp signal, the frequency of which is changed over time, into a radio frequency. As an estimation process using the FMCW signal, the estimation unit X108 superimposes the signal transmitted from the device X100 and the signal received by the receiving device X106 using a mixer. As a result, the superimposed signal becomes an intermediate frequency signal with a frequency corresponding to the time of flight of the received signal, and the distance to the object that reflected the FMCW signal is measured by detecting the frequency components contained in the superimposed signal.

[0164] As a different example of the sensing signal, the device X100 may use a signal obtained by frequency-converting a modulated signal of a predetermined frequency into a signal in a frequency band used for sensing. In this case, the estimation unit X108 may estimate the distance to the object that reflected the sensing signal, for example, based on the difference between the phase of the modulated component of the signal transmitted from the device X100 and the phase of the modulated component of the signal received by the receiving device X106. The estimation unit X108 may also compare the frequency of the transmitted modulated signal with the frequency of the received modulated signal to detect frequency fluctuations caused by the Doppler effect before the sensing signal is reflected and received, and estimate the moving speed and direction of the moving object. Note that the modulated signal may contain multiple frequency components. For example, an OFDM signal containing multiple frequency components may be used as the modulated signal described in Non-Patent Document 4.

[0165] The sensing signal is not limited to the above examples, and may be a signal modulated by a modulation method, an unmodulated carrier, or any other signal.

[0166] As described above, device X100 may use multiple antennas to simultaneously transmit multiple sensing signals, or may use multiple antenna units each including multiple antennas to simultaneously transmit multiple sensing signals.

[0167] In the present embodiment, the estimation process performed by the estimation unit X108 has been described as an example in which the distance is measured from the difference between the transmission time of the sensing signal and the reception time of the reflected signal. However, the estimation process performed by the estimation unit X108 is not limited to the above. For example, the estimation unit X108 may estimate the state of the transmission path from the received reflected signal and perform recognition processing based on a comparison with changes over time in the estimated transmission path state and average values ​​or feature quantities of previously estimated transmission path states, thereby determining whether an object is present in the sensing range or detecting whether the object is moving. The estimation unit X108 may also detect whether it is raining or not from the attenuation state of the received signal.

[0168] In addition, in this embodiment, an example has been described in which the reflected wave of a transmitted sensing signal is used for sensing. However, sensing using a sensing signal is not limited to the device that transmitted the sensing signal. For example, the receiving device X106 of the device X100 may receive a sensing signal transmitted from another device, and the estimation unit X108 may determine whether the other device is within the range of the sensing signal or estimate the direction of the other device based on the received signal. The distance to the other device may also be estimated based on the signal strength of the received sensing signal. The transmitting device X101 of the device X100 may transmit a sensing signal so that the other device can use it for sensing. The transmitted sensing signal may be a sensing signal transmitted by the device itself for sensing using a reflected wave, or a sensing signal may be periodically transmitted for sensing by the other device. Furthermore, when the device X100 receives a sensing signal transmitted from another device, the device X100 may transmit a sensing signal using the transmitting device X101 in the direction from which the received signal was received. Note that the sensing signal transmitted to the other device may be transmitted without controlling the directionality.

[0169] Also, while Figure 14 shows an example in which sensing device X100 receives signals reflected by objects #1 and #2, sensing device X100 may also obtain signals that are reflected from objects #1 and #2 and then by other objects or substances, and use these to detect the objects and estimate the distance and position of the objects.

[0170] Next, an example of a sensing method using radio waves different from that shown in FIG. 14 will be described.

[0171] Fig. 15 is a diagram showing an example of the configuration of device X200 that performs sensing using radio waves, for example. Of the components shown in Fig. 15, components having the same functions as the components shown in Fig. 14 are given the same reference numerals, and detailed descriptions of these components will be omitted.

[0172] Device X200 differs from device X100 in that it performs sensing using a modulated signal for sensing and / or a modulated signal for communication. A feature here is that, for example, device X200 transmits a signal, and a terminal, which is the communication partner, estimates the position, size, and distance to an object (e.g., object #1 in FIG. 15) by detecting changes in the signal transmitted by device X200. Note that when device X200 transmits a modulated signal for communication, data communication with the terminal is also possible. The following describes the case where sensing is performed using a modulated signal for communication.

[0173] Transmitting device X201 receives control signal X109 and transmission data X210 as input, and performs error correction coding, modulation, precoding, multiplexing, etc. to generate transmission signals X202_1 to X202_M for communication. Device X200 transmits transmission signals X202_1 to X202_M from antennas X103_1 to X103_M, respectively.

[0174] The number of transmission signals and antennas used for transmission is the same as in the description of Fig. 14, and may be two or more, or may be one. Compared to the description of Fig. 14, the transmission signal in the description of Fig. 14 includes a sensing signal component, whereas the transmission signal in this description includes a signal component obtained by modulating transmission data. However, the same applies in that transmitting device X201 can perform directivity control using coefficients used in weighting and combining processing to generate the transmission signal. Also, like device X100, device X200 may include only one antenna unit equipped with multiple antennas, or may include multiple antenna units.

[0175] When performing directivity control, the transmitting device X101 in the description of FIG. 14 controls the directivity of transmission in the direction in which sensing is desired, whereas the transmitting device X201 in FIG. 15 controls the directivity of transmission so as to improve the quality of communication with the terminal that is the communication partner. However, the transmitting device X201 may control the directivity of the transmission signal in the direction in which sensing is desired, or the terminal that is the communication partner may control the directivity so as to obtain desirable sensing results when performing sensing using the signal transmitted by the device X200. When the transmitting device X201 controls the directivity for sensing at the terminal, the transmitting device X201 transmits the signal using a coefficient specified by the terminal. The transmitted signal may or may not include a signal component modulated using transmission data. A signal that does not include a signal component modulated using transmission data is, for example, a signal modulated with a value known on the terminal side, such as a preamble or a reference signal. Furthermore, the transmitting device X201 may perform different directivity control when transmitting a signal including a signal component modulated using transmission data and when transmitting a signal not including a signal component modulated using transmission data.

[0176] The terminal receives the modulated signal transmitted by device X200, thereby obtaining data (performing communication) and also performing sensing.

[0177] Furthermore, when a terminal transmits a signal, device X200, the communication partner, may detect changes in the signal transmitted by the terminal and thereby estimate the position, size, distance to the object (e.g., object #1 in FIG. 15), type, material, etc. of the object (e.g., object #1 in FIG. 15). Note that when the terminal transmits a modulated signal for communication, data communication with device X200 is also possible.

[0178] For example, the receiver X206 receives modulated signals transmitted by the terminals using antennas X104_1 to X104_N. The receiver X206 receives the control signal X109 and received signals X205_1 to X205_N as inputs, performs demodulation and error correction decoding, and acquires received data. The receiver X206 also outputs the channel characteristics and other information obtained by the reception processing as an estimated signal X207.

[0179] The coefficients used in the weighting and combining process for the N received signals can be set for each of the received signals X105_1 to X105_N, and the reception directionality can be controlled by changing the coefficient values. The coefficients may be estimated in advance, or the received signals X105_1 to X105_N may be used to estimate a coefficient that will result in a greater amplitude or signal-to-noise ratio (CNR) of the sensing signal component after weighting and combining than when other coefficients are used, or that exceeds a predetermined threshold. Furthermore, the receiving device X206 may simultaneously acquire directional signals corresponding to each set of coefficients by using multiple sets of N coefficients corresponding to the received signals X105_1 to X105_N.

[0180] The estimation unit X208 receives the control signal X109 and the estimation signal X207 as input and performs estimation processing using the estimation signal X207. The estimation unit X208 estimates the surrounding environment, such as whether or not an object is present, based on, for example, the transmission path characteristics included in the estimation signal X207. The estimation unit X208 may also detect the movement or approach of an object based on temporal changes in the transmission path characteristics. The estimation unit X208 may estimate the direction of arrival of the received signal, i.e., the direction of the object that reflected the sensing signal, using, for example, a direction of arrival estimation method such as the MUSIC algorithm. The estimation unit X208 may estimate the position of the object by triangulation using, for example, information on the direction of arrival estimation method such as the MUSIC algorithm, the antenna position (e.g., the positions of the transmitting device and receiving device), and the direction of transmission directivity control. The estimation unit X208 may also detect the detection of an object, the movement of the object, the material of the object, etc., using the received signal.

[0181] The estimation unit X208 performs the estimation process by performing signal processing on the estimated signal X207 according to the event to be detected, such as the presence or absence of the object or the movement of the object. The estimation process is performed based on, for example, a determination result of whether a feature extracted by the signal processing exceeds a predetermined threshold. The estimation process may also be performed based on signal processing other than the above-described examples. For example, the estimation process may be performed using a model created by machine learning using a multilayer neural network. When using a model created by machine learning using a multilayer neural network for the estimation process, the estimation unit X208 may perform a predetermined preprocessing on the estimated signal X207 and then input the preprocessed data into the model created by machine learning using the multilayer neural network. The estimation unit X208 may also use information such as the frequency band used for communication or a channel number within the frequency band. The estimation unit X208 may also use the address of a communication device that transmitted the received communication signal or the address of a communication device that is the destination of the signal. In this way, by using information about the received communication signal, such as the frequency band and the address of the communication device, it is possible to compare communication signals that have the same or similar conditions, such as the location of the communication device that transmitted the signal or the directivity used when transmitting the signal, which may improve estimation accuracy.

[0182] The above description has been given of a case where sensing is performed using a communication signal transmitted by a communication partner. In FIG. 15 , device X200 is shown with different configurations: transmitting device X201 and antennas X103_1 to X103_M, which are components for performing transmission processing; and receiving device X206 and antennas X104_1 to X104_N, which are components for performing reception processing. However, the configuration of device X200 is not limited to this. For example, transmitting device X201 and receiving device X206 may be implemented as a single component, or multiple antennas may be shared for transmission and reception. Furthermore, as in the description of FIG. 14 , the multiple transmitting antennas in device X200 may be configured as multiple antenna units, and the multiple receiving antennas may be configured as multiple antenna units. Furthermore, the multiple transmitting antennas and the multiple receiving antennas in device X200 may be configured as a common transmitting / receiving antenna unit.

[0183] Furthermore, a sensing signal may be used instead of a communication signal. That is, the first device may use a sensing signal transmitted by another device to estimate the position, size, distance to the object (e.g., object #1 in FIG. 15), type, material, etc. of the object (e.g., object #1 in FIG. 15).

[0184] The sensing method using a communication signal can also be used for the same purpose as the example of transmitting a sensing signal to another device described with reference to Fig. 14. That is, device X200 may use a communication signal transmitted from another device such as a terminal not to sense the surrounding environment based on the transmission path characteristics of the signal, but to determine whether the other device is within the reach of the communication signal or to estimate the direction of the other device.

[0185] It should be noted that when device X200 receives a modulated signal for communication transmitted by a communication partner, for example, a terminal, it may perform only a demodulation operation without performing a sensing operation.

[0186] Next, a device that performs communication and sensing will be described.

[0187] Fig. 16 is a diagram showing an example of the configuration of a device that performs communication and sensing. Of the components shown in Fig. 16, components having the same functions as the components shown in Fig. 14 and Fig. 15 are given the same reference numerals, and detailed descriptions of these components will be omitted.

[0188] Apparatus X300 performs both sensing using a modulated signal for sensing and sensing using a modulated signal for communication. That is, transmitting device X301 of apparatus X300 has a function of transmitting a signal for sensing, similar to transmitting device X101, and a function of transmitting a signal for communication to another communication apparatus, similar to transmitting device X201. Furthermore, receiving device X306 of apparatus X300 has a function of receiving a signal for sensing, similar to receiving device X106, and a function of receiving a signal for communication transmitted by another communication apparatus, similar to receiving device X206. Furthermore, estimation unit X308 performs both estimation processing using a signal for sensing, similar to estimation unit X108, and estimation processing using a signal for communication, similar to estimation unit X208.

[0189] The processing performed by each component of device X300 is the same as that of device X100 in Figure 14 when transmitting and receiving sensing signals, and is the same as that of device X200 in Figure 15 when transmitting and receiving communication signals, so explanations will be omitted.

[0190] 16, the device X300 is shown with different configurations for the transmitting device X301 and antennas X103_1 to X103_M, which are components for performing transmission processing, and the receiving device X306 and antennas X104_1 to X104_N, which are components for performing reception processing, but the configuration of the device X300 is not limited to this. For example, the transmitting device X301 and the receiving device X306 may be realized as a single component, or one or more antennas or multiple antennas may be used in common for transmission and reception.

[0191] The device X300 may also include a transmitter for sensing separate from a transmitter for communication. In this case, the transmitter for communication and the transmitter for sensing may use the same one or more antennas or multiple antennas by switching between them, or may include one or more different antennas or multiple antennas for communication and sensing. The communication and sensing signal transmitter X301 may switch between transmitting sensing signals and modulated signals for communication based on mode information included in the control signal X309, and transmit the signals from the antenna. In other words, there may be a mode for transmitting sensing signals and a mode for transmitting modulated signals for communication. The communication and sensing transmitter X301 may also transmit a signal that combines a sensing signal and a modulated signal for communication.

[0192] Furthermore, device X300 may include a receiving device for sensing separate from a receiving device for communication. In this case, the receiving device for communication and the receiving device for sensing may use the same, one or more, or multiple antennas by switching between them, or may include one or more or multiple different antennas for communication and sensing. Furthermore, device X300 may include a transmitting device for communication, a transmitting device for sensing, a receiving device for communication, and a receiving device for sensing, respectively. Furthermore, device X300 may include a transmitting / receiving device for communication and a transmitting / receiving device for sensing. Furthermore, device X300 may include a transmitting / receiving device for communication, a transmitting device for sensing, and a receiving device for sensing.

[0193] 14 and 15, in this embodiment, one or more transmitting antennas may be configured with one or more antenna units, and one or more receiving antennas may be configured with one or more antenna units. Also, one or more transmitting antennas and one or more receiving antennas may be configured with a common transmitting / receiving antenna unit.

[0194] By using the above-described device configuration, it becomes possible to implement the embodiments described below, and thereby to obtain the effects described in each embodiment. Therefore, more specific examples of implementation methods will be described below.

[0195] (Fourth embodiment) In this embodiment, an example of a sensing method in which an access point (AP) or a terminal having a wireless communication function performs sensing will be described.

[0196] Fig. 17 is a diagram showing an example of a system configuration in this embodiment. In Fig. 17, Z100 is an access point (AP), and Z101_1, Z101_2, and Z101_3 are terminals that perform wireless communication with the AP. Note that Fig. 17 shows a case where each terminal communicates with the AP, but each terminal may have a function for performing communication between terminals.

[0197] Fig. 18 is a diagram showing an example of the configuration of a device capable of communication and sensing provided in AP Z100 and terminals Z101_1, Z101_2, and Z101_3. The device in Fig. 18 performs sensing using one or more frequency bands that can be used for communication and one or more channels in each frequency band, similar to device X300 in Fig. 16 described in the third embodiment.

[0198] The device of FIG. 18 includes a transceiver unit Z201, a sensing unit Z202, a transceiver antenna unit Z203, a transmission signal selection unit Z204, a reception signal selection unit Z205, and a control unit Z206.

[0199] The transmitting / receiving unit Z201 transmits and receives communication signals. The transmission and reception processing of communication signals performed by the transmitting / receiving unit Z201 is similar to the transmission processing performed by the transmitting device X201 and the reception processing performed by the receiving device X206 described in the third embodiment.

[0200] The sensing unit Z202 performs sensing based on the received signal. The sensing unit Z202 may implement a sensing method in which the sensing unit Z202 transmits a sensing signal and performs sensing using the received reflected signal, as described in the third embodiment, or may implement a sensing method in which the sensing unit Z202 performs sensing using a communication signal received from another communication device such as a terminal or AP, as described in the third embodiment. Furthermore, the sensing unit Z202 may implement both a sensing method in which the sensing signal is transmitted and a sensing method in which the received communication signal is used. When implementing a sensing method in which the sensing signal is transmitted, the sensing unit Z202 performs processing similar to that of the transmitting device X101 in the first embodiment to generate and output a sensing signal. On the other hand, when the sensing unit Z202 does not implement a sensing method in which the sensing signal is transmitted and only implements a sensing method in which the communication signal is used, the sensing unit Z202 does not need to transmit a signal.

[0201] The transmitting / receiving antenna unit Z203 transmits and receives signals. The transmitting signal selection unit Z204 transmits signals generated by the transmitting / receiving unit Z201 and the sensing unit Z202 from the transmitting / receiving antenna unit Z203. The receiving signal selection unit Z205 inputs signals received by the transmitting / receiving antenna unit Z203 to the transmitting / receiving unit Z201 and the sensing unit Z202. The control unit Z206 generates control signals to control the operation of the transmitting / receiving unit Z201, the sensing unit Z202, the transmitting signal selection unit Z204, and the receiving signal selection unit Z205, and controls the frequency and period for communication and the frequency and period for sensing. The transmitting signal selection unit Z204 may combine sensing signals and communication signals to generate and output signals based on a frame. The receiving signal selection unit Z205 may also generate communication signals and sensing signals from received signals and output both.

[0202] 19 to 37 show examples of frame configurations transmitted and received by a device according to the present embodiment. In FIGS. 19 to 37, the horizontal axis represents time, and in FIGS. 19, 20, and 22 to 37, the vertical axis represents frequency. The frequency on the vertical axis may represent, for example, the frequency of one channel in one frequency band, such as the subcarriers of an OFDM (Orthogonal Frequency Division Multiplexing) signal, or multiple channels in one or multiple frequency bands. The frequency on the vertical axis may also represent multiple subcarriers of an OFDM signal spanning multiple channels. Therefore, a single-carrier transmission scheme or a multi-carrier transmission scheme such as OFDM may be used. Furthermore, in the case of a frame or signal for sensing, for example, a band-limited pulse signal, a tone signal, or a carrier may be used. Therefore, when the frames of FIGS. 19 to 37 are used for communication and / or sensing, the frame of FIG. 19 does not necessarily represent a single-carrier transmission scheme or a multi-carrier transmission scheme signal.

[0203] The reference symbols shown in Figures 19 to 37 are symbols for the device in this embodiment to perform sensing. Note that the reference symbols may have functions for the communication partner to perform signal detection, time synchronization, frequency synchronization, channel estimation, etc. Also, the reference symbols may be control information required for the procedure to perform data communication, such as a MAC (Media Access Control) frame.

[0204] The control information symbol is assumed to be a symbol for notifying a communication method in a data symbol, for example. Therefore, the control information symbol includes information such as an error correction coding method (such as coding rate, block length (code length), and error correction code information), modulation method, transmission method (such as information on whether single-stream transmission or multiple-stream transmission is used) (MCS (Modulation and Coding Scheme) information), and data length.

[0205] A data symbol is a symbol for transmitting data to a communication partner.

[0206] The guard interval is an interval placed immediately after the reference signal to assist sensing. The guard interval is provided, for example, to avoid interference between signals or symbols placed before and after the guard interval, or to change the transmission directivity or reception directivity of the symbols before and after the guard interval. For example, the guard interval does not need to be free of signals. The guard interval does not need to be provided for a frame. The sensing unit Z202 is used in the guard interval, for example, to acquire the signal reception status when the communication device itself or another communication device is not transmitting a sensing signal or a communication signal.

[0207] 19 to 37 are merely examples, and the frame configuration is not limited to these. For example, other symbols may be included in the frame. Furthermore, when transmitting the frame, a single-carrier transmission method or a multi-carrier transmission method such as OFDM may be used.

[0208] The reference symbol may include a region for transmitting data, and the region for transmitting data may include data for identifying whether the transmitted signal is "transmitting a signal for communication" or "transmitting a signal for object detection."

[0209] Furthermore, although they are called reference symbols, the terminology is not limited to this and they may also be called pilot symbols, training symbols, training fields, preambles, control information symbols, midambles, etc. For example, sensing may be performed using pilot symbols, training symbols, training fields, preambles, control information symbols, midambles, etc.

[0210] FIG. 19 shows a frame structure when the device of this embodiment is performing communication.

[0211] The frame shown in FIG. 19 includes a preamble, control information symbols, and data symbols.

[0212] FIG. 20 shows a frame structure when the device of this embodiment is performing sensing.

[0213] The left diagram in Fig. 20 shows a frame configuration in which reference symbols are transmitted for sensing. The frame shown in the left diagram in Fig. 20 includes reference symbols.

[0214] In the diagram on the right of Fig. 20, a preamble and control information symbols are added to the reference symbols for sensing. The frame shown in the diagram on the right of Fig. 20 includes a preamble, control information symbols, and a reference signal.

[0215] A device that receives the frame on the right in Figure 20, even if it is a device equipped only with communication functions, can know that a sensing signal is present by receiving the preamble and control information symbols, and can thereby obtain the effect of being able to control the transmission timing of the modulated signal to prevent interference. An example of the presence of a signal at that time is shown in Figure 21.

[0216] As shown in FIG. 21, in a certain frequency band, it is possible to have a modulated signal for communication and a modulated signal for sensing.

[0217] The device of this embodiment may be a device equipped with only a communication function so as to be able to send and receive the frame shown in FIG.

[0218] Furthermore, the device of this embodiment may be a device equipped with only a sensing function so as to be able to send and receive the frame shown in FIG.

[0219] The device of this embodiment may be a device that processes communication and sensing in parallel so as to be able to transmit and receive the frames shown in Figures 22 to 37. The frames shown in Figures 22 to 37 will be described below. Note that Figures 22 to 37 are examples of frames that enable sensing, and communication may be enabled in conjunction with sensing. Also, although some of the frames shown in Figures 22 to 37 have guard intervals, implementation is possible even if the guard intervals are not present. Note that when considering the frames shown in Figures 22 to 37, the state shown in Figure 17 is considered. However, it is assumed that the AP can be considered as a repeater. The frame shown in Figure 22 includes a preamble, control information symbols, data symbols, midambles, and data symbols. In this case, the midamble is assumed to be a symbol for demodulating the data symbols and / or a symbol for sensing. This also applies to the other figures. Also, although the preamble is arranged to realize communication, the device may use the preamble and / or the control information symbols for sensing. This also applies to the other drawings.

[0220] The frame shown in Fig. 23 includes a preamble, a control information symbol, a data symbol, a midamble, and a data symbol. A guard interval is provided between the midamble and the data symbol. The transmission directivity may be different between the symbol before the guard interval and the symbol after the guard interval. This also applies to the other drawings. A guard interval may be provided before the midamble. Furthermore, the frame may have no guard interval. This also applies to the other drawings.

[0221] The frame shown in FIG. 24 includes a preamble, control information symbols, data symbols, and a reference signal. The data symbols are allocated in two different frequency bands F81 and F83. The reference signal is allocated in frequency band F82, which is different from the frequency band in which the data symbols are allocated. Three reference signals are allocated at time intervals in frequency band F82. A guard interval is provided between the reference signals. That is, the three reference signals are allocated with a guard interval sandwiched between them. The reference signals are used for sensing. This also applies to other drawings. The transmission directivity of the reference signal before the guard interval and the reference signal after the guard interval may be different. This also applies to other drawings.

[0222] When data symbols or reference signals are allocated to two or more frequency bands or two or more channels, multiple access using the Orthogonal Frequency Division Multiple Access (OFDMA) method may be adopted. In this case, the data symbols or references are allocated in ranges specified in both the time and frequency directions. Here, the ranges are also referred to as time-frequency resources, which are resources defined by time and frequency. Furthermore, time-frequency resources in which symbols containing communication data are allocated are also referred to as communication resources, and time-frequency resources in which symbols for sensing using radio waves are allocated are also referred to as sensing resources. The same applies hereinafter.

[0223] The frame shown in FIG. 25 includes a preamble, control information symbols, data symbols, and reference signals. The data symbols are allocated to five different frequency bands F91 to F95. The reference signals are allocated to frequency bands F92 and F94. In frequency bands F92 and F94, a guard interval is provided between the reference signals and data symbols. As in frequency bands F92 and F94, data symbols and reference signals may coexist in some frequency resources. This can achieve the effect of improving frequency utilization efficiency. Note that the transmission directivities of the reference signals and data symbols before and after the guard interval may be different. This also applies to other drawings.

[0224] The frame shown in Fig. 26 includes a preamble, control information symbols, data symbols, and a reference signal. The data symbols are allocated in frequency bands F101 and F103 to F105. Three reference signals are allocated in frequency band F102, with a guard interval sandwiched between them. Furthermore, reference signals are allocated across frequency bands F101 to F105. After these reference signals, data symbols are allocated in frequency bands F101 to F103 and F105, and two reference signals are allocated in frequency band F104, with a guard interval sandwiched between them. One distinctive feature is the presence of a wideband reference signal and a narrowband reference signal. This increases the possibility of highly accurate sensing.

[0225] The frame shown in Figure 27 includes a preamble, control information symbols, and data symbols addressed to users. The data symbols addressed to users #1, #2, #3, and #4 include data symbols addressed to users #1, #2, #3, and #4, and are allocated to frequency bands F111, F112, F113, and F114, respectively. This frame configuration can be considered an example of an OFDMA frame. For example, it can be considered a frame transmitted by an AP, a repeater, or the like.

[0226] The frame shown in Figure 28 includes a preamble, control information symbols, data symbols addressed to users, and a midamble. The data symbols addressed to users include data symbols addressed to users #1, #2, #3, and #4, and are allocated to frequency bands F121, F122, F123, and F124, respectively. The data symbols addressed to users are also allocated with a midamble sandwiched between them. This frame configuration can be considered an example of an OFDMA frame. For example, it can be considered a frame transmitted by an AP, a repeater, or the like.

[0227] The frame shown in FIG. 29 includes a preamble, control information symbols, data symbols addressed to users, and a midamble. The data symbols addressed to users include data symbols addressed to users #1, #2, #3, and #4, and are allocated to frequency bands F131, F132, F133, and F134, respectively. The data symbols addressed to users are allocated on either side of a midamble. A guard interval is provided between the midamble and the data symbols addressed to users. Note that a guard interval may be provided before the midamble. This frame configuration can be considered an example of an OFDMA frame. For example, it can be considered a frame transmitted by an AP, a repeater, etc.

[0228] The frame shown in FIG. 30 includes a preamble, control information symbols, data symbols addressed to users, and a reference signal. The data symbols addressed to users include data symbols addressed to users #1, #3, and #4, and are allocated to frequency bands F141, F143, and F144, respectively. Three reference signals are allocated to frequency band F142, with guard intervals sandwiched between them. When OFDMA is used, there may be a frequency band (for example, frequency band F142 in FIG. 30) that is not used for data symbols addressed to users. In such a case, a reference signal is allocated to the frequency band that is not used for data symbols addressed to a user. Note that frequency band F142 can also be used to transmit data symbols addressed to a certain user. Note that this can be considered a frame transmitted, for example, by an AP, a repeater, or the like. The frame shown in FIG. 31 includes a preamble, control information symbols, data symbols addressed to users, and a reference signal. Data symbols addressed to users include data symbols addressed to users #1, #2, #3, and #4, and are allocated to frequency bands F151, F152, F153, and F154, respectively. In frequency band F151, two reference signals are allocated with a guard interval after the data symbol addressed to user #1. In frequency band F152, a reference signal is allocated after the data symbol addressed to user #2, and a guard interval is provided after the reference signal.

[0229] When OFDMA is used, the time length of data symbols addressed to each user may differ from user to user. In such a case, a reference signal is allocated in a frequency band and time period not used for data symbols addressed to each user. For example, the time length of data symbols addressed to user #1 is shorter than the time length of data symbols addressed to user #4, and the reference symbol can be transmitted using the time after the data symbols addressed to user #1. Note that this can be considered as a frame transmitted by, for example, an AP, a repeater, or the like. The frame shown in FIG. 32 includes a preamble, control information symbols, data symbols addressed to users, and a reference signal. The data symbols addressed to users #1, #3, and #4 include data symbols addressed to users #1, #3, and #4, and are allocated to frequency bands F161, F163, and F164, respectively. Furthermore, reference signals are allocated across frequency bands F161 to F164, and a guard interval is provided after the reference signal. In frequency band F162, a guard interval is provided between reference signals. This can be considered as a frame transmitted by, for example, an AP, a repeater, or the like.

[0230] The frame shown in FIG. 33 includes a preamble, control information symbols, data symbols addressed to users, and reference signals. The data symbols addressed to users include data symbols addressed to users #1, #2, #3, and #4, and are allocated to frequency bands F171, F172, F173, and F174, respectively. Reference signals are allocated across frequency bands F171 to F174, and a guard interval is provided after the reference signals. In frequency band F171, a reference signal addressed to user #1 is allocated after the guard interval, followed by a guard interval. In frequency band F172, a reference signal addressed to user #2 is allocated after the guard interval, followed by a guard interval. The time lengths of the data symbols addressed to users, the reference signals, and the guard intervals are different between frequency bands F171 and F172. For example, the frame can be considered to be transmitted by an AP, a repeater, or the like.

[0231] The frame shown in FIG. 34 is an example of a frame configuration transmitted by a terminal, for example, a configuration example of a frame transmitted by a terminal owned by user #1, and includes a preamble, control information symbols, and data symbols (transmitted by the terminal owned by user #1). The data symbols (transmitted by the terminal owned by user #1) are allocated to frequency band F181. Frequency band F182 is not allocated to the terminal owned by user #1. However, frequency band F182 can be used by terminals owned by other users (terminals other than user #1) to transmit data symbols and reference signals.

[0232] The frame shown in FIG. 35 is an example of a frame configuration transmitted by a terminal, for example, a configuration example of a frame transmitted by a terminal owned by user #1, and includes a preamble, control information symbols, data symbols (transmitted by the terminal owned by user #1), and a reference signal. The data symbols (transmitted by the terminal owned by user #1) are allocated to frequency band F191. Frequency band F192 is not allocated to the terminal owned by user #1. Furthermore, a reference signal and a guard interval are provided after the data symbol (transmitted by the terminal owned by user #1). Furthermore, a reference signal and a guard interval are also provided after the next data symbol (transmitted by the terminal owned by user #1). However, frequency band F192 can be used by terminals owned by other users (terminals other than user #1) to transmit data symbols and reference signals.

[0233] The frame shown in FIG. 36 is an example of a frame configuration transmitted by a terminal, for example, a configuration example of a frame transmitted by a terminal owned by user #1, and includes a preamble, control information symbols, data symbols (transmitted by the terminal owned by user #1), and a reference signal. The data symbols (transmitted by the terminal owned by user #1) are allocated to frequency bands F201 and F202. Frequency band F203 is not allocated to the terminal owned by user #1. Furthermore, in frequency band F201, as with frequency band F191 in FIG. 35, a reference signal and a guard interval are provided after the data symbol (transmitted by the terminal owned by user #1), and a reference signal and a guard interval are also provided after the next data symbol (transmitted by the terminal owned by user #1). In frequency band F202, as with frequency band F181 in FIG. 34, data symbols (transmitted by the terminal owned by user #1) are allocated, and a reference signal and a guard interval are not provided. However, frequency band F203 can be used by terminals owned by other users (terminals other than user #1) to transmit data symbols and reference signals.

[0234] The frame shown in FIG. 37 is an example of a frame configuration transmitted by a terminal, for example, a configuration example of a frame transmitted by a terminal owned by user #1, and includes a preamble, control information symbols, data symbols (transmitted by the terminal owned by user #1), and a reference signal. The data symbols (transmitted by the terminal owned by user #1) are allocated to frequency bands F211 and F212. Frequency band F213 is not allocated to the terminal owned by user #1. Furthermore, in frequency band F211, as with frequency band F191 in FIG. 35, a reference signal and a guard interval are provided after the data symbol (transmitted by the terminal owned by user #1), and a reference signal and a guard interval are also provided after the next data symbol (transmitted by the terminal owned by user #1). In frequency band F212, data symbols (transmitted by the terminal owned by user #1) and a reference signal are allocated. However, frequency band F213 can be used by terminals owned by other users (terminals other than user #1) to transmit data symbols and reference signals. One distinctive feature is that it contains both wideband and narrowband reference signals. This increases the possibility of highly accurate sensing.

[0235] The device of this embodiment may be a device capable of processing transmission and reception of the frame of FIG. 19 or the frame of FIG.

[0236] The device of this embodiment may be a device capable of processing transmission and reception of the frame of FIG. 19 and the frames of FIGS.

[0237] The device of this embodiment may be a device capable of transmitting and receiving the frame of FIG. 20 and the frames of FIGS.

[0238] The device of this embodiment may be a device capable of transmitting and receiving the frame of FIG. 19, the frame of FIG. 20, and the frames of FIGS. 22 to 37.

[0239] In addition, when the device of this embodiment transmits, for example, frames of Figures 19 to 37, the device of this embodiment may transmit the frames of Figures 19 to 37 from one antenna, or the device of this embodiment may transmit the frames of Figures 19 to 37 from multiple antennas.

[0240] Furthermore, when the device of this embodiment receives modulated signals of the frames of Figures 19 to 37, for example, the device of this embodiment may receive the signals of the frames of Figures 19 to 37 using one antenna, or the device of this embodiment may receive the signals of the frames of Figures 19 to 37 using multiple antennas. Therefore, the transmission method may be any of SISO (Single-Input Single-Output), MISO (Multiple-Input Single-Output), SIMO (Single-Input Multiple-Output), and MIMO (Multiple-Input Multiple-Output).

[0241] By doing so, the AP (or repeater) and the terminal can achieve the effect of being able to perform sensing and communication.

[0242] As described above, the transmitting device includes a frame construction unit that constructs a frame that conforms to the OFDMA system and includes a plurality of time-frequency resources, which are resources defined by time and frequency, and a transmitting unit that transmits the frame constructed by the frame construction unit via radio waves, and the frame construction unit constructs the frame as the frame that includes communication resources, which are time-frequency resources in which symbols including communication data are arranged, and sensing resources, which are time-frequency resources in which symbols for sensing using the radio waves transmitted by the transmitting unit are arranged.

[0243] For example, the frame may include at least two sensing resources that are different in time but have the same frequency and are adjacent to each other in the time direction with a guard interval between them. For example, the reference signal in frequency band F142 in Fig. 30 corresponds to the two sensing resources.

[0244] For example, the frame may include two guard intervals with different time lengths and frequencies. For example, the guard intervals of frequency band F171 and frequency band F172 in Fig. 33 correspond to the two guard intervals.

[0245] For example, the frame may include at least two sensing resources having different frequencies and different time lengths. For example, the reference signal of frequency band F171 and the reference signal of frequency band F172 in FIG. 33 correspond to the two sensing resources.

[0246] Furthermore, the sensing process may include at least a process of detecting the position of an object, a process of detecting the presence or absence of an object, or a process of detecting the outer shape of an object by analyzing the reflected waves received by the receiving unit.

[0247] (Embodiment 5) In this embodiment, a control system that controls the operation of equipment using the detection devices and the like described in the first to fourth embodiments will be described.

[0248] The detection device in this embodiment further includes a control unit that controls the operation of the electric device based on the result of the sensing process performed by the processing unit. The control system in this embodiment includes the detection device and the electric device.

[0249] FIG. 38 is a diagram showing an example of the configuration of a control system according to this embodiment.

[0250] 38 includes a device A11 and a detection device A12. Here, the device A11 is an electric device that can operate under the control of the device A11, such as an electric motorcycle, an electric kick scooter, a vacuum cleaner, an electric car, etc. The device A11 is also an information processing device such as a smartphone or a tablet.

[0251] The device A11 includes interfaces #1 and #2, a control unit A111, a driving unit A112, a sensor A113, and a communication unit A114.

[0252] Interfaces #1 and #2 are interface devices for communication with the detecting device A12. In the example of FIG. 38, interfaces #1 and #2 are interfaces that perform communication without going through another device such as a relay device. Note that the communication here may be wireless communication such as Bluetooth (registered trademark), wireless LAN, or optical communication, or may be wired communication such as USB (Universal Serial Bus) or PLC (Power Line Communication). Also, unlike the example of FIG. 38, communication may go through another device such as a relay device. Interface #1 receives a signal (also referred to as a first signal) from interface #3 of the detecting device A12. Interface #2 transmits a signal (also referred to as a second signal) to interface #4 of the detecting device A12. Note that interface #1 may receive power from the detecting device A12.

[0253] The control unit A111 is a processing unit that controls the driving of the driving unit A112 and the sensor A113. The control unit A111 receives signals from the detection device A12 via interface #1, and controls the driving of the driving unit A112 and the sensor A113 based on the received signals. The control unit A111 also obtains information indicating the states of the driving unit A112 and the sensor A113, and transmits this information to the detection device A12 via interface #2. The control unit A111 may also obtain a software program for operating the control unit A111 via the communication unit A114.

[0254] The driving unit A112 is a processing unit that drives the device A11. The driving unit A112 includes, for example, wheels for moving the device A11, a steering device for controlling the direction of the wheels, and an acceleration device or braking device for controlling the rotation speed of the wheels. The driving unit A112 may further include a battery or the like that serves as a power source for driving the wheels. The driving unit A112 operates under the control of the control unit A111 to accelerate or decelerate the moving speed of the device A11 and change the moving direction. The driving unit A112 also outputs information indicating the state of the driving unit A112 to the control unit A111.

[0255] The sensor A113 is a sensor that senses the surroundings of the device A11. The sensor A113 is, for example, a sensor that measures temperature, humidity, or illuminance, or a distance sensor that measures the distance to a surrounding object. The sensor A113 operates under the control of the control unit A111, and outputs sensor values ​​and the like to the control unit A111.

[0256] The communication unit A114 is a communication interface for wireless communication with an AP (Access Point). Although the AP is referred to as an AP, it may also be a base station, a communication device, or the like.

[0257] The detection device A12 includes interfaces #3 and #4, a control unit A121, a processing unit A122, an image sensor A123, an application storage unit A124, communication units #1 and #2, and a sensing unit A125.

[0258] Interfaces #3 and #4 are interface devices for communication with device A11. In the example of FIG. 38, interfaces #3 and #4 are interfaces that perform communication without going through another device such as a relay device. Note that the communication here may be wireless communication such as Bluetooth (registered trademark), wireless LAN, or optical communication, or may be wired communication such as USB or PLC. Also, unlike the example of FIG. 38, communication may go through another device such as a relay device. Interface #3 transmits a signal to interface #1 of device A11. Interface #4 receives a signal from interface #2 of device A11. Note that interface #3 may supply power to device A11.

[0259] The control unit A121 is a processing unit that controls the operation of the device A11. The control unit A121 acquires, for example, a sensor value acquired by the sensor A113 or information indicating the state of the driving unit A112 from the device A11 via interface #4, and generates a signal that controls the driving of the driving unit A112 and / or the control unit A111 based on the acquired sensor value or information. The control unit A121 transmits the generated signal to the device A11 via interface #3 and controls, for example, the control unit A111 and / or the driving unit A112.

[0260] Another method will be described. The control unit A121 is a processing unit that controls the operation of the device A11. The control unit A121 acquires sensor values ​​obtained by the sensing unit A125 via the processing unit A122, and generates signals to drive the driving unit A112 and / or control the control unit A111 based on the acquired sensor values. Note that the control unit A121 may use information obtained from the driving unit A112 and / or the sensor A113 when generating signals to drive the driving unit A112 and / or control the control unit A111. The control unit A121 then transmits the generated signals to the device A11 via interface #3 to control, for example, the control unit A111 and / or the driving unit A112.

[0261] The processing unit A122 is a calculation unit that performs information processing in the detection device A 12. The processing unit A122 is configured by, for example, a CPU.

[0262] The image sensor A123 is a sensor that captures an image of the surroundings of the detection device A12 and generates an image. The image sensor A123 provides data related to the generated image to the processing unit A122.

[0263] The application storage unit A124 is a storage device that stores an application (software program) for controlling the device A11 or the detection device A12. The application storage unit A124 acquires the application (software program), for example, via the communication unit #1 or #2.

[0264] Furthermore, by updating the application, the accuracy of control of the control unit A121 is improved, which has the effect of improving safety in relation to the operation of the system configured by the device A11 and the detection device A12.

[0265] The communication unit #1 is a communication interface that performs wireless communication with an AP (access point).

[0266] The communication unit #2 is a communication interface for wireless communication with a base station of a mobile phone carrier network.

[0267] The sensing unit A125 is a processing unit that performs sensing using radio waves for communication. The sensing unit A125 detects objects around the detection device A12, more specifically, people, cars, bicycles, obstacles, etc., and recognizes the movements of the objects. The sensing unit A125 may be realized by transmitting radio waves at the same frequency as the communication unit #1.

[0268] In the control system shown in FIG. 38, the device A11 may be equipped with the detection device A12, or may be located at a position remote from the detection device A12.

[0269] When the device A11 is equipped with the detection device A12, the device A11 is controlled by signals transmitted and received via the communication unit A114 or the interface #1 or #2.

[0270] For example, when the device A11 is controlled via the interface #1 or #2 and is used when the user is out, the device A11 is equipped with the detection device A12.

[0271] Furthermore, for example, when device A11 is controlled via communication unit A114, device A11 is controlled based on a signal transmitted from detecting device A12 via an AP (access point). However, when controlling device A11 via communication unit A114, there is a possibility that functional limitations may be imposed. Note that when device A11 communicates with detecting device A12 via communication unit A114, direct communication with communication unit #1 of detecting device A12 may be performed instead of via an AP (access point).

[0272] According to this configuration, by updating the application on the side of the detection device A12, it is possible to update the functions and the control algorithm, which results in the effect that new functions can be provided by updating the application.

[0273] Furthermore, by installing a new application on the detection device A12, it becomes possible to use devices such as smartphones or tablets that have not previously been used to control the device A11 to control the device A11. This configuration makes it possible to use existing devices to control the device A11. As a result, for example, even devices that are not in use and are in an idle state can be used, which has the effect of potentially making it possible to utilize wasted resources such as the CPU, GPU, memory, storage, modem, or display of the unused devices.

[0274] It is also possible to configure the device A11 to be able to update firmware or applications, not just the detection device A12. This configuration makes it possible to provide new functions that cannot be addressed by simply updating the application on the detection device A12, or to provide firmware or applications that resolve security issues and quickly take countermeasures when a security issue is discovered on the device A11. The application is obtained from an external device, such as a cloud server, via communication units #1 and #2.

[0275] Next, an example of the configuration of a control system that controls the device A11 when the user is out and about, with the device A11 equipped with the detection device A12, will be described.

[0276] FIG. 39 is a diagram showing an example of the configuration of a control system according to this embodiment.

[0277] The control system shown in FIG. 39 includes a device A11, a detection device A12, an AP A21, a network A22, a base station A23, and a device A24.

[0278] The device A11 and the detector A12 transmit and receive signals via interfaces #1, #2, #3, and #4, the details of which have already been explained.

[0279] The AP of A21 is an access point that can be connected to the communication unit #1 of the detecting device A12.

[0280] Network A22 is a network connected to AP A21 and also connected to base station A23. Network A22 may include a part of a mobile phone carrier network or a part of the Internet. Network A22 may also be connected to a cloud including a server that executes information processing.

[0281] The base station A23 is, for example, a base station device connected to a mobile phone carrier network, and is connected to the device A24.

[0282] The device A24 is, for example, a portable information processing device such as a smartphone or tablet that is carried by a user.

[0283] In the control system, a detection device A12 is communicably connected to a device A24 via an access point A21, a network A22, and a base station A23 by a communication unit #1.

[0284] For example, a detection device A12 is connected to a device A11 such as a vacuum cleaner via interfaces #1, #2, #3, and #4. The device A24 accesses the detection device A12 installed in the device A11 via, for example, one or more networks, and controls the device A11 via the detection device A12.

[0285] One of the features of this control system is that the device A11 is controlled using data sensed by the sensing unit A125 included in the detection device A12. With this configuration, data sensed by a sensor not included in the device A11 can be used to control the device A11. This may enable the realization of functions that cannot be realized by the device A11 alone.

[0286] Furthermore, by having device A11 acquire data sensed by sensing unit A125 of detection device A12 via interface #1 and / or #2, security can be improved compared to acquiring data via a network via communication unit A114, and it may also become possible to provide functions that would not be possible if security constraints were present.

[0287] One of the features of this control system is that the device A11 is controlled via the detection device A12 installed in the device A11. This configuration allows the detection device A12 to determine whether or not to accept a control instruction transmitted from the device A24. For example, security against unauthorized access can be improved by performing processing such as authentication between the device A24 and the detection device A12.

[0288] Furthermore, as mentioned earlier, by updating the application in the application memory unit A124, the accuracy of control of the control unit A121 is improved, which has the effect of improving safety in relation to the operation of the system consisting of the equipment A11 and the detection device A12.

[0289] FIG. 40 is a diagram showing an example of the appearance of the control system according to this embodiment.

[0290] 40 is an electric scooter that corresponds to the device A11 and is equipped with a detection device A12. The detection device A12 controls, for example, the acceleration and deceleration of the device A11 based on the results of sensing by the sensing unit A125.

[0291] The electric kick scooter may be capable of controlling the drive unit A112 in accordance with user input using an input unit for operation input on the device A11 (i.e., the electric kick scooter) side or on the detection device A12 side, for example.

[0292] The control unit A121 controls the driving unit A112 to perform operations such as accelerating the device A11 (i.e., the electric kickboard) or changing the direction of travel. The control unit A121 may also control the driving unit A112 to decelerate the device A11. The deceleration may be achieved by a brake or by using the braking force of a generator or the like.

[0293] Here, the control unit A121 may control the drive unit A112 in accordance with input from the input unit, or may control the drive unit A112 based on sensing data acquired from the device A11 and / or the sensing unit A125.

[0294] For example, the sensing unit A125 performs sensing using radio waves. Note that the sensing method using radio waves is as described in other embodiments. For example, it is assumed that the sensing unit A125 performs sensing and detects a person. Based on the information that "a person has been detected" obtained from the sensing unit A125, the control unit A121 transmits a control signal including information to slow down the speed to the device A11 via the interface #3 and the interface #4. In response to this, the control unit A111 and the drive unit A112 of the device A11 perform control to slow down the speed.

[0295] Also, it is assumed that the sensing unit A125 performs sensing using radio waves and detects that there is no obstacle ahead. Based on the information that "there is no obstacle" obtained from the sensing unit A125, the control unit A121 accelerates the speed and then transmits a control signal including information to travel at a constant speed to the device A11 via the interface #3 and the interface #4. Accordingly, the control unit A111 and the drive unit A112 of the device A11 perform control to travel at a constant speed after accelerating.

[0296] Furthermore, it is assumed that the sensing unit A125 performs sensing using radio waves and detects that there is an obstacle on the right side. Based on the information obtained from the sensing unit A125 that "there is an obstacle on the right side," the control unit A121 transmits a control signal including information to perform an operation to avoid the obstacle to the device A11 via the interface #3 and the interface #4. In response to this, the control unit A111 and the drive unit A112 of the device A11 perform control to perform an operation to avoid the obstacle.

[0297] Another operation example will be described. The sensing unit A125 performs sensing using radio waves, and based on the information resulting from the sensing, the control unit A121 may estimate its own position or detect an obstacle and output a control signal for controlling acceleration or deceleration. At this time, the control unit A121 can also estimate its own position or detect an obstacle using image information obtained from the image sensor A123 and information obtained from the sensor A113. The control unit A121 transmits this control signal to the device A11 via interface #3 and interface #4. Accordingly, the control unit A111 and the drive unit A112 of the device A11 are controlled based on the control signal.

[0298] As another example of control based on sensing data acquired from the sensing unit A125, the control unit A121 detects a user's movement based on speed or acceleration data acquired by a speed sensor or acceleration sensor, and controls acceleration or deceleration. Here, the user's movement detected using the sensing unit A125 is the user kicking the ground, but other movements are also acceptable. For example, the movement may be the user's movement of the center of gravity of the body, or the direction of the user's face or a change in the direction of the user's face. In this case, the sensor that can be used as the sensing unit A125 may be a weight sensor, wireless radar, or radio wave sensing.

[0299] By controlling in the above manner, it is possible to obtain the effect of improving the safety of the user.

[0300] 41 is a diagram showing an example of processing by the control system in this embodiment. An example of control in the above example of control operation by the control unit A121 will be described using a flowchart.

[0301] The device A11 starts the control operation of the flowchart, for example, when the user instructs the start of the control operation using a switch or button, when the user turns on the power of the device A11, when the user attaches the detection device A12 to the device A11, or when it detects that the user has boarded the kick scooter that is the device A11.

[0302] In step S001, when the control unit A121 starts a control operation, it acquires acceleration data from the sensor A113 included in the device A11 and / or the sensing unit A125 included in the detection device A12.

[0303] In step S002, the control unit A121 detects an accelerating motion of the device A11 from the acceleration data. If an accelerating motion is detected (Yes in step S002), the process proceeds to step S003, and if an accelerating motion is not detected (No in step S002), the process proceeds to step S004.

[0304] In step S003, the control unit A121 controls the drive unit A112 to execute an assist process, for example, by rotating a motor to generate an acceleration force in the traveling direction. The acceleration force may be generated for a predetermined time period when it is determined from the acceleration data that acceleration has occurred in order to further increase the acceleration, or may be generated after acceleration has occurred, at the end of acceleration or the start of deceleration, so as to maintain the speed for a predetermined time period. When the assist process in step S003 ends, the process returns to step S001.

[0305] In step S004, the control unit A121 determines whether to end control of the drive unit A112, and ends the processing if the control of the drive unit A112 is to be ended, and returns to step S001 if the control of the drive unit A112 is not to be ended. The determination of whether to end control of the drive unit A112 may be made, for example, when the device A11 has completely stopped, or when the user has turned off the power of the device A11, when the user has removed the detection device A12 from the device A11, or when it has been detected that the user has got off the kick scooter that is the device A11.

[0306] Another operation of Fig. 41 will be described. The device A11 starts the control operation of the flowchart, for example, when the user issues an instruction to start the control operation using a switch or button, when the user turns on the power of the device A11, when the user attaches the detection device A12 to the device A11, or when it is detected that the user has boarded the kick scooter that is the device A11.

[0307] In step S001, when the control unit A111 starts a control operation, it acquires acceleration data from the sensor A113 provided in the device A11.

[0308] In step S002, the control unit A111 detects an accelerating motion of the device A11 from the acceleration data. If an accelerating motion is detected (Yes in step S002), the process proceeds to step S003, and if an accelerating motion is not detected (No in step S002), the process proceeds to step S004.

[0309] In step S003, the control unit A111 controls the drive unit A112 to execute an assist process, for example, by rotating a motor to generate an acceleration force in the traveling direction. The acceleration force may be generated for a predetermined time period when it is determined from the acceleration data that acceleration has occurred in order to further increase the acceleration, or may be generated after acceleration has occurred, at the end of acceleration or the start of deceleration, so as to maintain the speed for a predetermined time period. When the assist process in step S003 ends, the process returns to step S001.

[0310] In step S004, the control unit A111 determines whether to end control of the drive unit A112, and ends the process if the control of the drive unit A112 is to be ended, or returns to step S001 if the control of the drive unit A112 is not to be ended. The determination of whether to end control of the drive unit A112 may be made, for example, when the device A11 has completely stopped, or when the user has turned off the power of the device A11, when the user has removed the detection device A12 from the device A11, or when it has been detected that the user has got off the kick scooter that is the device A11.

[0311] By providing acceleration assistance based on the user's movements, detailed control according to the user's movements is possible, which makes it possible to prevent unintended acceleration and thereby improve safety.

[0312] (Embodiment 6) In this embodiment, a modification of the configuration and processing of the transmitting device in the above embodiment will be described.

[0313] FIG. 42 is a diagram showing an example of the configuration of the transmission device A3 in the sixth embodiment.

[0314] As shown in FIG. 42, the transmitting device A3 includes a frame configuration unit A31, a transmitting unit A32, a receiving unit A33, and a processing unit A34.

[0315] The frame configuration unit A31 configures a frame that conforms to a single-carrier system, a multi-carrier system such as OFDM, or an OFDMA (Orthogonal Frequency-Division Multiple Access) system, and that includes multiple time-frequency resources, which are resources defined by time and frequency.

[0316] The frame construction unit A31 constructs the frame as the above frame, which includes communication resources, which are time-frequency resources in which symbols containing communication data are placed, and sensing resources, which are time-frequency resources in which symbols for sensing using radio waves transmitted by the transmission unit A32 are placed.

[0317] The transmitting unit A32 transmits the frame constructed by the frame constructing unit A31 by radio waves.

[0318] FIG. 43 is a diagram illustrating an example of processing performed by the transmitting device according to the sixth embodiment.

[0319] 43, in step S1 (frame configuration step), a frame is configured that conforms to a single-carrier system, a multi-carrier system such as OFDM, or an OFDMA system, and that includes a plurality of time-frequency resources that are resources defined by time and / or frequency. Here, in the frame configuration step, the frame is configured to include communication resources that are time-frequency resources in which symbols including communication data are allocated, and sensing resources that are time-frequency resources in which symbols for sensing using radio waves transmitted in the transmission step are allocated.

[0320] In step S2 (transmission step), the frame constructed in the frame construction step is transmitted by radio waves.

[0321] This allows the transmitting device to sense the surroundings.

[0322] (Embodiment 7) In this embodiment, a specific example of an apparatus capable of performing sensing will be described.

[0323] FIG. 44 shows an example of the configuration of a device having both a communication function and a sensing function.

[0324] The transmitter / receiver N102 receives data N101 and a control signal N100 as input. When the control signal N100 indicates that "communication is to be performed," the transmitter / receiver N102 performs processes such as error correction coding and modulation on the data N101, and outputs a modulated signal N103. When the control signal N100 indicates that "sensing is to be performed," the transmitter / receiver N102 does not operate.

[0325] The sensing unit N104 receives the control signal N100 as input, and when the control signal N100 indicates that "sensing is to be performed," the sensing unit X204 outputs a sensing signal N105. Note that when the control signal N100 indicates that "communication is to be performed," the sensing unit N104, for example, does not operate.

[0326] The sensing unit N104 receives an operation-related signal N180 as input, determines a sensing operation based on the operation-related signal N180, and performs the operation based on the determination. Details will be explained later.

[0327] The transmission signal selection unit N106 receives as input the control signal N100, the modulated signal N103, and the sensing signal N105. If the control signal N100 indicates that "communication is to be performed," the transmission signal selection unit N106 outputs the modulated signal N103 as the selected signal N107. If the control signal N100 indicates that "sensing is to be performed," the transmission signal selection unit N106 outputs the sensing signal N105 as the selected signal N107.

[0328] The power adjustment unit N108 receives the selected signal N107 and the control signal N100 as input. If the control signal N100 indicates that communication is to be performed, the power adjustment unit N108 performs communication power adjustment on the selected signal N107 (for example, the coefficient by which the selected signal N107 is multiplied is α) and outputs a transmission signal N109.

[0329] Also, if the control signal N100 indicates that "sensing is to be performed," the selected signal N107 is subjected to power adjustment for communication (for example, the coefficient to be multiplied by the selected signal N107 is set to β), and the transmission signal N109 is output.

[0330] For example, α and β are assumed to be real numbers equal to or greater than 0. In this case, α>β (α is greater than β). By doing so, it is possible to reduce the transmission power during sensing, which makes it difficult to sense through walls, etc., increasing the possibility of ensuring privacy, and also achieving the effect of obtaining high data reception quality during communication.

[0331] Furthermore, α and β may be complex numbers. In this case, it is assumed that |α|>|β|. In this case, too, the transmission power during sensing can be reduced, which makes it difficult to sense through walls, etc., increases the possibility of ensuring privacy, and also provides the effect of obtaining high data reception quality during communication.

[0332] The transmission signal N109 is then output as a radio wave from the transmission / reception antenna unit N110.

[0333] Note that the power adjustment unit N108 may not be provided. At this time, the selected signal N107 is output as a radio wave from the transmitting / receiving antenna unit N110.

[0334] The transmitting / receiving antenna unit N110 outputs a receiving signal N111. The receiving signal selection unit N112 receives the control signal N100 and the receiving signal N111 as input. If the control signal N100 indicates that "communication is to be performed," the receiving signal selection unit N112 outputs the receiving signal N111 as a signal N113.

[0335] Furthermore, when the control signal N100 indicates that "sensing is to be performed," the reception signal selection unit N112 outputs the reception signal N111 as the signal N114.

[0336] The transmitter / receiver N102 receives the control signal N100 and the signal N113 as input. When the control signal N100 indicates that communication is to be performed, the transmitter / receiver N102 performs processes such as demodulation and error correction decoding on the signal N113, and outputs received data N115.

[0337] The sensing unit N104 receives the control signal N100 and the signal N114 as inputs. When the control signal N100 indicates that sensing is to be performed, the sensing unit N104 performs sensing using the signal N114 and the like, and outputs the sensing result N116.

[0338] The control unit N151 generates and outputs a control signal N100 based on an external signal N150, received data N115, and the like.

[0339] The control unit N151 generates and outputs a control signal N100 based on an external signal N150, received data N115, and the like.

[0340] The registration unit N180 receives the external signal N150 and the sensing result N116 as input. For example, if the external signal N150 indicates that an action for action registration is to be performed, the registration unit N180 outputs a signal N180 related to the action that includes information that action registration will be performed.

[0341] When the signal N180 relating to the action includes information indicating that action registration is to be performed, the sensing unit N104 generates and outputs a sensing signal N105 for sensing the gesture or the like of interest.

[0342] This sensing signal N105 is then transmitted as a radio wave, and thereafter, a device capable of sensing receives the signal, and the sensing unit N104 performs sensing estimation on the received signal and outputs a sensing result N116.

[0343] The registration unit N180 will register the sensing result N116.

[0344] A specific example will be described below.

[0345] Example 1: Assume that a first person takes a "sensing-capable device" or a "device capable of sensing and equipped with a communication function" into a home and leaves it somewhere. Here, the "sensing-capable device" or the "device capable of sensing and equipped with a communication function" is named device #A. Note that specific examples of device #A have already been described.

[0346] The first person requests that he or she easily find the device #A that he or she left behind. An example of the operation for this request will be described below.

[0347] Fig. 45 shows an example of the relationship between device #A and a first person. As shown in Fig. 45, first, device #A "decides to register a gesture (N201)." Therefore, device #A will transmit a sensing signal.

[0348] In response to this, the first person "performs a movement to be registered in device #A (N202)." This gesture will be called the "first gesture."

[0349] Then, device #A "registers a first gesture (N203)." Device #A may have a function to check whether the gesture has been registered correctly. Device #A may also have a function to edit the registered gesture. These functions may be used, for example, by the first person to correctly register the first gesture in device #A.

[0350] Next, "the registered gesture (for example, the first gesture) is linked to the action of device #A (N211)." For example, let us say that when a person performs the first gesture when device #A is missing, device #A performs the action of "playing a sound" or "vibrating." For example, let us say that the above ("playing a sound" or "vibrating") is named the "first action." It should be noted that device #A registers the content linked to the first action.

[0351] Thereafter, it is assumed that device #A performs sensing periodically, regularly, or irregularly.

[0352] For example, the first person can no longer find device #A, so he or she "performs a first gesture (N212)." Note that although the person who performs the first gesture is defined as the first person, another person may perform the first gesture.

[0353] Then, device #A performs sensing to recognize the first gesture and perform the first action (N213).

[0354] By doing so, it is possible to obtain the effect that device #A can be easily found, and it is also possible to obtain the effect that a person does not need to have a special device.

[0355] Next, a method for preventing erroneous gesture recognition will be described.

[0356] As described above, device #A can register gestures made by a person using N201, N202, and N203 in FIG. 45. In this manner, device #A is assumed to register multiple gestures. For example, device #A is assumed to have registered a first gesture, a second gesture, a third gesture, and a fourth gesture.

[0357] Incidentally, as mentioned above, if device #A is configured to "make a sound" or "vibrate" simply when a person makes the first gesture, there is a possibility that device #A may "make a sound" or "vibrate" simply by a person making the first gesture unintentionally (herein referred to as a malfunction).

[0358] To prevent such erroneous operation, a method is adopted in which multiple gestures are linked with the operation of device #A.

[0359] For example, if the first person performs the first and fourth gestures registered in device #A consecutively, device #A will "make a sound" or "vibrate," and this will be registered in device #A.

[0360] In this way, when the first person performs the first gesture and the fourth gesture, device #A will recognize these gestures and device #A will "make a sound" or "vibrate."

[0361] In this way, by combining multiple gestures, the probability that someone other than the first person who registered will accidentally perform the combined action is reduced, which has the effect of dramatically reducing malfunctions of device #A.

[0362] The number of gestures to be registered in device #A is not limited to the above example, and multiple gestures can be registered in the same manner. The number of gestures to be combined is also not limited to the above example, and multiple gestures can be combined and linked to the operation of device #A.

[0363] The number of gestures registered in device #A is not limited to the above example, and multiple gestures can be registered in the same manner. The number of gestures to be combined is also not limited to the above example, and multiple gestures can be combined and linked to the operation of device #A. While examples of "playing a sound" or "vibrating" have been described as examples of the operation of device #A, the link between multiple gestures and device operations is not limited to this, and multiple gestures may be linked to the operation of a device (terminal) as in the examples described below using Figures 46 to 49.

[0364] In addition, in the above example, when a person performs multiple gestures and device #A recognizes these multiple gestures, the action that device #A performs is described as "making a sound" or "vibrating," but this is not limited to this, and the device (terminal) may also operate as in the examples described below using Figures 46 to 49.

[0365] 45, a gesture is registered and then linked to an operation of device #A, but the procedure is not limited to this order. An operation of device #A may be specified and then the gesture to be linked may be registered. Also, the gesture to be linked to an operation of device #A may be a gesture that device #A already possesses. The important point here is to link one or more gestures to an operation of terminal #A.

[0366] Example 2: In Example 1, we explained the case where a "gesture" is linked to the "action of device #A making a sound or vibrating," but in this example, we will explain the case where a "gesture" is linked to the "operation of the communication function of device #A (terminal #A)."

[0367] Figure 46 shows an example of the state of a first person N301, a terminal #A of N302 that is "capable of sensing and has a communication function," and a device #B of N303. In Figure 46, it is assumed that terminal #A of N302 and device #B of N303 can communicate.

[0368] Figure 47 is an example different from Figure 46, and components that operate in the same way as in Figure 46 are assigned the same numbers. In Figure 47, there are a first person N301, a terminal #A of N302 that is "capable of sensing and has a communication device," an AP (access point) N401, and a device #B of N303. A network N402 may also exist. In Figure 47, it is assumed that terminal #A of N302 and the AP of N401 can communicate, and that device #B of N303 and the AP of N401 can communicate.

[0369] The operation of each device in FIG. 46 will be described with reference to FIG.

[0370] First, terminal #A of N302 "decides to register a gesture (N501)." Therefore, terminal #A of N302 transmits a sensing signal.

[0371] In response to this, the first person N301 "performs a movement to register in terminal #A of N302 (N502)." This gesture will be called the "second gesture."

[0372] Then, terminal #A of N302 "registers the second gesture (N503)." As described above, terminal #A of N302 may have a function to check whether the gesture was registered correctly. Terminal #A of N302 may also have a function to edit the registered gesture. These functions may be used, for example, by the first person N301 to correctly register the second gesture in terminal #A of N302.

[0373] Next, "the registered gesture (for example, the second gesture) is linked (N511) with the operation of terminal #A of N302." As an example, the following linkage is assumed: "When a person, including the first person N301, performs the second gesture, terminal #A of N302 instructs device #B of N303 to perform the 'second operation.' Therefore, terminal #A of N302 transmits information instructing device #B of N303 to perform the second operation." The linkage is assumed to be as described above.

[0374] After that, it is assumed that terminal #A of N302 performs sensing periodically, regularly, or irregularly.

[0375] Then, the first person N301 wants to request the device #B of N303 to perform a "second action," so it is assumed that the first person N301 "performs a second gesture (N512)." Note that although the person who performs the second gesture is the first person N301, another person may perform the second gesture.

[0376] Then, the terminal N0302 recognizes the second gesture by sensing (N513) and sends information instructing the device #B of N303 to perform the "second action" (N514).

[0377] Then, device #B of N303 will perform the second operation (N515).

[0378] By doing this, it becomes possible to easily give operation instructions to device #B, and there is an advantage that a person does not need to have a special device.

[0379] The operation of each device in Fig. 47 will be described using Fig. 49. In Fig. 49, the same numbers are used for components that operate in the same way as in Fig. 48.

[0380] First, terminal #A of N302 "decides to register a gesture (N501)." Therefore, terminal #A of N302 transmits a sensing signal.

[0381] In response to this, the first person N301 "performs a movement to register in terminal #A of N302 (N502)." This gesture will be called the "second gesture."

[0382] Then, terminal #A of N302 "registers the second gesture (N503)." As mentioned above, terminal #A of N302 may have a function to check whether the gesture has been registered correctly. Terminal #A of N302 may also have a function to edit the registered gesture. These functions may be used, for example, by the first person N301 to correctly register the second gesture in terminal A of N302.

[0383] Next, the registered gesture (e.g., the second gesture) is linked (N611) with the operation of terminal #A of N302. As an example, the following linkage is assumed: When a person, including the first person N301, performs the second gesture, terminal #A of N302 instructs device #B of N303 to perform a "third operation." To achieve this, terminal #A of N302 transmits information instructing the AP of N401 to perform the third operation. Then, the AP of N401 transmits a modulated signal including this information (instructing the AP to perform the third operation) to device #B of N303.

[0384] After that, it is assumed that terminal #A of N302 performs sensing periodically, regularly, or irregularly.

[0385] Then, the first person N301 wants to request the device #B of N303 to perform a "third action," so it is assumed that the first person N301 "performs a second gesture (N612)." Note that although the person who performs the second gesture is the first person N301, another person may perform the second gesture.

[0386] Then, terminal #A of N302 recognizes the second gesture by sensing (N613) and transmits information instructing the AP of N401 to perform the "third action" (N614).

[0387] Then, the AP of N401 transmits a modulated signal including this information (an instruction to perform the third operation) to device #B of N303 (N615).

[0388] As a result, device #B in N303 will perform the third operation (N616).

[0389] 47, the AP of N401 may communicate with the network N402. For example, the AP of N401 may communicate with a cloud server via the network N402. The AP of N401 may then receive instructions from the cloud server.

[0390] For example, the cloud server may obtain information about sensing from terminal #A of N302, device #B of N303, AP of N401, and the like.

[0391] In this case, the cloud server itself may grasp the registered gesture details and perform the gesture recognition calculations. However, terminal #A of N302 must upload this basic information to the cloud server. The cloud server may also grasp the details associated with the gestures and issue instructions to terminal #A of N302, device #B of N303, and the AP of N401 based on this information.

[0392] This has the effect of allowing a person to easily issue operational instructions to device #B, which allows the person to issue instructions to the device without having to hold a special device.

[0393] In this description, the term "gesture" is used, but instead of gesture, the term "personal movement," "shape of a part of a human body," "personal movement," "personal part detection," "personal authentication," "personal part authentication," "object movement," "object shape," "object detection," "object authentication," etc. may also be used in this description.

[0394] The sensing method may be the example described in this specification or may be another method.

[0395] (Embodiment 8) In this embodiment, a specific example of sensing in a space where a device capable of performing sensing exists will be described.

[0396] FIG. 50 shows an example of the state of each device in this embodiment. N700 represents a space called a home. As shown in FIG. 50, for example, home N700 may include an AP (access point) N701, audio N702, device N703, which is a terminal such as a "smartphone, a smart speaker, a tablet, a computer, or a mobile phone," and lighting fixture N704. Hereinafter, device N703 will be referred to as device #C.

[0397] Additionally, N705 people are living in the house N700.

[0398] As explained in the other embodiments, the AP of N701 is capable of sensing and communication.

[0399] The AP of N701 communicates with, for example, audio N702, device #C of N703, lighting N704, etc. Note that the AP of N701 may also communicate with devices other than these.

[0400] It is also assumed that the AP of N701 communicates with a (cloud) server N720 via a network N710.

[0401] Furthermore, it is assumed that the AP of N701 is communicating with a base station N730 via a network N710.

[0402] It is assumed that base station N730 is communicating with device N731, which is a terminal such as a smartphone, tablet, computer, or mobile phone. Device N731 will be referred to as device #D hereinafter.

[0403] Next, a detailed example of the operation of the AP of N701 in FIG. 50 will be described.

[0404] FIG. 51 shows a flowchart of an example of operation when an N701 AP is installed in a home.

[0405] First, the AP of N701 performs sensing (N801) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices (naturally including device #C of N703), information about the shape of electronic devices, etc. to the cloud server (N802).

[0406] As a result, the AP of N701 completes initial sensing (N803). Although it is described as initial sensing, after the AP of N701 is installed, the AP of N701 may perform the operation of Fig. 51 periodically, irregularly, regularly, or irregularly.

[0407] As another example, assume that Figure 51 shows the operation of device #C of N703.

[0408] First, device #C of N703 performs sensing (N801) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices, etc., to the cloud server via the AP of N701 (N802).

[0409] As a result, device #C of N703 ends initial sensing (N803). Although it is described as initial sensing, device #C of N703 may perform the operation of Fig. 51 periodically, irregularly, regularly, or irregularly.

[0410] FIG. 52 shows a flowchart of an example of the operation of the AP of N701.

[0411] The AP of N701 uploads information about devices in the home obtained through sensing, such as information about home appliances (naturally including audio N702 and lighting N704) and electronic devices (naturally including device #C of N703) to a cloud server (N901).

[0412] Then, information relating to the operation of the devices in the home and the cooperation of the sensing performed by the AP of N701 is uploaded to the cloud server (N902). Note that examples of cooperation between the operation of the devices in the home and the sensing of devices capable of sensing (here, the AP of N701), and registration of the cooperation are explained in the seventh embodiment. Below, an example of cooperation operation, in particular, which is different from the seventh embodiment, will be explained.

[0413] As another example, assume that Figure 52 shows the operation of device #C of N703.

[0414] Device #C of N703 uploads information about devices in the home obtained by sensing, such as information about home appliances (naturally including audio N702 and lighting N704) and electronic devices, to a cloud server via the AP of N701 (N901).

[0415] Then, information relating to the operation of the devices in the home and the cooperation of the sensing performed by device #C of N703 is uploaded to the cloud server (N902). Note that examples of cooperation between the operation of the devices in the home and the sensing of a device capable of sensing (device #C of N703 in this case), and the registration of cooperation are explained in embodiment 7. Below, we will explain examples of cooperation operations, particularly those that differ from embodiment 7.

[0416] Fig. 53 is a flowchart showing an example of the operation of the system. Below, examples of cooperation between audio N702 and sensing, cooperation between device #C and sensing, and cooperation between lighting N704 and sensing will be described using Fig. 53.

[0417] The AP of N701 performs sensing (N1001).

[0418] As a result of the sensing, the AP of N701 checks whether it has detected any moving objects, including people, inside the home (N1002).

[0419] If the AP of N701 does not detect any moving objects, including people, in the home (N1002 NO), it proceeds to "Perform sensing N1001".

[0420] When the AP of N701 detects a moving object, including a person, inside the home (N1002 YES), the AP of N701 uploads information about the situation inside the home to the cloud server N720 via the network N710 (N1003).

[0421] Accordingly, cloud server N720 transmits to AP N701 information relating to control of audio N702, information relating to control of device #C of N703, or information relating to control of lighting N704. As a result, AP 701 of N701 obtains information relating to control (control information) (N1004 YES).

[0422] Then, the AP of N701 transmits control information to the target device (in the case of FIG. 50, audio N702, or device #C of N703, or lighting N704) (N1005).

[0423] In response to this, the target device performs control based on this control information and ends the control (N1006).

[0424] Then, the AP of N701 is performing the next sensing (N1001).

[0425] On the other hand, if the AP of N701 detects a moving object, including a person, inside the home (N1002 YES), the AP of N701 uploads information about the situation inside the home to the cloud server N720 via the network N710 (N1003), but the AP of N701 does not obtain control information from the cloud server N720 (N1004 NO). In this case, the AP of N701 is performing the following sensing (N1001).

[0426] For example, consider a case in N1005 where the AP of N701 transmits control information to the audio N702. In this case, based on the information about the position of a person obtained by sensing by the AP of N701, the cloud server N720 transmits information about the sound / audio directionality control of the audio N702 to the AP of N701. Then, the AP of N701 transmits the information about the sound / audio directionality control to the audio N702, and the audio N702 performs sound / audio directionality control based on the information about the sound / audio directionality control.

[0427] As another example, consider a case in N1005 where the AP of N701 transmits control information to lighting N704. In this case, based on the information on the positions of people and moving objects obtained by sensing by the AP of N701, the cloud server N720 transmits information regarding "ON / OFF or light irradiation direction control" of lighting N704 to the AP of N701. The AP of N701 then transmits the information regarding "ON / OFF or light irradiation direction control" to lighting N704, and lighting N704 turns the lighting ON / OFF or controls the lighting direction based on the information regarding "ON / OFF or light irradiation direction control."

[0428] As described above, by controlling the devices present in the home based on the state of the home, it is possible to obtain the effect of providing a comfortable and safe lifestyle.

[0429] Another example of FIG. 53 will be described.

[0430] Device #C of N703 performs sensing (N1001).

[0431] As a result of the sensing, device #C of N703 checks whether it has detected any moving objects, including people, inside the home (N1002).

[0432] If device #C of N703 does not detect any moving objects, including people, in the home (N1002 NO), it proceeds to "Perform sensing N1001."

[0433] If device #C of N703 detects a moving object, including a person, inside the home (N1002 YES), device #C of N703 uploads information about the situation inside the home to cloud server N720 via N701's AP and network N710 (N1003).

[0434] Accordingly, cloud server N720 transmits information regarding the control of audio N702 or information regarding the control of lighting N704 to device #C of N703 via the APs of networks N710 and N701. As a result, device #C of N703 receives the information regarding the control (control information) (N1004 YES).

[0435] Then, device #C of N703 transmits control information to the target device (audio device N702 or lighting device N704 in the case of FIG. 50) (N1005), however, via the AP of N701.

[0436] In response to this, the target device performs control based on this control information and ends the control (N1006).

[0437] Then, device #C of N703 is performing the next sensing (N1001).

[0438] On the other hand, if device #C of N703 detects a moving object, including a person, inside the home (N1002 YES), device #C of N703 uploads information about the situation inside the home to cloud server N720 via the AP of N701 and network N710 (N1003), but device #C of N703 does not obtain control information from cloud server N720 (N1004 NO). In this case, device #C of N703 is performing the following sensing (N1001).

[0439] For example, consider the case where device #C of N703 in N1005 transmits control information to audio device N702. In this case, based on information about the person's position obtained by sensing device #C of N703, cloud server N720 transmits information about sound / audio directionality control for audio device N702 to device #C of N703 via the AP of N701. Device #C of N703 then transmits the information about sound / audio directionality control to audio device N702 via the AP of N701, and audio device N702 controls the sound / audio directionality based on the information about sound / audio directionality control.

[0440] As another example, consider the case where device #C of N703 in N1005 transmits control information to lighting N704. In this case, based on the information about the positions of people and moving objects obtained by device #C of N703 through sensing, cloud server N720 transmits information about "ON / OFF or light irradiation direction control" of lighting N704 to device #C of N703 via the AP of N701. Then, device #C of N703 transmits information about "ON / OFF or light irradiation direction control" to lighting N704 via the AP of N701, and lighting N704 turns the lighting on or off or controls the lighting direction based on the information about "ON / OFF or light irradiation direction control."

[0441] As described above, by controlling the devices present in the home based on the state of the home, it is possible to obtain the effect of providing a comfortable and safe lifestyle.

[0442] FIG. 54 shows a flowchart of an example of the operation of the AP N701 and the cloud server N720 in the home.

[0443] The AP of N701 performs sensing (N1101) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices (naturally including device #C of N703), information about the shape of electronic devices, etc. to the cloud server (N1102).

[0444] The N720 cloud server compares the stored information about the home situation with the newly acquired information about the home situation, and then checks whether a new situation has been detected (N1103).

[0445] When the cloud server of N720 confirms the new status (N1103 YES), it updates the information about the situation inside the home (N1104), and then performs the next sensing operation on the AP of N701 (N1101).

[0446] If the cloud server of N720 cannot confirm the new state (N1103 NO), it next performs sensing of the AP of N701 (N1101).

[0447] As another example, assume that FIG. 54 shows the operation of device #C of N703 and cloud server of N720.

[0448] Device #C of N703 performs sensing (N1101) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices, etc., to the cloud server via the AP of N701 (N1102).

[0449] The N720 cloud server compares the stored information about the home situation with the newly acquired information about the home situation, and then checks whether a new situation has been detected (N1103).

[0450] When the cloud server of N720 confirms the new status (N1103 YES), it updates the information about the situation inside the home (N1104), and then performs the next sensing of device #C of N703 (N1101).

[0451] If the cloud server of N720 cannot confirm the new status (N1103 NO), it next performs sensing of device #C of N703 (N1101).

[0452] As described above, by controlling the devices present in the home based on the condition of the home, it is possible to provide a comfortable and safe lifestyle, and by updating the information obtained through sensing as appropriate, it is possible to achieve more optimal control.

[0453] Note that operations involving the cloud server are indicated by dotted lines in Fig. 53. Similarly, operations involving the cloud server are indicated by dotted lines in Fig. 54.

[0454] (Embodiment 9) In this embodiment, a specific example of sensing in a space where a device capable of performing sensing exists will be described.

[0455] Figure 55 shows an example of the state of each device in this embodiment. Note that the same numbers are used for components that operate in the same way as in Figure 50.

[0456] N700 represents a space, for example, a home. As shown in FIG. 55, for example, home N700 may include an AP (access point) N701, audio N702, device N703, which may be a terminal such as a "smartphone, a smart speaker, a tablet, a computer, or a mobile phone," and lighting fixture N704. Hereinafter, device N703 will be referred to as device #C. For example, home N700 may include server Q101.

[0457] Additionally, N705 people are living in the house N700.

[0458] As explained in the other embodiments, the AP of N701 is capable of sensing and communication.

[0459] The AP of N701 communicates with, for example, audio N702, device #C of N703, lighting N704, etc. Note that the AP of N701 may also communicate with devices other than these.

[0460] It is also assumed that the AP of N701 is communicating with a server Q101 via a network Q102.

[0461] Furthermore, it is assumed that the AP of N701 is communicating with a base station N730 via a network N710.

[0462] It is assumed that base station N730 is communicating with device N731, which is a terminal such as a smartphone, tablet, computer, or mobile phone. Device N731 will be referred to as device #D hereinafter.

[0463] Next, a detailed example of the operation of the AP of N701 in FIG. 55 will be described.

[0464] FIG. 56 shows a flowchart of an example of operation when an N701 AP is installed in a home.

[0465] First, the AP of N701 performs sensing (Q201) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices (naturally including device #C of N703), information about the shape of electronic devices, etc. to server Q101 (Q202).

[0466] This completes the initial sensing of the AP of N701 (Q203). Note that although it is described as initial sensing, after the AP of N701 is installed, the AP of N701 may perform the operation of Fig. 56 periodically, irregularly, regularly, or irregularly.

[0467] As another example, assume that Figure 56 shows the operation of device #C of N703.

[0468] First, device #C of N703 performs sensing (Q201) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices, etc., to server Q101 via the AP of N701 (Q202).

[0469] As a result, device #C of N703 ends initial sensing (Q203). Although it is described as initial sensing, device #C of N703 may perform the operation of Fig. 56 periodically, irregularly, regularly, or irregularly.

[0470] FIG. 57 shows a flowchart of an example of the operation of the AP of N701.

[0471] The AP of N701 uploads information about devices in the home obtained by sensing, such as information about home appliances (naturally including audio N702 and lighting N704) and electronic devices (naturally including device #C of N703) to server Q101 (Q301).

[0472] Then, information regarding the operation of the devices in the home and the cooperation of the sensing performed by the AP of N701 is uploaded to the server Q101 (Q302). Note that examples of cooperation between the operation of the devices in the home and the sensing of the device capable of sensing (here, the AP of N701), and the registration of the cooperation are explained in the seventh embodiment. Below, we will explain examples of cooperation operations, particularly those that differ from the seventh embodiment.

[0473] As another example, assume that Figure 57 shows the operation of device #C of N703.

[0474] Device #C of N703 uploads information about devices in the home obtained by sensing, such as information about home appliances (naturally including audio N702 and lighting N704) and electronic devices, to server Q101 via the AP of N701 (Q301).

[0475] Then, server Q101 uploads information about the operation of the devices in the home and the cooperation of sensing performed by device #C of N703 to the cloud server (Q302). Note that examples of cooperation between the operation of the devices in the home and the sensing of a device capable of sensing (device #C of N703 in this case), and the registration of cooperation, are explained in embodiment 7. Below, we will explain examples of cooperation operations, particularly those that differ from embodiment 7.

[0476] Fig. 58 shows a flowchart of an example of operation related to the system series of Fig. 55. Below, examples of cooperation between audio N702 and sensing, cooperation between device #C and sensing, and cooperation between lighting N704 and sensing will be described using Fig. 58. Note that in Fig. 58, operations related to the server are indicated by dotted lines.

[0477] The AP of N701 performs sensing (Q401).

[0478] As a result of the sensing, the AP of N701 checks whether it has detected any moving objects, including people, inside the home (Q402).

[0479] If the AP of N701 has not detected any moving objects, including people, in the home (Q402 NO), it proceeds to "perform sensing Q401."

[0480] When the AP of N701 detects a moving object, including a person, inside the home (Q402 YES), the AP of N701 uploads information about the situation inside the home to the server Q101 via the network Q102 (Q403).

[0481] Accordingly, server Q101 transmits to AP N701 information relating to control of audio N702, information relating to control of device #C of N703, or information relating to control of lighting N704. As a result, AP 701 of N701 obtains information relating to control (control information) (Q404 YES).

[0482] Then, the AP of N701 transmits control information to the target device (in the case of FIG. 55, audio N702, or device #C of N703, or lighting N704) (Q405).

[0483] Accordingly, the target device performs control based on this control information and ends the control (Q406).

[0484] Then, the AP of N701 is performing the following sensing (Q401).

[0485] On the other hand, if the AP of N701 detects a moving object, including a person, inside the home (Q402 YES), the AP of N701 uploads information about the situation inside the home to server Q101 via network Q102 (Q403), but the AP of N701 does not receive control information from server Q101 (Q404 NO). In this case, the AP of N701 is performing the following sensing (Q401).

[0486] For example, consider the case where the AP of N701 transmits control information to the audio N702 in Q405. In this case, based on the information about the person's position obtained by the AP of N701 through sensing, the server Q101 transmits information about the sound / audio directionality control of the audio N702 to the AP of N701. Then, the AP of N701 transmits the information about the sound / audio directionality control to the audio N702, and the audio N702 performs sound / audio directionality control based on the information about the sound / audio directionality control.

[0487] As another example, consider the case where the AP of N701 transmits control information to lighting N704 in Q405. In this case, based on the information about the positions of people and moving objects obtained by the AP of N701 through sensing, server Q101 transmits information about "ON / OFF or light irradiation direction control" of lighting N704 to the AP of N701. Then, the AP of N701 transmits the information about "ON / OFF or light irradiation direction control" to lighting N704, and lighting N704 turns the lighting ON / OFF or controls the lighting direction based on the information about "ON / OFF or light irradiation direction control."

[0488] As described above, by controlling the devices present in the home based on the state of the home, it is possible to obtain the effect of providing a comfortable and safe lifestyle.

[0489] Another example of FIG. 58 will be described.

[0490] Device #C of N703 performs sensing (Q401).

[0491] As a result of the sensing, device #C of N703 checks whether it has detected any moving objects, including people, inside the home (Q402).

[0492] If device #C of N703 does not detect any moving objects, including people, in the home (Q402 NO), it proceeds to "Sensing execution Q401".

[0493] If device #C of N703 detects a moving object, including a person, inside the home (Q402 YES), device #C of N703 uploads information about the situation inside the home to server Q101 via the AP of N701 and network Q102 (Q403).

[0494] Accordingly, server Q101 transmits information regarding the control of audio N702 or information regarding the control of lighting N704 to device #C of N703 via network Q102 and the AP of N701. As a result, device #C of N703 receives the information regarding the control (control information) (Q404 YES).

[0495] Then, device #C of N703 transmits control information to the target device (audio device N702 or lighting device N704 in the case of FIG. 55) (Q405), but via the AP of N701.

[0496] Accordingly, the target device performs control based on this control information and ends the control (Q406).

[0497] Then, device #C of N703 is performing the following sensing (Q401).

[0498] On the other hand, if device #C of N703 detects a moving object, including a person, inside the home (Q402 YES), device #C of N703 uploads information about the situation inside the home to server Q101 via the AP of N701 and network Q102 (Q403), but device #C of N703 does not receive control information from server Q101 (Q404 NO). In this case, device #C of N703 is performing the following sensing (Q401).

[0499] For example, consider the case in Q405 where device #C of N703 transmits control information to audio device N702. In this case, based on the information about the person's position obtained by device #C of N703 through sensing, server Q101 transmits information about the sound / audio directionality control of audio device N702 to device #C of N703 via the AP of N701. Device #C of N703 then transmits the information about the sound / audio directionality control to audio device N702 via the AP of N701, and audio device N702 performs sound / audio directionality control based on the information about the sound / audio directionality control.

[0500] As another example, consider the case where device #C of N703 transmits control information to lighting N704 in Q405. In this case, based on the information about the positions of people and moving objects obtained by device #C of N703 through sensing, server Q101 transmits information about "ON / OFF or light irradiation direction control" of lighting N704 to device #C of N703 via the AP of N701. Then, device #C of N703 transmits information about "ON / OFF or light irradiation direction control" to lighting N704 via the AP of N701, and lighting N704 turns the lighting on or off or controls the lighting direction based on the information about "ON / OFF or light irradiation direction control."

[0501] As described above, by controlling the devices present in the home based on the state of the home, it is possible to obtain the effect of providing a comfortable and safe lifestyle.

[0502] Fig. 59 shows a flowchart of an example of the operation of the AP of N701 and the server Q 101 in the home. Note that in Fig. 59, operations related to the server are indicated by dotted lines.

[0503] The AP of N701 performs sensing (Q501) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices (naturally including device #C of N703), information about the shape of electronic devices, etc. to server Q101 (Q502).

[0504] The server Q101 compares the stored information about the home status with the newly acquired information about the home status, and then the server Q101 checks whether a new status has been detected (Q503).

[0505] When the server Q101 confirms the new status (Q503 YES), it updates the information about the situation inside the home (Q504), and then performs the next sensing operation on the AP of N701 (Q501).

[0506] If the server Q101 cannot confirm the new state (Q503 NO), it next performs sensing of the AP of N701 (Q501).

[0507] As another example, assume that FIG. 59 shows the operation of device #C and server Q101 of N703.

[0508] Device #C of N703 performs sensing (Q501) and uploads information about the situation inside the home, such as information about the number of rooms, information about the shapes of the rooms, information about what is installed, such as information about the location and shape of furniture, information about the location and shape of home appliances (naturally including audio N702 and lighting N704), information about the shape of home appliances, information about the location and shape of electronic devices, etc., to server Q101 via the AP of N701 (Q502).

[0509] The server Q101 compares the stored information about the home status with the newly acquired information about the home status, and then the server Q101 checks whether a new status has been detected (Q503).

[0510] When the server Q101 confirms the new status (Q503 YES), it updates the information about the situation inside the home (Q504), and then performs the next sensing of device #C of N703 (Q501).

[0511] If the server Q101 cannot confirm the new status (Q503 NO), it next performs sensing of device #C of N703 (Q501).

[0512] As described above, by controlling the devices present in the home based on the condition of the home, it is possible to provide a comfortable and safe lifestyle, and by updating the information obtained through sensing as appropriate, it is possible to achieve more optimal control.

[0513] 55, the AP N701, the network Q102, and the server Q101 may be configured as a single device. In this case, the network Q102 may be either wired or wireless, and therefore the AP N701 and the server Q101 are connected by wire or wirelessly within the device.

[0514] (Embodiment 10) In this embodiment, a specific example of how to use a device having at least a sensing function will be described.

[0515] Example 1: A character is generated based on the object obtained through sensing and displayed on the screen.

[0516] By using characters in an application, it is possible to realize diversity in the application, and to obtain such effects.

[0517] FIG. 60 is a diagram showing an example of the configuration of a system according to this embodiment.

[0518] In FIG. 60, the second device is assumed to be equipped with a monitor.

[0519] The first device has at least a sensing function, and performs sensing to, for example, capture characteristics of the first person to generate information and transmit the information to the second device.

[0520] The second device generates a first character to be displayed on the monitor of the second device from the information on the characteristics of the first person. The first character can then be displayed on the monitor of the second device. Note that customization of the first character, such as transformations, is also possible.

[0521] Alternatively, the first device may perform sensing, for example, to capture characteristics of the second object, generate information about the second character, and transmit the information to the second device.

[0522] The second device then displays the second character on a monitor of the second device based on the information about the second character. Note that the second character can also be customized by applying any transformations thereto.

[0523] The first device and the second device may be combined into one device.

[0524] FIG. 61 is a diagram showing an example of a system configuration according to this embodiment that is different from that shown in FIG.

[0525] In FIG. 61, the second device is assumed to be connectable to an external monitor.

[0526] The first device has at least a sensing function, and performs sensing to, for example, capture characteristics of the first person to generate information and transmit the information to the second device.

[0527] The second device generates a first character to be displayed on a monitor connected to the second device from the information on the characteristics of the first person. The first character can then be displayed on the monitor. Note that customization of the first character, such as transformations, is also possible.

[0528] Alternatively, the first device may perform sensing, for example, to capture characteristics of the second object, generate information about the second character, and transmit the information to the second device.

[0529] The second device then displays the second character on a monitor based on the information about the second character. Note that the second character can be customized by applying any transformations thereto.

[0530] Example 2: By using images (still images or videos) of an object obtained by a sensor that can capture images, such as a camera, and estimated information about the object obtained, for example, by wireless sensing, it is possible to reconstruct the three-dimensional space of the object.

[0531] FIG. 62 is a diagram showing an example of the configuration of a system according to this embodiment.

[0532] In FIG. 62, the third device is assumed to be equipped with a monitor.

[0533] The third device is equipped with a sensor capable of capturing images, such as a camera, and a wireless sensing unit.

[0534] The wireless sensing unit obtains three-dimensional spatial estimation information of the object.

[0535] By capturing an image using a sensor capable of capturing images, such as a camera, two-dimensional (or three-dimensional) image information and color information of an object can be obtained.

[0536] The 3D space estimation unit generates (colored) 3D space estimation information of the object from the "3D space estimation information of the object" and "2D (or 3D) image information and color information of the object" and displays it on the monitor.

[0537] Furthermore, since the estimated information of the (colored) three-dimensional space of an object is obtained as three-dimensional information, there is an advantage that when it is displayed on a monitor, the viewpoint for displaying the object can be freely changed.

[0538] FIG. 63 is a diagram showing an example of a system configuration according to this embodiment that is different from that shown in FIG.

[0539] In FIG. 63, the third device is assumed to be connectable to an external monitor.

[0540] The basic operation of each part is the same as that explained in FIG.

[0541] The sensing method in each embodiment will now be described in additional detail.

[0542] 64 and 65 are diagrams showing sensing methods in the respective embodiments.

[0543] FIG. 64 is a schematic diagram showing a space in three dimensions. As shown in FIG. 64, an object and a detection device exist in the space. The detection device senses the object wirelessly, for example, by radio waves. The object may have any shape.

[0544] Figure 65 is an example of a plane parallel to the xy plane in Figure 64 that passes through an object, and shows, for example, the path of radio waves transmitted by a detection device. Here, the radio waves that the detection device obtains from the object may be reflected waves that are waves that arrive at the object and are reflected by the object, or may be radio waves reflected by the object itself.

[0545] As shown in Figure 65, the detection device receives radio waves W1 that are reflected by the object itself and reach the detection device directly (called direct waves). The detection device also transmits radio waves, which are reflected by the wall to reach the object, and are then reflected by the object, reflected again by the wall, and reach the detection device, and the detection device receives radio waves W2, W3, and W4 (also called reflected waves).

[0546] 65 illustrates a single two-dimensional plane, the xy plane, cut out from three-dimensional space, but the same explanation as above can be applied to a two-dimensional plane cut out from three-dimensional space, so the detection device can detect the position and shape of an object using direct waves and reflected waves. In other words, the detection device can achieve the effect of being able to detect parts of an object that cannot be captured by a sensor that can capture images, such as a camera.

[0547] (Embodiment 11) In this embodiment, a specific example using sensing will be described.

[0548] 66 shows an example of a system configuration in this embodiment. A first device T101 communicates with a server (cloud server) T103 via a network T102. A second device T105 communicates with the server (cloud server) T103 via a network T104.

[0549] For example, the second device T105 provides (uploads) an application (software) created (developed) by a certain user to the server (cloud server) T103.

[0550] The server (cloud server) T103 is a server (cloud server) that stores applications (software) uploaded from devices including the second device. The server (cloud server) T103 may be configured from multiple devices, and the multiple devices may be connected via a network and distributed.

[0551] The server (cloud server) T103 is a device that provides applications to devices including the first device.

[0552] Examples of the first device include, but are not limited to, a mobile phone, a mobile phone, a smartphone, a tablet, a tablet PC (Personal Computer), a personal computer (however, it may have a monitor installed or may be able to connect to a monitor), a laptop computer, a television, a device connected to a monitor, a game console, a portable game console, AR (Augmented Reality) glasses, AR goggles, a monitor capable of displaying AR, a device connected to a monitor capable of displaying AR, VR (Virtual Reality) glasses, VR goggles, a monitor capable of displaying VR, a device connected to a monitor capable of displaying VR, MR (Mixed Reality) glasses, a monitor capable of displaying MR, a device connected to a monitor capable of displaying MR, a car navigation system, a head-mounted display, a device connected to a head-mounted display, a monitor, a device connected to a monitor, a projector, and a device connected to a projector.

[0553] Figure 67 shows a second example of a system configuration in this embodiment. In Figure 67, components that operate in the same way as in Figure 66 are given the same numbers. Figure 67 differs from Figure 66 in that a first device T101 communicates with a computer network T111.

[0554] Fig. 68 shows an example of the configuration of the first device in Fig. 66 and Fig. 67. A communication unit T201 communicates with other devices as indicated by T211 and T212. For example, the communication unit T201 communicates with a computer network T111 and a server (cloud server) T103. The communication unit T201 may also communicate with other devices.

[0555] The communication unit T201 is connected to the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, and the camera unit T206.

[0556] The storage unit T202 is connected to the communication unit T201, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, and the camera unit T206.

[0557] The monitor unit T203 is connected to the communication unit T201, the storage unit T202, the position estimation unit T204, the signal processing unit T205, and the camera unit T206.

[0558] The position estimation unit T204 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the signal processing unit T205, and the camera unit T206.

[0559] The signal processing unit T205 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, and the camera unit T206.

[0560] The camera unit (photographing unit) (image sensor unit) T206 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, and the signal processing unit T205.

[0561] Fig. 69 shows an example of a configuration of the first device in Fig. 66 and Fig. 67 that is different from that in Fig. 68. A communication unit T201 communicates with other devices as indicated by T211 and T212. For example, the communication unit T201 communicates with a computer network T111 and a server (cloud server) T103. The communication unit T201 may also communicate with other devices.

[0562] The communication unit T201 is connected to the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, the camera unit T206, and the sensing unit T207.

[0563] The storage unit T202 is connected to the communication unit T201, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, the camera unit T206, and the sensing unit T207.

[0564] The monitor unit T203 is connected to the communication unit T201, the storage unit T202, the position estimation unit T204, the signal processing unit T205, the camera unit T206, and the sensing unit T207.

[0565] The position estimation unit T204 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the signal processing unit T205, the camera unit T206, and the sensing unit T207.

[0566] The signal processing unit T205 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the camera unit T206, and the sensing unit T207.

[0567] The camera unit (photographing unit) (image sensor unit) T206 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, and the sensing unit T207.

[0568] The sensing unit T207 is connected to the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, and the camera unit T206.

[0569] As described in FIGS. 66 and 67, the first device T101 in FIGS. 66 and 67 downloads an application (software) from a (cloud) server T103 via a network T102.

[0570] A first device T101 having the configuration shown in either FIG. 68 or FIG. 69 stores downloaded applications (software) in a storage unit T202.

[0571] Therefore, the communication unit T201 provided in the first device T101 having the configuration of either Figure 68 or Figure 69 obtains an application (software) from the (cloud) server T103 via the network T102 and stores the application (software) in the memory unit T102.

[0572] 68 and 69, the signal processing unit T205 starts an application stored in the storage unit T202 based on, for example, a user instruction and executes the application. Note that the user instruction may be given by, for example, the user using the touch panel function of the monitor unit T203, or the user may give the instruction from an external device via the communication unit T201.

[0573] The signal processing unit T205 in Fig. 68 executes an application to access the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, and the camera unit T206 as necessary. Note that the configuration of the first device T101 is not limited to the configuration in Fig. 68, and may include other units accessible by the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, and the camera unit T206.

[0574] By executing an application, the signal processing unit T205 in Fig. 69 accesses the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the camera unit T206, and the sensing unit T207 as necessary. Note that the configuration of the first device T101 is not limited to the configuration in Fig. 69, and may include other units accessible by the communication unit T201, the storage unit T202, the monitor unit T203, the position estimation unit T204, the signal processing unit T205, the camera unit T206, and the sensing unit T207.

[0575] FIG. 70 shows an example of the flow of operations when the signal processing unit T205 of the first device T101 having the configuration shown in FIG. 68 or 69 executes an application (software).

[0576] First, the application (software) starts to run.

[0577] Next, check whether the application you are running has an expiration date (see T301).

[0578] If the answer is "No" (if the application is not valid), "the application is terminated" or "a prompt to update the application is made (for example, a message prompting the user to update the application is displayed on the monitor unit T203)."

[0579] If the answer is "YES" (if the application is enabled), then a check is made to see if "location information can be acquired" (see T302). (For example, if location information can be acquired by the location estimation unit T204, this means that location information can be acquired.)

[0580] If the answer is "No" (if location information cannot be obtained (for example, if the location estimation unit T204 is not operating or if location information cannot be obtained)), "the application will be terminated" or "a change will be made to enable location information acquisition (for example, a display will be displayed on the monitor unit T203 urging the user to enable location information acquisition)."

[0581] If the answer is "Yes" (when position information is available (for example, when the position estimation unit T204 is operating or when position information is available)), the process moves to the next operation.

[0582] The location information can be obtained, for example, by the following methods.

[0583] The location estimation unit T204 is equipped with a GPS (Global Positioning System), and obtains location information by operating the GPS.

[0584] The communication unit T201 obtains information from a base station of a cellular system, and the location estimation unit T204 uses this information to estimate the location and obtain location information.

[0585] The communication unit T201 obtains information from a wireless LAN access point, and the location estimation unit T204 uses this information to estimate the location and obtain location information.

[0586] The communication unit T201 obtains information from a base station or access point of an optical communication system using visible light or the like, and the location estimation unit T204 uses this information to estimate the location and obtain location information.

[0587] The position estimation unit T204 estimates the position using information obtained by the sensing unit T207 performing the sensing operation, and obtains position information.

[0588] However, the sensing operation of the sensing unit T207 has been described in detail in other embodiments of this specification, so a description thereof will be omitted. Sensing may be performed using radio waves or light such as visible light.

[0589] Furthermore, if the acquisition of location information is set to off while the signal processing unit T205 is executing an application (software) (when location information can no longer be acquired), the signal processing unit T205 may either "terminate the application" or "prompt the user to make changes so that location information can be acquired (for example, display a message on the monitor unit T203 urging the user to make changes so that location information can be acquired)."

[0590] FIG. 71 is a first example of the processing from FIG. 70 onward when the signal processing unit T205 of the first device T101 is executing an application, and FIG. 72 is a second example.

[0591] First example: As a first example, examples 1-1 and 1-2 will be described.

[0592] Example 1-1: As shown in FIG. 71, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0593] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0594] The signal processing unit T205 then performs analysis, such as detecting distinctive objects, from the "still image or video" obtained from the camera unit T206 (see T402), thereby obtaining "surrounding information based on the location information" (see T403).

[0595] The signal processing unit T205 uses the "location information," "still image or video," and "surrounding information based on the location information" to, for example, display "still image or video" around the location, as well as "display 'Advertisement, first character group, and game available,' or display 'Advertisement and first character group,' or display 'Advertisement and game available,' or display 'Advertisement'" on the monitor unit T203 (see T403).

[0596] It should be noted that the term "group of characters" refers to one or more characters.

[0597] Example 1-2: As shown in FIG. 71, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0598] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0599] The signal processing unit T205 then performs analysis, such as detecting distinctive objects, from the "still image or video" obtained from the camera unit T206 (see T402), thereby obtaining "surrounding information based on the location information" (see T403).

[0600] The signal processing unit T205 causes the monitor unit T203 to display "'Advertisement, first character group and game available', or display "Advertisement and first character group', or display "Advertisement and game available', or display "Advertisement"" based on the "location information," "still image or video," and "surrounding information based on the location information" (see T403).

[0601] In Example 1-1, the user sees the peripheral information in the form of "still images or videos," but in Example 1-2, the user sees the peripheral information in the form of real images. In other words, the user sees the "real image" and "a display of 'Advertisement, first character group, and game available,' or a display of 'Advertisement and first character group,' or a display of 'Advertisement and game available,' or a display of 'Advertisement.'"

[0602] Also, the term "group of characters" means one or more characters.

[0603] Second example: As a second example, examples 2-1, 2-2, and 2-3 will be described.

[0604] Example 2-1: As shown in FIG. 72, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0605] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0606] Then, the signal processing unit T205 obtains "periphery information based on the position information" from the position information (see T411).

[0607] The signal processing unit T205 uses the "location information," "still image or video," and "surrounding information based on the location information" to, for example, display on the monitor unit T203 "a still image or video" around the location, as well as "display 'Advertisement, first character group, and game available,' or display 'Advertisement and first character group,' or display 'Advertisement and game available,' or display 'Advertisement'" (see T411).

[0608] It should be noted that the term "group of characters" refers to one or more characters.

[0609] Example 2-2: As shown in FIG. 72, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0610] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0611] Then, the signal processing unit T205 obtains "periphery information based on the position information" from the position information (see T411).

[0612] The signal processing unit T205 causes the monitor unit T203 to display "'Advertisement, first character group and game available', or display "Advertisement and first character group', or display "Advertisement and game available', or display "Advertisement"" based on the "location information," "still image or video," and "surrounding information based on the location information" (see T411).

[0613] In Example 2-1, the user sees the peripheral information in the form of "still images or videos," but in Example 2-2, the user sees the peripheral information in the form of real images. In other words, the user sees the "real image" and "a display of 'Advertisement, first character group, and game available,' or a display of 'Advertisement and first character group,' or a display of 'Advertisement and game available,' or a display of 'Advertisement.'"

[0614] Also, the term "group of characters" means one or more characters.

[0615] Example 2-3: As shown in FIG. 72, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0616] Then, the signal processing unit T205 obtains "periphery information based on the position information" from the position information (see T411).

[0617] The signal processing unit T205 causes the monitor unit T203 to display "'Advertisement, first character group and game available', or display "Advertisement and first character group', or display "Advertisement and game available', or display "Advertisement"" based on the "location information" and "surrounding information based on the location information" (see T411).

[0618] In Example 2-1, the user sees the peripheral information in the form of "still images or videos," but in Example 2-2, the user sees the peripheral information in the form of real images. In other words, the user sees the "real image" and "a display of 'Advertisement, first character group, and game available,' or a display of 'Advertisement and first character group,' or a display of 'Advertisement and game available,' or a display of 'Advertisement.'"

[0619] Also, the term "group of characters" means one or more characters.

[0620] In this case, the camera unit T206 may not be present in FIGS.

[0621] A specific example of the display on the monitor unit T203 will be described below.

[0622] An example will be explained using FIG.

[0623] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0624] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0625] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0626] In this case, the application is assumed to be an application that "displays advertisements for store A," "displays information for playing game B," and "displays a group of characters (called character group C)" in the vicinity of "Tokyo Tower."

[0627] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area, such as Tokyo Tower, as well as "displaying advertisements for store A," "displaying information for playing game B," and "displaying a group of characters (group of characters C)" (see T503).

[0628] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A," "display for playing game B," and "display a group of characters (group of characters C)" (see T503), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can see it.

[0629] In the above, examples have been described in which "display of an advertisement for retailer A," "display for playing game B," and "display of a group of characters (group of characters C)" are displayed, but it is also possible to display at least one of "display of an advertisement for retailer A," "display for playing game B," and "display of a group of characters (group of characters C)."

[0630] Alternatively, the following may be done.

[0631] - Display may be set by location and characteristics.

[0632] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on information obtained by the position estimation unit T204 and / or information obtained by the camera unit T206.

[0633] In this case, the application is assumed to be an application that "displays advertisements for company D," "displays for playing game E," and "displays a group of characters (named character group F)" around the "Sky Tree."

[0634] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as the Skytree, as well as "displaying advertisements for company D," "displaying information for playing game E," and "displaying a group of characters (group of characters F)."

[0635] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company D," "display a display for playing game E," and "display a group of characters (group of characters F)," and these displays may be superimposed on real images of the surrounding area, such as the Skytree, so that the user can see them.

[0636] In the above, examples have been described in which "display of advertisements for company D," "display for playing E game," and "display of character group (character group F)" are displayed, but it is also possible to display at least one of "display of advertisements for company D," "display for playing E game," and "display of character group (character group F)."

[0637] The display may be different for each application.

[0638] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0639] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0640] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0641] In this case, this other application is assumed to be an application that "displays advertisements for Company G," "displays for playing H games," and "displays a group of characters (called character group I)" in the vicinity of "Tokyo Tower."

[0642] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, as well as "displaying advertisements for Company G," "displaying information for playing the H game," and "displaying a group of characters (group of characters I)."

[0643] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company G," "display for playing the H game," and "display a group of characters (group of characters I)," and these displays may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can view them.

[0644] In the above, examples have been described in which "display of advertisements for company G," "display for playing the H game," and "display of a group of characters (group of characters I)" are displayed, but it is also possible to display at least one of "display of advertisements for company G," "display for playing the H game," and "display of a group of characters (group of characters I)."

[0645] The display may be set according to time fluctuations.

[0646] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0647] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0648] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0649] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0650] In this case, the application is assumed to be an application that displays advertisements for company J, displays information for playing game K, and displays a group of characters (named character group L) around Tokyo Tower at time #2.

[0651] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, as well as ``displaying an advertisement for company J,'' ``displaying information for playing game K,'' and ``displaying a group of characters (group of characters L).''

[0652] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company J," "display for playing game K," and "display a group of characters (group of characters L)," and these displays may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can view them.

[0653] In the above, examples have been described in which "display of advertisements for company J," "display for playing K game," and "display of character group (character group L)" are performed, but it is also possible to display at least one of "display of advertisements for company J," "display for playing K game," and "display of character group (character group L)."

[0654] An example will be explained using FIG.

[0655] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0656] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0657] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0658] In this case, the application is assumed to be an application that "displays an advertisement for store A" and "displays a group of characters (called character group C)" in the vicinity of "Tokyo Tower."

[0659] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays an advertisement for store A" and "displays a group of characters (group of characters C)" (see T513).

[0660] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" and "display a group of characters (group of characters C)" (see T513), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can view it.

[0661] In the above, examples of "displaying an advertisement for Retailer A" and "displaying a group of characters (character group C)" have been described, but it is also possible to display at least one of "displaying an advertisement for Retailer A" and "displaying a group of characters (character group C)".

[0662] Alternatively, the following may be done.

[0663] - Display may be set by location and characteristics.

[0664] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on information obtained by the position estimation unit T204 and / or information obtained by the camera unit T206.

[0665] In this case, the application is assumed to be an application that "displays advertisements for company D" and "displays a group of characters (named character group F)" around the "Sky Tree."

[0666] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Sky Tree, and also "displays advertisements for company D" and "displays a group of characters (group of characters F)."

[0667] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company D" and "display a group of characters (group of characters F)", and this display may be superimposed on real images of the surrounding situation, such as the Skytree, so that the user can view it.

[0668] In the above, examples of "displaying an advertisement for company D" and "displaying a group of characters (character group F)" are described, but it is also possible to display at least one of "displaying an advertisement for company D" and "displaying a group of characters (character group F)".

[0669] The display may be different for each application.

[0670] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0671] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0672] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0673] In this case, this other application is assumed to be an application that "displays advertisements for company G" and "displays a group of characters (named character group I)" in the vicinity of "Tokyo Tower."

[0674] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays advertisements for company G" and "displays a group of characters (group of characters I)."

[0675] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for Company G" and "display a group of characters (group of characters I)", and this display may be superimposed on a real image of the surrounding situation, such as Tokyo Tower, so that the user can view it.

[0676] In the above, examples of "displaying an advertisement for company G" and "displaying a group of characters (character group I)" are described, but it is also possible to display at least one of "displaying an advertisement for company G" and "displaying a group of characters (character group I)".

[0677] The display may be set according to time fluctuations.

[0678] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0679] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0680] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0681] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0682] In this case, the application is assumed to be an application that "displays an advertisement for company J" and "displays a group of characters (named character group L)" around "Tokyo Tower" at time #2.

[0683] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays advertisements for company J" and "displays a group of characters (group of characters L)."

[0684] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for Company J" and "display a group of characters (group of characters L)", and this display may be superimposed on a real image of the surrounding situation, such as Tokyo Tower, so that the user can view it.

[0685] In the above, examples of "displaying an advertisement for company J" and "displaying a group of characters (character group L)" are described, but it is also possible to display at least one of "displaying an advertisement for company J" and "displaying a group of characters (character group L)".

[0686] An example will be explained using FIG.

[0687] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0688] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0689] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0690] In this case, the application is assumed to be an application that "displays advertisements for store A" and "displays for playing game B" in the vicinity of "Tokyo Tower."

[0691] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also displays an advertisement for store A and a display for playing game B (see T523).

[0692] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" and "display for playing game B" (see T523), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can see it.

[0693] Although the above describes an example of "displaying an advertisement for retailer A" and "displaying a display for playing game B," it is also possible to display at least one of "displaying an advertisement for retailer A" and "displaying a display for playing game B."

[0694] Alternatively, the following may be done.

[0695] - Display may be set by location and characteristics.

[0696] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on information obtained by the position estimation unit T204 and / or information obtained by the camera unit T206.

[0697] In this case, the application is assumed to be an application that "displays advertisements for company D" and "displays for playing game E" around the "Sky Tree."

[0698] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Skytree, and also performs "display of advertisements for company D" and "display for playing game E."

[0699] As another method, the monitor unit T203 in Figures 68 and 69 may display "advertisements for company D" and "displays for playing game E," and superimpose these displays on real images of the surrounding area, such as the Skytree, so that the user can see them.

[0700] In the above, an example was described in which "company D's advertisement display" and "display for playing E-game" were displayed, but it is also possible to display at least one of "company D's advertisement display" and "display for playing E-game."

[0701] The display may be different for each application.

[0702] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0703] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0704] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 analyzes the still image or video captured by the camera unit T206 and recognizes that one building is "Tokyo Tower."

[0705] In this case, this other application is assumed to be an application that "displays advertisements for Company G" and "displays for playing H game" in the vicinity of "Tokyo Tower." Note that, in this case, the signal processing unit T205 may perform processing using location information.

[0706] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also performs "display of advertisements for company G" and "display for playing the H game."

[0707] As another method, the monitor unit T203 in Figures 68 and 69 may display "advertisements for Company G" and "displays for playing the H game," and this display may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can see them.

[0708] In the above, an example of "displaying an advertisement for Company G" and "displaying for playing the H game" is described, but it is also possible to display at least one of "displaying an advertisement for Company G" and "displaying for playing the H game."

[0709] The display may be set according to time fluctuations.

[0710] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0711] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0712] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0713] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0714] In this case, the application is assumed to be an application that "displays an advertisement for company J" and "displays a display for playing game K" around "Tokyo Tower" at time #2.

[0715] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also performs "display of advertisements for company J" and "display for playing game K."

[0716] As another method, the monitor unit T203 in Figures 68 and 69 may display "an advertisement for Company J" and "a display for playing Game K," and the user may view these displays overlaid with real images of the surrounding area, such as Tokyo Tower.

[0717] In the above, an example was described in which "company J's advertisement display" and "display for playing K game" were displayed, but it is also possible to display at least one of "company J's advertisement display" and "display for playing K game."

[0718] An example will be explained using FIG.

[0719] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0720] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0721] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0722] In this case, the application is assumed to be an application that "displays advertisements for a retailer A" in the vicinity of "Tokyo Tower."

[0723] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for Dealer A" (see T533).

[0724] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" (see T533), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can view it.

[0725] Alternatively, the following may be done.

[0726] - Display may be set by location and characteristics.

[0727] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on information obtained by the position estimation unit T204 and / or information obtained by the camera unit T206.

[0728] In this case, the application is assumed to be an application that displays "advertisements for company D" around "Sky Tree."

[0729] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Skytree, and also "displays an advertisement for company D."

[0730] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company D" and superimpose this display on the actual image of the surrounding area such as the Sky Tree so that the user can view it.

[0731] The display may be different for each application.

[0732] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0733] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0734] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0735] In this case, it is assumed that this other application is an application for "displaying advertisements for company G" around "Tokyo Tower."

[0736] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for company G."

[0737] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company G" and superimpose this display on a real image of the surrounding area such as Tokyo Tower so that the user can view it.

[0738] The display may be set according to time fluctuations.

[0739] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0740] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0741] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0742] 68 and 69 analyzes the still or video images captured by the camera unit T206 and recognizes that one building is "Tokyo Tower." Note that, at this time, the signal processing unit T205 may perform processing using location information.

[0743] In this case, the application is assumed to be an application that "displays advertisements for Company J" around "Tokyo Tower" at time #2.

[0744] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for Company J."

[0745] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company J" and superimpose this display on a real image of the surrounding area such as Tokyo Tower so that the user can view it.

[0746] An example will be explained using FIG.

[0747] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0748] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0749] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0750] In this case, the application is assumed to be an application that "displays advertisements for store A," "displays information for playing game B," and "displays a group of characters (called character group C)" in the vicinity of "Tokyo Tower."

[0751] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area, such as Tokyo Tower, as well as "displaying advertisements for store A," "displaying information for playing game B," and "displaying a group of characters (group of characters C)" (see T603).

[0752] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A," "display for playing game B," and "display a group of characters (group of characters C)" (see T603), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can see it.

[0753] In the above, examples have been described in which "display of an advertisement for retailer A," "display for playing game B," and "display of a group of characters (group of characters C)" are displayed, but it is also possible to display at least one of "display of an advertisement for retailer A," "display for playing game B," and "display of a group of characters (group of characters C)."

[0754] Alternatively, the following may be done.

[0755] - Display may be set by location and characteristics.

[0756] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on the information obtained by the location estimation unit T204.

[0757] In this case, the application is assumed to be an application that "displays advertisements for company D," "displays for playing game E," and "displays a group of characters (named character group F)" around the "Sky Tree."

[0758] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as the Skytree, as well as "displaying advertisements for company D," "displaying information for playing game E," and "displaying a group of characters (group of characters F)."

[0759] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company D," "display a display for playing game E," and "display a group of characters (group of characters F)," and these displays may be superimposed on real images of the surrounding area, such as the Skytree, so that the user can see them.

[0760] In the above, examples have been described in which "display of advertisements for company D," "display for playing E game," and "display of character group (character group F)" are displayed, but it is also possible to display at least one of "display of advertisements for company D," "display for playing E game," and "display of character group (character group F)."

[0761] The display may be different for each application.

[0762] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0763] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0764] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0765] In this case, this other application is assumed to be an application that "displays advertisements for Company G," "displays for playing H games," and "displays a group of characters (called character group I)" in the vicinity of "Tokyo Tower."

[0766] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, as well as "displaying advertisements for Company G," "displaying information for playing the H game," and "displaying a group of characters (group of characters I)."

[0767] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company G," "display for playing the H game," and "display a group of characters (group of characters I)," and these displays may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can view them.

[0768] In the above, examples have been described in which "display of advertisements for company G," "display for playing the H game," and "display of a group of characters (group of characters I)" are displayed, but it is also possible to display at least one of "display of advertisements for company G," "display for playing the H game," and "display of a group of characters (group of characters I)."

[0769] The display may be set according to time fluctuations.

[0770] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0771] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0772] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0773] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0774] In this case, the application is assumed to be an application that displays advertisements for company J, displays information for playing game K, and displays a group of characters (named character group L) around Tokyo Tower at time #2.

[0775] Therefore, the monitor unit T203 in Figures 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, as well as ``displaying an advertisement for company J,'' ``displaying information for playing game K,'' and ``displaying a group of characters (group of characters L).''

[0776] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company J," "display for playing game K," and "display a group of characters (group of characters L)," and these displays may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can view them.

[0777] In the above, examples have been described in which "display of advertisements for company J," "display for playing K game," and "display of character group (character group L)" are performed, but it is also possible to display at least one of "display of advertisements for company J," "display for playing K game," and "display of character group (character group L)."

[0778] An example will be explained using FIG.

[0779] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0780] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0781] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0782] In this case, the application is assumed to be an application that "displays an advertisement for store A" and "displays a group of characters (called character group C)" in the vicinity of "Tokyo Tower."

[0783] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays advertisements for store A" and "displays a group of characters (group of characters C)" (see T613).

[0784] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" and "display a group of characters (group of characters C)" (see T613), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can view it.

[0785] In the above, examples of "displaying an advertisement for Retailer A" and "displaying a group of characters (character group C)" have been described, but it is also possible to display at least one of "displaying an advertisement for Retailer A" and "displaying a group of characters (character group C)".

[0786] Alternatively, the following may be done.

[0787] - Display may be set by location and characteristics.

[0788] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on the information obtained by the location estimation unit T204.

[0789] In this case, the application is assumed to be an application that displays "advertisements for company D" and "character group (named character group F)" around "Sky Tree."

[0790] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Sky Tree, and also "displays advertisements for company D" and "displays a group of characters (group of characters F)."

[0791] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for company D" and "display a group of characters (group of characters F)", and this display may be superimposed on real images of the surrounding situation, such as the Skytree, so that the user can view it.

[0792] In the above, examples of "displaying an advertisement for company D" and "displaying a group of characters (character group F)" are described, but it is also possible to display at least one of "displaying an advertisement for company D" and "displaying a group of characters (character group F)".

[0793] The display may be different for each application.

[0794] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0795] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0796] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0797] In this case, this other application is assumed to be an application that "displays advertisements for company G" and "displays a group of characters (named character group I)" in the vicinity of "Tokyo Tower."

[0798] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays advertisements for company G" and "displays a group of characters (group of characters I)."

[0799] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for Company G" and "display a group of characters (group of characters I)", and this display may be superimposed on a real image of the surrounding situation, such as Tokyo Tower, so that the user can view it.

[0800] In the above, examples of "displaying an advertisement for company G" and "displaying a group of characters (character group I)" are described, but it is also possible to display at least one of "displaying an advertisement for company G" and "displaying a group of characters (character group I)".

[0801] The display may be set according to time fluctuations.

[0802] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0803] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0804] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0805] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0806] In this case, the application is assumed to be an application that "displays an advertisement for company J" and "displays a group of characters (named character group L)" around "Tokyo Tower" at time #2.

[0807] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also "displays advertisements for company J" and "displays a group of characters (group of characters L)."

[0808] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for Company J" and "display a group of characters (group of characters L)", and this display may be superimposed on a real image of the surrounding situation, such as Tokyo Tower, so that the user can view it.

[0809] In the above, examples of "displaying an advertisement for company J" and "displaying a group of characters (character group L)" are described, but it is also possible to display at least one of "displaying an advertisement for company J" and "displaying a group of characters (character group L)".

[0810] An example will be explained using FIG.

[0811] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0812] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0813] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0814] In this case, the application is assumed to be an application that "displays advertisements for store A" and "displays for playing game B" in the vicinity of "Tokyo Tower."

[0815] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also displays an advertisement for store A and a display for playing game B (see T623).

[0816] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" and "display for playing game B" (see T623), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can see it.

[0817] Although the above describes an example of "displaying an advertisement for retailer A" and "displaying a display for playing game B," it is also possible to display at least one of "displaying an advertisement for retailer A" and "displaying a display for playing game B."

[0818] Alternatively, the following may be done.

[0819] - Display may be set by location and characteristics.

[0820] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on the information obtained by the location estimation unit T204.

[0821] In this case, the application is assumed to be an application that "displays advertisements for company D" and "displays for playing game E" around the "Sky Tree."

[0822] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Skytree, and also performs "display of advertisements for company D" and "display for playing game E."

[0823] As another method, the monitor unit T203 in Figures 68 and 69 may display "advertisements for company D" and "displays for playing game E," and superimpose these displays on real images of the surrounding area, such as the Skytree, so that the user can see them.

[0824] In the above, an example was described in which "company D's advertisement display" and "display for playing E-game" were displayed, but it is also possible to display at least one of "company D's advertisement display" and "display for playing E-game."

[0825] The display may be different for each application.

[0826] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0827] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0828] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0829] In this case, this other application is assumed to be an application that "displays advertisements for Company G" and "displays for playing H game" in the vicinity of "Tokyo Tower." Note that, in this case, the signal processing unit T205 may perform processing using location information.

[0830] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also performs "display of advertisements for company G" and "display for playing the H game."

[0831] As another method, the monitor unit T203 in Figures 68 and 69 may display "advertisements for Company G" and "displays for playing the H game," and this display may be superimposed on real images of the surrounding area, such as Tokyo Tower, so that the user can see them.

[0832] In the above, an example of "displaying an advertisement for Company G" and "displaying for playing the H game" is described, but it is also possible to display at least one of "displaying an advertisement for Company G" and "displaying for playing the H game."

[0833] The display may be set according to time fluctuations.

[0834] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0835] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0836] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0837] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0838] In this case, the application is assumed to be an application that "displays an advertisement for company J" and "displays a display for playing game K" around "Tokyo Tower" at time #2.

[0839] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as Tokyo Tower, and also performs "display of advertisements for company J" and "display for playing game K."

[0840] As another method, the monitor unit T203 in Figures 68 and 69 may display "an advertisement for Company J" and "a display for playing Game K," and the user may view these displays overlaid with real images of the surrounding area, such as Tokyo Tower.

[0841] In the above, an example was described in which "company J's advertisement display" and "display for playing K game" were displayed, but it is also possible to display at least one of "company J's advertisement display" and "display for playing K game."

[0842] An example will be explained using FIG.

[0843] For example, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0844] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0845] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0846] In this case, the application is assumed to be an application that "displays advertisements for a retailer A" in the vicinity of "Tokyo Tower."

[0847] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for Dealer A" (see T633).

[0848] As another method, the monitor unit T203 in Figures 68 and 69 may "display an advertisement for retailer A" (see T633), and this display may be superimposed on a real image of the surrounding area, such as Tokyo Tower, so that the user can view it.

[0849] Alternatively, the following may be done.

[0850] - Display may be set by location and characteristics.

[0851] For example, it is assumed that the signal processing unit T205 recognizes that the location is near the "Sky Tree" based on the information obtained by the location estimation unit T204.

[0852] In this case, the application is assumed to be an application that "displays advertisements for company D" around the "Sky Tree."

[0853] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays images or videos of the surrounding area such as the Skytree, and also "displays an advertisement for company D."

[0854] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company D" and superimpose this display on the actual image of the surrounding area such as the Sky Tree so that the user can view it.

[0855] The display may be different for each application.

[0856] For example, suppose that a first device T101 having the configuration shown in Fig. 68 or 69 executes an application different from that described above. Then, for example, the location estimation unit T204 in Fig. 68 or 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0857] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0858] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0859] In this case, it is assumed that this other application is an application for "displaying advertisements for company G" around "Tokyo Tower."

[0860] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for company G."

[0861] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company G" and superimpose this display on a real image of the surrounding area such as Tokyo Tower so that the user can view it.

[0862] The display may be set according to time fluctuations.

[0863] For example, at time #1, the display of T503 in FIG. 73 is performed as described above.

[0864] Then, at time #2, similarly, the location estimation unit T204 in FIGS. 68 and 69 estimates that the location is around "4-2-8 Shibakoen, Minato-ku, Tokyo" (see T501).

[0865] It is assumed that the camera unit T206 in FIGS. 68 and 69 has acquired a still image or video as shown in T502.

[0866] Accordingly, the signal processing unit T205 in FIGS. 68 and 69 recognizes that the location is "near Tokyo Tower" based on the location information.

[0867] In this case, the application is assumed to be an application that "displays advertisements for Company J" around "Tokyo Tower" at time #2.

[0868] Therefore, the monitor unit T203 in FIGS. 68 and 69 displays an image or video of the surrounding area such as Tokyo Tower, and also "displays an advertisement for Company J."

[0869] As another method, the monitor unit T203 in FIGS. 68 and 69 may "display an advertisement for company J" and superimpose this display on a real image of the surrounding area such as Tokyo Tower so that the user can view it.

[0870] FIG. 81 is a third example of the processing from FIG. 70 onward when the signal processing unit T205 of the first device T101 is executing an application, and FIG. 82 is a fourth example.

[0871] Third example: As a third example, examples 3-1 and 3-2 will be described.

[0872] Example 3-1: As shown in FIG. 81, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0873] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0874] The signal processing unit T205 then performs analysis, such as detecting distinctive objects, from the "still image or video" obtained from the camera unit T206 (see T402), thereby obtaining "surrounding information based on the location information" (see T703).

[0875] The signal processing unit T205 uses the "location information," "still image or video," and "surrounding information based on the location information" to, for example, display on the monitor unit T203 "a still image or video" around the location, as well as "display 'Store, first character group, and game available,' or display 'Store and first character group,' or display 'Store and game available,' or display 'Store'" (see T703).

[0876] It should be noted that the term "group of characters" refers to one or more characters.

[0877] Example 3-2: As shown in FIG. 81, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0878] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0879] The signal processing unit T205 then performs analysis, such as detecting distinctive objects, from the "still image or video" obtained from the camera unit T206 (see T402), thereby obtaining "surrounding information based on the location information" (see T403).

[0880] The signal processing unit T205 displays "'Store, first character group, and game available,' or "Store and first character group,' or "Store and game available,' or "Store"" on the monitor unit T203 based on the "location information," "still image or video," and "surrounding information based on the location information" (see T703).

[0881] In Example 3-1, the user sees the surrounding information in the form of "still images or videos," but in Example 3-2, the user sees the surrounding information in the form of real images. In other words, the user sees the "real image" and a display of "store, first character group, and game available," or a display of "store and first character group," or a display of "store and game available," or a display of "store."

[0882] Also, the term "group of characters" means one or more characters.

[0883] Fourth example: As a fourth example, examples 4-1, 4-2, and 4-3 will be described.

[0884] Example 4-1: As shown in FIG. 82, after the processing in FIG. 70, first, the position estimation unit T204 in FIGS. 68 and 69 obtains position information, and this position information is obtained by the signal processing unit T205 (see T401).

[0885] Thereafter, the camera unit T206 in FIGS. 68 and 69 obtains "still images or moving images" (which may or may not be real-time).

[0886] Then, the signal processing unit T205 obtains "periphery information based on the position information" from the position information (see T711).

[0887] The signal processing unit T205 uses the "location information," "still image or video," and "surrounding information based on the location information" to, for example, display on the monitor unit T203 "a still image or video" around the location, as well as "a display of 'Store, ...

Claims

1. A management device comprising: a communication unit that communicates with a plurality of sensing devices arranged in a building; and a processor; The processor: receiving capability information from the plurality of sensing devices that indicates sensing capabilities of the sensing devices; Selecting a sensing device to be used to measure a predetermined area within the building; Estimating the presence or position of a target in the predetermined area based on the sensing results received from the sensing device; Management device.

2. The capability information includes information indicating whether the sensing can be performed. The management device according to claim 1 .

3. The capability information includes information indicating whether the sensing request can be accepted. The management device according to claim 1 .

4. The processor selects the sensing device to be used for measuring the predetermined area based on the capability information. The management device according to claim 1 .

5. The processor transmits request information indicating a sensing request to the selected sensing device, and receives sensing result information from the sensing device. The management device according to claim 1 .

6. The processor transmits triangulation request information and receives triangulation result related information when two or more of the sensing devices are used to measure the predetermined area, and estimates the position of the target by triangulation. The management device according to claim 5 .

7. The processor transmits a control information symbol including sensing method information and frame type information as at least a part of the sensing-related information before transmitting the request information. The management device according to claim 5 .

8. A management method implemented by a management device that communicates with a plurality of sensing devices arranged in a building, comprising: receiving capability information from the plurality of sensing devices that indicates sensing capabilities of the sensing devices; Selecting a sensing device to be used to measure a predetermined area within the building; Estimating the presence or position of a target in the predetermined area based on the sensing results received from the sensing device; Management method.

Images