Information processing device, detection sensor, information processing system, and information processing method

Abstract

This invention comprises: an image acquisition unit (102) that acquires, from a detection sensor (2, 2a) which is provided inside a vehicle and includes a camera (22) and a radio wave sensor (21), a captured image captured by the camera (22); a camera error detection unit (103) that detects a camera (22) installation error, on the basis of the captured image; a radio wave sensor error calculation unit (104) that calculates a radio wave sensor (21) installation error, on the basis of the camera (22) installation error detected by the camera error detection unit (103); and an output unit (106, 106a, 106b) that outputs error consideration information which is for converting moving body distribution information generated by the radio wave sensor (21) into moving body distribution information which takes into account the radio wave sensor (21) installation error calculated by the radio wave sensor error calculation unit (104).

Description

Information processing device, detection sensor, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device, a detection sensor, an information processing system, and an information processing method.

[0002] A conventional technology for monitoring vehicle occupants is known, which is a radio wave sensor installed in a vehicle cabin, which emits radio waves into the vehicle cabin and receives reflected waves from objects to detect the position and movement of a moving object in the vehicle cabin, based on information generated from the reflected waves received by the radio wave sensor. As a technology for correcting a deviation in the installation position or orientation of a camera installed in the vehicle cabin, a technology is known in which a marker or the like whose direction or distance from the camera is known is captured by the camera, and the camera's installation position or orientation is adjusted or the captured image is corrected based on the deviation between the position of the marker or the like in the captured image and the position of the marker or the like that should have been captured in the captured image (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2007-006175

[0004] Generally, when a radio wave sensor is installed in a vehicle cabin, it is fixed to the vehicle cabin via some structure, such as the ceiling or dashboard, between the sensor and the vehicle frame. Therefore, there is a possibility that the installation position of the radio wave sensor or the direction of the radio waves emitted by the radio wave sensor may be misaligned. Hereinafter, a misalignment in the installation position of the radio wave sensor or the direction of the radio waves emitted by the radio wave sensor is referred to as a "radio wave sensor installation error." As a result, there is a problem that the information output from the radio wave sensor may have been generated when the radio wave sensor had an installation error. If vehicle occupants are monitored based on information generated when the radio wave sensor had an installation error, the accuracy of the monitoring results may be reduced. Note that the technology typified by the technology disclosed in Patent Document 1 adjusts the installation position or orientation of the camera or corrects the captured image, but does not take into account the installation error of the radio wave sensor, and therefore remains unable to solve the above problem.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an information processing device that can provide information (hereinafter referred to as "error-taking information") that can be used as information obtained without any installation error of the radio wave sensor, generated based on the reflected waves received by a radio wave sensor that is installed in the vehicle cabin, emits radio waves into the vehicle cabin, and can detect the position and movement of moving objects in the vehicle cabin by receiving reflected waves of the emitted radio waves reflected by an object.

[0006] The information processing device according to the present disclosure includes an image acquisition unit that acquires an image captured by the camera from a detection sensor having a camera installed within the vehicle cabin that captures an image of a target area within the vehicle cabin where at least an occupant may be present, and a radio wave sensor that generates moving object distribution information that three-dimensionally represents the distribution of areas within the vehicle cabin where moving objects are present, based on radio waves emitted toward the target area within the vehicle cabin and reflected by objects within the vehicle cabin; a camera error detection unit that detects an installation error of the camera based on the image acquired by the image acquisition unit; a radio wave sensor error calculation unit that calculates an installation error of the radio wave sensor based on the camera installation error detected by the camera error detection unit; and an output unit that outputs error consideration information to convert the moving object distribution information generated by the radio wave sensor into moving object distribution information that takes into account the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit.

[0007] According to the present disclosure, it is possible to provide error-taking information that allows information generated based on the reflected waves received by a radio wave sensor that is installed within the vehicle cabin, emits radio waves into the vehicle cabin, and is capable of detecting the position and movement of moving objects within the vehicle cabin by receiving reflected waves of the emitted radio waves reflected by an object, to be used as information obtained without any installation error of the radio wave sensor.

[0008] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to a first embodiment. FIG. 2 is a diagram for explaining a configuration example of a detection sensor according to the first embodiment. FIG. 3 is a diagram for explaining other configuration examples of the detection sensor according to the first embodiment. FIG. 4 is a configuration diagram showing a radio wave sensor mounted on a detection sensor in the first embodiment. FIG. 5 is a block diagram showing the function of a radio wave sensor mounted on a vehicle in the first embodiment. FIG. 6 is an explanatory diagram showing an example of an FM signal Tx(k) generated by a high frequency signal generating circuit. FIG. 7 is an explanatory diagram showing an example of an FM transmission wave and an FM reception wave. FIG. 8 is an explanatory diagram showing a sweep time T of an FM transmission wave and a frequency difference f ｄ and FIG. 10C are diagrams illustrating an example of a moving object map generated by the radio wave sensor in embodiment 1. FIG. 12A and FIG. 12B are diagrams illustrating an example of an influence on a monitoring device that monitors occupants using a moving object map generated by the radio wave sensor, assuming that a deviation occurs in the position and angle of a moving object detected by the radio wave sensor. FIG. 13A and FIG. 13B are diagrams illustrating an example of a hardware configuration of an information processing device according to embodiment 1. FIG. 13B is a diagram illustrating an example of a configuration of an information processing device according to embodiment 2. FIG. 13B is a flowchart illustrating the operation of an information processing device according to embodiment 2. FIG. 13C is a diagram illustrating an example of a configuration of an information processing device according to embodiment 3. FIG. 13C is a flowchart illustrating the operation of an information processing device according to embodiment 3.

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1. An information processing device according to embodiment 1 provides information (hereinafter referred to as "error-considered information") that a monitoring device for monitoring occupants in a vehicle cabin uses, based on information (hereinafter referred to as "moving object distribution information") that is generated by a radio wave sensor installed in the vehicle cabin and that represents in three dimensions the distribution of areas where moving objects exist within the vehicle cabin, as information obtained without any installation error of the radio wave sensor. The radio wave sensor detects moving objects based on waves reflected by objects within the vehicle cabin after emitting radio waves toward an area within the vehicle cabin where at least an occupant may be present (hereinafter referred to as the "target area"), and generates the moving object distribution information. In the following description, "radio wave sensor installation error" refers to a deviation in the installation position of the radio wave sensor in the real space within the vehicle cabin (installation position error) or a deviation in the direction of the radio waves emitted by the radio wave sensor in the real space within the vehicle cabin (radio wave direction error). More specifically, "radio wave sensor installation error" refers to an error in the installation position of the radio wave sensor relative to a predetermined installation position, or an error in the direction of the radio waves emitted by the radio wave sensor relative to a predetermined direction. Note that the "error" referred to here does not include errors that are not problematic in practice, but refers to errors that affect a monitoring device that performs processing using moving object distribution information generated by the radio wave sensor. In the following description, when referring to the "direction of the radio wave sensor," this "direction of the radio wave sensor" refers to the "direction of the radio waves emitted by the radio wave sensor."

[0011] In the first embodiment, the error-considered information is, in detail, moving object distribution information after correcting the moving object distribution information generated by the radio wave sensor so that it becomes moving object distribution information obtained without any installation error of the radio wave sensor (hereinafter referred to as "corrected moving object distribution information"). The information processing device according to the first embodiment generates the corrected moving object distribution information and outputs it to the monitoring device. In the first embodiment, the moving object distribution information is a so-called "moving object map." In the following description, the moving object distribution information is also referred to as a moving object map, and the corrected moving object distribution information is also referred to as a corrected moving object map. Details of the moving object map will be described later.

[0012] FIG. 1 is a diagram illustrating an example configuration of an information processing device 1 according to a first embodiment. The information processing device 1 is connected to a detection sensor 2 and a monitoring device 3. The information processing device 1 and the detection sensor 2 constitute an information processing system 4. The detection sensor 2 is equipped with a radio wave sensor 21 and a camera 22. The information processing device 1 corrects a moving object map based on an image captured by the camera 22 equipped on the detection sensor 2 and a moving object map generated by the radio wave sensor 21 equipped on the detection sensor 2, and outputs the corrected moving object map to the monitoring device 3 as error-considered information. The monitoring device 3 includes, for example, an occupant detection device 31 and a physique determination device 32, and monitors occupants in the vehicle cabin based on the corrected moving object map output from the information processing device 1. In the first embodiment, the information processing device 1, the detection sensor 2, and the monitoring device 3 are mounted on, for example, a vehicle (not shown). The information processing device 1, the occupant detection device 31, and the physique determination device 32 will be described in detail later, and first, the detection sensor 2 according to the first embodiment will be described in detail.

[0013] The detection sensor 2 is provided in the vehicle interior, for example, on an overhead console. Note that this is merely an example, and the detection sensor 2 may be provided in a location other than the overhead console, such as on the ceiling or dashboard. The detection sensor 2 is provided in a location in the vehicle interior where the camera 22 can capture an image of the target area and the radio wave sensor 21 can detect a moving object present in the target area; in other words, where the radio wave sensor 21 and the camera 22 can view the target area in the vehicle interior. Note that the target area captured by the camera 22 and the target area to which the radio wave sensor 21 emits radio waves do not have to coincide with each other.

[0014] The detection sensor 2 is installed so that the radio wave sensor 21 emits radio waves in a predetermined direction (hereinafter referred to as the "reference sensor direction") at a predetermined installation position (hereinafter referred to as the "reference sensor position") in the vehicle cabin, and the camera 22 faces a predetermined installation position (hereinafter referred to as the "camera reference position") and a predetermined direction (hereinafter referred to as the "reference camera direction"). However, as described above, the detection sensor 2 is fixed in the vehicle cabin via some kind of structure, such as the dashboard, between it and the vehicle body frame, and therefore installation errors of the radio wave sensor 21 installed in the vehicle cabin may occur.

[0015] In the first embodiment, the position of the radio wave sensor 21 is represented by the center position of the radio wave sensor 21. In addition, the position of the camera 22 is represented by the center position of the camera 22. The positions of the radio wave sensor 21 and the camera 22 are represented by a three-dimensional coordinate system that represents the real space inside the vehicle cabin. In the three-dimensional coordinate system that represents the real space inside the vehicle cabin, the x-axis is an axis parallel to the vehicle width direction, the y-axis is an axis parallel to the vehicle height direction, and the z-axis is an axis parallel to the vehicle length direction. In the first embodiment, "parallel" is not limited to being strictly "parallel" and includes "approximately parallel."

[0016] 2 and 3 are diagrams for explaining a configuration example of the detection sensor 2 according to the first embodiment. The upward direction in the drawings of the detection sensor 2 shown in FIGS. 2 and 3 corresponds to the upward direction when the detection sensor 2 is installed in the vehicle cabin. For convenience, FIG. 2 illustrates only the radio wave sensor 21, the camera 22, and the fixed part 23 of the components of the detection sensor 2. For convenience, FIG. 3 illustrates only the camera 22, the transmitting antenna 11, the receiving antenna 12, and the board 24 of the components of the detection sensor 2.

[0017] 2 , the detection sensor 2 includes a fixed part 23, a camera 22 mounted on the fixed part 23, and a radio wave sensor 21 mounted on the fixed part 23. When the detection sensor 2 is installed in the vehicle cabin, at least two of the three-dimensional coordinate axes of the camera 22 and the three-dimensional coordinate axes of the radio wave sensor 21 are configured to coincide. The three-dimensional coordinate axes of the camera 22 are the x-axis, y-axis, and z-axis specific to the camera 22, and the three-dimensional coordinate axes of the radio wave sensor 21 are the x-axis, y-axis, and z-axis specific to the radio wave sensor 21. In the first embodiment, the x-axis and y-axis specific to the camera 22 refer to axes on the plane of the imaging element of the camera 22, and the z-axis refers to an axis normal to the imaging element. In the first embodiment, the x-axis and y-axis specific to the radio wave sensor 21 refer to axes on the surface of the substrate 24 on which the antennas (transmitting antenna 11 and receiving antenna 12) are arranged, and the z-axis refers to an axis normal to the surface of the substrate 24. In the first embodiment, "match" is not limited to a perfect match, but also includes a match within an allowable error range.

[0018] As shown in FIG. 3 , the detection sensor 2 may include a camera 22, a radio wave sensor 21, and a substrate 24 on which the camera, the transmitting antenna 11 (see FIG. 4 described later) of the radio wave sensor 21, and the receiving antenna 12 (see FIG. 4 described later) of the radio wave sensor 21 are provided on a common surface, with the camera 22 being provided between the transmitting antenna 11 and the receiving antenna 12 on the common surface of the substrate 24. The transmitting antenna 11 shown in FIG. 3 is more specifically the transmitting antenna elements 11-1 and 11-2 (see FIG. 4 described later) of the transmitting antenna 11, and the receiving antenna 12 shown in FIG. 3 is more specifically the receiving antenna elements 12-1 to 12-4 (see FIG. 4 described later) of the receiving antenna 12. While only one transmitting antenna element 11-1 or 11-2 is shown in FIG. 3 , the transmitting antenna 11 in the radio wave sensor 21 may include multiple transmitting antenna elements 11-1 or 11-2. 3 shows three receiving antenna elements 12-1 to 12-4, the number of receiving antenna elements 12-1 to 12-4 included in the receiving antenna 12 in the radio wave sensor 21 is not limited to three. The receiving antenna 12 has a plurality of receiving antenna elements 12-1 to 12-4. Details of the radio wave sensor 21 will be described later.

[0019] The camera 22 captures an image of a target area in the vehicle cabin. The camera 22 is, for example, a near-infrared camera or a visible light camera. In the first embodiment, the camera 22 is assumed to be shared with a so-called DMS (Driver Monitoring System), which is installed for the purpose of monitoring the interior of the vehicle.

[0020] The radio wave sensor 21 generates moving object distribution information, i.e., a moving object map, which represents the distribution of areas where moving objects exist in the vehicle cabin in three dimensions, based on the waves that are generated when radio waves emitted toward a target area in the vehicle cabin are reflected by objects in the vehicle cabin. The radio wave sensor 21 is, for example, a millimeter wave radar.

[0021] The detection sensor 2 outputs the captured image captured by the camera 22 and the moving object map generated by the radio wave sensor 21 to the information processing device 1 .

[0022] Here, the radio wave sensor 21 according to the first embodiment will be described in detail.

[0023] Fig. 4 is a configuration diagram showing the radio wave sensor 21 mounted on the detection sensor 2 in embodiment 1. Fig. 5 is a block diagram showing the functions of the radio wave sensor 21 mounted on the vehicle in embodiment 1. Of the three-dimensional coordinate axes in Fig. 4, the x-axis is an axis parallel to the vehicle width direction, the y-axis is an axis parallel to the vehicle height direction, and the z-axis is an axis parallel to the vehicle length direction.

[0024] The transmitting antenna 11 is a planar antenna configured on an electronic circuit board and has a plurality of transmitting antenna elements 11-1 and 11-2 that radiate radio waves toward a target area.

[0025] The installation positions of the transmitting antenna element 11-1 and the transmitting antenna element 11-2 are different from each other in the vehicle height direction. In the radio wave sensor 21 shown in Figure 4, the installation position of the transmitting antenna element 11-1 in the vehicle height direction is higher than the installation position of the transmitting antenna element 11-2 in the vehicle height direction.

[0026] The receiving antenna 12 is a planar antenna constructed on an electronic circuit board, and is installed on the same plane as the transmitting antenna 11. However, the term "same plane" here does not mean that the plane on which the transmitting antenna 11 and the plane on which the receiving antenna 12 are installed are strictly the same, but also includes different planes as long as there are no practical problems. The receiving antenna 12 has multiple receiving antenna elements 12-1 to 12-4 that receive reflected waves of the radio waves radiated from the transmitting antenna 11.

[0027] The receiving antenna elements 12-1, 12-2, 12-3, and 12-4 are installed at different positions in the width direction of the vehicle.

[0028] The radio wave sensor circuit unit 13 includes a high-frequency signal generating circuit 14, a radio wave transmitting unit 15, a radio wave receiving unit 16, an analog-to-digital conversion circuit (hereinafter referred to as the "A / D conversion circuit") 17, a map generating unit 18, a communication circuit 19, and a power supply circuit 200. The high-frequency signal generating circuit 14, the radio wave transmitting unit 15, and the radio wave receiving unit 16 constitute a radio wave transmitting / receiving unit 130 that radiates radio waves from the transmitting antenna 11 and acquires a received signal based on the reflected waves from the receiving antenna 12.

[0029] The high-frequency signal generating circuit 14 generates an FM (Frequency Modulation) signal, whose frequency changes over time, as a sensing signal and outputs the FM signal to the radio wave transmitting unit 15 and the radio wave receiving unit 16. In the radio wave sensor 21 shown in FIG. 4 , the FM-CW (Frequency Modulation-Continuous Wave) method is used as the modulation method, and the high-frequency signal generating circuit 14 generates the FM signal. However, the modulation method is not limited to the FM-CW method, and for example, the FCM (Fast-Chirp Modulation) method may also be used. When the FCM method is used as the modulation method, the high-frequency signal generating circuit 14 generates the FCM signal and outputs the FCM signal to the radio wave transmitting unit 15 and the radio wave receiving unit 16.

[0030] The radio wave transmitting unit 15 has a transmitting circuit 15-1 and a transmitting circuit 15-2. The radio wave transmitting unit 15 radiates radio waves toward a target area from one of the transmitting antenna elements 11-1 and 11-2. That is, the radio wave transmitting unit 15 alternately switches between the transmitting antenna elements 11-1 and 11-2 to radiate radio waves. When radiating radio waves from the transmitting antenna element 11-1, the radio wave transmitting unit 15 outputs an FM signal from the transmitting circuit 15-1 to the transmitting antenna element 11-1. When radiating radio waves from the transmitting antenna element 11-2, the radio wave transmitting unit 15 outputs an FM signal from the transmitting circuit 15-2 to the transmitting antenna element 11-2.

[0031] The transmitting circuit 15-1 amplifies the FM signal output from the high frequency signal generating circuit 14 and outputs the amplified FM signal to the transmitting antenna element 11-1, thereby causing the transmitting antenna element 11-1 to radiate an FM transmission wave, which is an electric wave, toward the target area. The transmitting circuit 15-2 amplifies the FM signal output from the high frequency signal generating circuit 14 and outputs the amplified FM signal to the transmitting antenna element 11-2, thereby causing the transmitting antenna element 11-2 to radiate an FM transmission wave toward the target area.

[0032] The radio wave receiving unit 16 has receiving circuits 16-1, 16-2, 16-3, and 16-4. When an FM reception wave, which is a reflected wave, is received by the receiving antenna element 12-m, the receiving circuit 16-m (m=1, 2, 3, 4) acquires a reception signal of the FM reception wave from the receiving antenna element 12-m. The receiving circuit 16-m calculates the difference f between the frequency of the FM signal output from the high frequency signal generating circuit 14 and the frequency of the reception signal (hereinafter referred to as the "frequency difference"). ｄ The receiving circuit 16-m extracts the frequency difference f ｄ an intermediate frequency signal IF having ｍ and generates an intermediate frequency signal IF ｍ is output to the A / D conversion circuit 17.

[0033] The A / D conversion circuit 17 converts the intermediate frequency signal IF output from the receiving circuit 16-m (m=1, 2, 3, 4) ｍ converting the analog signal to a digital signal D ｍ The A / D conversion circuit 17 converts the digital signal D ｍ is output to the map generating unit 18.

[0034] The map generating unit 18 is realized by, for example, a digital signal processing circuit. The map generating unit 18 receives from the A / D conversion circuit 17 a digital signal D relating to the FM reception wave received by the receiving circuit 16-m (m=1, 2, 3, 4). ｍ The map generating unit 18 obtains the digital signal D ｍ A moving object map is generated based on m (m=1, 2, 3, 4). The map generating unit 18 outputs the generated moving object map to the communication circuit 19.

[0035] The digital signal processing circuit may be realized, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The digital signal processing circuit is not limited to being realized by dedicated hardware, but may also be realized by software, firmware, or a combination of software and firmware. The software or firmware is stored as a program in the memory of a computer. The computer refers to hardware that executes a program, and includes, for example, a CPU (Central Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor).

[0036] Here, we will explain the specific operation of the radio wave sensor 21 to generate a moving object map, and an example of the moving object map. While signals modulated by various modulation methods can be used as the sensing signal of the radio wave sensor 21, an example using a signal modulated by the FM-CW method will be explained. As shown in FIG. 6 , the high-frequency signal generating circuit 14 generates an FM signal Tx(k) (k = 1, ..., K), which is a chirp signal whose frequency changes over time, as the sensing signal. K is an integer equal to or greater than 2. FIG. 6 is an explanatory diagram showing an example of the FM signal Tx(k) generated by the high-frequency signal generating circuit 14. In the FM signal Tx(k) shown in FIG. 6 , the initial frequency state is the lower limit frequency, and the frequency increases over time until it reaches the upper limit frequency. In FIG. 6 , the sweep time T is the time it takes for the frequency of the FM signal Tx(k) to reach the upper limit frequency from the lower limit frequency, and the frequency bandwidth BW is the frequency difference between the upper limit frequency and the lower limit frequency. In the example of FIG. 6, the FM signal Tx(k) has a constant period T ｃ, and is generated K times. The high frequency signal generating circuit 14 outputs the FM signal Tx(k) to the transmitting circuits 15-1 and 15-2 of the radio wave transmitting unit 15, and outputs the FM signal Tx(k) to the receiving circuits 16-1 to 16-4 of the radio wave receiving unit 16. The amplitude waveform of the FM signal Tx(k) is also shown in FIG.

[0037] The transmission circuits 15-1 and 15-2 alternately perform the output operation of the FM signal Tx(k). When the transmission circuit 15-1 receives the FM signal Tx(k) from the high-frequency signal generating circuit 14 at the timing of the output operation of the FM signal Tx(k), it amplifies the FM signal Tx(k) and outputs the amplified FM signal Tx(k) to the transmitting antenna element 11-1. When the amplified FM signal Tx(k) is output from the transmission circuit 15-1 to the transmitting antenna element 11-1, an FM transmission wave is radiated from the transmitting antenna element 11-1 toward the target area. When the transmission circuit 15-1 is performing the output operation of the FM signal Tx(k), the transmission circuit 15-2 stops the output operation of the FM signal Tx(k). The timing at which the transmission circuit 15-1 performs the output operation of the FM signal Tx(k) is, for example, k=1, 3, 5, etc.

[0038] When the transmission circuit 15-2 receives the FM signal Tx(k) from the high frequency signal generating circuit 14 at the timing of outputting the FM signal Tx(k), it amplifies the FM signal Tx(k) and outputs the amplified FM signal Tx(k) to the transmitting antenna element 11-2. When the transmission circuit 15-2 outputs the amplified FM signal Tx(k) to the transmitting antenna element 11-2, an FM transmission wave is radiated from the transmitting antenna element 11-2 toward the target area. When the transmission circuit 15-2 is outputting the FM signal Tx(k), the transmission circuit 15-1 stops outputting the FM signal Tx(k). The timing at which the transmission circuit 15-2 outputs the FM signal Tx(k) is, for example, k=2, 4, 6, etc.

[0039] When transmitting antenna element 11-1 receives FM signal Tx(k) from transmitting circuit 15-1, it radiates an FM transmission wave whose frequency transitions over time toward the target area as shown in Fig. 7. When transmitting antenna element 11-2 receives FM signal Tx(k) from transmitting circuit 15-2, it radiates an FM transmission wave as shown in Fig. 7 toward the target area. Fig. 7 is an explanatory diagram showing an example of an FM transmission wave and an FM reception wave. The FM transmission waves radiated from transmitting antenna element 11-1 and transmitting antenna element 11-2 are reflected by occupants and the like present in the target area.

[0040] As shown in FIG. 7, the FM reception wave, which is a reflected wave of the FM transmission wave reflected by the passengers or the like, is transmitted from the transmitting antenna element 11-1 or the transmitting antenna element 11-2 for t ｄ After the time has elapsed, the FM transmission wave is received by the receiving antenna elements 12-1 to 12-4. At this time, the frequency of the FM transmission wave and the frequency of the FM reception wave are different from each other by f ｄ The frequency difference f between the FM transmitted wave and the FM received wave, which are chirp signals, is ｄ increases in proportion to the distance between the radio wave sensor 21 and a reflecting object such as a passenger. ｄ FIG.

[0041] When the receiving antenna element 12-m (m=1, 2, 3, 4) receives the FM reception wave, which is a reflected wave, it outputs a reception signal Rx(k) of the FM reception wave to the receiving circuit 16-m. The receiving circuit 16-m outputs a frequency difference f, which is the difference between the frequency of the FM signal Tx(k) output from the high frequency signal generating circuit 14 and the frequency of the reception signal Rx(k) output from the receiving antenna element 12-m. ｄ The receiving circuit 16-m extracts the frequency difference f ｄ (k) ｍ (k) and generates an intermediate frequency signal IF ｍ (k) is output to the A / D conversion circuit 17.

[0042] The A / D conversion circuit 17 receives the intermediate frequency signal IF from the receiving circuit 16-m (m=1, 2, 3, 4). ｍ(k), the intermediate frequency signal IF ｍ (k) is converted from an analog signal to a digital signal D ｍ The A / D conversion circuit 17 converts the digital signal D ｍ (k) is output to the map generating unit 18.

[0043] The map generating unit 18 receives the digital signal D ｍ (k) (m=1, 2, 3, 4). The map generating unit 18 obtains the digital signal D ｍ A moving object map is generated based on (k) (m=1, 2, 3, 4).

[0044] An example of the flow of operations up to the generation of the operation map by the map generating unit 18 will be described below. The map generating unit 18 receives the digital signal D ｍ (k) (m=1, 2, 3, 4) each time a digital signal D ｍ The map generator 18 converts the four digital signals D １ (k) to D ４ The first frequency spectrum Sp is obtained by combining the Fourier transform results of (k). １ The digital signal D ｍ By Fourier transforming (k), the spectrum value of the received signal Rx(k) (k=1, . . . , K) of the reflected wave from a reflecting object such as a passenger is calculated based on the beat frequency Fs １，ｎ (n=1, ..., N) signal. Beat frequency Fs １，ｎ The signal strength of １，ｎ The signal strength of the first frequency spectrum Sp １ This is the peak value. In formula (1), R ｎ is the distance between the radio wave sensor 21 and the passenger, and c is the propagation speed of the radio wave.

[0045] The map generator 18 generates a first frequency spectrum Sp １ The beat frequency Fs at which the signal strength reaches its peak value is １，ｎ The map generating unit 18 searches for the beat frequency Fs１，ｎ By substituting into equation (1), the distance R ｎ Calculate.

[0046] The map generator 18 generates a first frequency spectrum Sp １ Each time the first frequency spectrum Sp １ is Fourier transformed in the time direction of the FM transmission wave periodically radiated from the transmitting antenna 11, to obtain a second frequency spectrum Sp ２ The first frequency spectrum Sp １ is Fourier transformed in the time direction of the FM transmission wave, the spectrum value of the received signal Rx(k) (k=1, . . . , K) of the wave reflected from the reflecting object is calculated based on the relative velocity v between the radio wave sensor 21 and the reflecting object. ｎ The beat frequency Fs shown in the following equation (2) corresponds to ２，ｎ Beat frequency Fs ２，ｎ The signal strength of ２，ｎ The signal strength of the second frequency spectrum Sp ２ This is the peak value. In formula (2), f ０ is the center frequency of the FM signal Tx(k).

[0047] The map generator 18 generates a second frequency spectrum Sp ２ The beat frequency Fs at which the signal strength reaches its peak value is ２，ｎ The map generating unit 18 searches for the beat frequency Fs ２，ｎ By substituting into equation (2), the relative velocity v ｎ Calculate.

[0048] The installation positions of the receiving antenna elements 12-1 to 12-4 are different from each other in the vehicle width direction. Therefore, even if a wave is reflected from the same reflecting object, the propagation distance to reach the receiving antenna element 12-1 is different from the propagation distance to reach the receiving antenna element 12-m (m = 2, 3, 4). Therefore, a phase difference Δφ occurs between the phase of the FM reception wave received by the receiving antenna element 12-1 and the phase of the FM reception wave received by the receiving antenna element 12-m (m = 2, 3, 4), as shown in the following equation (3). In formula (3), d ｘ is the distance between the receiving antenna elements 12-1 to 12-4 in the direction parallel to the vehicle width direction, and θ ｘ，ｎ is the angle of incidence of the FM received wave on the xz plane to the receiving antenna element 12-m, and λ is the wavelength of the FM transmitted wave.

[0049] The map generating unit 18 receives the digital signal D ｍ (k) (m=1, 2, 3, 4) each time a digital signal D ｍ The map generator 18 converts the four digital signals D １ (k) to D ４ By combining the Fourier transform results of (k), a third frequency spectrum Sp ３，ｍ The map generator 18 calculates the third frequency spectrum Sp ３，ｍ Each time a third frequency spectrum Sp ３，ｍ is Fourier transformed in the time direction of the periodically radiated FM transmission wave to obtain a fourth frequency spectrum Sp ４，ｍ The map generator 18 calculates the fourth frequency spectrum Sp ４，ｍ Each time a fourth frequency spectrum Sp ４，ｍ is Fourier transformed to obtain the fifth frequency spectrum Sp ５ The fourth frequency spectrum Sp ４，ｍis Fourier transformed across the vehicle width direction (direction parallel to the x-axis), which is the direction in which the receiving antenna elements 12-1 to 12-4 are arranged, the spectral value of the received signal Rx(k) (k=1, . . . , K) of the wave reflected from the reflecting object is expressed as ｘ，ｎ The frequency component Fs corresponding to ５，ｎ The frequency component Fs ５，ｎ The signal strength of the frequency component Fs ５，ｎ The signal intensity of the fifth frequency spectrum Sp ５ This is the peak value.

[0050] The map generator 18 generates a fifth frequency spectrum Sp ５ The frequency component Fs at which the signal strength reaches its peak value is ５，ｎ The map generator 18 searches for the searched frequency component Fs ５，ｎ By substituting into the following equation (4), the incident angle θ in the k-th transmission and reception on the xz plane is ｘ，ｎ Calculate.

[0051] The radio wave sensor 21 shown in FIG. 4 includes four receiving antenna elements 12-1 to 12-4. The radio wave sensor 21 also includes two transmitting antenna elements 11-1 and 11-2, and the transmitting antenna elements 11-1 and 11-2 alternately radiate FM transmission waves. Therefore, as shown in FIG. 9, the radio wave sensor 21 configures virtual antennas 12-1' to 12-8' equivalent to the four receiving antenna elements 12-1 to 12-4 being lined up in two rows in the vehicle height direction. In the example of FIG. 9, the virtual antennas 12-1' to 12-4' and the virtual antennas 12-5' to 12-8' are lined up in the vehicle height direction. The distance between the virtual antennas 12-1' to 12-4' and the virtual antennas 12-5' to 12-8' in the vehicle height direction is d. ｙ 9 is an explanatory diagram showing an example of the virtual antennas 12-1' to 12-8' configured in the radio wave sensor 21.

[0052] The map generator 18 generates a fourth frequency spectrum Sp ４，ｍis Fourier transformed in the vehicle height direction (direction parallel to the y-axis), which is the direction in which the virtual antennas 12-1′ etc. and 12-5′ etc. are aligned, to obtain the frequency component Fs ５，ｎ By substituting into the following equation (5), the incident angle θ in the k-th transmission and reception on the yz plane is ｙ，ｎ Calculate.

[0053] Then, the map generating unit 18 calculates the distance R ｎ , the angle of incidence θ in the xz plane ｘ，ｎ and the angle of incidence θ in the yz plane ｙ，ｎ A three-dimensional spatial distribution is generated with the dimensions of each of the above. The process of generating the three-dimensional spatial distribution itself is a known technique, and therefore a detailed description thereof will be omitted. In the first embodiment, the three-dimensional spatial distribution generated by the map generation unit 18 is a "moving object map." The moving object map represents the distribution of areas in the vehicle cabin where moving objects exist in three dimensions. In detail, the moving object map represents the minute movements of moving objects, in other words, objects that reflect the radio waves irradiated by the radio wave sensor 21, in the vehicle cabin using a plurality of grids that correspond to the reflection points of the radio waves in three-dimensional space.

[0054] 10A, 10B, and 10C are diagrams illustrating an example of a moving object map generated by the radio wave sensor 21 in the first embodiment. FIG. 10A illustrates an example of a top view of the moving object map, FIG. 10B illustrates an example of a side view of the moving object map, and FIG. 10C illustrates an example of a front view of the moving object map. For ease of explanation, the moving object map is illustrated in a three-view format in FIGS. 10A, 10B, and 10C. The moving object maps illustrated in FIGS. 10A, 10B, and 10C are moving object maps for the case where an occupant is present in the rear right seat in the vehicle cabin.

[0055] In the moving object map, each grid is assigned a numerical value corresponding to the speed of an object present at the grid position. Specifically, the faster the speed of an object present at a given grid position, the higher the numerical value assigned to that grid. That is, in the moving object map, a moving object with slight movement, in other words, a grid included in an area where an occupant is present, is assigned a higher numerical value than a grid where no occupant is present. In the moving object maps shown in FIGS. 10A, 10B, and 10C, a grid with a higher assigned numerical value is shown darker. That is, in the moving object maps shown in FIGS. 10A, 10B, and 10C, a grid included in an area where an occupant is present is shown darker. Note that stationary objects, in other words, reflected wave components with a speed of zero, do not appear in the moving object map.

[0056] Furthermore, each grid in the moving object map is associated with a coordinate in the real space inside the vehicle cabin, which is expressed in a three-dimensional coordinate system that represents the real space inside the vehicle cabin. For example, assuming that the radio wave sensor 21 is installed at a reference sensor position, the map generator 18 associates each grid with a coordinate in the real space inside the vehicle cabin based on the reference position.

[0057] The communication circuit 19 transfers the moving object map output from the map generation unit 18 to the information processing device 1 via an interface unit 210, which will be described later. The power supply circuit 200 receives power from a control unit (not shown) or the like via the interface unit 210. The power supply circuit 200 distributes the received power as driving power to the high-frequency signal generation circuit 14, the radio wave transmission unit 15, the radio wave reception unit 16, the A / D conversion circuit 17, the map generation unit 18, and the communication circuit 19. The interface unit 210 is an interface for connecting the radio wave sensor circuit unit 13 to the information processing device 1 or a control unit (not shown) or the like.

[0058] As described above, the radio wave sensor 21 in the first embodiment is a radio wave sensor 21 capable of detecting the distance and angle to an object present in the vehicle cabin and whether the object is moving, and is a radio wave sensor capable of generating a moving object map based on the distance and angle to the detected object (more specifically, the moving object). In the above description, the receiving antenna 12 in the radio wave sensor 21 has four receiving antenna elements 12-1 to 12-4. However, this is merely an example. The receiving antenna 12 may have multiple receiving antenna elements, and the receiving antenna 12 may have two receiving antenna elements, three receiving antenna elements, or five or more receiving antenna elements. In the above description, the transmitting antenna 11 in the radio wave sensor 21 has two transmitting antenna elements 11-1 and 11-2. However, this is merely an example. The transmitting antenna 11 in the radio wave sensor 21 may have only one transmitting antenna element, or may have three or more transmitting antenna elements.

[0059] 1 , a description will be given of an example configuration of the information processing device 1. The information processing device 1 includes a moving object distribution information acquisition unit 101, an image acquisition unit 102, a camera error detection unit 103, a radio wave sensor error calculation unit 104, a correction unit 105, and an output unit 106.

[0060] The moving object distribution information acquisition unit 101 acquires moving object distribution information, i.e., a moving object map, generated by the radio wave sensor 21 from the detection sensor 2. The moving object distribution information acquisition unit 101 outputs the acquired moving object map to the correction unit 105.

[0061] The image acquisition unit 102 acquires the captured image captured by the camera 22 from the detection sensor 2. The image acquisition unit 102 outputs the acquired captured image to the camera error detection unit 103.

[0062] The camera error detection unit 103 detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102. In the first embodiment, the "installation error of the camera 22" refers to a deviation in the installation position of the camera 22 in the real space inside the vehicle cabin (installation position error) or a deviation in the orientation of the camera 22 in the real space inside the vehicle cabin (orientation error of the camera 22). More specifically, the "installation error of the camera 22" refers to an error in the installation position of the camera 22 with respect to a predetermined installation position (hereinafter referred to as a "reference camera position"), or an error in the orientation of the camera 22 with respect to a predetermined orientation (hereinafter referred to as a "reference camera orientation"). More specifically, the installation position error of the camera 22 refers to an error in the vehicle width direction, vehicle height direction, and vehicle length direction of the installation position of the camera 22 within the vehicle cabin.

[0063] For example, the camera error detection unit 103 can detect an error in the installation position of the camera 22, among the installation errors of the camera 22, by using a known image recognition technique to compare the position and size of a predetermined target object in the captured image with the position (hereinafter referred to as the "target position") and size (hereinafter referred to as the "target size") of the target object in the captured image where the target object should be. Note that the target object may be, for example, a structure inside the vehicle, such as a window frame, a vehicle body, or a seat frame inside the vehicle, or a marker that is previously attached to the vehicle interior by printing or the like. In detail, the target position and target size of the target object in the captured image are the position and size of the target object in the captured image captured by the camera 22 when the camera 22 is installed at a reference camera position, facing the reference camera, and without any misalignment.

[0064] For example, an administrator or the like may conduct a test in advance to obtain information on the target position and target size of the target, and store the obtained information on the target position and target size in the camera error detection unit 103 in association with information indicating the target. The target position and target size are represented by coordinates on the captured image. The target position is represented, for example, by the coordinates of the center of the target in the captured image. The target size is represented, for example, by the coordinates of the four corners of the smallest rectangle that surrounds the target in the captured image.

[0065] The camera error detection unit 103 can detect an error in the vehicle width direction in the real space of the installation position of the camera 22 based on the difference in the vehicle width direction between the position of the target object captured in the captured image acquired by the image acquisition unit 102 and the target position. Note that, for example, a calculation formula for determining the difference in the vehicle width direction between the position of the target object and the target position in the real space when the difference in the vehicle width direction between the position of the target object and the target position in the captured image is large is defined in advance and stored in the camera error detection unit 103.

[0066] Furthermore, the camera error detection unit 103 can detect an error in the vehicle height direction in the real space of the installation position of the camera 22 based on the difference in the vehicle height direction between the position of the target object captured in the captured image acquired by the image acquisition unit 102 and the target position. Note that, for example, a calculation formula for determining the difference in the vehicle height direction between the position of the target object and the target position in the real space when the difference in the vehicle height direction between the position of the target object and the target position in the captured image is large is defined in advance and stored in the camera error detection unit 103.

[0067] Furthermore, the camera error detection unit 103 can detect an error in the vehicle length direction in real space of the installation position of the camera 22 based on the difference between the target size and the size of the target object captured in the captured image acquired by the image acquisition unit 102. Note that a calculation formula for determining, for example, the difference in the vehicle length direction between the target position and the target position in real space when the difference in the vehicle length direction between the target position and the target position in the captured image is large is defined in advance and stored in the camera error detection unit 103.

[0068] In addition, the camera error detection unit 103 can detect an error in the orientation of the camera 22, which is one of the installation errors of the camera 22, based on the difference between the positions of multiple targets captured on the captured image acquired by the image acquisition unit 102 and the target positions of the multiple targets.

[0069] For example, the camera error detection unit 103 may detect an error in the installation position of the camera 22 and an error in the orientation of the camera 22, i.e., an installation error of the camera 22, based on the captured image acquired by the image acquisition unit 102, information indicating the target object, the target position, the target size, and a trained model (hereinafter referred to as a "machine learning model"). The machine learning model is a model that receives the captured image, information indicating the target object, the target position, and the target size as input, and outputs an error in the installation position of the camera 22 and an error in the orientation of the camera 22.

[0070] The camera error detection unit 103 outputs information about the detected installation error of the camera 22, i.e., information about an error in the installation position of the camera 22 and an error in the orientation of the camera 22 (hereinafter referred to as "camera installation error information"), to the radio wave sensor error calculation unit 104. In the camera installation error information, the error in the installation position of the camera 22 is represented, for example, by distances in real space within the vehicle cabin (distances in the vehicle width direction, vehicle height direction, and vehicle length direction). The error in the orientation of the camera 22 is represented, for example, by an angle indicating the difference in the orientation of the camera 22 from a reference camera orientation.

[0071] The radio wave sensor error calculation unit 104 calculates the installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103. As described above, in the first embodiment, the "installation error of the radio wave sensor 21" refers to an error in the installation position of the radio wave sensor 21 relative to a predetermined installation position (hereinafter referred to as the "reference sensor position") in the real space within the vehicle cabin, or an error in the orientation of the radio wave sensor 21 relative to a predetermined orientation (hereinafter referred to as the "reference sensor orientation"). More specifically, the error in the installation position of the radio wave sensor 21 refers to an error in the vehicle width direction, vehicle height direction, and vehicle length direction of the installation position of the radio wave sensor 21 within the vehicle cabin.

[0072] Because the positional relationship between the radio wave sensor 21 and the camera 22 in the detection sensor 2 and the relationship between the orientation of the radio wave sensor 21 and the orientation of the camera 22 are known, the radio wave sensor error calculation unit 104 can calculate the installation error of the radio wave sensor 21 based on the installation error of the camera 22. The radio wave sensor error calculation unit 104 may calculate the installation error of the radio wave sensor 21, for example, using a table in which the installation error of the camera 22 corresponds to the installation error of the radio wave sensor 21 (hereinafter referred to as the "radio wave sensor error calculation table"). The radio wave sensor error calculation table is generated in advance by an administrator or the like and stored in the radio wave sensor error calculation unit 104.

[0073] The radio wave sensor error calculation unit 104 outputs the calculated information on the installation error of the radio wave sensor 21 (hereinafter referred to as "sensor installation error information") to the correction unit 105. In the sensor installation error information, the error in the installation position of the radio wave sensor 21 is expressed, for example, by distances in the real space within the vehicle cabin (distances in the vehicle width direction, vehicle height direction, and vehicle length direction). The error in the orientation of the radio wave sensor 21 is expressed, for example, by the difference in the orientation of the radio wave sensor 21 from the orientation of a reference sensor.

[0074] The correction unit 105 corrects the moving object map acquired by the moving object distribution information acquisition unit 101 based on the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. In detail, for example, the correction unit 105 first calculates parameters (hereinafter referred to as "map correction parameters") for correcting the moving object map based on the installation error of the radio wave sensor 21. The map correction parameters are parameters that define the amount of rotation or horizontal movement of the moving object map depending on, for example, the installation position error of the radio wave sensor 21 (i.e., the error in the installation position of the radio wave sensor 21 in the vehicle width direction, the error in the vehicle height direction, and the error in the vehicle length direction) and the orientation error of the radio wave sensor 21 (the difference in orientation from the reference sensor orientation).

[0075] The map correction parameters may be, for example, a table (hereinafter referred to as a "map correction table") that converts coordinates associated with each grid in the moving object map. The coordinates associated with each grid are, for example, coordinates in the real space inside the vehicle cabin. More specifically, the map correction table is information that associates the error in the installation position of the radio wave sensor 21 (i.e., the error in the installation position of the radio wave sensor 21 in the vehicle width direction, the error in the vehicle height direction, and the error in the vehicle length direction), the error in the orientation of the radio wave sensor 21 (the difference in orientation from the orientation of a reference sensor), the amount of movement (e.g., how many pixels or how many millimeters) of which grid in the moving object map should be moved in the vehicle width direction, the vehicle height direction, and the vehicle length direction, and the angle by which the moving object map should be rotated after the movement.

[0076] A plurality of map correction tables may be prepared, for example, depending on the type of installation error of the radio wave sensor 21. For example, a plurality of map correction tables may be prepared, such as a map correction table for when there is an error in the installation position of the radio wave sensor 21 only in the vehicle width direction, a map correction table for when there is an error in the vehicle height direction, and a map correction table for when there is an error in the vehicle length direction, etc. The correction unit 105 determines which map correction table to use to correct the moving body map, depending on the type of installation error of the radio wave sensor 21.

[0077] The correction unit 105 outputs the corrected moving object map to the output unit 106. If correction is not required, in other words, if there is no installation error of the radio wave sensor 21, the correction unit 105 outputs the moving object map acquired by the moving object distribution information acquisition unit 101 to the output unit 106 as is as the corrected moving object map.

[0078] The output unit 106 outputs the corrected moving object map output from the correction unit 105, that is, the moving object map corrected by the correction unit 105, to the monitoring device 3 as error consideration information.

[0079] The monitoring device 3 according to the first embodiment will be described. The monitoring device 3 monitors occupants in the vehicle cabin based on the moving object map, specifically the corrected moving object map, output from the information processing device 1. The monitoring device 3 includes an occupant detection device 31 and a physique determination device 32.

[0080] The occupant detection device 31 detects whether a child has been left behind in the vehicle cabin based on the corrected moving object map. In the first embodiment, a child is assumed to be an occupant whose physique makes it difficult for the child to get out of the vehicle on his or her own if left behind. In the first embodiment, the child includes an infant.

[0081] For example, the occupant detection device 31 first detects an occupant present in the vehicle cabin and determines whether the occupant is an adult or a child. Specifically, the occupant detection device 31 determines whether a spatial distribution included in the corrected moving object map (for example, the spatial distribution shown by the colored grid in FIG. 10 ) is a spatial distribution corresponding to an occupant (hereinafter referred to as an "occupant spatial distribution") based on the shape of the spatial distribution. Any method may be used for the determination based on the shape of the spatial distribution. For example, a model that has learned the shape of the occupant spatial distribution may be used to determine whether the spatial distribution included in the moving object map is an occupant spatial distribution. In addition, the occupant detection device 31 determines whether the relative velocity v corresponding to the spatial distribution included in the corrected moving object map is an occupant spatial distribution. ｎ is greater than a preset threshold value (hereinafter referred to as a "speed determination threshold value"), the spatial distribution may be determined to be an occupant spatial distribution. ｎ are associated with the velocity determination threshold. The velocity determination threshold is stored in a location that can be referenced by the occupant detection. Even if an occupant is not moving their hands or feet, they are breathing and therefore moving to some extent. If the occupant detection device 31 determines that the spatial distribution included in the corrected moving object map is an occupant spatial distribution, it detects that an occupant is present in the vehicle cabin.

[0082] Furthermore, if the occupant detection device 31 determines that the spatial distribution included in the corrected moving object map is an occupant spatial distribution, the occupant detection device 31 calculates the distance R ｎ , incident angle θ ｘ，ｎ and the angle of incidence θ ｙ，ｎ The position of the occupant in the occupant space distribution is identified from each of the distances R ｎ and the incident angle θ with respect to the radio wave sensor 21 ｘ，ｎ and the angle of incidence θ ｙ，ｎ If the distance R from the radio wave sensor 21 is known, the occupant detection device 31 can identify the position of the occupant. ｎ and the incident angle θ with respect to the radio wave sensor 21 ｘ，ｎ and the angle of incidence θ ｙ，ｎ The seating position of the occupant is indicated by, for example, the seat (for example, the rear right seat, the rear left seat, or the rear center seat).

[0083] The occupant detection device 31 also determines whether the detected occupant is an adult or a child for each seat based on the size of the occupant space distribution included in the corrected moving object map. Any method may be used for the determination based on the size of the occupant space distribution. For example, a virtual area for determining physique (hereinafter referred to as a "physique determination area") in a pre-defined space within the vehicle cabin may be used to determine whether the occupant is an adult or a child. The physique determination area is pre-defined by an administrator or the like and stored in a location accessible by the monitoring device 3. The physique determination area is, for example, a rectangular area on the seat surface of each seat, with the bottom surface including the seat surface and the height measured from the seat surface to the top of the headrest. The occupant detection device 31 determines whether the occupant is an adult or a child based on, for example, the height of the occupant space distribution in the physique determination area. Another possible method of determining whether an occupant is an adult or a child is to use a model that has learned the size of the occupant space distribution for adults and children.

[0084] The occupant detection device 31 then detects whether a child has been left behind in the vehicle cabin based on the determination result of whether the detected occupant is an adult or a child. For example, when the occupant detection device 31 determines that no adult is present in the vehicle cabin but that a child is present, i.e., when it determines that only a child is present in the vehicle cabin, the occupant detection device 31 detects that a child has been left behind in the vehicle cabin. For example, when an adult is present in the vehicle cabin or when neither an adult nor a child is present, the occupant detection device 31 detects that a child has not been left behind in the vehicle cabin. When the occupant detection device 31 detects that a child has been left behind in the vehicle cabin, it outputs an alarm to an output device (not shown). The output device is, for example, a mobile terminal carried by the vehicle owner or a horn or light provided in the vehicle.

[0085] Furthermore, when the occupant detection device 31 detects an occupant present in the vehicle compartment, the occupant detection device 31 may cause the output device to output a sound or the like urging the occupant to fasten their seat belt.

[0086] The physique determination device 32 determines the physique of an occupant present in the vehicle cabin based on the corrected moving object map. For example, the physique determination device 32 determines whether the occupant is an "adult" or a "child." In this case, the physique determination device 32 determines the physique of the occupant using, for example, a method similar to the method used by the occupant detection device 31 described above to determine whether the occupant is an adult or a child upon detecting the occupant present in the vehicle cabin.

[0087] Furthermore, the physique determination device 32 may further subdivide the occupant's physique into categories such as "large adult," "small adult," or "child." In this case, for example, the physique determination region is divided into three regions in the height direction. The three regions are designated, in order from highest to lowest, as a first physique determination region, a second physique determination region, and a third physique determination region. The physique determination device 32 determines whether the occupant spatial distribution included in the corrected dynamic object map is included in the first physique determination region, the second physique determination region, or the third physique determination region. For example, if the occupant spatial distribution is included in all of the first physique determination region, the second physique determination region, and the third physique determination region, the physique determination device 32 determines that the occupant is a "large adult." For example, if the occupant spatial distribution is included only in the third physique determination region, the physique determination device 32 determines that the occupant is a "child."

[0088] The physique determination device 32 outputs information relating to the determined physique of the occupant (hereinafter referred to as the "physique determination result") to, for example, a seat belt control device (not shown) and a seat control device (not shown). The seat belt control device controls, for example, the amount of seat belt withdrawal based on the physique determination result output from the physique determination device 32. The seat control device adjusts, for example, the seat position or the backrest based on the physique determination result output from the physique determination device 32.

[0089] The operation of the information processing device 1 according to the first embodiment will be described. FIG. 11 is a flowchart for describing the operation of the information processing device 1 according to the first embodiment. For example, when the information processing device 1 receives an operation instruction, it performs the operation shown in the flowchart of FIG. 11. Note that a control unit (not shown) of the information processing device 1 receives the operation instruction, and when the control unit receives the operation instruction, it operates each component of the information processing device 1. The operation instruction is input, for example, by an administrator or the like. For example, the administrator or the like inputs the operation instruction when installing the detection sensor 2 in a vehicle.

[0090] The moving object distribution information acquisition unit 101 acquires moving object distribution information, i.e., a moving object map, generated by the radio wave sensor 21 from the detection sensor 2 (step ST1). The moving object distribution information acquisition unit 101 outputs the acquired moving object map to the correction unit 105.

[0091] The image acquisition unit 102 acquires the captured image captured by the camera 22 from the detection sensor 2 (step ST2). The image acquisition unit 102 outputs the acquired captured image to the camera error detection unit 103.

[0092] The camera error detection unit 103 detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102 in step ST2 (step ST3). The camera error detection unit 103 outputs camera installation error information to the radio wave sensor error calculation unit 104.

[0093] The radio wave sensor error calculation unit 104 calculates the installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103 in step ST3 (step ST4). The radio wave sensor error calculation unit 104 outputs the sensor installation error information to the correction unit 105.

[0094] The correction unit 105 corrects the moving object map acquired by the moving object distribution information acquisition unit 101 based on the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104 in step ST4 (step ST5). The correction unit 105 outputs the corrected moving object map to the output unit 106. Note that if correction is not necessary, in other words, if there is no installation error of the radio wave sensor 21, the correction unit 105 outputs the moving object map acquired by the moving object distribution information acquisition unit 101 to the output unit 106 as is as the corrected moving object map.

[0095] The output unit 106 outputs the corrected moving object map output from the correction unit 105 in step ST5, i.e., the moving object map corrected by the correction unit 105, as error consideration information to the monitoring device 3 (step ST6).

[0096] In the flowchart of Fig. 11, the process of step ST1 is performed before the process of step ST2, but this is merely an example. For example, in the flowchart of Fig. 11, the process of step ST1 and the process of step ST2 may be performed in parallel. It is sufficient that the process of step ST1 is performed before the process of step ST5 is performed.

[0097] As described above, the information processing device 1 according to the first embodiment acquires an image captured by the camera 22 from the detection sensor 2, and detects an installation error of the camera 22 based on the acquired image. When the information processing device 1 detects an installation error of the camera 22, it calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22. The information processing device 1 corrects the moving object distribution information generated by the radio wave sensor 21, i.e., the moving object map, acquired from the detection sensor 2, based on the calculated installation error of the radio wave sensor 21. The information processing device 1 then outputs the corrected moving object map to the monitoring device 3 as error consideration information. The monitoring device 3 monitors the interior of the vehicle based on the corrected moving object map output from the information processing device 1.

[0098] As described above, when the radio wave sensor 21 is installed in the vehicle cabin, it is generally fixed to the vehicle cabin via some structure, such as the ceiling or dashboard, between the sensor and the vehicle frame. Therefore, there is a possibility that an installation error occurs in the radio wave sensor 21 installed in the vehicle cabin. If an installation error occurs in the radio wave sensor 21, deviations also occur in the position and angle of a moving object detected by the radio wave sensor 21. In particular, the position of the body or head of an occupant seated far from the radio wave sensor 21 will be significantly deviated relative to the radio wave sensor 21, resulting in a large deviation in the position and angle of the occupant detected by the radio wave sensor 21. Deviations in the position and angle of a moving object detected by the radio wave sensor 21 have a significant impact on the monitoring results of the monitoring device 3, which monitors the occupants using a moving object map generated by the radio wave sensor 21.

[0099] 12A and 12B are diagrams illustrating an example of the effect on the monitoring device 3, which monitors occupants using a moving object map generated by the radio wave sensor 21, of a hypothetical deviation in the position and angle of a moving object detected by the radio wave sensor 21. Note that FIGS. 12A and 12B illustrate, as an example, a case in which a deviation in the orientation of the radio wave sensor 21 causes a deviation in the position and angle of a moving object detected by the radio wave sensor 21, thereby affecting the monitoring device 3. Here, the monitoring device 3 is a physique determination device 32, which uses a physique determination region to determine whether the occupant's physique is a "large adult," a "small adult," or a "child." In FIGS. 12A and 12B, the physique determination region is indicated by "S." In addition, in FIGS. 12A and 12B, the vehicle is indicated by "C," and the radio wave radiation range of the radio wave sensor 21 is indicated by "R." 12A and 12B are side views of the vehicle. Fig. 12A shows an example of the interior of the vehicle when the radio wave sensor 21 is installed at the reference sensor position so as to emit radio waves in the direction of the reference sensor. Fig. 12B shows an example of the interior of the vehicle when the radio wave sensor 21 is installed at the reference sensor position but is shifted upward from the reference sensor position.

[0100] As shown in FIG. 12B , if the orientation of the radio wave sensor 21 is displaced upward, the position and angle of the moving object detected by the radio wave sensor 21 are displaced upward accordingly. This is because the radio wave sensor 21 calculates the position and angle of the detected moving object assuming that the installation position and orientation of the radio wave sensor 21 are the reference sensor position and orientation, respectively. If the physique determination device 32 were to determine the occupant's physique using the moving object map generated by the radio wave sensor 21 in such a state, the physique determination device 32 could erroneously determine the occupant's physique. This is because the coordinates in the real space within the vehicle cabin corresponding to each grid on the moving object map are displaced by the above-mentioned displacement. If a physique determination region were set according to the coordinates in the real space within the vehicle cabin corresponding to each grid on the moving object map, the physique determination region would have to be set within the range shown in FIG. 12B . However, the physique determination area is actually set based on coordinates in the real space of the vehicle interior without any deviation (for example, as shown in FIG. 12A ). As a result, the physique determination device 32 may erroneously determine the physique of an occupant, for example, determining that the physique of an occupant who should be determined as a "child" is a "small adult."

[0101] Therefore, it is very important to detect installation errors of the radio wave sensor 21 and to enable the moving object map generated by the radio wave sensor 21 to be used as a moving object map obtained without the installation error of the radio wave sensor 21. To address this issue, as described above, the information processing device 1 according to embodiment 1 acquires images captured by the camera 22 from the detection sensor 2 and detects installation errors of the camera 22 based on the captured images. Upon detecting an installation error of the camera 22, the information processing device 1 calculates the installation error of the radio wave sensor 21 based on the installation error of the camera 22. The information processing device 1 corrects the moving object map obtained from the detection sensor 2 and generated by the radio wave sensor 21 based on the calculated installation error of the radio wave sensor 21. The information processing device 1 then outputs the corrected moving object map to the monitoring device 3 as error consideration information. This allows the information processing device 1 to provide error consideration information, specifically, the corrected moving object map, for using information on the moving object map generated by the radio wave sensor 21 as information obtained without the installation error of the radio wave sensor 21. The monitoring device 3 can monitor the occupants using the corrected moving object map provided by the information processing device 1, and as a result, the monitoring device 3 can prevent incorrect monitoring of the occupants (for example, false detection of a child being left behind or false determination of the occupant's physique).

[0102] In the first embodiment, the radio wave sensor 21 is mounted on the detection sensor 2, and the detection sensor 2 includes a fixed part 23, a camera 22 provided on the fixed part 23, and the radio wave sensor 21 provided on the fixed part 23. When the detection sensor 2 is installed in the vehicle cabin, at least two of the x-axis, y-axis, and z-axis specific to the camera 22 coincide with the x-axis, y-axis, and z-axis specific to the radio wave sensor 21. Alternatively, the detection sensor 2 includes the camera 22, the radio wave sensor 21, and a substrate 24 on which the transmitting antenna 11 and the receiving antenna 12 of the camera 22 and the radio wave sensor 21 are provided on a common surface. The information processing device 1 acquires an image captured by the camera 22 from the detection sensor 2, and detects an installation error of the camera 22 based on the captured image. When the information processing device 1 detects an installation error of the camera 22, it calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22. The information processing device 1 corrects the moving object map generated by the radio wave sensor 21, which is acquired from the detection sensor 2, based on the calculated installation error of the radio wave sensor 21. Then, the information processing device 1 outputs the corrected moving object map to the monitoring device 3 as error consideration information. This allows the information processing device 1 to provide error consideration information, specifically, the corrected moving object map, for using the information of the moving object map generated by the radio wave sensor 21 as information obtained in a state without the installation error of the radio wave sensor 21. The monitoring device 3 can monitor the occupants using the corrected moving object map provided by the information processing device 1, and as a result, the monitoring device 3 can prevent erroneous monitoring of the occupants (for example, erroneous detection of a child being left behind or erroneous determination of the occupant's physique).

[0103] As described above, a technique is generally known in which a marker whose exact direction or distance from the camera 22 is known is captured by the camera 22, a deviation from the position and direction of the image that should be captured in the captured image is detected, and the installation position and direction of the camera 22 are adjusted or the captured image is corrected. For example, with respect to the radio wave sensor 21, it is not impossible to place an object serving as a marker whose exact direction or distance from the radio wave sensor 21 is known, detect the object, detect a deviation from the position and direction of the object that should be detected, and adjust the installation position of the radio wave sensor 21 and the direction of radio wave emission. However, adjusting the radio wave sensor 21 in this manner within a vehicle cabin is not easy. For example, this is because it is difficult for the radio wave sensor 21 to distinguish between the marker and other structures within the vehicle cabin. Furthermore, the radio wave sensor 21 suffers from a so-called multipath problem in which the marker appears double due to the reflected radio waves from structures within the vehicle cabin. The information processing device 1 of embodiment 1 is able to use a moving object map as a moving object map obtained without any installation error of the radio wave sensor 21, without being affected by the difficulty of the radio wave sensor 21 in distinguishing between markers and structures other than the markers inside the vehicle cabin, or the so-called multipath problem that occurs in the radio wave sensor 21.

[0104] In the first embodiment, the information processing device 1 is provided outside the detection sensor 2, but this is merely an example. The information processing device 1 may be provided in the detection sensor 2.

[0105] In the first embodiment described above, the information processing device 1 is an in-vehicle device mounted on a vehicle, and the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) are provided in the in-vehicle device. However, the present invention is not limited to this. Some of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) may be mounted in the in-vehicle device of the vehicle, and the rest may be provided in a server connected to the in-vehicle device via a network, thereby constituting a system with the in-vehicle device and the server. In addition, the moving object distribution information acquisition unit 101, image acquisition unit 102, camera error detection unit 103, radio wave sensor error calculation unit 104, correction unit 105, output unit 106, and a control unit (not shown) may all be provided on the server.

[0106] 2, the detection sensor 2 is configured so that, when the detection sensor 2 is installed in the vehicle cabin, at least two of the x-axis, y-axis, and z-axis specific to the camera 22 coincide with the x-axis, y-axis, and z-axis specific to the radio wave sensor 21. However, this is merely an example, and it is not essential that, in the detection sensor 2, at least two of the x-axis, y-axis, and z-axis specific to the camera 22 coincide with the x-axis, y-axis, and z-axis specific to the radio wave sensor 21. However, since the detection sensor 2 is configured so that at least two of the x-axis, y-axis, and z-axis specific to the camera 22 coincide with the x-axis, y-axis, and z-axis specific to the radio wave sensor 21, the information processing device 1 can easily correct the moving object map compared to a configuration in which only one or none of the x-axis, y-axis, and z-axis specific to the camera 22 coincide with the x-axis, y-axis, and z-axis specific to the radio wave sensor 21.

[0107] In the first embodiment described above, when the detection sensor 2 has the configuration shown in Fig. 3, the camera 22 is configured to be provided between the transmitting antenna 11 and the receiving antenna 12 on the common surface of the substrate 24. However, this is merely an example, and it is not essential that the camera 22 be provided between the transmitting antenna 11 and the receiving antenna 12 on the common surface of the substrate 24 in the detection sensor 2. However, by configuring the detection sensor 2 so that the camera 22 is provided between the transmitting antenna 11 and the receiving antenna 12 on the common surface of the substrate 24, the information processing device 1 can more easily correct the moving object map compared to a configuration in which the camera 22 is not provided between the transmitting antenna 11 and the receiving antenna 12 on the common surface of the substrate 24.

[0108] 13A and 13B are diagrams illustrating an example of the hardware configuration of the information processing device 1 according to embodiment 1. In embodiment 1, the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) are realized by a processing circuit 1001. That is, the information processing device 1 includes a processing circuit 1001 for performing control to provide error consideration information for using the moving object map information generated by the radio wave sensor 21 as information obtained without installation error of the radio wave sensor 21. The processing circuit 1001 may be dedicated hardware as shown in FIG. 13A, or a processor 1004 that executes a program stored in memory as shown in FIG. 13B.

[0109] If the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0110] When the processing circuit is the processor 1004, the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005, thereby executing the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown). In other words, the information processing device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST6 of FIG. 11 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and the control unit (not shown). Here, the memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0111] Note that the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of the moving object distribution information acquisition unit 101 and the image acquisition unit 102 may be implemented by a processing circuit 1001 as dedicated hardware, while the functions of the camera error detection unit 103, the radio wave sensor error calculation unit 104, the correction unit 105, the output unit 106, and a control unit (not shown) may be implemented by a processor 1004 reading and executing programs stored in a memory 1005. The information processing device 1 also includes an input interface device 1002 and an output interface device 1003 that communicate with devices such as the detection sensor 2 or the monitoring device 3 via wired or wireless communication.

[0112] As described above, according to the first embodiment, the information processing device 1 is configured to include an image acquisition unit 102 that acquires an image captured by the camera 22 from a detection sensor 2 having the camera 22 that is provided within the vehicle cabin and captures an image of a target area within the vehicle cabin where at least an occupant may be present, and the radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of areas within the vehicle cabin where moving objects are present, based on radio waves radiated toward the target area within the vehicle cabin and reflected by objects within the vehicle cabin; a camera error detection unit 103 that detects an installation error of the camera 22 based on the image acquired by the image acquisition unit 102; a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103; and an output unit 106 that outputs error consideration information to convert the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. Therefore, the information processing device 1 can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects of the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0113] In detail, the information processing device 1 includes a moving object distribution information acquisition unit 101 that acquires moving object distribution information from the detection sensor 2, and a correction unit 105 that corrects the moving object distribution information acquired by the moving object distribution information acquisition unit 101 based on the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and the output unit 106 is configured to output the moving object distribution information (corrected moving object map) corrected by the correction unit 105 as error consideration information. Therefore, the information processing device 1 can provide error consideration information for using information generated based on the reflected waves received by the radio wave sensor 21, which is provided in the vehicle cabin and is capable of detecting the position and movement of moving objects in the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by objects, i.e., moving object distribution information, as information obtained without the installation error of the radio wave sensor 21.

[0114] Furthermore, as described above, according to the first embodiment, the detection sensor 2 is provided in a vehicle interior and includes a fixed part 23, a camera 22 provided on the fixed part 23 and configured to capture an image of a target area in the vehicle interior where at least an occupant may be present, a radio wave sensor 21 provided on the fixed part 23 and configured to generate moving object distribution information that three-dimensionally represents a distribution of an area where moving objects exist in the vehicle interior based on radio waves that are emitted toward the target area in the vehicle interior and reflected by an object in the vehicle interior, an image acquisition unit 102 that acquires the captured image captured by the camera 22, and a camera error detection unit 103 that detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102. The detection sensor 2 includes a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103, and an output unit 106 that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and can be configured so that when the detection sensor 2 is installed in the vehicle cabin, at least two of the three-dimensional coordinate axes of the camera 22 (x-axis, y-axis, and z-axis specific to the camera 22) and the three-dimensional coordinate axes of the radio wave sensor 21 (x-axis, y-axis, and z-axis specific to the radio wave sensor 21) coincide. Therefore, the detection sensor 2 can provide error consideration information that allows the information generated by the radio wave sensor 21, i.e., the moving object distribution information, to be used as information obtained without the installation error of the radio wave sensor 21.

[0115] Furthermore, as described above, according to the first embodiment, the detection sensor 2 is provided in the vehicle interior and includes a camera 22 that captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on radio waves that are radiated toward the target area in the vehicle interior and reflected by objects in the vehicle interior, the radio wave sensor 21 having a transmitting antenna 11 that radiates radio waves and a receiving antenna 12 that receives reflected waves of the radio waves radiated from the transmitting antenna 11, and the camera 22, the transmitting antenna 11, and the receiving antenna 12. 2 are provided on a common surface, an image acquisition unit 102 that acquires an image captured by the camera 22, a camera error detection unit 103 that detects an installation error of the camera 22 based on the image captured by the image acquisition unit 102, a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103, and an output unit 106 that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. Therefore, the detection sensor 2 can provide error consideration information that allows the information generated by the radio wave sensor 21, i.e., the moving object distribution information, to be used as information obtained without the installation error of the radio wave sensor 21.

[0116] Furthermore, according to the first embodiment, the information processing system 4 includes a detection sensor 2 having a camera 22 that is provided in the vehicle interior and captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on a radio wave radiated toward the target area in the vehicle interior and reflected by an object in the vehicle interior, and an image acquisition unit 102 that acquires the captured image captured by the camera 22, a camera error detection unit 103 that detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102, and a radio wave sensor error detection unit 103 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103. The information processing device 1 has a calculation unit 104 and an output unit 106 that outputs error consideration information to convert moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and a detection sensor 2 that is installed in a vehicle cabin and has a fixed part 23, a camera 22 installed on the fixed part 23, and a radio wave sensor 21 installed on the fixed part 23, and is characterized in that when installed in the vehicle cabin, at least two of the three-dimensional coordinate axes of the camera 22 (x-axis, y-axis, and z-axis specific to the camera 22) and the three-dimensional coordinate axes of the radio wave sensor 21 (x-axis, y-axis, and z-axis specific to the radio wave sensor 21) are coincident. Therefore, the information processing system 4 can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects that reflect the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0117] Furthermore, according to the first embodiment, the information processing system 4 is provided in a vehicle interior, and includes a camera 22 that captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on a radio wave that is emitted toward the target area in the vehicle interior and reflected by an object in the vehicle interior. The information processing system 4 includes an image acquisition unit 102 that acquires an image captured by the camera 22 from the detection sensor 2, a camera error detection unit 103 that detects an installation error of the camera 22 based on the image acquired by the image acquisition unit 102, and a detection unit 104 that detects an installation error of the camera 22 detected by the camera error detection unit 103. and an output unit 106 that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and a detection sensor 2 that is provided in a vehicle cabin and has a camera 22, a transmitting antenna 11 that radiates radio waves, and a receiving antenna 12 that receives reflected waves of the radio waves radiated from the transmitting antenna 11, and a substrate 24 on which the camera 22, the transmitting antenna 11, and the receiving antenna 12 are provided on a common surface. Therefore, the information processing system 4 can provide error consideration information for using information generated based on reflected waves received by the radio wave sensor 21 that is provided in the vehicle cabin and that is capable of detecting the position and movement of moving objects in the vehicle cabin by radiating radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by an object, i.e., moving object distribution information, as information obtained without the installation error of the radio wave sensor 21.

[0118] Embodiment 2 In Embodiment 1, the information processing device has a function of correcting the moving object map and outputs the corrected moving object map as error consideration information. In Embodiment 2, an embodiment will be described in which the information processing device does not have a function of correcting the moving object map.

[0119] 14 is a diagram showing a configuration example of an information processing device 1a according to embodiment 2. In embodiment 2, the information processing device 1a is connected to a detection sensor 2 and a monitoring device 3. The information processing device 1a and the detection sensor 2 constitute an information processing system 4a. In embodiment 2, the information processing device 1a, the detection sensor 2, and the monitoring device 3 are mounted on, for example, a vehicle (not shown).

[0120] The detection sensor 2 according to the second embodiment has a configuration similar to that of the detection sensor 2 according to the first embodiment described with reference to FIGS. 2 and 3 in the first embodiment. Furthermore, the monitoring device 3 includes an occupant detection device 31 and a physique determination device 32, similar to the monitoring device 3 described with reference to FIG. 1 in the first embodiment. In FIG. 14 , components similar to those of the information processing device 1 described with reference to FIG. 1 in the first embodiment are denoted by the same reference numerals, and redundant description will be omitted. The information processing device 1a according to the second embodiment differs from the information processing device 1 according to the first embodiment in that it does not include the correction unit 105. Furthermore, the specific operation of the output unit 106a in the information processing device 1a according to the second embodiment differs from the specific operation of the output unit 106 in the information processing device 1 according to the first embodiment.

[0121] In the second embodiment, the output unit 106a outputs the moving object distribution information acquired by the moving object distribution information acquisition unit 101 and the sensor installation error information indicating the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104 as error consideration information to the monitoring device 3. That is, the output unit 106a associates the moving object distribution information output from the moving object distribution information acquisition unit 101 with the sensor installation error information output from the radio wave sensor error calculation unit 104, and outputs the associated information as error consideration information to the monitoring device 3. In the second embodiment, the moving object distribution information acquisition unit 101 outputs the acquired moving object distribution information to the output unit 106a. In the second embodiment, the radio wave sensor error calculation unit 104 outputs the sensor installation error information to the output unit 106a.

[0122] In the second embodiment, the information processing device 1a does not have a function for correcting the moving object map. Therefore, the moving object map included in the error consideration information output by the information processing device 1a to the monitoring device 3 may be a moving object map generated by the radio wave sensor 21 when the radio wave sensor 21 has an installation error. Therefore, in the second embodiment, when monitoring an occupant, the monitoring device 3 performs a process (hereinafter referred to as a "monitoring correction process") based on the sensor installation error information included in the error consideration information to enable the moving object map to be used as a moving object map generated when the radio wave sensor 21 has no installation error, and then monitors the occupant. An example of the monitoring correction process is a process for correcting a physique determination area. For example, the occupant detection device 31 and the physique determination device 32 correct the position or orientation of the physique determination area to account for the installation error of the radio wave sensor 21 based on the sensor installation error information. This allows the monitoring device 3 to monitor occupants using the moving object map generated by the radio wave sensor 21 provided by the information processing device 1a as a moving object map generated without any installation error of the radio wave sensor 21, and as a result, the monitoring device 3 can prevent incorrect monitoring of occupants (for example, false detection of a child being left behind or false determination of the physique of an occupant).

[0123] The operation of the information processing device 1a according to the second embodiment will be described. FIG. 15 is a flowchart for describing the operation of the information processing device 1a according to the second embodiment. For example, when the information processing device 1a receives an operation instruction, it performs the operation shown in the flowchart of FIG. 15. Note that a control unit (not shown) of the information processing device 1a receives the operation instruction, and when the control unit receives the operation instruction, it operates each component of the information processing device 1a. The operation instruction is input, for example, by an administrator or the like. The administrator or the like inputs the operation instruction, for example, when installing the detection sensor 2 in the vehicle.

[0124] In the flowchart of Fig. 15, the specific contents of the processes of steps ST11 to ST14 are the same as the specific contents of the processes of steps ST1 to ST4 in the flowchart of Fig. 11 already explained in embodiment 1, and therefore redundant explanations will be omitted. However, in step ST11, the moving object distribution information acquisition unit 101 outputs the acquired moving object map to the output unit 106a. Also, in step ST14, the radio wave sensor error calculation unit 104 outputs sensor installation error information to the output unit 106a.

[0125] The output unit 106a outputs the moving body distribution information acquired by the moving body distribution information acquisition unit 101 in step ST11 and the sensor installation error information indicating the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104 in step ST14 as error consideration information to the monitoring device 3 (step ST15).

[0126] In the flowchart of Fig. 15, the process of step ST11 is performed before the process of step ST12, but this is merely an example. For example, in the flowchart of Fig. 15, the process of step ST11 and the process of step ST12 may be performed in parallel. It is sufficient that the process of step ST11 is performed before the process of step ST15 is performed.

[0127] In this way, the information processing device 1a according to the second embodiment acquires, from the detection sensor 2, an image captured by the camera 22, and detects an installation error of the camera 22 based on the acquired image. When the information processing device 1a detects an installation error of the camera 22, it calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22. The information processing device 1a outputs, as error consideration information, the moving object distribution information generated by the radio wave sensor 21 acquired from the detection sensor 2, i.e., the moving object map, and sensor installation error information indicating the calculated installation error of the radio wave sensor 21, to the monitoring device 3. In this way, the information processing device 1a can provide error consideration information, specifically, the moving object map and sensor installation error information, for using the information of the moving object map generated by the radio wave sensor 21 as information obtained in a state where there was no installation error of the radio wave sensor 21. Based on the moving object map and sensor installation error information provided by the information processing device 1a, the monitoring device 3 performs a monitoring correction process to enable the moving object map to be used as a moving object map generated without installation error of the radio wave sensor 21, and then monitors the occupants. As a result, the monitoring device 3 can prevent erroneous monitoring of occupants (for example, erroneous detection of a child being left behind or erroneous determination of the physique of an occupant).

[0128] In the second embodiment, the information processing device 1a is provided outside the detection sensor 2, but this is merely an example. The information processing device 1a may be provided in the detection sensor 2.

[0129] In the second embodiment described above, the information processing device 1a is an in-vehicle device mounted on a vehicle, and the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown) are provided in the in-vehicle device. However, the present invention is not limited to this. A system may be configured by the in-vehicle device and the server, with some of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown) being mounted in the in-vehicle device of the vehicle, and the rest being provided in a server connected to the in-vehicle device via a network. Furthermore, the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown) all being provided in the server.

[0130] The hardware configuration of the information processing device 1a according to embodiment 2 is the same as the hardware configuration of the information processing device 1 described in embodiment 1 using Figures 13A and 13B, and is therefore not shown in the figures. In embodiment 2, the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown) are realized by a processing circuit 1001. That is, the information processing device 1a includes a processing circuit 1001 for performing control to provide error consideration information for using moving object map information generated by the radio wave sensor 21 as information obtained without installation error of the radio wave sensor 21. The processing circuit 1001 may be dedicated hardware as shown in Figure 13A, or a processor 1004 that executes a program stored in a memory 1005 as shown in Figure 13B.

[0131] The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown). That is, the information processing device 1a includes a memory 1005 for storing a program that, when executed by the processing circuit 1001, results in the execution of steps ST11 to ST15 of FIG. 15 . The program stored in the memory 1005 can also be said to cause a computer to execute the processing procedures or methods of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106a, and a control unit (not shown). The information processing device 1a also includes an input interface device 1002 and an output interface device 1003 that communicate via wired or wireless communication with devices such as the detection sensor 2 or the monitoring device 3.

[0132] As described above, according to the second embodiment, the information processing device 1a is configured to include: an image acquisition unit 102 that acquires an image captured by the camera 22 from a detection sensor 2 that has the camera 22 that is provided in the vehicle cabin and captures an image of a target area in the vehicle cabin where at least an occupant may be present; and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of areas in the vehicle cabin where moving objects are present, based on radio waves that are emitted toward the target area in the vehicle cabin and reflected by objects in the vehicle cabin; a camera error detection unit 103 that detects an installation error of the camera 22 based on the image acquired by the image acquisition unit 102; a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103; and an output unit 106a that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. Therefore, the information processing device 1a can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects of the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0133] In detail, the information processing device 1a includes a moving object distribution information acquisition unit 101 that acquires moving object distribution information from the detection sensor 2, and the output unit 106a outputs, as error consideration information, the moving object distribution information acquired by the moving object distribution information acquisition unit 101 and sensor installation error information that indicates an installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. Therefore, the information processing device 1a can provide error consideration information for using information generated based on reflected waves received by the radio wave sensor 21, which is provided in the vehicle cabin and is capable of detecting the position and movement of moving objects in the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by objects, i.e., moving object distribution information, as information obtained without the installation error of the radio wave sensor 21.

[0134] Furthermore, according to the second embodiment, the information processing system 4 a includes a detection sensor 2 that is provided in the vehicle interior and has a camera 22 that captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on a radio wave radiated toward the target area in the vehicle interior and reflected by an object in the vehicle interior, and an image acquisition unit 102 that acquires the captured image captured by the camera 22, a camera error detection unit 103 that detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102, and a radio wave sensor error detection unit 103 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103. The information processing device 1a has a calculation unit 104 and an output unit 106a that outputs error consideration information for converting moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104; and a detection sensor 2 that is installed in a vehicle cabin and has a fixed part 23, a camera 22 installed on the fixed part 23, and a radio wave sensor 21 installed on the fixed part 23, wherein when installed in the vehicle cabin, at least two of the three-dimensional coordinate axes of the camera 22 (x-axis, y-axis, and z-axis specific to the camera 22) and the three-dimensional coordinate axes of the radio wave sensor 21 (x-axis, y-axis, and z-axis specific to the radio wave sensor 21) are coincident. Therefore, the information processing system 4a can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects of the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0135] Furthermore, according to the second embodiment, the information processing system 4 a is provided in the vehicle interior, and includes a camera 22 that captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on a radio wave that is emitted toward the target area in the vehicle interior and reflected by an object in the vehicle interior. The information processing system 4 a includes an image acquisition unit 102 that acquires an image captured by the camera 22 from the detection sensor 2, a camera error detection unit 103 that detects an installation error of the camera 22 based on the image captured by the image acquisition unit 102, and a detection unit 104 that detects an installation error of the camera 22 based on the installation error of the camera 22 detected by the camera error detection unit 103. and an information processing device 1a having a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the above-mentioned error, and an output unit 106a that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and a detection sensor 2 that is provided in a vehicle cabin and has a camera 22, a transmitting antenna 11 that radiates radio waves, and a receiving antenna 12 that receives reflected waves of the radio waves radiated from the transmitting antenna 11, and a substrate 24 on which the camera 22, the transmitting antenna 11, and the receiving antenna 12 are provided on a common surface. Therefore, the information processing system 4a can provide error consideration information for using information generated based on reflected waves received by the radio wave sensor 21 that is provided in the vehicle cabin and is capable of detecting the position and movement of moving objects in the vehicle cabin by radiating radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by an object, i.e., moving object distribution information, as information obtained without the installation error of the radio wave sensor 21.

[0136] Embodiment 3 In embodiment 1, the information processing device has a function of correcting the moving object map and outputs the corrected moving object map as error consideration information. In embodiment 3, another embodiment will be described in which the information processing device does not have a function of correcting the moving object map.

[0137] 16 is a diagram showing a configuration example of an information processing device 1b according to embodiment 3. In embodiment 3, the information processing device 1b is connected to a detection sensor 2a and a monitoring device 3. The information processing device 1b and the detection sensor 2a constitute an information processing system 4b. In embodiment 3, the information processing device 1b, the detection sensor 2a, and the monitoring device 3 are mounted on, for example, a vehicle (not shown).

[0138] The detection sensor 2a according to the second embodiment includes the same configuration as the detection sensor 2 according to the first embodiment described with reference to FIGS. 2 and 3 , but also includes a control unit 25 that changes the orientation of the radio wave sensor 21, i.e., the direction of radio wave radiation from the radio wave sensor 21, based on instruction information output from the information processing device 1b. Details of the instruction information will be described later. In the third embodiment, the detection sensor 2a includes, for example, a servo motor (not shown) that changes the orientation of the radio wave sensor 21. The control unit 25 changes the orientation of the radio wave sensor 21 by controlling the servo motor based on the instruction information output from the information processing device 1b. Furthermore, for example, the radio wave sensor 21 in the detection sensor 2a may be a radio wave sensor 21 that can change the direction of radio wave radiation by applying signals of different phases to multiple antennas, such as a phased array antenna. In this case, the control unit 25 instructs the radio wave sensor 21 to change the direction of radio wave radiation based on the instruction information output from the information processing device 1b. The control unit 25 corrects the installation error of the radio wave sensor 21 by changing the orientation of the radio wave sensor 21 based on the instruction information output from the information processing device 1b. As a result, the moving object map generated by the radio wave sensor 21 without any installation error of the radio wave sensor 21 is output from the detection sensor 2a to the information processing device 1b.

[0139] In the third embodiment, the monitoring device 3 includes an occupant detection device 31 and a physique determination device 32, similar to the monitoring device 3 described in the first embodiment.

[0140] 16, components similar to those of the information processing device 1 described in accordance with Embodiment 1 are denoted by the same reference numerals, and redundant description will be omitted. The information processing device 1b according to Embodiment 3 differs from the information processing device 1 according to Embodiment 1 in that it does not include a correction unit 105. Furthermore, in the information processing device 1b according to Embodiment 3, the specific operation of the output unit 106b differs from the specific operation of the output unit 106 in the information processing device 1 according to Embodiment 1.

[0141] In the third embodiment, the output unit 106b outputs, as error consideration information, instruction information instructing the detection sensor 2a to change the orientation of the radio wave sensor 21, based on the sensor installation error information output from the radio wave sensor error calculation unit 104. The instruction information includes information that makes it possible to determine the installation error of the radio wave sensor 21. Furthermore, the output unit 106b outputs the moving object distribution information acquired by the moving object distribution information acquisition unit 101 to the monitoring device 3. Note that in the third embodiment, the moving object distribution information acquisition unit 101 outputs the acquired moving object distribution information to the output unit 106b. Also in the third embodiment, the radio wave sensor error calculation unit 104 outputs the sensor installation error information to the output unit 106b.

[0142] In the third embodiment, the information processing device 1b does not have a function for correcting the moving object map. Instead, the information processing device 1b informs the detection sensor 2a of how much the installation position or orientation of the radio wave sensor 21 needs to be changed so that the radio wave sensor 21 is in a state free of installation error, and outputs instruction information for correcting the installation error of the radio wave sensor 21. This enables the information processing device 1b to acquire the moving object map generated by the radio wave sensor 21 as a moving object map obtained without installation error of the radio wave sensor 21. As a result, the moving object map output by the information processing device 1b to the monitoring device 3 is also a moving object map obtained without installation error of the radio wave sensor 21. Therefore, the monitoring device 3 can prevent erroneous monitoring of occupants (for example, erroneous detection of a child being left behind or erroneous determination of the occupant's physique).

[0143] The operation of the information processing device 1b according to the third embodiment will be described. FIG. 17 is a flowchart for describing the operation of the information processing device 1b according to the third embodiment. For example, when the information processing device 1b receives an operation instruction, the information processing device 1b performs the operation shown in the flowchart of FIG. 17. Note that a control unit (not shown) of the information processing device 1b receives the operation instruction, and when the control unit receives the operation instruction, it operates each component of the information processing device 1b. The operation instruction is input, for example, by an administrator or the like. The administrator or the like inputs the operation instruction, for example, when installing the detection sensor 2a in a vehicle.

[0144] In the flowchart of Fig. 17, the specific contents of the processes in steps ST111 to ST114 are the same as the specific contents of the processes in steps ST1 to ST4 in the flowchart of Fig. 11, which have been explained in embodiment 1, and therefore redundant explanations will be omitted. However, in step ST111, the moving object distribution information acquisition unit 101 outputs the acquired moving object map to the output unit 106b. Also, in step ST114, the radio wave sensor error calculation unit 104 outputs sensor installation error information to the output unit 106b.

[0145] The output unit 106b outputs, as error consideration information, instruction information instructing the detection sensor 2a to change the orientation of the radio wave sensor 21 based on the sensor installation error information indicating the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104 in step ST114 (step ST115-1).The output unit 106b also outputs the moving object distribution information acquired by the moving object distribution information acquisition unit 101 in step ST111, i.e., the moving object map, to the monitoring device 3 (step ST115-2).

[0146] In the flowchart of FIG. 17, the processing of step ST111 is performed before the processing of step ST112, but this is merely an example. For example, in the flowchart of FIG. 17, the processing of step ST111 and the processing of step ST112 may be performed in parallel. The processing of step ST111 only needs to be performed before the processing of step ST115-2 is performed. In addition, in the flowchart of FIG. 17, the processing of step ST115-1 and the processing of step ST115-2 are performed in parallel, but this is merely an example. For example, in the flowchart of FIG. 17, the processing of step ST115-1 and the processing of step ST115-2 may be performed in succession.

[0147] 17 , for example, the output unit 106b in the information processing device 1b may output instruction information to the detection sensor 2a, and then output the motion object map generated after the detection sensor 2a corrects the installation error of the radio wave sensor 21, and then output the motion object map to the monitoring device 3. In this case, for example, when the control unit 25 corrects the installation error of the radio wave sensor 21 in the detection sensor 2a, the control unit 25 outputs completion information indicating that the correction has been completed to the output unit 106b of the information processing device 1b. After outputting the instruction information to the detection sensor 2a, the output unit 106b suspends output of the motion object map to the monitoring device 3 until it acquires completion information from the detection sensor 2a. Once it acquires the completion information from the detection sensor 2a, the output unit 106b starts outputting the motion object map to the monitoring device 3. This allows the information processing device 1b to further reduce the occurrence of erroneous occupant monitoring (e.g., erroneous detection of a child being left behind or erroneous determination of the occupant's physique) by the monitoring device 3.

[0148] In this way, the information processing device 1b according to the third embodiment acquires an image captured by the camera 22 from the detection sensor 2a and detects an installation error of the camera 22 based on the captured image. When the information processing device 1a detects an installation error of the camera 22, it calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22. Based on the calculated installation error of the radio wave sensor 21, the information processing device 1b outputs instruction information, as error consideration information, to the detection sensor 2a, instructing the detection sensor 2a to change the direction of radio wave emission by the radio wave sensor 21. This allows the information processing device 1b to provide error consideration information for using information on the moving object map generated by the radio wave sensor 21 as information obtained without the installation error of the radio wave sensor 21, specifically, instruction information instructing the detection sensor 2a to change the direction of radio waves. The monitoring device 3 can monitor occupants using the moving object map generated without the installation error of the radio wave sensor 21, provided by the information processing device 1b. As a result, the monitoring device 3 can prevent erroneous monitoring of occupants (for example, erroneous detection of a child being left behind or erroneous determination of the occupant's physique).

[0149] In the third embodiment, the information processing device 1b is provided outside the detection sensor 2a, but this is merely an example. The information processing device 1b may be provided in the detection sensor 2a.

[0150] In the above third embodiment, the information processing device 1b is an in-vehicle device mounted on a vehicle, and the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown) are provided in the in-vehicle device. However, the present invention is not limited to this. A system may be configured by the in-vehicle device and the server, with some of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown) being mounted in the in-vehicle device of the vehicle, and the rest being provided in a server connected to the in-vehicle device via a network. Furthermore, the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown) all being provided in the server.

[0151] Furthermore, in the above-described third embodiment, the monitoring device 3 acquires the moving object map generated by the radio wave sensor 21 via the information processing device 1b, but this is merely an example. For example, the monitoring device 3 may acquire the moving object map directly from the detection sensor 2a. In this case, the output unit 106b in the information processing device 1b does not output the moving object map to the monitoring device 3. Furthermore, the information processing device 1b may be configured without the moving object distribution information acquisition unit 101. If the information processing device 1b is configured without the moving object distribution information acquisition unit 101, the processing of step ST111 and the processing of step ST115-2 can be omitted in the operation of the information processing device 1b described using the flowchart of FIG. 17.

[0152] The hardware configuration of the information processing device 1b according to embodiment 3 is the same as the hardware configuration of the information processing device 1 described in embodiment 1 using Figures 13A and 13B, and therefore is not illustrated. In embodiment 3, the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown) are realized by a processing circuit 1001. That is, the information processing device 1b includes a processing circuit 1001 for performing control to provide error consideration information for using the moving object map information generated by the radio wave sensor 21 as information obtained without installation error of the radio wave sensor 21. The processing circuit 1001 may be dedicated hardware as shown in Figure 13A, or a processor 1004 that executes a program stored in a memory 1005 as shown in Figure 13B.

[0153] The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown). That is, the information processing device 1a includes a memory 1005 for storing a program that, when executed by the processing circuit 1001, results in the execution of steps ST111 to ST115-1 and ST115-2 in FIG. 17 described above. The program stored in the memory 1005 can also be said to cause a computer to execute the processing procedures or methods of the moving object distribution information acquisition unit 101, the image acquisition unit 102, the camera error detection unit 103, the radio wave sensor error calculation unit 104, the output unit 106b, and a control unit (not shown). The information processing device 1b includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the detection sensor 2a or the monitoring device 3.

[0154] As described above, according to the third embodiment, the information processing device 1b is configured to include an image acquisition unit 102 that acquires an image captured by the camera 22 from the detection sensor 2a, which has the camera 22 that is provided in the vehicle cabin and captures an image of a target area in the vehicle cabin where at least an occupant may be present, and the radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of areas in the vehicle cabin where moving objects are present, based on radio waves that are emitted toward the target area in the vehicle cabin and reflected by objects in the vehicle cabin; a camera error detection unit 103 that detects an installation error of the camera 22 based on the image acquired by the image acquisition unit 102; a radio wave sensor error calculation unit 104 that calculates an installation error of the radio wave sensor 21 based on the installation error of the camera 22 detected by the camera error detection unit 103; and an output unit 106b that outputs error consideration information to convert the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104. Therefore, the information processing device 1b can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects that reflect the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0155] Specifically, in the information processing device 1b, the output unit 106b outputs, as error consideration information, instruction information to the detection sensor 2a that instructs the detection sensor 2a to change the direction of radio wave emission by the radio wave sensor 21. Therefore, the information processing device 1b can provide error consideration information for using information generated based on the reflected waves received by the radio wave sensor 21, which is provided in the vehicle cabin and is capable of detecting the position and movement of moving objects in the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by objects, i.e., moving object distribution information, as information obtained without an installation error of the radio wave sensor 21.

[0156] Furthermore, according to the third embodiment, the information processing system 4b is provided with a detection sensor 2a including a camera 22 that captures an image of a target area in the vehicle cabin where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle cabin based on a radio wave radiated toward the target area in the vehicle cabin and reflected by an object in the vehicle cabin, and an image acquisition unit 102 that acquires the captured image captured by the camera 22, a camera error detection unit 103 that detects an installation error of the camera 22 based on the captured image acquired by the image acquisition unit 102, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle cabin based on a radio wave radiated toward the target area in the vehicle cabin and reflected by an object in the vehicle cabin. The information processing device 1b has a radio wave sensor error calculation unit 104 that calculates an installation error, and an output unit 106b that outputs error consideration information to convert moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104; and a detection sensor 2a that is installed in the vehicle cabin and has a fixed part 23, a camera 22 installed on the fixed part 23, and a radio wave sensor 21 installed on the fixed part 23, and is characterized in that when installed in the vehicle cabin, at least two of the positions of the camera 22 and the radio wave sensor 21 in the width direction, height direction, or depth direction in the three-dimensional space within the vehicle cabin coincide. Therefore, the information processing system 4b can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects that reflect the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0157] Furthermore, according to the third embodiment, the information processing system 4b is provided with a camera 22 that is provided in the vehicle interior and captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor 21 that generates moving object distribution information (moving object map) that three-dimensionally represents the distribution of an area where moving objects exist in the vehicle interior based on a radio wave that is emitted toward the target area in the vehicle interior and reflected by an object in the vehicle interior. The information processing system 4b includes an image acquisition unit 102 that acquires an image captured by the camera 22 from the detection sensor 2a, a camera error detection unit 103 that detects an installation error of the camera 22 based on the image acquired by the image acquisition unit 102, and a detection unit 2103 that detects an installation error of the camera 22 based on the installation error of the camera 22 detected by the camera error detection unit 103. Based on this, the information processing device 1b is configured to include a radio wave sensor error calculation unit 104 that calculates the installation error of the radio wave sensor 21 and an output unit 106b that outputs error consideration information to convert the moving object distribution information generated by the radio wave sensor 21 into moving object distribution information that takes into account the installation error of the radio wave sensor 21 calculated by the radio wave sensor error calculation unit 104, and a detection sensor 2a that is provided in the vehicle cabin and has a radio wave sensor 21 that has a camera 22, a transmitting antenna 11 that radiates radio waves, and a receiving antenna 12 that receives reflected waves of the radio waves radiated from the transmitting antenna 11, and a substrate 24 on which the camera 22, the transmitting antenna 11, and the receiving antenna 12 are provided on a common surface. Therefore, the information processing system 4b can provide error-taking information that allows the information generated based on the reflected waves received by the radio wave sensor 21, which is installed in the vehicle cabin and can detect the position and movement of moving objects within the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves from objects that reflect the radiated radio waves, i.e., moving object distribution information, to be used as information obtained without any installation error of the radio wave sensor 21.

[0158] Specifically, in the information processing system 4b, the output unit 106b of the information processing device 1b outputs instruction information to the detection sensor 2a instructing the radio wave sensor 21 to change the direction of radio wave emission, and the detection sensor 2a is configured to include a control unit 25 that changes the direction of radio wave emission from the radio wave sensor 21 based on the instruction information output from the output unit 106b. Therefore, the information processing system 4b can provide error consideration information for using information generated based on the reflected waves received by the radio wave sensor 21, which is provided in the vehicle cabin and is capable of detecting the position and movement of a moving object in the vehicle cabin by emitting radio waves into the vehicle cabin and receiving reflected waves of the radiated radio waves reflected by an object, i.e., moving object distribution information, as information obtained without any installation error of the radio wave sensor 21.

[0159] In the first to third embodiments described above, the occupant detection device 31 or the physique determination device 32 may be configured to change the threshold value used to determine the moving body, in other words, the physique of the occupant, depending on the installation error of the radio wave sensor 21. Specifically, the information processing device 1, 1a, or 1b includes information capable of identifying the installation error of the radio wave sensor 21 in the error consideration information provided to the occupant detection device 31 or the physique determination device 32. The occupant detection device 31 or the physique determination device 32 identifies the installation error of the radio wave sensor 21 from the information capable of identifying the installation error of the radio wave sensor 21 output from the information processing device 1, 1a, or 1b, and changes the threshold value used to determine the occupant's physique, for example, by increasing or decreasing it, depending on the installation error. This allows the occupant detection device 31 or the physique determination device 32 to determine the occupant's physique, taking into account the moving body map originally output from the radio wave sensor 21 that includes an error. More specifically, if there is an installation error in the radio wave sensor 21, the characteristics of the moving object reflection change, and therefore the occupant detection device 31 or the physique determination device 32 can determine the occupant's physique while taking into account the fact that errors also occur in the information associated with the grid on the moving object map.

[0160] In the above first to third embodiments, the monitoring device 3 includes the occupant detection device 31 and the physique determination device 32, but this is merely an example. The monitoring device 3 may include either the occupant detection device 31 or the physique determination device 32. In the above first to third embodiments, the information processing devices 1, 1a, and 1b can also provide error consideration information to devices other than the monitoring device 3.

[0161] Furthermore, the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0162] The information processing device of the present disclosure can provide error-taking information that allows information generated based on the reflected waves received by a radio wave sensor that is installed in the vehicle cabin, emits radio waves into the vehicle cabin, and receives reflected waves from objects to detect the position and movement of moving objects in the vehicle cabin, as information obtained without any installation error of the radio wave sensor.

[0163] 1, 1a, 1b Information processing device, 101 Moving object distribution information acquisition unit, 102 Image acquisition unit, 103 Camera error detection unit, 104 Radio wave sensor error calculation unit, 105 Correction unit, 106, 106a, 106b Output unit, 2, 2a Detection sensor, 21 Radio wave sensor, 22 Camera, 23 Fixed part, 24 Board, 25 Control unit, 3 Monitoring device, 31 Occupant detection device, 32 Physical size determination device, 4, 4a, 4b Information processing system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. an image acquisition unit that acquires an image captured by the camera from a detection sensor having a camera that is provided in a vehicle interior and captures an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor that generates moving object distribution information that represents a three-dimensional distribution of an area in the vehicle interior where moving objects are present, based on a radio wave that is emitted toward the target area in the vehicle interior and reflected by an object in the vehicle interior; a camera error detection unit that detects an installation error of the camera based on the captured image acquired by the image acquisition unit; a radio wave sensor error calculation unit that calculates the installation error of the radio wave sensor based on the installation error of the camera detected by the camera error detection unit; an output unit that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor into moving object distribution information in which the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit is taken into consideration; An information processing device comprising:

2. a moving object distribution information acquisition unit that acquires the moving object distribution information from the detection sensor; a correction unit that corrects the moving object distribution information acquired by the moving object distribution information acquisition unit based on the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit, The output unit outputs the moving object distribution information corrected by the correction unit as the error consideration information.

2. The information processing apparatus according to claim 1,

3. a moving object distribution information acquisition unit that acquires the moving object distribution information from the detection sensor, The output unit outputs, as the error consideration information, the moving object distribution information acquired by the moving object distribution information acquisition unit and sensor installation error information indicating the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit.

2. The information processing apparatus according to claim 1,

4. The output unit outputs, as the error consideration information, instruction information for instructing the detection sensor to change a direction of radiation of the radio wave by the radio wave sensor.

2. The information processing apparatus according to claim 1,

5. The installation error of the camera includes an error in the installation position of the camera or an error in the orientation of the camera, The installation error of the radio wave sensor includes an error in the installation position of the radio wave sensor or an error in the direction of radiation of the radio wave by the radio wave sensor.

2. The information processing apparatus according to claim 1,

6. Located inside the vehicle cabin, A fixing part; a camera that is provided on the fixed part and captures an image of a target area in the vehicle interior where at least an occupant may be present; a radio wave sensor that is provided on the fixed part and that generates moving object distribution information that represents a three-dimensional distribution of an area in which moving objects exist within the vehicle cabin based on a radio wave that is emitted toward the target area within the vehicle cabin and reflected by an object within the vehicle cabin; an image acquisition unit that acquires an image captured by the camera; a camera error detection unit that detects an installation error of the camera based on the captured image acquired by the image acquisition unit; a radio wave sensor error calculation unit that calculates the installation error of the radio wave sensor based on the installation error of the camera detected by the camera error detection unit; an output unit that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor into moving object distribution information that takes into account the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit, When the camera is installed in the vehicle cabin, at least two of the three-dimensional coordinate axes of the camera and the three-dimensional coordinate axes of the radio wave sensor are aligned. A detection sensor characterized by:

7. Located inside the vehicle cabin, a camera configured to capture an image of a target area within the vehicle cabin where at least an occupant may be present; a radio wave sensor that generates moving object distribution information that represents a three-dimensional distribution of an area in which moving objects exist within the vehicle cabin based on a radio wave radiated toward the target area within the vehicle cabin and reflected by an object within the vehicle cabin, the radio wave sensor having a transmitting antenna that radiates the radio waves and a receiving antenna that receives the reflected waves of the radio waves radiated from the transmitting antenna; a substrate on which the camera, the transmitting antenna, and the receiving antenna are provided on a common surface; an image acquisition unit that acquires an image captured by the camera; a camera error detection unit that detects an installation error of the camera based on the captured image acquired by the image acquisition unit; a radio wave sensor error calculation unit that calculates the installation error of the radio wave sensor based on the installation error of the camera detected by the camera error detection unit; an output unit that outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor into moving object distribution information in which the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit is taken into consideration; A detection sensor comprising:

8. The camera is disposed between the transmitting antenna and the receiving antenna on a common surface of the substrate.

8. The detection sensor according to claim 7.

9. a moving object distribution information acquisition unit for acquiring the moving object distribution information; a correction unit that corrects the moving object distribution information acquired by the moving object distribution information acquisition unit based on the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit, The output unit outputs the moving object distribution information corrected by the correction unit as the error consideration information. The detection sensor according to any one of claims 6 to 8.

10. a moving object distribution information acquisition unit for acquiring the moving object distribution information, The output unit outputs, as the error consideration information, the moving object distribution information acquired by the moving object distribution information acquisition unit and sensor installation error information indicating the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit. The detection sensor according to any one of claims 6 to 8.

11. a control unit that changes a direction of radiation of the radio wave by the radio wave sensor; The output unit outputs, to the control unit, instruction information for instructing the control unit to change a direction of radiation of the radio wave by the radio wave sensor, as the error consideration information. The detection sensor according to any one of claims 6 to 8.

12. The installation error of the camera includes an error in the installation position of the camera or an error in the orientation of the camera, The installation error of the radio wave sensor includes an error in the installation position of the radio wave sensor or an error in the direction of radiation of the radio wave by the radio wave sensor. The detection sensor according to any one of claims 6 to 8.

13. An information processing device according to any one of claims 1 to 5; The detection sensor is provided in the vehicle interior and includes a fixed part, the camera provided on the fixed part, and the radio wave sensor provided on the fixed part, wherein at least two of the three-dimensional coordinate axes of the camera and the three-dimensional coordinate axes of the radio wave sensor coincide with each other when the detection sensor is installed in the vehicle interior. An information processing system comprising:

14. An information processing device according to any one of claims 1 to 5; the radio wave sensor is provided in the vehicle interior and has the camera, a transmitting antenna that radiates the radio wave, and a receiving antenna that receives the reflected wave of the radio wave radiated from the transmitting antenna; and a substrate on which the camera, the transmitting antenna, and the receiving antenna are provided on a common surface. An information processing system comprising:

15. The camera is disposed between the transmitting antenna and the receiving antenna on a common surface of the substrate.

15. The information processing system according to claim 14.

16. In the information processing device, the output unit outputs instruction information to the detection sensor to instruct the radio wave sensor to change a direction of radiation of the radio wave, The detection sensor includes a control unit that changes a direction of radiation of the radio wave by the radio wave sensor based on the instruction information output from the output unit.

15. The information processing system according to claim 14.

17. a step of acquiring an image captured by the camera from a detection sensor, the image acquisition unit being provided in a vehicle interior and including a camera for capturing an image of a target area in the vehicle interior where at least an occupant may be present, and a radio wave sensor for generating moving object distribution information that represents a three-dimensional distribution of an area in the vehicle interior where moving objects are present, based on a radio wave radiated toward the target area in the vehicle interior and reflected by an object in the vehicle interior; A camera error detection unit detects an installation error of the camera based on the captured image acquired by the image acquisition unit; A radio wave sensor error calculation unit calculates the installation error of the radio wave sensor based on the installation error of the camera detected by the camera error detection unit; an output unit outputs error consideration information for converting the moving object distribution information generated by the radio wave sensor into moving object distribution information in which the installation error of the radio wave sensor calculated by the radio wave sensor error calculation unit is taken into consideration; An information processing method comprising: