Reporting device, reporting method, and reporting program

Abstract

To make it possible to transmit image information on a vehicle compartment using an existing audio communication system.SOLUTION: A notification system is installed in a vehicle, and includes an imaging unit that is incorporated in the vehicle and acquires plane image on a vehicle compartment of the vehicle, a conversion unit that converts the plane image into an audio signal so as to generate a first audio signal, and a communication unit that is incorporated in the vehicle and that, when the first audio signal is generated by the conversion unit, transmits the first audio signal.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a reporting device, a reporting method, and a reporting program. [Background technology]

[0002] US Patent No. 5,399,633 discloses a method and apparatus for monitoring at least one occupant of a vehicle. The invention described in US Patent No. 5,399,633 includes capturing images of the vehicle occupant using at least one image capture device and determining at least one vital sign of the vehicle occupant by analysis of the captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-101967 A Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 states that the results of image analysis are provided during medical treatment in an emergency, but it is unclear in what form the results are provided. For example, it would be effective to transmit images when a passenger is alive but unable to talk, but this is not easy because it would be impossible to transmit images without changing the specifications of the voice communication system used for emergency calls.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a reporting device, a reporting method, and a reporting program that can transmit image information inside the vehicle cabin using an existing voice communication system. [Means for solving the problem]

[0006] In order to solve the above problems, the reporting device of the present invention is, for example, a reporting device mounted on a vehicle, and is characterized in that it includes an imaging unit provided in the vehicle for acquiring a planar image of the interior of the vehicle's cabin, a conversion unit for converting the planar image into an audio signal to generate a first audio signal, and a communication unit provided in the vehicle for transmitting the first audio signal.

[0007] In order to solve the above problem, the reporting method of the present invention is characterized by including, for example, a step of capturing a planar image of the interior of a vehicle's cabin, a step of converting the planar image into an audio signal to generate a first audio signal, and a step of transmitting the first audio signal.

[0008] In order to solve the above problems, the reporting program of the present invention is characterized in that it causes a computer to function, for example, as a conversion unit that converts a planar image of the interior of the vehicle captured by an imaging unit provided in the vehicle into an audio signal to generate a first audio signal, and a communication unit that transmits the first audio signal. The computer program can be provided by downloading via a network such as the Internet, or can be provided by recording it on various computer-readable recording media such as a CD-ROM.

[0009] In any of the above aspects of the present invention, a planar image of the vehicle interior is acquired by an imaging unit provided in the vehicle, and the planar image is converted into an audio signal to generate and transmit a first audio signal. This allows image information in the vehicle interior to be transmitted using an existing audio communication system. Therefore, there is no need to change the specifications of the communication equipment including the TCU or the communication infrastructure. In addition, even if the occupant cannot make a voice call, the information in the vehicle interior can be grasped.

[0010] The device may further include an image transmission trigger detection unit that detects an image transmission trigger that requests transmission of the planar image, and the conversion unit may generate the first audio signal when the image transmission trigger is detected. This allows the audio signal generated by converting the planar image to be transmitted only when necessary. The first audio signal generated by converting the planar image sounds like noise and may interfere with telephone conversations, but transmitting the first audio signal only when necessary prevents the audio signal from interfering with telephone conversations.

[0011] The image transmission trigger may be a sound, the communication unit may receive the sound and output it to the image transmission trigger detection unit, and the image transmission trigger detection unit may detect the image transmission trigger when the sound is detected. In this way, by transmitting a first audio signal to an external device in response to an instruction from an external device (e.g., a center side device), it is easy for the external device that receives the audio signal to know the timing at which the first audio signal was transmitted, and it is easy to create a spectrogram based on the first audio signal.

[0012] The vehicle may further include a microphone provided in the vehicle cabin, and the image transmission trigger detection unit may detect speech from a voice acquired by the microphone, and the image transmission trigger may be detected when no speech is detected for a predetermined period of time. This allows the notification device to automatically generate and transmit a first voice signal when an occupant is unable to make a voice call (e.g., faints).

[0013] The conversion unit may generate the first audio signal by treating the planar image in which real numbers are two-dimensionally arranged as a spectrogram, adding phase information to the spectrogram to generate complex data in which complex numbers are two-dimensionally arranged, and performing an inverse Fourier transform on the complex data. The conversion unit may also generate the phase information randomly using random numbers. This makes it possible to generate the first audio signal from the planar image. Furthermore, by using random numbers, it is possible to appropriately add phase information.

[0014] The apparatus may include a time-series data acquisition unit that acquires time-series data, the conversion unit converting the time-series data into an audio signal to generate a second audio signal, and the communication unit transmitting the second audio signal. The conversion unit may also generate the second audio signal by analog-modulating the time-series data so that the frequency range falls within the audio band. This makes it possible to transmit not only planar images (two-dimensional data) but also time-series data (one-dimensional data) using an existing audio communication system.

[0015] The time-series data acquisition unit may be a biometric information acquisition unit that acquires biometric information including at least one of a heart rate, a respiratory rate, a blood pressure, a blood glucose level, a body temperature, and an arterial blood oxygen saturation, and the conversion unit may convert the biometric information into an audio signal to generate a second audio signal. This makes it possible to transmit not only a planar image but also the biometric information using an existing audio communication system.

[0016] The vehicle may further include a biological information acquisition unit that acquires biological information including at least one of a heart rate, a respiratory rate, a blood pressure, a blood glucose level, a body temperature, and an arterial blood oxygen saturation level, and the image transmission trigger detection unit may detect the image transmission trigger when the biological information is smaller than a first threshold value or larger than a second threshold value that is larger than the first threshold value. This allows the reporting device to automatically generate and transmit a first voice signal in a situation where a passenger is unable to make a voice call (e.g., fainting).

[0017] The imaging unit may continuously capture the planar images, and the conversion unit may continuously generate the first audio signal. This allows an external device or the like to more reliably grasp a change in the situation inside the vehicle cabin. Effect of the Invention

[0018] According to the present invention, image information within the vehicle cabin can be transmitted using an existing audio communication system. [Brief description of the drawings]

[0019] [Figure 1]FIG. 1 is a diagram showing a schematic diagram of a vehicle 10 provided with a reporting device 1 according to a first embodiment, and a center side device 20 provided in a call center. [Diagram 2] 1 is an example of a spectrogram. [Diagram 3] FIG. 2 is a diagram illustrating a schematic flow of a process for generating a spectrogram. [Figure 4] 1 is a block diagram illustrating an example of a hardware configuration of a notification device 1. FIG. [Diagram 5] 2 is a block diagram showing an outline of the functional configuration of a notification device 1. FIG. [Figure 6] FIG. 2 is a diagram illustrating a schematic diagram of a process performed by a first conversion unit 102a. [Figure 7] FIG. 2 is a diagram illustrating a schematic diagram of a process performed by a first conversion unit 102a. [Figure 8] (A) is an example of a planar image P1, and (B) is an example of an audio signal A1 and a spectrogram |X(m, k)|. [Figure 9] 4 is a flowchart showing the flow of processing performed by the notification device 1. [Figure 10] 13 is a flowchart showing a process flow in a modified example of the reporting device 1. [Figure 11] 2 is a block diagram showing an outline of the functional configuration of a reporting device 2. FIG. [Figure 12] 4 is a flowchart showing the flow of processing performed by the notification device 2. [Figure 13] 2 is a block diagram showing an outline of the functional configuration of a reporting device 3. FIG. [Figure 14] 4 is a flowchart showing a flow of processing performed by a reporting device 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of a notification device according to the present invention will be described in detail with reference to the drawings. The notification device according to the present invention is a device mounted in a vehicle, which notifies a call center from the vehicle and starts a voice conversation. Hereinafter, the present invention will be described using an emergency notification device as an example, which automatically notifies a call center when an accident occurs such as when an airbag is deployed, when an emergency notification button is pressed, etc., but the notification device according to the present invention is not limited to an emergency notification device.

[0021] <First embodiment> 1 is a diagram showing a vehicle 10 equipped with a reporting device 1 according to a first embodiment, and a center-side device 20 installed in a call center. The reporting device 1 is connected to the center-side device 20 wirelessly or via a network (a communication network for mobile phones in this embodiment). The reporting device 1 mainly includes an imaging unit 11, a microphone 12, a speaker 13, a biometric information acquisition unit 14, a processing unit 15 that performs processing such as converting images into sound, a TCU (telematics control unit) 16, and antennas 18 and 19.

[0022] The imaging unit 11, the microphone 12, and the speaker 13 are provided in the passenger compartment 10a of the vehicle 10. The imaging unit 11 is, for example, an RGB camera that captures color images, a visible light camera such as a black and white camera, an infrared camera that detects electromagnetic waves in the infrared wavelength range and creates an image, or a thermograph, and acquires a planar image P1 of the inside of the passenger compartment 10a.

[0023] The biological information acquiring unit 14 is, for example, a heart rate monitor provided on the steering wheel, and acquires the driver's heart rate. Note that the biological information acquiring unit 14 is not limited to a heart rate monitor, and may acquire biological information including at least one of the heart rate, respiratory rate, blood pressure, blood glucose level, body temperature, and arterial blood oxygen saturation.

[0024] The processing unit 15 converts the planar image P1 (two-dimensional data) into an audio signal to generate an audio signal A1. The processing unit 15 also converts the biometric information (one-dimensional data) acquired by the biometric information acquisition unit 14 into an audio signal to generate an audio signal A2. The process of converting the planar image P1 and the biometric information into an audio signal will be described in detail later.

[0025] The TCU 16 controls two-way communication in telephone calls, ETC (Electronic Toll Collection System), etc. The TCU 16 also transmits the voice signals A1 and A2 and the voice signal input from the microphone 12 to the center side device 20 via the antenna 18.

[0026] The processing unit 15 and the TCU 16 may be constructed as a dedicated board mounted on an in-vehicle device provided in the vehicle 10, for example.

[0027] The antenna 18 receives radio waves transmitted from the center side device 20 via a mobile phone network, and transmits radio waves to the center side device 20. The antenna 19 receives radio waves transmitted from a GNSS (Global Navigation Satellite System).

[0028] The center side device 20 mainly includes a microphone 21, a speaker 22, a communication device 23, a processing unit 24 for performing processing such as converting voice into an image, a display unit 25, and an antenna 26.

[0029] The communication device 23 receives radio waves transmitted from the reporting device 1 via an antenna 26 and a mobile phone network, and also transmits radio waves to the reporting device 1. The communication device 23 is connected to a microphone 21, a speaker 22, and a processing unit 24, and receives an audio signal from the microphone 21 and outputs an audio signal to the speaker 22 and the processing unit 24. The speaker 22 outputs an audio signal input from the microphone 12, and outputs audio signals A1 and A2 to the processing unit 24.

[0030] The processing unit 24 generates a spectrogram P2 based on the input audio signal A1, and acquires biological information based on the input audio signal A2. The spectrogram P2 is output from the processing unit 24 to the display unit 25 and displayed on the display unit 25. A spectrogram is a two-dimensional graph (time axis, frequency axis) that represents the results of frequency analysis performed continuously over time.

[0031] FIG. 2 is an example of a spectrogram. The upper part of FIG. 2 is an audio signal x(t), with the horizontal axis representing time and the vertical axis representing amplitude. Note that (t) indicates that it is a function of time. The lower part of FIG. 2 is a spectrogram |X(m, k)|, with the horizontal axis representing time and the vertical axis representing frequency, and the color intensity of each point represents the magnitude (amplitude) at a certain frequency at a certain point in time. Note that m is the frame number and k is the discrete frequency number. Note that the method of displaying the spectrogram is not limited to a two-dimensional graph, and it may also be a three-dimensional graph (waterfall plot).

[0032] Figure 3 is a diagram showing a schematic flow of the process for generating a spectrogram. First, a window function (see dotted line in Figure 3) is multiplied with the audio signal x(t) to extract a portion of the audio signal x(t). Hereinafter, the extracted section is called a frame. At this time, overlap processing is applied to overlap a portion of the previous frame with a portion of the current frame.

[0033] Next, a Fast Fourier Transform (FFT) is performed on each frame, and the calculation results are arranged along the time axis. As a result, a two-dimensional array of complex numbers is obtained. Hereafter, the two-dimensional array obtained here is called STFT data X(m, k).

[0034] Since the STFT data X(m, k) is a complex number, the absolute value of the STFT data |X(m, k)| or the power of the STFT data |X(m, k)| 2 In this embodiment, the spectrogram |X(m, k)| is found by acquiring a two-dimensional array of the absolute values ​​|X(m, k)| of the STFT data.

[0035] 2 and 3, the spectrogram is drawn as a black and white image, but the spectrogram may be displayed as a color image. Also, the color intensity of the spectrogram may be logarithmically converted (dB converted) and displayed.

[0036] 4 is a block diagram showing an example of a hardware configuration of the notification device 1. The notification device 1 includes a control device 50, a communication device 52, and a storage device .

[0037] The control device 50 is mainly configured with a CPU (Central Processing Unit) 56 and a memory 58. The control device 50 functions as various functional units by the CPU 56 executing a predetermined program stored in the storage device 54 or the memory 58. The program may be temporarily or permanently stored (memorized) in a removable storage medium such as a semiconductor memory, a memory card, an optical disk, a magneto-optical disk, or a magnetic disk.

[0038] The communication device 52 is composed of a communication interface for communicating with an external device, etc. The communication device 52 transmits and receives various information to and from, for example, the center side device 20, etc. The storage device 54 is composed of a hard disk, etc. This storage device 54 stores various programs and various information required for executing processing in the control device 50, and information on the processing results.

[0039] 5 is a block diagram showing an outline of the functional configuration of the reporting device 1. Functionally, the reporting device 1 includes a control unit 100. The control unit 100 includes at least an image transmission trigger detection unit 101, a conversion unit 102, a communication unit 103, a speaker amplifier 104, a location information acquisition unit 105, and an emergency report initiation unit 106 as software resources, using a calculation unit such as a CPU (Central Processing Unit) 56 for executing information processing and a storage device 54.

[0040] The image transmission trigger detection unit 101 is a functional unit that detects an image transmission trigger that requests the transmission of an image. In this embodiment, the image transmission trigger is a sound, and for example, the image transmission trigger is a sound of 440 Hz and 880 Hz that continues for 0.5 seconds in sequence. Note that the form of the image transmission trigger is not limited to this, and may be, for example, a voice such as "Please send me an image."

[0041] The image transmission trigger is transmitted from the center side device 20, received by the communication unit 103, and output from the communication unit 103 to the image transmission trigger detection unit 101. The image transmission trigger detection unit 101 acquires a sound indicating an image transmission trigger in advance, and detects the image transmission trigger by comparing the previously acquired sound with the voice output from the communication unit 103.

[0042] The conversion unit 102 mainly has a first conversion unit 102a that converts a planar image P1 captured by the imaging unit 11 into an audio signal to generate an audio signal A1, and a second conversion unit 102b that converts the biometric information acquired by the biometric information acquisition unit 14 into an audio signal to generate an audio signal A2.

[0043] The first conversion unit 102a is a functional unit that converts a planar image P1 in which real numbers are arranged two-dimensionally (two-dimensional data) into a spectrogram, adds phase information to the spectrogram to generate complex data (STFT data X(m, k)) in which complex numbers are arranged two-dimensionally, and generates an audio signal A1 by performing an inverse Fourier transform on the complex data.

[0044] 6 and 7 are diagrams that show the process performed by the first conversion unit 102a. Specifically, as shown in Fig. 6, the first conversion unit 102a first adds phase information to the planar image P1, converts it into a complex number, and generates STFT data X(m, k), performs an inverse short-time Fourier transform (iSTFT, inverse STFT), and then combines the data to generate an audio signal x(t).

[0045] As shown in FIG. 7, the inverse short-time Fourier transform is the inverse transform of the short-time Fourier transform, and by performing the short-time Fourier transform and the inverse short-time Fourier transform, it is possible to convert between the audio signal x(t) and the STFT data X(m, k) and vice versa.

[0046] In the conversion from STFT data X(m, k) to spectrogram |X(m, k)|, the absolute value is obtained, so phase information can be discarded. In contrast, in the conversion from spectrogram |X(m, k)| to STFT data X(m, k), phase information must be added and complex numbered. In this embodiment, the phase information is generated randomly using random numbers, as shown in the following formula 1.

[0047]

number

[0048] Fig. 8(A) is an example of a planar image P1, and Fig. 8(B) is an example of an audio signal A1 and a spectrogram |X(m, k)|. The upper part of Fig. 8(B) is the audio signal A1, and the lower part of Fig. 8(B) is the spectrogram |X(m, k)|. In the spectrogram |X(m, k)|, the contour line of the planar image P1 is clearly illustrated, and by looking at the spectrogram |X(m, k)|, the same effect as looking at the planar image P1 can be obtained.

[0049] Returning to the explanation of FIG. 5, the second conversion unit 102b is a functional unit that converts bioinformation, which is a time-series signal (one-dimensional data) in which real numbers are arranged in a time series, into an audio signal to generate an audio signal A2. The audio signal A2 can be generated by analog modulation, such as FM modulation, AM modulation, or PM modulation. However, it is necessary to ensure that the frequency after modulation falls within the audio band for emergency calls. For example, when a narrowband audio codec is used in communication, if the FM carrier frequency is set to 2 kHz and the maximum frequency deviation is set to 1 kHz for FM modulation, the frequency range will be 1 kHz to 3 kHz, which falls within the audio band of 0.3 kHz to 3.4 kHz.

[0050] The generation of the audio signal A2 is not limited to analog modulation. For example, the biometric information may be digitized into binary data, and the audio signal A2 may be generated by digital modulation such as QAM (quadrature amplitude modulation).

[0051] The communication unit 103 is a functional unit that transmits and receives signals via antennas 18 and 19. When the audio signals A1 and A2 are generated by the conversion unit 102, the communication unit 103 transmits the audio signals A1 and A2. In addition, the communication unit 103 transmits the audio signal input from the microphone 12.

[0052] Furthermore, the communication unit 103 receives audio transmitted from the center side device 20. The communication unit 103 outputs the received audio to the image transmission trigger detection unit 101 and the speaker amplifier 104.

[0053] In addition, the communication unit 103 receives radio waves transmitted from the GNSS 30 via the antenna 19 and outputs them to the position information acquisition unit 105.

[0054] The speaker amplifier 104 is a functional unit that amplifies a signal output from the communication unit 103 and outputs the signal to the speaker 13. The location information acquisition unit 105 is a functional unit that acquires radio waves transmitted from the GNSS 30 via the communication unit 103 and acquires location information based on the acquired radio waves. The speaker amplifier 104 and the location information acquisition unit 105 are already known, and therefore a description thereof will be omitted.

[0055] The emergency call initiation unit 106 is a functional unit that starts an emergency call when an emergency call start trigger is input. The emergency call start trigger is input from an in-vehicle device (not shown) or the like. The emergency call initiation unit 106 is connected to the image transmission trigger detection unit 101, the communication unit 103, the speaker amplifier 104, and the position information acquisition unit 105, and outputs an instruction to start an emergency call to these functional units when the emergency call start trigger is input.

[0056] Note that FIG. 4 shows only a part of the main hardware configuration of the notification device 1, and the notification device 1 may have other configurations that are generally included in a server device. The notification device 1 may be realized using an information processing device such as a dedicated or general-purpose server computer. The notification device 1 may be configured by a single information processing device, or may be configured by multiple information processing devices distributed on a communication network. The functional components of the notification device 1 shown in FIG. 5 may be further classified into more components depending on the processing content, or one component may execute the processing of multiple components.

[0057] 9 is a flowchart showing the flow of processing performed by the reporting device 1. First, when an emergency call start trigger is input from an in-vehicle device or the like, the emergency call start unit 106 outputs an emergency call start instruction to the image transmission trigger detection unit 101, the communication unit 103, the speaker amplifier 104, and the position information acquisition unit 105, and starts an emergency call (step SP11).

[0058] Specifically, the communication unit 103 receives voice and the like from the center-side device 20 via the antenna 18 based on an instruction to start an emergency call, and outputs the voice to the speaker amplifier 104. As a result, the voice output from the center-side device 20 is output from the speaker 13. Also, the communication unit 103 outputs the voice input from the microphone 12 (emergency call voice) to the center-side device 20 via the antenna 18 based on an instruction to start an emergency call. As a result, the emergency call voice becomes audible at the center-side device 20. Also, when an instruction to start an emergency call is input, the image transmission trigger detection unit 101 waits for input of an image transmission trigger. These processes are performed continuously while the emergency call is being made.

[0059] Furthermore, based on an instruction to start an emergency call, the location information acquisition unit 105 receives radio waves from the GNSS 30 via the antenna 19 and outputs the radio waves to the location information acquisition unit 105. The location information acquisition unit 105 obtains location information and outputs the information to the communication unit 103. The communication unit 103 outputs the location information to the center side device 20 via the antenna 18. Note that the location information acquisition unit 105 and the transmission of the location information are not essential.

[0060] Next, the image transmission trigger detection section 101 determines whether or not an image transmission trigger has been input via the communication section 103 (step SP13).

[0061] If an image transmission trigger has not been input (NO in step SP13), the image transmission trigger detection unit 101 repeats the process of step SP13. If an image transmission trigger has been input (YES in step SP13), the image transmission trigger detection unit 101 outputs to the conversion unit 102 that an image transmission trigger has been input, and the first conversion unit 102a acquires the planar image P1 from the imaging unit 11 (step SP15).

[0062] Next, the first conversion unit 102a converts the planar image P1 acquired in step SP13 into audio to generate an audio signal A1 (step SP17). The first conversion unit 102a outputs the generated audio signal A1 to the communication unit 103. Then, the communication unit 103 superimposes the audio signal A1 on the emergency call audio and transmits it (step SP19).

[0063] After that, the emergency call initiation unit 106 ends the series of processes. Furthermore, if an emergency call initiation trigger is input from an in-vehicle device or the like while the emergency call initiation unit 106 is performing the process shown in FIG. 9, the emergency call initiation unit 106 ends the process shown in FIG.

[0064] After performing this, the center-side device 20 receives the voice signal A1 transmitted from the reporting device 1. As shown in Fig. 1, the processing unit 24 generates a spectrogram P2 based on the input voice signal A1. As a result, the spectrogram P2 is displayed on the display unit 25 of the center-side device 20, and the state inside the vehicle compartment 10a can be confirmed at the call center.

[0065] In addition, when an image transmission trigger is input from the image transmission trigger detection unit 101 to the conversion unit 102, the second conversion unit 102b acquires the biometric information from the biometric information acquisition unit 14 at the same time as step SP13, the second conversion unit 102b converts the biometric information into audio to generate an audio signal A2 at the same time as step SP17, and the communication unit 103 may superimpose the audio signal A2 on the emergency call audio and transmit it at the same time as step SP19.

[0066] According to this embodiment, the planar image P1 captured by the imaging unit 11 is converted into an audio signal to generate the audio signal A1, and when the audio signal A1 is generated, the audio signal A1 is transmitted, thereby making it possible to transmit image information in the vehicle interior using an existing audio communication system. Therefore, there is no need to change the specifications of the communication equipment and communication infrastructure including the TCU 16. The TCU 16 relays the notification audio and does not have an image transmission function, so in order to transmit an image via the TCU 16, an interface with the imaging unit 11 and firmware for image transmission must be provided, which requires time and effort. On the other hand, when transmitting the audio signal A1 obtained by converting a planar image as in this embodiment, there is no need to change the TCU 16, which does not require time or cost, and in addition, an image communication function can be added later to a vehicle equipped with a TCU 16 that does not have an image transmission function.

[0067] Furthermore, according to this embodiment, by transmitting the audio signal A1 obtained by converting the planar image P1 into audio, the occupant can grasp information about the inside of the vehicle compartment 10a even when he or she is unable to make a voice call.

[0068] According to this embodiment, when the image transmission trigger transmitted from the center-side device 20 is detected by the image transmission trigger detection unit 101, the first conversion unit 102a converts the planar image P1 into audio to generate the audio signal A1, so that the audio signal A1 can be transmitted only when necessary. If the audio signal A1 is transmitted all the time, the audio signal A1 may sound like noise and interfere with the call. In contrast, by transmitting the audio signal A1 using the image transmission trigger as a trigger as in this embodiment, the audio signal A1 can be transmitted as necessary without interfering with the call. In particular, in this embodiment, the image transmission trigger transmitted from the center-side device 20 is used as a trigger for transmitting the audio signal A1, so that the timing at which the audio signal A1 is transmitted by the center-side device 20 is easy to understand, and a spectrogram can be easily created based on the audio signal A1.

[0069] Furthermore, according to the present embodiment, the image transmission trigger from the center-side device 20 is made into a sound, so that the image transmission trigger can be carried over the voice of the call. Therefore, the image transmission trigger can be sent from the center-side device 20 without changing the specifications of the TCU 16.

[0070] Furthermore, according to this embodiment, by converting the biometric information acquired by the biometric information acquisition unit 14 into an audio signal to generate the audio signal A2, it is possible to transmit not only the planar image P1 but also the biometric information using an existing audio communication system.

[0071] In this embodiment, the reporting device 1 is provided with the microphone 12, the speaker 13, and the speaker amplifier 104 to perform two-way communication with the center side device 20, but the microphone 12, the speaker 13, and the speaker amplifier 104 are not essential. In order to transmit the audio signal A1, it is sufficient for the reporting device 1 to be provided with at least the image transmission trigger detection unit 101, the conversion unit 102, and the communication unit 103.

[0072] It is also not essential that the image transmission trigger detection unit 101, i.e., the voice signal A1 is generated and transmitted based on an image transmission trigger. For example, the voice signal A1 may be generated and transmitted when the notification device 1 starts a notification. However, it is preferable to transmit the voice signal A1 only when necessary based on an image transmission trigger so as not to interfere with the call.

[0073] In the present embodiment, the biometric information acquired by the biometric information acquisition unit 14 is converted into the audio signal A2 by the second conversion unit 102b and transmitted from the communication unit 103, but the biometric information acquisition unit 14 and the second conversion unit 102b are not essential. For example, the reporting device 1 may acquire biometric information via the communication unit 103 from a device other than the biometric information acquisition unit 14, for example, a wearable device.

[0074] In the present embodiment, the notification device 1 converts the biometric information acquired by the biometric information acquisition unit 14 into the audio signal A2 and transmits it from the communication unit 103, but the notification device may transmit various forms of time-series data other than biometric information as audio signals. For example, the communication device may include at least one of a temperature sensor, a humidity sensor, and an air pressure sensor, and may acquire time-series data such as temperature data, humidity data, and air pressure data from the temperature sensor, humidity sensor, and air pressure sensor, respectively, convert the data into audio signals by the second conversion unit 102b, and transmit the audio signals from the communication unit 103. For example, the communication device may include a gas sensor (e.g., a CO2 sensor), acquire time-series data such as substance concentration data (e.g., carbon dioxide concentration data) from the gas sensor, convert the data into audio signals by the second conversion unit 102b, and transmit the audio signals from the communication unit 103. Time-series data such as temperature data, humidity data, air pressure data, and substance concentration data can also be converted into audio signals in the same manner as biometric information.

[0075] In the present embodiment, in step SP15, the communication unit 103 transmits the voice signal A1 superimposed on the emergency call voice, but the communication unit 103 may transmit only the voice signal A1.

[0076] In addition, in this embodiment, the voice signal A1 is transmitted only once and the process is terminated, but the voice signal A1 may be transmitted multiple times. Fig. 10 is a flowchart showing the flow of the process in the modified example of the reporting device 1. Steps SP11 to SP19 are the same, so the explanation will be omitted.

[0077] After the audio signal A1 is superimposed on the emergency call voice and transmitted (step SP19), the image transmission trigger detection unit 101 judges whether the emergency call has been terminated or whether a transmission end trigger has been inputted via the communication unit 103 (step SP21). For example, the image transmission trigger detection unit 101 assumes that the emergency call has been terminated when a signal instructing the end of the emergency call is outputted from an in-vehicle device or the like and inputted to the emergency call initiation unit 106, and inputted from the emergency call initiation unit 106 to the image transmission trigger detection unit 101. Also, for example, the image transmission trigger detection unit 101 assumes that the transmission end trigger has been inputted when the transmission end trigger is transmitted from the center side device 20 and inputted to the image transmission trigger detection unit 101 via the communication unit 103. The image transmission trigger detection unit 101 detects the transmission end trigger by acquiring a sound indicating the transmission end trigger in advance and comparing the previously acquired sound with the voice outputted from the communication unit 103. For example, sounds of 440 Hz and 880 Hz continuing in sequence for one second each can be used as a transmission end trigger, but the form of the transmission end trigger is not limited to this.

[0078] If the emergency call has not ended or the transmission end trigger has not been input (NO in step SP21), the image transmission trigger detection unit 101 returns the process to step SP15. That is, the first conversion unit 102a continuously performs the processes of acquiring the flat image P1 (step SP15), converting the flat image P1 into audio to generate the audio signal A1 (step SP17), and the communication unit 103 superimposing the audio signal A1 on the emergency call audio and transmitting it (step SP19) until the transmission end trigger is input.

[0079] When the emergency call is ended or a transmission end trigger is input (YES in step SP21), the emergency call starting unit 106 ends the series of processes. Also, when an emergency call start trigger is input from an in-vehicle device or the like while performing the process shown in Fig. 10, the emergency call starting unit 106 ends the process shown in Fig. 10.

[0080] According to this modification, the voice signal A1 can be sent continuously, so that the center side device 20 can more reliably grasp the change in the situation inside the vehicle compartment 10a.

[0081] <Second embodiment> In the first embodiment of the present invention, when an image transmission trigger transmitted from the center side device 20 is input to the image transmission trigger detection unit 101 via the communication unit 103, the first conversion unit 102a converts the planar image P1 into audio to generate the audio signal A1, and the communication unit 103 transmits the audio signal A1 by superimposing it on the emergency call voice. However, the trigger for generating the audio signal A1 from the planar image P1 is not limited to this.

[0082] In the second embodiment of the present invention, a voice signal A1 is generated from a planar image P1 when a person's speech is not detected in the vehicle interior 10a. A notification device 2 according to the second embodiment of the present invention will be described below. Note that the same reference numerals are used to designate the same parts as those in the first embodiment, and description thereof will be omitted.

[0083] 11 is a block diagram showing an outline of the functional configuration of the reporting device 2. Similar to the reporting device 1, the reporting device 2 includes, as hardware components, a control device 50, a communication device 52, and a storage device 54. Furthermore, functionally, the reporting device 2 includes a control unit 100A.

[0084] The control unit 100A has at least an image transmission trigger detection unit 101A, a conversion unit 102, a communication unit 103, a speaker amplifier 104, a location information acquisition unit 105, and an emergency call initiation unit 106 as software resources, using a calculation device such as a CPU (Central Processing Unit) 56 for executing information processing and a memory device 54.

[0085] The image transmission trigger detection unit 101A is a functional unit that detects an image transmission trigger that requests the transmission of an image. Voice acquired by the microphone 12 is input to the image transmission trigger detection unit 101A. For example, the image transmission trigger detection unit 101A detects speech from the voice acquired by the microphone 12, and assumes that an image transmission trigger is detected when no speech is detected for a predetermined time. The predetermined time is, for example, 10 seconds, but can be set to any time.

[0086] In this way, when the reporting device 2 cannot detect any speech for a predetermined period of time, it can automatically determine that the occupant is in a situation where he or she is unable to make a voice call (unconscious, dead, etc.) and that it is time to transmit an image of the inside of the vehicle cabin 10a.

[0087] Various known methods can be used to detect speech from the sound acquired by the microphone 12. In this embodiment, the image transmission trigger detection unit 101A determines that speech is occurring when the sound pressure level acquired by the microphone 12 exceeds a threshold, and determines that speech is not occurring when the sound pressure level does not exceed the threshold.

[0088] 12 is a flowchart showing the flow of processing performed by the reporting device 2. First, when an emergency call start trigger is input, the emergency call start unit 106 outputs an emergency call start instruction to the image transmission trigger detection unit 101A, the communication unit 103, the speaker amplifier 104, and the position information acquisition unit 105, and starts an emergency call (step SP11).

[0089] Next, the image transmission trigger detection unit 101A detects speech from the sound picked up by the microphone 12, and determines whether or not speech has been detected for a predetermined period of time (step SP12).

[0090] If speech is detected (YES in step SP12), the image transmission trigger detection unit 101A repeats the process of step SP12. If speech is not detected for a predetermined time (NO in step SP12), the image transmission trigger detection unit 101A outputs to the conversion unit 102 that an image transmission trigger has been input, and the first conversion unit 102a acquires the planar image P1 from the imaging unit 11 (step SP15).

[0091] Next, the first conversion unit 102a converts the planar image P1 acquired in step SP13 into audio to generate an audio signal A1 (step SP17). The first conversion unit 102a outputs the generated audio signal A1 to the communication unit 103. Then, the communication unit 103 superimposes the audio signal A1 on the emergency call audio and transmits it (step SP19). Then, the emergency call initiation unit 106 ends the series of processes. Furthermore, if an emergency call initiation trigger is input from an in-vehicle device or the like while the emergency call initiation unit 106 is performing the process shown in FIG. 12, the emergency call initiation unit 106 ends the process shown in FIG. 12.

[0092] According to this embodiment, speech is detected from the sound picked up by the microphone 12, and if no speech is detected for a predetermined period of time, an audio signal A1 is generated. Therefore, in a situation where an occupant is unable to make a voice call (e.g., fainting), the reporting device 2 can automatically generate and transmit the audio signal A1.

[0093] <Third embodiment> In the second embodiment of the present invention, an audio signal A1 is generated from a planar image P1 in response to the detection of no human speech within the vehicle cabin 10a, but the trigger for generating an audio signal A1 from a planar image P1 is not limited to this.

[0094] The third embodiment of the present invention is an embodiment in which an audio signal A1 is generated from a flat image P1 when an abnormality occurs in biological information. A reporting device 3 according to the third embodiment of the present invention will be described below. Note that the same parts as those in the first embodiment are given the same reference numerals and description thereof will be omitted.

[0095] 13 is a block diagram showing an outline of the functional configuration of the reporting device 3. Similar to the reporting device 1, the reporting device 3 includes, as hardware components, a control device 50, a communication device 52, and a storage device 54. Furthermore, functionally, the reporting device 3 includes a control unit 100B.

[0096] The control unit 100B has at least an image transmission trigger detection unit 101B, a conversion unit 102, a communication unit 103, a speaker amplifier 104, a location information acquisition unit 105, and an emergency call initiation unit 106 as software resources, using a calculation device such as a CPU (Central Processing Unit) 56 for executing information processing and a memory device 54.

[0097] The image transmission trigger detection unit 101B is a functional unit that detects an image transmission trigger that requests the transmission of an image. Biometric information acquired by the biometric information acquisition unit 14 is input to the image transmission trigger detection unit 101B. In this embodiment, the biometric information acquisition unit 14 is a heart rate monitor, and the image transmission trigger detection unit 101B continuously acquires the heart rate.

[0098] Then, the image transmission trigger detection unit 101B compares the biometric information with a first threshold and a second threshold, and assumes that an image transmission trigger is detected when the biometric information is smaller than the first threshold or larger than the second threshold. Note that the second threshold is larger than the first threshold. For example, assume that the image transmission trigger detection unit 101B detects an image transmission trigger when the heart rate is smaller than 50 (first threshold) or when the heart rate is larger than 120 (second threshold).

[0099] In this way, when the heart rate is lower than the first threshold value or higher than the second threshold value, the reporting device 3 can automatically determine that the occupant is in a situation where he or she cannot make a voice call (fainted, dead, etc.) and that it is time to transmit an image of the inside of the vehicle cabin 10a.

[0100] 14 is a flowchart showing the flow of processing performed by the reporting device 3. First, when an emergency call start trigger is input, the emergency call start unit 106 outputs an emergency call start instruction to the image transmission trigger detection unit 101B, the communication unit 103, the speaker amplifier 104, and the position information acquisition unit 105, and starts an emergency call (step SP11).

[0101] Next, the image transmission trigger detection unit 101B acquires the biometric information from the biometric information acquisition unit 14, and determines whether or not there is an abnormality in the biometric information, that is, whether or not the biometric information is smaller than the first threshold value or larger than the second threshold value (step SP14).

[0102] If the biometric information is not smaller than the first threshold or not larger than the second threshold (NO in step SP14), the image transmission trigger detection unit 101B repeats the process of step SP14. If the biometric information is smaller than the first threshold or larger than the second threshold (YES in step SP14), the image transmission trigger detection unit 101B outputs to the conversion unit 102 that an image transmission trigger has been input, and the first conversion unit 102a acquires the planar image P1 from the imaging unit 11 (step SP15).

[0103] Next, the first conversion unit 102a converts the planar image P1 acquired in step SP13 into audio to generate an audio signal A1 (step SP17). The first conversion unit 102a outputs the generated audio signal A1 to the communication unit 103. Then, the communication unit 103 superimposes the audio signal A1 on the emergency call audio and transmits it (step SP19). Then, the emergency call initiation unit 106 ends the series of processes. Furthermore, if an emergency call initiation trigger is input from an in-vehicle device or the like while the emergency call initiation unit 106 is performing the process shown in FIG. 14, the emergency call initiation unit 106 ends the process shown in FIG. 14.

[0104] According to this embodiment, an audio signal A1 is generated when the biometric information is smaller than the first threshold value or larger than the second threshold value, so that the reporting device 2 can automatically generate and transmit the audio signal A1 in a situation where an occupant is unable to make a voice call (e.g., fainting).

[0105] In this embodiment, the biometric information acquisition unit 14 is a heart rate monitor, and an image transmission trigger is detected when the biometric information falls below a threshold value, but the biometric information acquisition unit 14 is not limited to a heart rate monitor. For example, the biometric information acquisition unit 14 may be a non-contact vital sensor that measures the number of breaths and pulse, a wearable device that measures the heart rate, arterial blood oxygen saturation (SpO2), body temperature, etc., or a blood pressure monitor (for example, a watch-type wearable blood pressure monitor) that measures blood pressure. In addition, the image transmission trigger detection unit 101B may detect an image transmission trigger when, for example, the number of breaths, arterial blood oxygen saturation, body temperature, blood pressure, etc. are smaller than a first threshold value or larger than a second threshold value according to the acquired biometric information.

[0106] Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the gist of the present invention are also included. Also, it is possible to replace a part of the configuration of each embodiment described in this specification with the configuration of another embodiment, and it is also possible to add, delete, or replace other configurations to the configuration of an embodiment. [Explanation of symbols]

[0107] 1, 2, 3: Notification device 10: Vehicle 10a: Vehicle interior 11: Imaging section 12: Microphone 13: Speaker 14: Biometric information acquisition unit 15: Processing section 18, 19: Antenna 20: Center side device 21: Microphone 22: Speaker 23: Communication equipment 24: Processing section 25:Display section 26: Antenna 50: Control device 52: Communication equipment 54: Storage device 56 :CPU 58: Memory 100, 100A, 100B: Control unit 101, 101A, 101B: Image transmission trigger detection unit 102: Conversion section 102a: First conversion unit 102b: Second conversion section 103: Communications Department 104: Speaker amplifier 105: Location information acquisition unit 106:Emergency call start section

Claims

1. A notification device mounted on a vehicle, An imaging unit provided in the vehicle and configured to capture a planar image of an interior of the vehicle; a conversion unit that generates a first audio signal, the first audio signal being an audio signal obtained by converting the two-dimensional image into audio; a communication unit provided in the vehicle and configured to transmit the first audio signal to a device connected wirelessly or via a network; A reporting device comprising:

2. an image transmission trigger detection unit that detects an image transmission trigger that requests transmission of the planar image, The converter generates the first audio signal when the image transmission trigger is detected. The reporting device according to claim 1 .

3. the image transmission trigger is sound; The communication unit receives the sound and outputs it to the image transmission trigger detection unit; The image transmission trigger detection unit detects the image transmission trigger when the sound is detected. The reporting device according to claim 2 .

4. A microphone is provided in the vehicle interior, The image transmission trigger detection unit detects speech from the sound acquired by the microphone, and determines that the image transmission trigger has been detected when no speech is detected for a predetermined period of time.

4. The reporting device according to claim 2 or 3.

5. the conversion unit converts the planar image in which real numbers are two-dimensionally arranged into a spectrogram, and adds phase information to the spectrogram to generate complex number data in which complex numbers are two-dimensionally arranged; The first audio signal is generated by performing an inverse Fourier transform on the complex data. The reporting device according to any one of claims 1 to 4.

6. The conversion unit randomly generates the phase information by using a random number. The reporting device according to claim 5 .

7. A time series data acquisition unit for acquiring time series data, The converter converts the time-series data into an audio signal to generate a second audio signal; The communication unit transmits the second audio signal. The reporting device according to any one of claims 1 to 6.

8. The conversion unit generates the second audio signal by analog-modulating the time-series data so that the frequency range falls within a voice band. The reporting device according to claim 7 .

9. the time-series data acquisition unit is a biological information acquisition unit that acquires biological information including at least one of a heart rate, a respiratory rate, a blood pressure, a blood glucose level, a body temperature, and an arterial blood oxygen saturation; The conversion unit converts the bioinformation into an audio signal to generate the second audio signal.

9. The reporting device according to claim 7 or 8.

10. a biological information acquiring unit that acquires biological information including at least one of a heart rate, a respiratory rate, a blood pressure, a blood glucose level, a body temperature, and an arterial blood oxygen saturation level; The image transmission trigger detection unit determines that the image transmission trigger is detected when the biological information is smaller than a first threshold value or larger than a second threshold value that is larger than the first threshold value. The reporting device according to any one of claims 2 to 4.

11. The imaging unit continuously captures the planar images, The converter continuously generates the first audio signal. The reporting device according to any one of claims 1 to 10.

12. capturing a planar image of an interior of a vehicle; generating a first audio signal, the first audio signal being an audio signal obtained by converting the planar image into audio; transmitting the first audio signal; A reporting method comprising:

13. Computer, a conversion unit that generates a first audio signal, which is an audio signal generated from a planar image of the interior of the vehicle cabin captured by an imaging unit provided in the vehicle, and is an audio signal obtained by converting the planar image into audio; a communication unit that transmits the first audio signal to a device connected wirelessly or via a network; A reporting program characterized by functioning as a

Images