Remote support device, remote support method, remote support program, and remote support system

Abstract

To reduce the monitoring effort of a supporter in a remote support of a support recipient.SOLUTION: A storage control unit performs control so that a past image captured in the past by a camera device carried by a support recipient, imaging position information for indicating a geographical imaging position of the past image, and marking information about a predetermined mark attached to the past image are associated with each other and stored in a storage unit. A reading control unit receives a real-time captured image captured in real time by the camera device carried by the support recipient, and reads the past image and marking information corresponding to a capturing position of the real-time captured image from the storage unit. Then, a support image generation unit generates a support image in which a predetermined mark indicated by marking information read from the storage unit is reflected in either or both of the real-time captured image and the past image, which is a support image to be displayed on a display unit monitored by a support person who supports the support recipient.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present invention relates to a remote assistance device, a remote assistance method, a remote assistance program, and a remote assistance system. [Background technology]

[0002] 2. Description of the Related Art Currently, there are known devices that support a person requiring assistance, such as a visually impaired person, so that the person can act safely and independently.

[0003] For example, Patent Document 1 (JP Patent Publication No. 2000-312356) discloses a guidance system for visually impaired people in which a support person checks the situation around the visually impaired person based on images captured by a camera device attached to the visually impaired person's body and remotely guides the visually impaired person in a safe direction. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2000-312356 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of the visually impaired person guidance system of Patent Document 1, the supporter needs to constantly monitor the images captured by the camera device carried by the visually impaired person with high concentration in order to safely guide the visually impaired person. This poses the problem that the supporter needs to put in a lot of monitoring effort. This problem becomes more pronounced when one supporter supports multiple visually impaired people.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a remote support device, a remote support method, a remote support program, and a remote support system that reduce the monitoring effort of a supporter when remotely supporting a supported person. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the present invention comprises a memory control unit that controls the storage in a memory unit of past images captured in the past by a camera device carried by the person being supported, imaging location information indicating the geographical imaging location of the past images, and marking information regarding a predetermined mark affixed to the past images, in association with each other; a read control unit that receives real-time captured images captured in real time by a camera device carried by the person being supported, and reads out from the memory unit the past images and marking information corresponding to the imaging location of the real-time captured images; and a support image generation unit that generates a support image to be displayed on a display unit monitored by a supporter supporting the person being supported, the support image reflecting a predetermined mark indicated by the marking information read out from the memory unit for either or both of the real-time captured images and the past images. Effect of the Invention

[0008] According to the present invention, the monitoring effort of a supporter who remotely supports a supportee can be reduced. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a system configuration of a remote support system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of a mobile terminal device provided in the remote support system according to the embodiment. [Diagram 3] FIG. 3 is a functional block diagram of each function realized by the CPU of the mobile terminal device executing a voice support program stored in the built-in memory. [Figure 4] FIG. 4 is a perspective view showing the external configuration of the headset device. [Diagram 5] FIG. 5 is a block diagram showing the hardware configuration of the left and right unit housings. [Figure 6] FIG. 6 is a functional block diagram of each function realized by the microcomputers in the left and right unit housings operating based on a control program. [Figure 7] FIG. 7 is a block diagram showing a hardware configuration of the center server device. [Figure 8] FIG. 8 is a functional block diagram showing the functional configuration of the center server device. [Figure 9] FIG. 9 is a block diagram showing a hardware configuration of the remote monitoring terminal device. [Figure 10] FIG. 10 is a functional block diagram showing the functional configuration of the remote monitoring terminal device. [Figure 11] FIG. 11 is a flowchart showing the flow of operations for generating a privacy protection image in the center server device. [Figure 12] FIG. 12 is a diagram showing an example of a privacy protection image. [Figure 13] FIG. 13 is a flowchart showing the flow of the marking information processing operation in the remote monitoring terminal device and the warning area display processing based on the marking information. [Figure 14] FIG. 14 is a diagram showing an example of a privacy protection image in which a marking process for a cautionary region has been performed by a supporter. [Figure 15] FIG. 15 is a flowchart showing the flow of the marking information processing operation in the center server device that receives the marking information from the remote monitoring terminal device. [Figure 16] FIG. 16 is a diagram showing regions requiring attention and feature points designated for a past image. [Figure 17] FIG. 17 is a diagram for explaining the order in which feature point IDs are assigned to the feature points. [Figure 18] FIG. 18 is a diagram for explaining the vector values ​​of each feature point and the region feature vector. [Figure 19] FIG. 19 is a diagram showing an example of a support screen displayed on the monitor device of the remote monitoring terminal device. [Figure 20] FIG. 20 is a flowchart showing the flow of the operation of reflecting an attention-required region specified in a past image in a real-time video sequence. [Figure 21]FIG. 21 is a diagram showing an example of a real-time video image in which an attention-requiring area of ​​a past image is reflected. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a remote support system according to an embodiment of the present invention will be described with reference to the drawings. In this embodiment of the remote support system, a supporter remotely supports a supportee based on an image captured by a camera device attached to the supportee. The remote support system according to this embodiment can be used, for example, for remote support of visually impaired people, remote support of children, or guidance of people moving through unfamiliar areas. Hereinafter, an example of remote support of a visually impaired person will be described as an example of a supportee.

[0011] (System Configuration) Fig. 1 is a diagram showing a system configuration of a remote support system according to an embodiment. As shown in Fig. 1, the remote support system according to the embodiment includes a visually impaired person device 1 worn by a visually impaired person, a remote monitoring terminal device 3 operated by a supporter, and a center server device 2 that relays communication between the visually impaired person device 1 and the remote monitoring terminal device 3. The center server device 2 and the remote monitoring terminal device 3 are connected to each other via a predetermined network 4. As the network 4, for example, a wide area network such as the Internet or a private network such as a LAN (Local Area Network) can be used. The center server device 2 is an example of a remote support device.

[0012] The visually impaired person device 1 performs wireless communication with the wireless base station 5. As described later, the visually impaired person device 1 is carried by the visually impaired person, and the visually impaired person's surroundings are captured as moving images by a camera device provided in the visually impaired person device 1, and the visually impaired person's voice is collected by a microphone device also provided in the visually impaired person device 1. The captured images of the visually impaired person's surroundings and the voice of the visually impaired person are transmitted to the center server device 2 via the wireless base station 5 and the network 4, and the captured images are subjected to privacy protection processing described later in the center server device 2, and then transmitted to the remote monitoring terminal device 3 together with the voice of the visually impaired person. In addition, the support voice of the supporter inputted in the remote monitoring terminal device 3 is transmitted to the center server device 2, and further transmitted from the center server device 2 to the visually impaired person device 1 via the network 4 and the wireless base station 5. This enables the visually impaired person to be remotely supported by the support voice from the supporter.

[0013] (Hardware configuration of devices for visually impaired people) Fig. 2 is a block diagram of the visually impaired person device 1 provided in the remote support system of the embodiment. As shown in Fig. 2, the visually impaired person device 1 has a headset device 7 and a mobile terminal device 8. As the mobile terminal device 8, a smartphone or a tablet terminal device, which is a mobile device equipped with a wireless communication function, can be used.

[0014] The headset device 7 has an earphone device 25 for the visually impaired person to hear the support voice of the supporter, a camera device 26 for capturing an image of the surroundings of the visually impaired person, and a microphone device 27 for collecting the voice generated by the visually impaired person. Details will be described later.

[0015] The mobile terminal device 8 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, an internal memory 13, a communication unit 14, a short-range wireless communication unit 15, a touch panel 16, and a GPS (Global Positioning System) sensor 17. The mobile terminal device 8 also includes an acceleration sensor 18, a timer 19, an external interface (external I / F) 20, a camera unit 21, a microphone unit 22, and a speaker unit 23.

[0016] The built-in memory 13 stores a voice support program. As described later, the CPU 11 executes the voice support program to realize each function of the mobile terminal device 8. After the mobile terminal device 8 is powered on, the visually impaired person operates a touch panel 16 described later, causing the CPU 11 to start execution of the voice support program. Alternatively, the CPU 11 may start execution of the voice support program in conjunction with the powering on of the mobile terminal device 8. The CPU 11 receives an image of the surroundings of the visually impaired person captured by the headset device 7 based on the voice support program, adds GPS information acquired by a GPS sensor 17 described later as the image capture position and the time counted by a timer 19 as image capture time information (time stamp) to the image, and transmits the image to the center server device 2. The CPU 11 also receives the voice of the visually impaired person collected by the headset device 7 based on the voice support program and transmits it to the center server device 2. The CPU 11 also receives the support voice of the supporter from the center server device 2 based on the voice support program and transmits it to the headset device 7. Furthermore, the CPU 11 transmits to the center server device 2 GPS information acquired by a GPS sensor 17 and acceleration information acquired by an acceleration sensor 18, which will be described later, based on the voice support program.

[0017] The communication unit 14 performs wireless communication with the wireless base station 5. The short-distance wireless communication unit 15 performs short-distance wireless communication, such as Bluetooth (registered trademark), with the headset device 7. The touch panel 16 is a device in which a display unit and an operation unit are integrated. A user can perform a desired input operation by touching the touch panel 16. The GPS sensor 17 outputs GPS information indicating the current location of the visually impaired person carrying the mobile terminal device 8 based on signals from multiple GPS satellites in the sky. The acceleration sensor 18 outputs acceleration information for detecting the moving direction of the visually impaired person carrying the mobile terminal device 8. The timer 19 counts the imaging time added to the captured image captured by the camera device 26 of the headset device 7. The communication unit 14 receives an image of the surroundings of the visually impaired person captured by the headset device 7 from the headset device 7, adds the imaging time counted by the timer 19 to the image, and transmits it to the center server device 2.

[0018] 2, the external I / F 20 is an interface used when connecting the earphone device 25, camera device 26, and microphone device 27 of the headset device 7 by wired connection, for example, via a USB cable. The camera unit 21 is an imaging function built into the portable terminal device 8. The microphone unit 22 is a sound collection function built into the portable terminal device 8. The speaker unit 23 is a sound output function built into the portable terminal device 8.

[0019] (Functional configuration of mobile terminal device) Fig. 3 is a functional block diagram of each function realized by the CPU 11 of the mobile terminal device 8 executing the voice support program stored in the built-in memory 13. As shown in Fig. 3, the CPU 11 realizes each function of a communication control unit 31, an acquisition unit 32, and a voice control unit 33 based on the voice support program.

[0020] The communication control unit 31 controls wireless communication with the wireless base station 5 and also controls short-distance wireless communication with the headset device 7. The acquisition unit 32 acquires the captured image of the surroundings of the visually impaired person captured by the camera device 26 of the headset device 7 and the voice of the visually impaired person collected by the microphone device 27 of the head device 7, which are received by the short-distance wireless communication control of the communication control unit 31, and transmits them to the center server device 2 via the wireless base station 5 and the network 4. The communication control unit 31 also receives the support voice of a supporter supporting the visually impaired person, which is transmitted from the center server device 2, via the network 4 and the wireless base station 5, and transmits it to the headset device 7 by short-distance wireless communication. When the microphone device 27 of the head device 7 is not used, the voice control unit 33 collects the voice of the visually impaired person with the microphone unit 22 of the mobile terminal device 8, and controls transmission to the center server device 2 via the communication control unit 31. In addition, when the earphone device 25 of the head device 7 is not used, the voice control unit 33 receives the support voice of the supporter supporting the visually impaired person transmitted from the center server device 2 via the network 4, the wireless base station 5, and the communication control unit 31, and controls the output via the speaker unit 23 of the mobile terminal device 8.

[0021] (External configuration of headset device) Fig. 4 is a perspective view showing the external configuration of the headset device 7. As shown in Fig. 4, the headset device 7 is a so-called eyeglass-type headset device. A left unit housing part 36L is provided on the left temple part (arm) 35L, which is the part that is hung on the ear of the visually impaired person, and a right unit housing part 36R is provided on the right temple part 35R.

[0022] The left unit housing 36L is provided with a camera device 26LF for capturing an image in front of the visually impaired person and a camera device 26LR for capturing an image behind the visually impaired person. Similarly, the right unit housing 36R is provided with a camera device 26RF for capturing an image in front of the visually impaired person and a camera device 26RR for capturing an image behind the visually impaired person.

[0023] Each of the camera devices 26LF, 26LR, 26RF, and 26RR is provided with a lens with a wide angle of view (wide-angle lens), such as a fisheye lens. As an example, the angle of view of each of the camera devices 26LF, 26LR, 26RF, and 26RR is about 180 degrees. Therefore, each of the camera devices 26LF, 26LR, 26RF, and 26RR can capture an image of a 360-degree range including the left and right sides and feet of a visually impaired person. Here, by combining the camera device 26LF and the camera device 26RF and using them as a stereo camera, the distance to an object in the image captured by these camera devices can be detected. In the embodiment, the center server device 2 described later utilizes the detection of the distance based on the images captured by the camera device 26LF and the camera device 26RF.

[0024] The right unit housing 36R is provided with an earphone device 25 that is attached to the right ear of the visually impaired person. A support voice from the supporter is output through the earphone device 25 to instruct the visually impaired person on what to do or to alert him / her to dangerous objects. The earphone device 25 is an example of a voice output device carried by the supported person.

[0025] In the example of FIG. 4, the earphone device 25 is provided only in the right unit housing portion 36R, but the earphone device 25 may be provided only in the left unit housing portion 36L, or the earphone device 25 may be provided in each of the both unit housing portions 36L, 36R.

[0026] The left unit housing 36L is provided with a microphone device 27 that collects the voice of the visually impaired person. The voice of the visually impaired person collected by the microphone device 27 is converted into a data format (voice data) that can be transmitted to the center server device 2 and the remote monitoring terminal device 3 based on a predetermined communication protocol, transmitted to the mobile terminal device 8 by short-distance wireless communication, transmitted to the center server device 2 via the wireless base station 5 according to a wireless communication protocol provided by a predetermined communication carrier, and further transmitted from the center server device 2 to the remote monitoring terminal device 3. In the remote monitoring terminal device 3, the received voice data is converted into an analog signal and output to the supporter. This allows the supporter to recognize the intention of the visually impaired person through the voice of the visually impaired person and provide appropriate support according to the visually impaired person's request.

[0027] In this example, the left and right unit housing parts 36L, 36R are provided, but either one of the unit housing parts may be provided. The camera device 26 provided in the unit housing part may be only a camera device that captures an image in front of the visually impaired person. Even in this case, the camera device 26 has a wide-angle lens, so that it can capture an image of a wide range in front of the visually impaired person. However, by capturing an image in front of and behind the visually impaired person with the left and right unit housing parts 36L, 36R, it is possible to obtain a wider range of captured images around the visually impaired person, enabling more accurate support.

[0028] When the microphone device 27 of the headset device 7 is not used, the visually impaired person may carry the portable terminal device 8, for example, in a breast pocket, and the voice of the visually impaired person may be picked up by the microphone unit 22 of the portable terminal device 8.

[0029] In addition, when the earphone device 25 of the headset device 7 is not used, the visually impaired person may carry the portable terminal device 8, for example, in a breast pocket, and the support voice of the supporter may be output from the speaker unit 23 of the portable terminal device 8.

[0030] In addition, in this embodiment, the headset device 7 and the mobile terminal device 8 are described as transmitting and receiving images and audio by short-distance wireless communication. However, as shown by the dotted line block in Fig. 2, the headset device 7 and the mobile terminal device 8 may be connected via a wired connection, for example, a USB cable, via the external I / F 20 of the mobile terminal device 8 to transmit and receive images and audio.

[0031] (Hardware configuration of unit housing) Fig. 5 is a block diagram showing the hardware configuration of the left and right unit housings 36L, 36R. As shown in Fig. 5, each unit housing 36L, 36R has a microcomputer 40, a RAM 41, a built-in memory 42, a camera device 26, a short-range wireless communication unit 44, an earphone device 25, and a microphone device 27. A control program is stored in the built-in memory 42. The microcomputer 40 operates based on this control program to control transmission of an image of the surroundings of the visually impaired person captured by the camera device 26 to the mobile terminal device 8 via the short-range wireless communication unit 44.

[0032] Furthermore, the microcomputer 40 operates based on the control program to convert the voice of the visually impaired person collected by the microphone device 27 into a data format (voice data) that can be transmitted to the center server device 2 and the remote monitoring terminal device 3 based on a predetermined communication protocol, and controls the transmission to the mobile terminal device 8 via the short-range wireless communication unit 44. Furthermore, the microcomputer 40 operates based on the control program to convert the support voice data of the supporter received from the mobile terminal device 8 via the short-range wireless communication unit 44 into an analog signal, and controls the output via the earphone device 25.

[0033] (Functional configuration of the unit housing) 6 is a functional block diagram of the functions realized by the microcomputers 40 of the left and right unit housings 36L, 36R operating based on a control program. As shown in FIG. 6, the microcomputer 40 functions as a camera control unit 48, an audio control unit 49, and a short-range wireless communication control unit 50 by executing the control program.

[0034] The camera control unit 48 controls the camera device 26 to capture an image of the surroundings of the visually impaired person. The audio control unit 49 acquires the voice of the visually impaired person collected by the microphone device 27, and converts the voice into a data format (audio data) that can be transmitted to the center server device 2 and the remote monitoring terminal device 3 based on a predetermined communication protocol. The audio control unit 49 also converts the support voice data of the supporter received from the mobile terminal device 8 into an analog signal, and controls output via the earphone device 25. The short-distance wireless communication control unit 50 controls the short-distance wireless communication unit 44 to transmit the image of the surroundings of the visually impaired person captured by the camera device 26 and the voice data of the visually impaired person collected by the microphone device 27 to the mobile terminal device 8.

[0035] (Hardware configuration of center server device) 7 is a block diagram showing a hardware configuration of the center server device 2. As shown in this figure, the center server device 2 includes a CPU 51, a ROM 52, a RAM 53, a HDD (Hard Disk Drive) 54, and a communication unit 55.

[0036] The communication unit 55 communicates with the remote monitoring terminal device 3 via the network 4 , and also communicates with the mobile terminal device 8 via the network 4 and the wireless base station 5 .

[0037] The HDD 54 is an example of a storage unit. The HDD 54 stores a remote assistance control program, map data, voice memo data, marking information, past images, a feature point management table, and an attention area management table. When the CPU 51 receives an image (real-time video) captured by the camera device 26 of the visually impaired person device 1 carried by the visually impaired person from the visually impaired person device 1 based on the remote assistance control program, the CPU 51 performs a process (privacy protection process) for protecting the privacy of the person in the image on the image. Here, the real-time video is an example of a real-time captured image. Also, based on the remote assistance control program, the CPU 51 reads out past images and marking information corresponding to the capture position of the real-time video from the storage unit, and provides the past images and marking information together with the real-time video that has been subjected to the privacy protection process to the remote monitoring terminal device 3, thereby assisting the supporter in supporting the visually impaired person.

[0038] Further, the CPU 51 displays the current position of the visually impaired person on the map data based on the remote assistance control program and provides it to the remote monitoring terminal device 3. Further, the CPU 51 receives voice memo data or text information, etc., input by the supporter to the remote monitoring terminal device 3, for example, "Be careful because there is a step ahead," for making it easier for the supporter to recognize the presence of an area requiring attention, from the remote monitoring terminal device 3 based on the remote assistance control program, and controls storage in the HDD 54. Further, the CPU 51 receives, in detail later, from the remote monitoring terminal device 3, marking information indicating an area requiring attention designated by the supporter on the real-time video image that has been subjected to privacy protection processing and provided to the remote monitoring terminal device 3, together with the real-time video image, and controls storage in the HDD 54 of the received real-time video image as a past image together with the received marking information based on the remote assistance control program. Note that the past image is a video image that has been subjected to privacy protection processing similarly to the real-time video image.

[0039] Furthermore, the CPU 51 extracts feature points having image characteristics, such as edge parts and corner parts, from the past image based on the remote assistance control program, and stores them in the feature point management table, and also calculates an area feature vector, which is a direct sum of the feature points corresponding to each attention area, and stores the area feature vector in the attention area management table in association with the past image. The calculation of the feature points and area feature vector will be described later.

[0040] (Functional configuration of the center server device) Fig. 8 is a block diagram showing the functional configuration of the center server device 2. As shown in Fig. 8, the CPU 51 of the center server device 2 executes a remote assistance control program stored in the HDD 54 to realize the functions of an acquisition unit 60, a feature point extraction unit 61, a calculation unit 62, a current position identification unit 63, and a privacy protection image generation unit 64. The CPU 51 also executes the remote assistance control program to realize a matching processing unit 65, an assistance image generation unit 66, a communication control unit 67, a color selection unit 68, a person area detection unit 69, a storage control unit 70, a marking information processing unit 71, and an attention area change processing unit 72.

[0041] The acquisition unit 60 acquires, from the visually impaired person device 1 carried by the visually impaired person, an image of the visually impaired person's surroundings captured by the visually impaired person device 1, collected voice data of the visually impaired person, GPS information acquired by the GPS sensor 17, and acceleration information acquired by the acceleration sensor 18. The acquisition unit 60 also acquires, from the remote monitoring terminal device 3, the support voice of the supporter and marking information indicating an attention-required area designated by the supporter on the real-time video. The feature point extraction unit 61 extracts feature points having image characteristics such as edge parts and corner parts from past images, which are past real-time videos, and real-time videos in which images captured by the visually impaired person device 1 carried by the visually impaired person have been subjected to privacy protection processing. The calculation unit 62 calculates an area feature vector, which is a direct sum of feature points corresponding to each attention-required area, based on the extracted feature points.

[0042] The current location identification unit 63 identifies the current location, moving direction, and moving speed of the visually impaired person based on GPS information and acceleration information acquired from the mobile terminal device 8 of the visually impaired person. The privacy protection image generation unit 64 is an example of a privacy protection unit, and protects privacy by representing the area (outline) of a person's figure in an image (real-time video) captured by the visually impaired person in a predetermined color by filling it with a predetermined color or other method, and generates a real-time video as a privacy protection image (privacy protection processing). The matching processing unit 65 performs matching processing on the past image and the real-time video based on the area feature vector, and detects an attention-requiring area specified by the supporter in the real-time video.

[0043] The support image generating unit 66 generates a support image that the supporter sees when monitoring the visually impaired person. The communication control unit 67 communicates with the remote monitoring terminal device 3 via the network 4. The communication control unit 67 also communicates with the visually impaired person device 1 via the network 4 and the wireless base station 5. The color selecting unit 68 selects a color to be used when filling in a person according to a predicted danger level for the visually impaired person when generating a privacy protection image. The person area detecting unit 69 detects an area in which a person is shown from an image (real-time moving image) captured by the visually impaired person device 1. The storage control unit 70 controls the storage of marking information indicating an attention-required area designated by the supporter, and voice memo data or text information for making it easier for the supporter to recognize the presence of the attention-required area, in the HDD 54. The voice memo data and text information are examples of auxiliary information.

[0044] The marking information processing unit 71 generates marking information indicating the attention area designated by the supporter. The attention area changing processing unit 72 changes (enlarges or reduces) the size of the attention area designated on the past image, thereby improving the matching accuracy of the matching processing unit 65.

[0045] (Hardware configuration of remote monitoring terminal device) Fig. 9 is a block diagram showing a hardware configuration of the remote monitoring terminal device 3. As shown in Fig. 9, the remote monitoring terminal device 3 has a CPU 81, a ROM 82, a RAM 83, a communication unit 84, a HDD 85, an input / output interface (input / output I / F) 86, a microphone unit 87, and a speaker unit 88. A monitor device 89, an operation unit 90, etc. are connected to the input / output I / F 86.

[0046] The monitor device 89 is an example of a display unit or a display device. The monitor device 89 displays real-time moving images that have been subjected to privacy protection processing and that are acquired from the center server device 2, past images and marking information corresponding to the capture positions of the real-time moving images, and map information indicating the current location of the visually impaired person. An input device such as a keyboard device and a mouse device is connected as the operation unit 90. The supporter operates the operation unit 90 to input areas requiring attention in the real-time moving images. The microphone unit 87 collects the support voice of the supporter alerting the visually impaired person. The speaker unit 88 outputs the voice of the visually impaired person acquired from the center server device 2.

[0047] (Functional configuration of remote monitoring terminal device) A remote assistance program is stored in the HDD 85. The CPU 81 executes the remote assistance program to realize the functions of an acquisition unit 91, a display control unit 92, a marking processing unit 93, a communication control unit 94, and a voice processing unit 95, as shown in FIG.

[0048] The acquisition unit 91 acquires the real-time video, past images and marking information corresponding to the capture positions of the real-time video, map information indicating the current location of the visually impaired person, and voice data of the visually impaired person, transmitted from the center server device 2, via the network 4 and a communication control unit 94 described later. The display control unit 92 controls the display on the monitor device 89 of the real-time video, past images and marking information corresponding to the capture positions of the real-time video, map information indicating the current location of the visually impaired person, and the like acquired by the acquisition unit 91.

[0049] The marking processing unit 93 identifies an area requiring attention designated by the supporter in the real-time video acquired by the acquisition unit 91. The communication control unit 94 receives the real-time video, past images and marking information corresponding to the capturing positions of the real-time video, map information indicating the current location of the visually impaired person, and voice data of the visually impaired person, all of which are transmitted from the center server device 2. The communication control unit 94 also transmits marking information indicating the area requiring attention on the real-time video designated by the supporter together with the real-time video to the center server device 2 via the network 4. The communication control unit 94 also transmits the support voice (voice data) of the supporter converted by a voice processing unit 95 (described later) to the center server device 2 via the network 4.

[0050] The voice processing unit 95 converts the support voice of the supporter alerting the visually impaired person into a data format that can be transmitted to the center server device 2 and the visually impaired person device 1 based on a predetermined communication protocol. In addition, the voice processing unit 95 converts the voice data of the visually impaired person received via the center server device 2 into an analog signal that can be output via the speaker unit 88.

[0051] (Generation of Privacy Protection Images in the Center Server Device) Next, a generating operation of a privacy protection image by the center server device 2 in the remote support system of this embodiment will be described. In the remote support system of the embodiment, an image (real-time video) captured by the visually impaired person device 1 is transmitted from the center server device 2 to the remote monitoring terminal device 3. Then, a supporter who operates the remote monitoring terminal device 3 provides support to the visually impaired person to avoid danger while viewing the real-time video transmitted from the center server device 2.

[0052] However, there is a possibility that various people may appear in the image captured by the device for visually impaired persons 1. For this reason, it is necessary to protect the privacy of people appearing in the image captured by the device for visually impaired persons 1. For this reason, the center server device 2 generates a privacy protection image in which people appearing in the image captured by the device for visually impaired persons 1 are blacked out (privacy protection processing), and transmits the generated image to the remote monitoring terminal device 3. This makes it possible to provide support to the visually impaired person while protecting the privacy of people appearing in the image captured by the device for visually impaired persons 1.

[0053] In the following description, the subject of privacy protection processing is assumed to be a person, but other subjects such as automobiles, whose owners may be identified due to the faces of passengers or license plates being captured, may be painted over as subjects of privacy protection processing. In this case, if the entire automobile is painted over, the painted-over automobile may occupy most of the privacy protection screen, making it difficult to determine what is being captured. Therefore, the passengers and license plates that are captured may be painted over as subjects of privacy protection processing, and one or more representative feature points may be displayed in color for the entire automobile.

[0054] In addition, the privacy protection processing may be a method of directly and irreversibly processing the original real-time video, or a method of separately creating a fill-in pattern as additional information for the original real-time video and superimposing the fill-in pattern on the real-time video when it is displayed.

[0055] Fig. 11 is a flowchart showing the flow of the operation of generating real-time moving images in the center server device 2. First, an image captured by the device for visually impaired persons 1 carried by a visually impaired person is transmitted to the center server device 2 via the wireless base station 5 and the network 4 together with GPS information and acceleration information indicating the current location of the visually impaired person. In step S1, the acquisition unit 60 of the center server device 2 shown in Fig. 8 acquires the image (real-time moving image) captured by the device for visually impaired persons 1, the GPS information, and the acceleration information via the communication unit 55 and the communication control unit 67.

[0056] In step S2, the feature point extraction unit 61 extracts a plurality of feature points, such as edge and corner parts, of an object captured in the real-time video. In step S3, the calculation unit 62 calculates the distance and relative speed from the visually impaired person to each feature point of the object captured in the real-time video using the function of the stereo camera described in FIG. 4. Specifically, the two camera devices 26LF and 26RF provided in the headset device 7 of the device for visually impaired person 1 are used as stereo cameras, and the distance to the object captured in the image and its change over time (relative speed) are detected based on the images captured by each camera device. Here, the relative speed of the object captured in the image with the visually impaired person as the reference is defined as positive in the direction approaching the visually impaired person.

[0057] Next, in step S4, the color selection unit 68 selects a color corresponding to the direction of movement of each feature point relative to the visually impaired person (positive / negative relative: movement toward the visually impaired person (positive movement) or movement away from the visually impaired person (negative movement)) and the magnitude of the relative speed. Also, in step S5, the person area detection unit 69 detects the area of ​​the person image in the real-time video based on each feature point detected in step S2. Then, in step S6, the privacy protection image generation unit 64 generates a privacy-protected real-time video in which the area of ​​the person image detected in step S5 is filled with the color selected in step S4.

[0058] Specifically, Fig. 12 is a diagram showing an example of a real-time video before and after privacy protection processing is performed. Fig. 12(a) shows an example of an image (real-time video before privacy protection processing is performed) captured by the device 1 for visually impaired persons carried by a visually impaired person. As an example, the real-time video shown in Fig. 12(a) can be a parallax image formed by the privacy protection image generating unit 64 performing so-called parallax calculation based on the left and right captured images captured by the left and right camera devices 26 of the headset device 7 described with reference to Figs. 4 and 5.

[0059] In the real-time video shown in Fig. 12(a), a bronze statue holding a cane is shown in front of a visually impaired person, and on the left side of the statue, two people are shown walking away from the visually impaired person. Also, on the right side of the statue, two people are shown walking toward the visually impaired person. Of the two people walking toward the visually impaired person, the person on the right side of the real-time video walks faster, and the person next to the person on the right side walks slower.

[0060] In this example, the relative velocity of the two people walking to the left of the statue in a direction away from the visually impaired person is negative. People walking in a direction away from the visually impaired person are less likely to come into contact with the visually impaired person, and are less dangerous to the visually impaired person. Therefore, the color selection unit 68 selects "blue," which means safety for the visually impaired person. Then, the privacy protection image generation unit 64 generates a privacy protection image in which the images of the two people walking to the left of the statue are filled with blue, as shown in FIG. 12(b).

[0061] In contrast, the relative speed of two people walking to the right of the bronze statue in a direction approaching the visually impaired person is a positive relative speed. People walking in a direction approaching the visually impaired person are likely to come into contact with the visually impaired person, and are therefore at high risk. Also, the faster the walking speed, the greater the damage the visually impaired person will receive in the event of contact. For this reason, the color selection unit 68 selects "red," which indicates a high risk, for people whose positive relative speed is equal to or greater than a predetermined value, and selects "yellow," which indicates a medium risk, for people whose positive relative speed is less than the predetermined value.

[0062] The privacy protection image generating unit 64 generates a privacy protection image in which the rightmost person, who has a positive relative velocity image equal to or greater than a predetermined value, among the two people walking on the right side of the bronze statue, is painted red as shown in Fig. 12(b).The privacy protection image generating unit 64 generates a privacy protection image in which the rightmost person, who has a positive relative velocity image equal to or greater than a predetermined value, among the two people walking on the right side of the bronze statue, is painted yellow as shown in Fig. 12(b).

[0063] For a person whose movement is not detected based on the relative speed, such as a person sitting on a chair or leaning against a pillar, the color selection unit 68 selects "green" indicating that the risk is almost zero. For a person whose movement is not detected, the privacy protection image generation unit 64 generates a privacy protection image in which the person is filled with green as shown in FIG. 12(b).

[0064] In addition, in determining the above-mentioned risk level, the current position, moving direction, and moving speed of the visually impaired person identified by the current position identification unit 63 based on the GPS information and acceleration information received from the visually impaired person device 1 may be taken into consideration. For example, even if the same positive relative speed is detected for a person appearing in an image, the risk level may be determined to be medium if the visually impaired person is stationary, and high if the visually impaired person is moving toward the person. Also, for example, if the current position is in a place with heavy traffic, the risk level may be determined to be higher than in a place with light traffic.

[0065] In this way, the privacy protection image generating unit 64 generates a privacy protection image (privacy protection processing) in which the image of a person in the image (real-time moving image) captured by the device 1 for visually impaired persons is filled in with a predetermined color, thereby protecting the privacy of the person. The real-time moving image subjected to the privacy protection processing is transmitted to the remote monitoring terminal device 3 of the supporter by the communication control unit 67 through the communication unit 55 in step S7 of the flowchart in FIG. 11. As a result, an image in which the privacy of the person in the image captured by the device 1 for visually impaired persons is protected is displayed on the monitor device 89 of the remote monitoring terminal device 3, as illustrated in FIG. 12(b). In addition, the color for filling in the image of the person in the image (real-time moving image) captured by the device 1 for visually impaired persons is selected according to the degree of danger of the person to the visually impaired person, so that the supporter can accurately grasp the situation around the visually impaired person.

[0066] As will be described in detail later, the assistant performs a marking process on the privacy-protected real-time video displayed on the monitor device 89, attaching a predetermined mark to a caution area that is predicted to be dangerous to a visually impaired person. FIG. 12(c) shows a privacy-protected image in which the assistant has marked a caution area that is predicted to be dangerous to a visually impaired person with an oval mark. In the example of FIG. 12(c), the step at the lower left, the paper bag placed at the lower left of the bronze statue, and the bronze statue have been marked by the assistant and designated as caution areas. Note that other marks such as arrows may be used as the marks used in the marking process.

[0067] In addition to inputting the areas requiring attention, the assistant can also input voice memo data (or text information) to serve as a reminder. The example in Fig. 12(c) shows that voice memo data "Watch your step, don't trip" was input for the step, voice memo data "Watch your step, don't bump into things" was input for the paper bag at the lower left of the bronze statue, and voice memo data "Watch out for obstacles" was input for the bronze statue.

[0068] The marking information indicating such a region requiring attention and the voice memo data (or text information) serving as a memo are transmitted to the center server device 2 together with the real-time moving image that has been subjected to privacy protection processing, and are stored in the HDD 54. The real-time moving image is stored as a past image. The marking information and the voice memo data (or text information) are then read out from the HDD 54 the next time assistance is provided at the same location, and the image captured by the device 1 for visually impaired persons is transmitted to the remote monitoring terminal device 3 together with the real-time moving image that has been subjected to privacy protection processing.

[0069] As a result, a mark indicating the previously inputted caution area corresponding to the imaging position of the real-time video image that has been subjected to privacy protection processing is displayed on the monitor device 89 of the remote monitoring terminal device 3 together with the privacy-protected real-time video image, and a voice message such as "Watch out for bumps at your feet" is output via the speaker unit 88 based on the voice memo data. Alternatively, when text information is received instead of the voice memo data, a mark indicating the caution area and text information such as "Watch out for bumps at your feet" are also displayed on the monitor device 89 of the remote monitoring terminal device 3. The supporter recognizes the caution area in the place where the visually impaired person is walking by visually recognizing the caution area displayed together with the privacy-protected real-time video image and listening to the voice message based on the voice memo data.

[0070] Then, the support person inputs a support voice, for example, "There is a step up ahead. Be careful not to trip," via the microphone unit 87. This support voice is transmitted to the device 1 for the visually impaired person and output via the earphone device 25. This support voice enables the visually impaired person to recognize that there is a step up ahead from the current location, enabling safe walking.

[0071] (Processing of marking information and display of areas requiring attention) Next, the above-mentioned marking information processing operation and the attention area display operation based on the marking information will be described in detail. Fig. 13 is a flowchart showing the flow of the marking information processing operation in the remote monitoring terminal device 3 and the attention area display process based on the marking information. In the flowchart of Fig. 13, first, in step S11, the communication control unit 94 and the acquisition unit 91 of the remote monitoring terminal device 3 shown in Fig. 10 receive and acquire the above-mentioned privacy-protected real-time video from the center server device 2. In step S12, the display control unit 92 controls the display of the privacy-protected real-time video on the monitor device 89.

[0072] In step S13, the marking processing unit 93 generates marking information indicating a dangerous object (area requiring caution) specified by an input operation, such as by surrounding a desired area of ​​the real-time video that has been subjected to privacy protection processing with an oval-shaped mark.

[0073] The marking information can be input while a supporter is monitoring the road on which the visually impaired person is walking.

[0074] Fig. 14 is a diagram showing an example of a privacy-protected real-time video in which a marking process for a caution area is performed by a supporter in step S13. Fig. 14 shows an example in which the image of a person to be subjected to privacy protection processing is not shown, and the steps at the boundary between the sidewalk and the roadway are marked with oval marks and designated as caution areas DZ1 and DZ2, respectively, and voice memo data such as "Watch out for tripping under your feet" and "Watch out for someone jumping out from the left" is input. When such a marking operation is performed, in step S14, the communication control unit 94 controls the communication unit 84 to transmit the privacy-protected real-time video in which the marking process is performed, the marking information indicating the designated caution area, and the voice memo data (or text information) to the center server device 2.

[0075] Here, the flowchart in FIG. 15 is a flowchart showing the flow of processing operation of the marking information in the center server device 2 that has received the marking information from the remote monitoring terminal device 3. When the center server device 2 receives the real-time video that has been subjected to the marking process and the privacy protection process, the marking information, and the voice memo data (or text information) from the remote monitoring terminal device 3, in step S21, the center server device 2 associates the received real-time video that has been subjected to the privacy protection process, the marking information, and the voice memo data (or text information) and stores them in the HDD 54. The "past image" stored in the HDD 54 shown in FIG. 7 indicates the real-time video that has been subjected to the privacy protection process stored in the HDD 54 in this step S21, that is, the real-time video that has been subjected to the privacy protection process received by the center server device 2 together with the marking information from the remote monitoring terminal device 3. Hereinafter, the real-time video that has been subjected to the privacy protection process and stored in the HDD 54 in step S21 will be referred to as a past image.

[0076] Next, the feature point extraction unit 61 extracts feature points from the past image based on the past image stored in the HDD 54 and the marking information associated therewith. To be more specific, first, the feature point extraction unit 61 shown in FIG. 8 extracts feature points in and near the attention area indicated by the marking information from the past image. FIG. 16 is a diagram showing the attention area and feature points specified for the past image. In FIG. 16, the area surrounded by an elliptical mark indicates the attention area DZ2. The feature point extraction unit 61 extracts feature points inside and outside the attention area DZ2 as indicated by the x marks in FIG. 16. Then, the feature point extraction unit 61 assigns a feature point identification number (feature point ID) of "1" to the feature point located at the center of gravity of the attention area DZ2 or closest to the center of gravity (ID=1).

[0077] Next, the feature point extraction unit 61 searches for segments in ascending order of distance from the segment containing the feature point with ID=1 to detect other feature points, and assigns feature point IDs to the feature points of each detected segment in counterclockwise (left-handed) order.

[0078] FIG. 17 is a diagram for explaining the order of assigning feature point IDs to each feature point. In FIG. 17, the regions divided by concentric ellipses shown by dotted lines indicate segments. The feature point with ID=1, which corresponds to the center of gravity of the attention area DZ2, is included in the innermost segment. When the feature point extraction unit 61 determines the feature point with ID=1, the feature point extraction unit 61 assigns feature point IDs to feature points present in a segment one segment outside the segment including this ID=1, in a counterclockwise (left) direction. Next, the feature point extraction unit 61 assigns feature point IDs to feature points present in a segment two segments outside the segment including ID=1, in a counterclockwise (left) direction. By repeating such an operation, the feature point extraction unit 61 assigns feature point IDs to each feature point in a counterclockwise (left) direction starting from ID=1, as shown by the spiral arrow in FIG. 17.

[0079] Next, the feature point extraction unit 61 stores information about each feature point to which a feature point ID has been assigned in this way in a feature point management table (see FIG. 7) shown in Table 1 below.

[0080] [Table 1]

[0081] As shown in Table 1, the feature point extraction unit 61 stores in the feature point management table the time stamp (TS), GPS position coordinate mesh area, and feature point ID attached to the past image in which the attention area is specified. The feature point extraction unit 61 also stores in the feature point management table the feature point image coordinates, attention area inside / outside flag, feature point vector value, and relative coordinates from the attention area origin (feature point ID1).

[0082] The time stamp (TS) is the imaging time information of the past image. This imaging time information was originally counted by the timer 19 shown in FIG. 2 and was attached to the image (real-time moving image) captured by the device for visually impaired persons 1, and the past image is also stored in the HDD 54 with the time stamp (TS) attached. The GPS position coordinate mesh area is the position information (one example of imaging position information) of the location where the visually impaired person was located at the time the past image was originally captured by the device for visually impaired persons 1. This position information is stored as, for example, the first mesh, the second mesh, etc., of the divided areas obtained by dividing the past image in which the caution area is specified into a plurality of meshes.

[0083] The feature point ID is an ID given to each feature point in order in a counterclockwise direction (left-handed direction) starting from ID=1 as described above. The feature point image coordinates are the X-axis and y-axis coordinate values ​​of each feature point. The attention area inside / outside flag is information indicating whether the feature point of each feature point ID is located inside or outside the attention area. If the feature point of that feature point ID is located inside the attention area, a attention area inside / outside flag of "1" is stored in the feature point management table. On the other hand, if the feature point of that feature point ID is located outside the attention area, a attention area inside / outside flag of "0" is stored in the feature point management table.

[0084] The feature point vector value is the vector value of the feature point of each feature point ID as shown in Fig. 18. The relative coordinates from the origin of the attention area (feature point ID 1) indicate the X and y coordinate values ​​of each other feature point, which indicate the distance away from the feature point with ID = 1.

[0085] Next, the feature point extraction unit 61 extracts feature points from the frames of the past images from which the feature points were extracted in this manner for a certain time period of past frames and a certain time period of future frames in the same manner as described above, and stores the coordinates of the feature points in the image in a feature point management table. The marking information processing unit 71 refers to this feature point management table and detects an attention area for each frame based on the center of gravity and distribution range of each feature point. The marking information processing unit 71 stores the coordinates of the detected attention area in the attention area management table shown in Table 2 below in association with each frame (see FIG. 7).

[0086] [Table 2]

[0087] As shown in Table 2, the marking information processing unit 71 stores the time stamp (TS) attached to the past image and the attention area identification number (attention area ID) attached to the attention area specified on that frame in the attention area management table. The marking information processing unit 71 also stores the feature point image coordinates of each feature point inside and outside the attention area, the attention area inside and outside flag, weighting coefficient, feature point vector value, and relative coordinates from the attention area origin (feature point ID1) in the attention area management table.

[0088] The caution area inside / outside flag is information indicating whether the feature point of each feature point ID is located inside or outside the caution area. If the feature point of that feature point ID is located inside the caution area, the caution area inside / outside flag of "1" is stored in the caution area management table. If the feature point of that feature point ID is located outside the caution area, the caution area inside / outside flag of "0" is stored in the caution area management table.

[0089] The weighting coefficient is a coefficient used when calculating a region feature vector, which will be described later. If the feature point of the feature point ID is located inside the region requiring attention, a weighting coefficient of "1" is used. If the feature point of the feature point ID is located outside the region requiring attention, a weighting coefficient of "0" is used.

[0090] The feature point vector value is the vector value of the feature point of each feature point ID as shown in Fig. 18. The relative coordinates from the origin of the attention area (feature point ID 1) indicate the X and y coordinate values ​​of each other feature point, which indicate the distance away from the feature point with ID = 1.

[0091] By referring to such an attention-required region management table, the position and size of the attention-required region marked by the supporter can be detected, and can be reflected in the corresponding real-time video image.

[0092] That is, when generating a real-time video reflecting an attention-requiring area, the current position identification unit 63 identifies the current position of the visually impaired person based on the GPS information transmitted from the visually impaired person device 1 in step S22. In step S23, the storage control unit 70, which is an example of a read control unit, reads out a previous image corresponding to the imaging position of the real-time video from the HDD 54. The storage control unit 70 also detects the coordinate position and size of the attention-requiring area associated with the read out previous image by referring to the attention-requiring area management table. Furthermore, the storage control unit 70 reads out voice memo data (or text information) associated with the read out previous image from the HDD 54.

[0093] The support image generating unit 66 generates a "support image" including a real-time video reflecting the attention-required area of ​​the position and size detected by referring to the attention-required area management table, voice memo data (or text information), and map data indicating the current location of the visually impaired person. Then, the communication control unit 67 transmits such a support image to the remote monitoring terminal device 3. Note that the past image and the real-time video are subjected to the above-mentioned privacy protection processing.

[0094] 13, the communication control unit 94 of the remote monitoring terminal device 3 controls the communication unit 84 so as to receive the support image transmitted from the center server device 2. Then, the display control unit 92 of the remote monitoring terminal device 3 displays the support image received from the center server device 2 on the monitor device 89.

[0095] The supporter recognizes the surroundings of the visually impaired person's current location based on the privacy-protected real-time video included in the support image, and recognizes the attention-required area corresponding to the visually impaired person's current location based on the past image reflecting the attention-required area and the voice memo data (or text information). Then, the supporter inputs a support voice for avoiding the attention-required area via the microphone unit 87. In step S16, this support voice is transmitted by the communication control unit 94 to the center server device 2, and then transmitted from the center server device 2 via the network 4 and the wireless base station 5 to the visually impaired person device 1, and output via the earphone device 25 of the visually impaired person device 1. This enables the visually impaired person to avoid danger.

[0096] FIG. 19 is a diagram showing an example of a support image displayed on the monitor device 89 of the remote monitoring terminal device 3. FIG. 19(a) is a map image showing the current location of the visually impaired person detected based on GPS information received from the visually impaired person device 1. FIG. 19(a) shows, from the top, the movement of the current location of the visually impaired person over time. The current location identification unit 63 of the center server device 2 identifies the current location of the visually impaired person based on the GPS information received from the visually impaired person device 1. Then, the support image generation unit 66 generates a map image showing the current location of the visually impaired person identified by the current location identification unit 63.

[0097] FIG. 19(b) shows past images (real-time video captured in the past and subjected to privacy protection processing) corresponding to the current position of the visually impaired person identified by the current position identification unit 63, that is, the capturing position of the real-time video. From the top, FIG. 19(b) shows images of scenery such as sidewalks, roadways, and stores captured over time in the past (TS=0001 to TS=0005). If a person is captured, the image is stored after undergoing the above-mentioned privacy protection processing. The example of FIG. 19(b) also shows that marking information of caution areas DZ1 and DZ2 is added to the step at the end of the sidewalk, and voice memo data (or text information) saying "Watch your step" is added to each past image.

[0098] The support image generating unit 66 of the center server device 2 reads out a past image corresponding to the imaging position of the real-time video from the HDD 54. The support image generating unit 66 also refers to the attention area management table to reflect the attention area corresponding to the read out past image on the past image, for example, with an oval mark. As a result, a past image is generated in which the attention area corresponding to the moving position of the visually impaired person is reflected, as shown in Figures 19(a) and 19(b).

[0099] Fig. 19(c) shows real-time video that has undergone privacy protection processing of the scenery around the current location of the visually impaired person, captured by the device 1 for visually impaired people. Fig. 19(c) shows, from top to bottom, real-time video that has undergone privacy protection processing, captured over time (TS=1001 to TS=1005). Fig. 19(b) and Fig. 19(c) show video captured at the same location at different times, in the past (TS=0001 to TS=0005) and the present (TS=1001 to TS=1005).

[0100] The support image generating unit 66 of the center server device 2 generates support images including a map image (FIG. 19(a)) showing the current location of the visually impaired person, a past image (FIG. 19(b)) reflecting an oval mark indicating an area requiring attention, and a real-time video image (FIG. 19(c)) captured by the device for visually impaired people 1 and subjected to privacy protection processing. The communication control unit 67 controls the communication unit 55 to add voice memo data (or text information) associated with each past image to the support images generated in this way and transmit them to the remote monitoring terminal device 3.

[0101] The display control unit 93 of the remote monitoring terminal device 3 controls the display of the support image received from the center server device 2 on the monitor device 89. In addition, the audio processing unit 95 controls the output of audio corresponding to the audio memo data added to the support image via the speaker unit 88. This allows the supporter to recognize the surroundings of the visually impaired person based on the current position of the visually impaired person (FIG. 19(a)) and real-time video images (FIG. 19(c)) that have been captured by the device 1 for visually impaired people and have undergone privacy protection processing. In addition, the supporter can recognize the surroundings of the visually impaired person where the visually impaired person is currently walking based on the past image (FIG. 19(b)) reflecting the attention-requiring area and the audio from the audio memo data, and transmit audio to support the visually impaired person in walking.

[0102] The voice of the support person providing support to the visually impaired person is collected by the microphone unit 87 of the remote monitoring terminal device 3, transmitted to the device for visually impaired person 1 via the communication unit 84 by the communication control unit 94, and output via the earphone device 25 of the device for visually impaired person 1. This allows the visually impaired person to hear the support voice of the support person via the earphone device 25, and can take action to avoid danger before it occurs.

[0103] (An example of reflecting areas of interest specified in past images in real-time video) Next, the example described with reference to Fig. 19 is an example of generating a support image including the current location of the visually impaired person, a real-time video, and a past image. However, an elliptical mark or the like indicating a region requiring attention specified in a past video may be superimposed on the real-time video. In this case, a support image including both the current location of the visually impaired person and the real-time video on which an elliptical mark indicating a region requiring attention is superimposed is displayed on the monitor device 89 of the remote monitoring terminal device 3.

[0104] Here, when displaying an attention area designated on a past image on a real-time video, it is preferable to accurately detect the attention area designated on the past image from the real-time video. In this case, the remote support system of the embodiment detects an area feature vector from the real-time video that is the same as the area feature vector of the attention area designated on the past image (area feature vector matching process). This allows the attention area designated on the past image to be accurately reflected on the real-time video.

[0105] Fig. 20 is a flow chart showing the flow of the operation of reflecting the attention area specified in the past image in the real-time video. In this case, the calculation unit 62 of the remote monitoring terminal device 3 shown in Fig. 8 calculates in advance the area feature vector of the attention area indicated by the above-mentioned feature point marking information shown in Table 2 based on the following formulas 1 and 2, and stores it in a storage unit such as the RAM 53 or the HDD 54 (step S31).

[0106]

number

[0107]

number

[0108] FIG. 18 shows the region feature vector of the attention-required region calculated by the formula 1 and the formula 2. As shown in FIG. 18, the calculation unit 62 calculates the direct sum of the feature point vector values ​​corresponding to each attention-required region based on the formula 1 as the region feature vector "V i The number of dimensions of the region feature vector is the sum of the number of dimensions of each feature point vector corresponding to each attention-requiring region.

[0109] That is, in Equation 1, "cj" is a weighting coefficient that is assigned depending on whether the feature point is inside or outside the attention area, as shown in Table 2. "vj" is the vector value of each feature point. "Xdj" is the distance on the X coordinate away from the feature point with ID=1 located at the center of gravity of the attention area. "ydj" is the distance on the y coordinate away from the feature point with ID=1 located at the center of gravity of the attention area.

[0110] The calculation unit 62 performs direct sum processing on each feature point vector value “vj” as shown in the formula 2 to obtain a region feature vector “V i " is calculated and stored in a storage unit such as the RAM 53 or the HDD 54. As a result, the storage unit stores area feature vectors corresponding to the respective attention-required areas on the respective past images.

[0111] Next, in step S32 of FIG. 20, the storage control unit 70 reads out from the HDD 54 a past image corresponding to the imaging position of the real-time moving image based on the imaging time of the real-time moving image.

[0112] In step S33, the calculation unit 62 calculates region feature vectors of all regions in the real-time video based on the above-mentioned formula 1 and formula 2. Also in step S33, the matching processing unit 65 performs a matching process between each region feature vector of the real-time video and the region feature vector of the attention region of the past image read from the HDD 54. Then, the matching processing unit 65 detects the region feature vector in the real-time video that has the highest degree of matching with the region feature vector of the attention region of the past image.

[0113] In step S34, the matching processing unit 65 determines whether the region feature vector of the real-time video image having the highest matching degree to the region feature vector of the attention region of the past image has a matching degree equal to or greater than a certain value.

[0114] If the matching degree is equal to or greater than a certain value (step S34: Yes), this means that there is almost no geographical discrepancy between the real-time video and the past image, and the process proceeds to step S38 to generate a real-time video that reflects the attention-required areas of the past image.

[0115] The real-time video has been subjected to the above-mentioned privacy protection processing. The real-time video is then transmitted by the communication control unit 67 to the remote monitoring terminal device 3 via the communication unit 55 together with the voice memo data (or text information) corresponding to the attention-requiring area and the map data indicating the current location of the visually impaired person. The supporter of the remote monitoring terminal device 3 recognizes the presence of the attention-requiring area based on the attention-requiring area reflected in the real-time video and the voice indicating the precautions for the attention-requiring area by the voice memo data, and transmits a support voice to the visually impaired person device 1 to support the visually impaired person.

[0116] On the other hand, if a matching degree of a certain value or more is not obtained (step S34: No), it means that an area of ​​a size matching the attention area of ​​the past image is not detected in the real-time video. In this case, the process proceeds to step S35, and the attention area change processing unit 72 changes (enlarges or reduces) the size of the attention area by a certain factor, and changes the coordinates of the feature points in the area feature vector by the same ratio.

[0117] In step S36, the matching processor 65 performs matching processing on the real-time video again based on the region feature vector of the attention region changed in this way. Then, in step S37, the matching processor 65 detects the region feature vector of the real-time video that matches the region feature vector of the attention region of the changed past image most closely.

[0118] In step S37, the matching processing unit 65 determines whether the area feature vector in the real-time video that has the highest matching degree to the area feature vector of the attention area of ​​the changed past image has a matching degree equal to or greater than a certain value.

[0119] If a matching degree of a certain value or more is not obtained (step S37: No), it means that an area of ​​a size matching the attention area of ​​the past image is not detected in the real-time video. Therefore, the process returns to step S35, and the attention area change processing unit 72 again changes (enlarges or reduces) the size of the attention area by a certain number of times, and changes the coordinates of the feature points in the area feature vector by the same ratio. The matching processing unit 65 again performs the above-mentioned matching process based on the area feature vector of the attention area of ​​the past image that has been changed.

[0120] As a result of repeating the matching process in this manner, if a region feature vector with a matching degree equal to or greater than a certain value is detected from the real-time video (step S37: Yes), in step S38, the region of interest from the past image is reflected in the region detected from the real-time video.

[0121] The real-time video has been subjected to the above-mentioned privacy protection processing. The real-time video, together with the voice memo data (or text information) corresponding to the reflected attention-requiring area and the map data indicating the current location of the visually impaired person, is transmitted as a support image by the communication control unit 67 to the remote monitoring terminal device 3 via the communication unit 55. The supporter of the remote monitoring terminal device 3 recognizes the presence of the attention-requiring area from the attention-requiring area reflected in the real-time video and the voice memo data (or text information), and transmits a support voice for supporting the visually impaired person to the device 1.

[0122] FIG. 21 shows an example of a real-time video in which an area requiring attention from a past image is reflected. FIG. 21(a) shows a real-time video captured by the device 1 for visually impaired persons over time (TS=1001 to TS=1005). FIG. 21(b) is map data showing the movement of the current location of a visually impaired person. FIG. 21(c) shows a past image (TS=0001 to TS=0005) in which an area requiring attention is specified. The text such as "Watch your step" displayed in this past image is a visual representation of the voice memo data.

[0123] As described above, when the attention-requiring area of ​​the past image is reflected in the real-time video, the support image generating unit 66 generates a support image, which is a real-time video reflecting the attention-requiring area of ​​the past image, together with map data showing the current position of the visually impaired person and voice memo data shown in Fig. 21(b). This support image is transmitted to the remote monitoring terminal device 3 by the communication control unit 67 via the communication unit 55.

[0124] As a result, as shown in FIG. 21(d), a real-time video reflecting the attention-requiring area designated on the past image is displayed on the monitor device 89 of the remote monitoring terminal device 3. Also, a voice corresponding to the voice memo data is outputted through the speaker unit 88 of the remote monitoring terminal device 3. The supporter recognizes an area where the visually impaired person needs to pay attention based on the attention-requiring area displayed in the real-time video and the voice related to the attention-requiring area outputted through the speaker unit 88. Then, the supporter inputs a support voice to call the attention of the visually impaired person through the microphone unit 87. This support voice is subjected to a predetermined data processing by the voice processing unit 95, transmitted to the visually impaired person device 1 through the communication unit 84 by the communication control unit 94, and outputted through the earphone device 25 of the visually impaired person device 1. As a result, the visually impaired person can hear the support voice of the supporter, and can walk safely, for example, by avoiding a step ahead.

[0125] (Effects of the embodiment) As is clear from the above description, the remote support system according to the embodiment can provide the following effects.

[0126] 1. A caution area is displayed on a past image or a real-time moving image corresponding to the current position of a supportee such as a visually impaired person, and displayed on the supporter's remote monitoring terminal device 3. This allows the supporter to easily recognize things that require attention at the current position of the supportee, such as a jump from the side, a step or a hole at the feet, without requiring high concentration for a long time and danger prediction skills, and enables accurate support of the supportee. Therefore, for example, a visually impaired person can go out without relying on the accompaniment of a guide dog or a guide helper, dealing with various walking risks. In addition, since the supporter can support the supportee without requiring high concentration for a long time and danger prediction skills, the monitoring effort of the supportee can be reduced, and one supporter can easily support multiple supportees.

[0127] 2. People and the like appearing in past images or real-time moving images displayed on the supporter's remote monitoring terminal device 3 are filled in with a predetermined color and provided to the remote monitoring terminal device 3. This makes it possible to protect the privacy of people and the like appearing in past images or real-time moving images.

[0128] 3. The color used to fill in people and other objects is changed according to the degree of danger to the person being supported. This allows the supporter to be informed of the degree of danger to the person being supported from each person around the person being supported. This allows the supporter to be able to avoid danger in advance.

[0129] 4. A wide-angle lens such as a fisheye lens is used for the camera device 26 that captures the surrounding scenery of the person being supported. This allows a wider range of the surroundings of the person being supported to be captured. The supporter can then provide support to the person being supported while monitoring a wider range based on the captured images of such a wider range.

[0130] Finally, the above-described embodiment is presented as an example, and is not intended to limit the scope of the present invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention.

[0131] For example, in the above-described embodiment, the functions realized by the various programs may be realized in whole or in part by hardware such as an integrated circuit (IC).

[0132] The above-mentioned various programs may be provided by recording the file information in an installable or executable format on a computer-readable recording medium such as a CD-ROM or a flexible disk (FD). The above-mentioned various programs may be provided by recording the file information in an installable or executable format on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disc), a Blu-ray (registered trademark) disk, or a semiconductor memory. The above-mentioned various programs may be provided by installing the programs via a network such as the Internet. The above-mentioned various programs may be provided by being pre-installed in a ROM or the like in the device.

[0133] Furthermore, the system configuration of the remote support system includes a visually impaired person device 1 carried by a visually impaired person, a remote monitoring terminal device 3 operated by a supporter, and a center server device 2 that mediates communication between the visually impaired person device 1 and the remote monitoring terminal device 3. However, the functions of the center server device 2 may be distributed among the portable terminal device 1 and the remote monitoring terminal device 3. Alternatively, some or all of the functions of the center server device 2 may be provided in the remote monitoring terminal device 3. When all of the functions of the center server device 2 are provided in the remote monitoring terminal device 3, the center server device 2 can be omitted from the system configuration of the remote support system, and the above-mentioned effects can be obtained.

[0134] In the above embodiment, a support image is generated including a past image in which an attention area is reflected by, for example, an elliptical mark, or a support image is generated including a real-time video image in which an attention area is reflected. However, a support image may be generated by reflecting the attention area in both the past image and the real-time image. In this case, the same effect as described above can be obtained.

[0135] Furthermore, the accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions. [Explanation of symbols]

[0136] 1. Equipment for the visually impaired 2. Center server device 3. Remote monitoring terminal device 4. Network 5. Radio base station 7 Headset device 17 GPS Sensor 18 Acceleration Sensor 19 Timer 25 Earphone device 26 Camera Equipment 27 Microphone Equipment 36L Left unit housing 36R Right unit housing 60 Acquisition Department 61 Feature point extraction unit 62 Calculation section 63 Current position identification part 64 Privacy protection image generation unit 65 Matching processing section 66 Support Image Generation Unit 67 Communication control section 68 Color Selection 69 Person area detection unit 70 Memory control unit 71 Marking information processing section 72 Attention area change processing section 87 Microphone section 88 Speaker section 89 Monitor Device 93 Marking Processing Section

Claims

1. A storage control unit that controls storage in a storage unit of a past image captured in the past by a camera device carried by the support recipient, image capture location information indicating the geographical image capture location of the past image, and marking information regarding a predetermined mark added to the past image, in association with each other; A read control unit that receives a real-time captured image captured in real time by the camera device carried by the support recipient, and reads out the past image and the marking information corresponding to the capturing position of the real-time captured image from the storage unit; a support image generating unit that generates a support image to be displayed on a display unit monitored by a supporter who supports the supportee, the support image being generated by reflecting a predetermined mark indicated by the marking information read out from the storage unit on either or both of the real-time captured image and the past image; A remote support device having the above configuration.

2. The support image generation unit reflects, on the real-time captured image, the predetermined mark indicated by the marking information corresponding to the imaging position of the real-time captured image read out by the read control unit from the storage unit.

2. The remote support device according to claim 1,

3. The storage control unit further controls the storage unit to store auxiliary information that allows the supporter to recognize information related to the marking information, The support image generating unit adds the auxiliary information related to the marking information to the support image.

3. The remote support device according to claim 1 or 2,

4. The image capturing apparatus further includes a privacy protection unit that performs a predetermined privacy protection process on privacy protection objects included in the past image and the real-time captured image.

4. The remote support device according to claim 1, wherein the remote support device is a remote support device.

5. The privacy protection unit performs processing to display the privacy protection target in a predetermined color.

5. The remote support device according to claim 4.

6. The privacy protection unit performs processing to represent the privacy protection target in a color corresponding to a degree of danger that the privacy protection target poses to the support recipient.

6. The remote support device according to claim 5,

7. a matching processing unit that performs a matching process for searching, when reflecting the mark in the real-time captured image, an area having the same image characteristics as an area in the past image in which the mark is reflected, from the real-time captured image; The support image generating unit generates the support image by reflecting the mark in the area on the real-time captured image searched by the matching processing unit.

7. The remote support device according to claim 1, wherein the remote support device is a remote support device.

8. When an area having a degree of matching equal to or higher than a predetermined degree of matching with image characteristics of the area on the past image where the mark is reflected is not found from the real-time captured image, the matching processing unit performs the matching process based on image characteristics obtained by changing a size of the area on the past image where the mark is reflected.

8. The remote support device according to claim 7,

9. The camera device is provided with a wide-angle lens having a wide angle of view, or a stereo camera installed on the left and right in front of the support recipient.

9. The remote support device according to claim 1, wherein the remote support device is a remote support device.

10. a storage control step in which a storage control unit controls storage in a storage unit of a past image captured in the past by a camera device carried by the support recipient, image capture location information indicating a geographical image capture location of the past image, and marking information regarding a predetermined mark attached to the past image, in association with each other; A read control step in which a read control unit receives a real-time captured image captured in real time by the camera device carried by the support recipient, and reads out the past image and the marking information corresponding to the capturing position of the real-time captured image from the storage unit; a support image generating step in which a support image generating unit generates a support image to be displayed on a display unit monitored by a supporter who supports the supportee, the support image being generated by reflecting a predetermined mark indicated by the marking information read out from the storage unit on either or both of the real-time captured image and the past image; A remote support method comprising the steps of:

11. Computer, A storage control unit that controls storage in a storage unit of a past image captured in the past by a camera device carried by the support recipient, image capture location information indicating the geographical image capture location of the past image, and marking information regarding a predetermined mark added to the past image, in association with each other; A read control unit that receives a real-time captured image captured in real time by the camera device carried by the support recipient, and reads out the past image and the marking information corresponding to the capturing position of the real-time captured image from the storage unit; The support image is to be displayed on a display unit monitored by a supporter who supports the support recipient, and the support image generating unit generates a support image in which a predetermined mark indicated by the marking information read out from the storage unit is reflected on either or both of the real-time captured image and the past image. A remote support program featuring:

12. A remote support system having a support recipient device carried by the support recipient, the support recipient device including a camera device for capturing an image of the surroundings of the support recipient and an audio output device for outputting audio for supporting the support recipient, a display device for displaying an image captured by the camera device, and a remote monitoring terminal device including a transmission unit for collecting the support voice of a supporter supporting the support recipient and transmitting the support voice to the support recipient device, Further comprising a remote support device that relays communication between the support recipient device and the remote monitoring terminal device, The remote support device includes: A storage control unit that controls storage in a storage unit of a past image captured in the past by the camera device carried by the support recipient, image capture location information indicating a geographical image capture location of the past image, and marking information regarding a predetermined mark added to the past image, in association with each other; A read control unit that receives a real-time captured image captured in real time by the camera device carried by the support recipient, and reads out the past image and the marking information corresponding to the capturing position of the real-time captured image from the storage unit; and a support image generating unit that generates a support image to be displayed on a display unit monitored by a supporter who supports the support recipient, the support image being generated by reflecting a predetermined mark indicated by the marking information read out from the storage unit on either or both of the real-time captured image and the past image. A remote support system characterized by the above.

Images