Image processing device and image processing method

Abstract

To provide an image processing device and a method thereof that can make a user clearly understand a display target even when a movement or a moving object is displayed at a remote location.SOLUTION: An image processing device 220 according to the present invention includes an image acquiring unit that acquires images captured by an outside camera 110 and an inside camera 120, an object detection unit that detects an object that a user 230 of a display device 240 needs to see, a display determination unit that determines whether the object is unclear, and a display image generation unit that changes the display mode of the object and generates a display image when the object is unclear, and determines that the object is unclear in at least one of a case in which the size of the object is less than or equal to a predetermined threshold, a case in which a time in which the object is displayed is less than or equal to the predetermined threshold, and a case in which the rate of change in the size of the object is greater than or equal to the predetermined threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus and an image processing method for generating a display image to be displayed on a display device located at a remote location away from an imaging device, based on an image captured by the imaging device. [Background technology]

[0002] Conventionally, a spatial display device that displays a virtual space three-dimensionally is disclosed in Patent Document 1. The spatial display device disclosed in Patent Document 1 appropriately displays parts that are difficult to see from the current field of view by displaying parts in areas with good visibility in the virtual space in different forms from parts in areas with poor visibility. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-205974 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, although the above-mentioned conventional spatial display device can accommodate changes in the field of view, the display object is stationary, and therefore when a moving or dynamic object is displayed at a remote location, the display object may be displayed unclearly due to communication delays, etc. Therefore, the conventional spatial display device has a problem in that when a moving or dynamic object is displayed at a remote location, the display object cannot be clearly understood by the user.

[0005] Therefore, the present invention has been made in consideration of the above problems, and aims to provide an image processing device and an image processing method that can allow a user to clearly understand the displayed object even when a moving or dynamic object is displayed at a remote location. [Means for solving the problem]

[0006] According to an aspect of the present invention, an image processing device and a method thereof acquire an image captured by an imaging device, detect an object that a user of a display device needs to view in the captured image, determine whether the detected object is unclear, and if it is determined that the object is unclear, change the display mode of the object to generate a display image. generating a sound to inform a user of the detected object; When there is an object that is captured in the captured image but not displayed in the displayed image , and if the user does not respond to the voice If there is an object that is captured in the captured image but not displayed in the display image, the display mode is changed to one in which an image indicating the direction of the object is superimposed, and a display image is generated. According to another aspect of the present invention, an image processing device and a method thereof acquire an image captured by an imaging device, detect an object that a user of a display device needs to view in the captured image, determine whether the detected object is unclear, and if it is determined that the object is unclear, change the display mode of the object to generate a display image. generating a sound to inform a user of the detected object; When there is an error in the display position of an object , and if the user does not respond to the voice Then, it determines that the object is obscured. Effect of the Invention

[0007] According to the present invention, even when an object that moves or is in motion is displayed at a remote location, the user can clearly understand the displayed object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of a virtual space display system including an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. [Diagram 3]FIG. 3 is a diagram showing an example of a display screen when the display mode of an object is changed to highlighted display by the image processing device according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a display screen when the display mode of objects by the image processing device according to one embodiment of the present invention is changed to a superimposed display of icons. [Diagram 5] FIG. 5 is a diagram showing an example of a display screen when the display mode of an object by the image processing device according to one embodiment of the present invention is changed to a superimposed display of an image indicating the direction of the object. [Figure 6] FIG. 6 is a diagram showing an example of a display screen when correcting the display position of an object by an image processing device according to an embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart showing a processing procedure for changing the display mode of an object by an image processing device according to an embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart showing a processing procedure of a user attention drawing process performed by an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0010] [Virtual space display system configuration] Fig. 1 is a block diagram showing the configuration of a virtual space display system including an image processing device according to this embodiment. As shown in Fig. 1, the virtual space display system 1 captures images of the interior of a vehicle in a vehicle space 100 and the surroundings of the vehicle, and provides a display image of a virtual space 300 to a user 230 in a remote space 200 away from the vehicle space 100. The display image shows the interior of the vehicle and the surroundings of the vehicle in the virtual space 300.

[0011] The vehicle space 100 is a real space inside and outside the vehicle, and is equipped with an exterior camera 110, an interior camera 120, a controller 130, and a communication device 140. In the vehicle space 100, an occupant 160 is riding in the interior of a vehicle 150, and the occupant 160 is wearing a display device 170.

[0012] The exterior camera 110 is an imaging device mounted on the vehicle 150 and captures images of the surroundings of the vehicle 150. The exterior camera 110 outputs captured images of the surroundings of the vehicle 150 to the controller 130. The captured images include not only images of the area in front of the vehicle, but also 360-degree images including the sides and rear of the vehicle 150. The exterior camera 110 is also equipped with a microphone, which records audio of the surroundings of the vehicle 150 along with the images and outputs the audio to the controller 130 together with the captured images.

[0013] The in-vehicle camera 120 is an imaging device mounted on the vehicle 150 and captures an image of the situation inside the vehicle cabin of the vehicle 150. The in-vehicle camera 120 outputs an image of the situation inside the vehicle cabin of the vehicle 150 to the controller 130. The captured image captures not only images of the driver's seat and the passenger seat, but also images of the entire interior of the vehicle cabin, including images of the rear seats. The in-vehicle camera 120 is also equipped with a microphone, and audio from inside the vehicle cabin of the vehicle 150 is recorded along with the image and output to the controller 130 together with the captured image.

[0014] The controller 130 can communicate with the exterior camera 110, the interior camera 120, the communication device 140, and the display device 170 wirelessly or by wire. The controller 130 acquires captured images and audio from the exterior camera 110 and the interior camera 120, and transmits them to the remote space 200 via the communication device 140. The controller 130 also includes a main memory configured with a central processing unit (CPU) and a random access memory (RAM). The controller 130 executes a program using the main memory as a working area, and performs information processing related to the display of a virtual space such as an augmented reality (AR) or a virtual reality (VR). The controller 130 may be a personal computer (PC), a game machine, a server device, a smartphone, or the like, as long as it is capable of executing a program and performing arithmetic processing. The controller 130 may be mounted on the vehicle 150, or may be incorporated in the display device 170.

[0015] The communication device 140 is an interface that connects to a network such as the Internet and transmits and receives data to and from the remote space 200 .

[0016] The display device 170 is a so-called head-mounted display (HMD). The display device 170 is configured to be worn on the head of the occupant 160, thereby holding a display for displaying an image in front of the eyes of the occupant 160. The display device 170 also includes a sensor for detecting the movement of the line of sight of the occupant 160. Note that the display device 170 may be a wearable display, or may be a glasses-type device.

[0017] As a head-mounted display applied to the display device 170, a see-through HMD such as AR goggles is applied.

[0018] The see-through HMD holds a virtual image optical system using, for example, a half mirror or a transparent light guide plate in front of the eyes of the occupant 160, and displays an image inside the virtual image optical system. Therefore, the occupant 160 wearing the see-through HMD can view the outside scenery even while viewing the image displayed on the virtual image optical system. Note that a specific example of the see-through HMD is a glasses-type device in which the part equivalent to the lens of glasses is configured as the virtual image optical system.

[0019] Further, the display device 170 is provided with, for example, an acceleration sensor and a gyro sensor, and is configured to be able to detect the movement of the head (head posture) of the occupant 160 wearing the display device 170. As an example, the display device 170 detects each of the components in the yaw direction, pitch direction, and roll direction as the movement of the head of the occupant 160. Then, the display device 170 recognizes a change in the direction indicated by the line of sight of the occupant 160 based on the detected movement of the head of the occupant 160, and presents an image according to the recognized change in the line of sight direction to the occupant 160. Based on such a configuration, for example, an image in a virtual space is presented to the occupant 160 according to the change in the line of sight direction of the occupant 160. This makes it possible to give the occupant 160 a sense of immersion as if he or she were in a virtual space.

[0020] Moreover, the display device 170 may include an optical sensor (camera). The display device 170 can track the gaze of the occupant 160 by using the optical sensor. The display device 170 may include a proximity sensor. The proximity sensor is installed in the goggles and is used to determine whether or not the display device 170 is in contact with the face of the occupant 160.

[0021] Next, the remote space 200 is a real space in a remote location away from the vehicle space 100, and is equipped with a communication device 210 and an image processing device 220. In the remote space 200, a user 230 wears a display device 240 and views an image of the virtual space 300.

[0022] The communication device 210 is an interface that connects to a network such as the Internet and transmits and receives data to and from the vehicle space 100 .

[0023] The image processing device 220 generates a display image to be displayed on the display device 240 in the remote space 200 based on the captured images captured by the exterior camera 110 and the interior camera 120. Specifically, the image processing device 220 acquires the captured images captured by the exterior camera 110 and the interior camera 120, and detects an object that the user 230 needs to view in the acquired captured images. Then, the image processing device 220 determines whether the detected object is unclear, and if it is determined that the object is unclear, changes the display mode of the object to generate a display image.

[0024] The image processing device 220 can communicate with the communication device 210 and the display device 240 wirelessly or by wire. The image processing device 220 includes a controller, which includes a main memory configured with a CPU (Central Processing Unit) and a RAM (Random Access Memory) and the like. The image processing device 220 executes a program using the main memory as a working area, and performs information processing related to the display of a virtual space such as AR (Augmented Reality) or VR (Virtual Reality). The image processing device 220 may be a personal computer (PC), a game machine, a server device, a smartphone, or the like, as long as it is capable of executing a program and performing arithmetic processing. The image processing device 220 may be incorporated in the display device 240.

[0025] Next, the configuration of the image processing device 220 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the image processing device according to this embodiment. As shown in Fig. 2, the image processing device 220 includes a captured image acquisition unit 221, an object detection unit 223, a display determination unit 225, a display image generation unit 227, and a sound generation unit 229.

[0026] The captured image acquisition unit 221 acquires, via a communication network, captured images captured by the exterior camera 110 and the interior camera 120 in the vehicle space 100. The captured images captured by the exterior camera 110 and the interior camera 120 installed in the vehicle space 100 are transmitted from the vehicle space 100 by the communication device 140, received by the communication device 210 in the remote space 200, and acquired by the captured image acquisition unit 221.

[0027] The object detection unit 223 detects an object that the user 230 needs to view on the captured image acquired by the captured image acquisition unit 221. The detected object is an object to be noted that the vehicle occupant needs to view, such as a pedestrian, a bicycle, a motorbike, a vehicle, other obstacles, and further a traffic light, a sign, etc. Also, an avatar of the occupant 160 or the user 230 may be detected as an object. The object detection unit 223 detects an object from the captured image by performing image analysis on the captured image.

[0028] The display determination unit 225 determines whether or not an object detected by the object detection unit 223 is unclear on the captured image. Since the captured image is captured in the vehicle space 100 and transmitted to the remote space 200, there may be a portion of the captured image that is unclearly displayed due to the performance of the communication device, communication delay, etc. Therefore, the display determination unit 225 determines whether or not the detected object is unclear.

[0029] Specifically, when the size of an object on a captured image is equal to or smaller than a predetermined threshold, the display determination unit 225 determines that the object is unclear. For example, when a preceding vehicle is detected as an object, if the number of pixels on one side of a rectangular bounding box surrounding the preceding vehicle is equal to or smaller than a predetermined threshold, the object is too small for the user 230 to visually recognize. Therefore, the display determination unit 225 determines that the object is unclear.

[0030] In addition, the display determination unit 225 determines that an object is unclear when the time that the object is displayed on the captured image is equal to or less than a predetermined threshold. For example, when a bicycle is detected as an object, if the bicycle moves laterally on the captured image and is displayed for a time equal to or less than a predetermined threshold, the bicycle is displayed only for a short time, and it becomes difficult for the user 230 to see due to communication delays, etc. Therefore, the display determination unit 225 determines that the object is unclear. Also, the object may be determined to be unclear when the moving speed of the object is equal to or greater than a predetermined threshold. For example, when a bicycle is detected as an object, if the speed at which the bicycle moves laterally on the captured image is equal to or greater than a predetermined threshold, the bicycle moves too fast and it becomes difficult for the user 230 to see due to communication delays, etc. Therefore, the display determination unit 225 determines that the object is unclear.

[0031] Furthermore, the display determination unit 225 determines that an object is unclear when the rate of change in size of the object on the captured image is equal to or greater than a predetermined threshold. For example, when an oncoming vehicle is detected as an object, the oncoming vehicle may approach fast on the captured image and the rate of change in size of the oncoming vehicle may be equal to or greater than a predetermined threshold. In such a case, the change in size of the oncoming vehicle may be too fast, making it difficult for the user 230 to visually recognize it due to communication delays, etc. Therefore, the display determination unit 225 determines that the object is unclear.

[0032] In this way, the display determination unit 225 determines that an object is unclear when at least one of the following occurs: the size of the object is equal to or smaller than a predetermined threshold, the time during which the object is displayed is equal to or smaller than a predetermined threshold, and the rate of change in the size of the object is equal to or larger than a predetermined threshold. Furthermore, the display determination unit 225 may also determine that an object is unclear in the following cases:

[0033] The display determination unit 225 determines that an object is unclear when the color difference between the object and its surroundings in the captured image is equal to or less than a predetermined threshold. For example, when a pedestrian is detected as an object, if the color difference between the pedestrian's clothes and the color of the scenery in the captured image is equal to or less than a predetermined threshold, the pedestrian cannot be distinguished from the scenery behind him / her, making it difficult for the user 230 to see. Therefore, the display determination unit 225 determines that the object is unclear.

[0034] In addition, the display determination unit 225 determines that an object is unclear when there is an object that is captured in the captured image but not displayed on the display image. The captured image captures not only the front of the vehicle, but also the sides and rear of the vehicle. However, the display image displayed to the user 230 displays only the range of the user's field of view. Therefore, when the user 230 is looking ahead, a bicycle running on the side of the vehicle is captured in the captured image but not displayed on the display image, making it difficult for the user 230 to see it. Therefore, the display determination unit 225 determines that an object is unclear when there is an object that is captured in the captured image but not displayed on the display image.

[0035] Furthermore, the display determination unit 225 determines that an object is unclear when there is an error in the display position of the object on the captured image. For example, when an oncoming vehicle is detected as an object, the display position of the object may be displayed with a delay due to communication delays or the like. In such a case, since the position of the oncoming vehicle is not displayed accurately, it becomes difficult for the user 230 to accurately visually recognize the position of the object. Therefore, the display determination unit 225 determines that the object is unclear. As a method of determining the error, an estimated position of the oncoming vehicle is calculated from the communication delay time and the vehicle speed of the oncoming vehicle, and when there is a difference between this estimated position and the display position of the oncoming vehicle, it is determined that there is an error.

[0036] In this way, the display determination unit 225 determines that the object is unclear in at least one of the above cases.

[0037] The display image generating unit 227 generates a display image to be displayed on the display device 240 based on the captured images captured by the exterior camera 110 and the interior camera 120. The captured images captured by the exterior camera 110 include images of not only the front of the vehicle 150 but also the sides and rear. The captured images captured by the interior camera 120 include images of not only the driver's seat and passenger seat but also the back seat. Therefore, the display image generating unit 227 acquires the direction of the head and the line of sight of the user 230 from the display device 240, extracts an image in the direction in which the user 230 is looking from the captured images, and generates a display image.

[0038] The display image is an image of virtual space 300, and displays the situation inside the vehicle 150 and the situation around the vehicle 150 in virtual space 300. In addition, the display image includes recorded audio around the vehicle 150 and audio inside the vehicle, which are played back in sync with the image.

[0039] If the display determination section 225 determines that the object is unclear, the display image generation section 227 changes the display mode of the object and generates a display image.

[0040] Specifically, when it is determined that an object is unclear, the display image generating unit 227 changes the display mode of the object to a highlighted display and generates a display image. For example, as shown in Fig. 3, when an oncoming vehicle is detected as an object, the display mode is changed to a highlighted display mode in which a rectangular bounding box 30 surrounding the oncoming vehicle is blinked. The highlighting may be performed not only by blinking the bounding box, but also by increasing the brightness of the bounding box.

[0041] Furthermore, when it is determined that an object is unclear, the display image generating unit 227 generates a display image by changing the display mode of the object to a superimposed display of an image showing the object. For example, as shown in Fig. 4, when an oncoming vehicle is detected as an object, the display mode is changed from a display mode in which a bounding box 40 surrounding the oncoming vehicle is displayed to a display mode in which an image of an icon 42 showing the oncoming vehicle is superimposed.

[0042] Furthermore, when there is an object captured in the captured image but not displayed in the display image, the display image generating unit 227 changes the display mode to one in which an image indicating the direction of the object is superimposed, and generates a display image. For example, as shown in FIG. 5, a bicycle is captured at position 50 in the captured image, but is not displayed because it is outside the range of the display image. In this case, the display mode is changed to one in which an image 52 of an arrow indicating the direction of the bicycle is superimposed, and a display image is generated. At this time, an image 54 of "bicycle" indicating the type of object may be added and superimposed.

[0043] Furthermore, when there is an error in the display position of an object, the display image generating unit 227 changes the display mode of the image showing the object to one in which it is superimposed at a position where the error has been corrected, and generates a display image. For example, as shown in Fig. 6, when an oncoming vehicle is detected as an object, the oncoming vehicle is displayed at a position 62, even though the current estimated position of the oncoming vehicle is at a position 60. Therefore, the display image generating unit 227 changes the display mode of the image 64 of the icon showing the oncoming vehicle to one in which it is superimposed at a position where the error has been corrected, and generates a display image.

[0044] At this time, as shown in FIG. 6, when the position of the estimated position 60 is closer to the vehicle 150 than the display position 62 of the oncoming vehicle, the display image generating unit 227 changes the size of the image 64 of the icon indicating the oncoming vehicle and displays it. Specifically, the display image generating unit 227 enlarges and displays the image 64 of the icon indicating the oncoming vehicle, and the enlargement ratio at this time is set based on the distance between the oncoming vehicle and the vehicle 150 (or the vehicle exterior camera 110) (the difference between the position of the oncoming vehicle and the position of the vehicle 150). In other words, the image 64 of the icon indicating the oncoming vehicle is enlarged and displayed according to how it appears when the oncoming vehicle is located at the estimated position 60. Also, instead of the image 64 of the icon indicating the oncoming vehicle, the image of the oncoming vehicle may be enlarged and superimposed. Since the estimated position of the oncoming vehicle has been calculated, the distance between the oncoming vehicle and the vehicle 150 (or the vehicle exterior camera 110) can be obtained from the estimated position and the position of the vehicle 150.

[0045] The sound generation unit 229 generates a sound for making the user 230 aware of an object detected by the object detection unit 223. For example, when an object is highlighted, a sound or a warning sound is generated to alert the user 230. The generated sound is output to the display device 240 and provided to the user 230 from the display device 240.

[0046] The display device 240 is a so-called head-mounted head-mounted display (HMD). The display device 240 is configured to hold a display for displaying images in front of the eyes of the user 230 by being worn on the head of the user 230. The display device 240 also includes a sensor for detecting the movement of the line of sight of the user 230. Note that the display device 240 may be a wearable display or may be a glasses-type device.

[0047] As a head-mounted display applied to the display device 240, an immersive HMD such as VR goggles is applied.

[0048] When the immersive HMD is worn on the head or face of the user 230, it is worn so as to cover the eyes of the user 230, and a display is held in front of the eyes of the user 230. Therefore, it is difficult for the user 230 wearing the immersive HMD to directly view the outside scenery (scenery of the real world), and only the image displayed on the display comes into view. With this configuration, the immersive HMD can give a sense of immersion to the user 230 viewing the image.

[0049] Further, the display device 240 is provided with, for example, an acceleration sensor and a gyro sensor, and is configured to be able to detect the movement of the head (head posture) of the user 230 wearing the display device 240. As an example, the display device 240 detects each of the components in the yaw direction, pitch direction, and roll direction as the movement of the head of the user 230. Then, the display device 240 recognizes a change in the direction indicated by the line of sight of the user 230 based on the detected movement of the head of the user 230, and presents an image according to the recognized change in the line of sight direction to the user 230. Based on such a configuration, for example, an image in a virtual space is presented to the user 230 according to the change in the line of sight direction of the user 230. This makes it possible to give the user 230 a sense of immersion as if he or she were in a virtual space.

[0050] Furthermore, the display device 240 may include an optical sensor (camera). The display device 240 can track the gaze of the user 230 using the optical sensor. The display device 240 may include a proximity sensor. The proximity sensor is installed in the goggles and is used to determine whether or not the display device 240 is in contact with the face of the user 230.

[0051] Next, the virtual space 300 will be described. As shown in Fig. 1, the virtual space 300 is a space virtually showing the periphery of the vehicle 150 and the interior of the vehicle, and an avatar 310 of the occupant 160 and an avatar 320 of the user 230 are displayed superimposed on a camera image captured by the in-vehicle camera 120. The avatar 310 is a virtual object in the virtual space 300, and the movement of the occupant 160 is reflected in the avatar 310, so that the avatar 320 moves in the same way as the occupant 160. The avatar 320 is also a virtual object in the virtual space 300, and the movement of the user 230 is reflected in the avatar 320, so that the avatar 320 moves in the same way as the user 230.

[0052] The occupant 160 is wearing a display device 170, which is an AR goggle, and while actually viewing the scenery around the vehicle 150, which is the real space, and the situation inside the vehicle cabin, he or she can also see the avatar 320 of the user 230 sitting next to him or her in the virtual space 300 via a virtual image optical system.

[0053] Meanwhile, the user 230 is wearing a display device 240 that is a pair of VR goggles, and is viewing a display image of a virtual space 300 that is generated from images captured by the exterior camera 110 and the interior camera 120. In the display image viewed by the user 230, icons and bounding boxes are superimposed on the scenery outside the vehicle, and when the user 230 moves his / her head sideways, an avatar 310 of the passenger 160 is displayed sitting next to him / her.

[0054] [Object display mode change processing] Next, a process of changing the display mode of an object by the image processing device 220 according to this embodiment will be described. Fig. 7 is a flowchart showing a processing procedure of the process of changing the display mode of an object by the image processing device 220 according to this embodiment. The process of changing the display mode of an object shown in Fig. 7 starts when the user 230 starts using the virtual space display system 1.

[0055] 7, in step S10, when the captured image acquisition unit 221 acquires a captured image from the communication device 210, the object detection unit 223 determines whether or not an object that needs to be visually recognized by the user 230 is present in the acquired captured image. If no object is present, the process proceeds to step S50, and if an object is present, the object is detected and the process proceeds to step S20.

[0056] In step S20, the display determination unit 225 determines whether the object detected in step S10 is unclear, i.e., whether the object is displayed unclearly on the captured image. Specifically, the display determination unit 225 determines that the object is unclear when the size of the object on the captured image is equal to or smaller than a predetermined threshold. Alternatively, the display determination unit 225 determines that the object is unclear when the time during which the object is displayed is equal to or smaller than a predetermined threshold, or when the rate of change in the size of the object is equal to or larger than a predetermined threshold.

[0057] Furthermore, the display determination unit 225 may determine that an object is unclear when the color difference between the object and its surroundings in the captured image is equal to or less than a predetermined threshold value. Furthermore, when there is an object captured in the captured image but not displayed in the display image, or when there is an error in the display position of the object, the display determination unit 225 may determine that the object is unclear.

[0058] If the object is not unclear, the display determination unit 225 proceeds to step S50, and if the object is unclear, the display determination unit 225 proceeds to step S30.

[0059] In step S30, when it is determined that the object is unclear in step S20, the display image generating unit 227 changes the display mode of the object to generate a display image. Specifically, the display image generating unit 227 changes the display mode of the object to a highlighted display to generate a display image. The display image generating unit 227 may also change the display mode of the object to a superimposed display of an image showing the object to generate a display image.

[0060] Furthermore, when there is an object that is captured in the captured image but not displayed on the display image, the display image generating unit 227 changes the display mode to one in which an image indicating the direction of the object is superimposed and generates the display image.

[0061] Furthermore, when there is an error in the display position of an object, the display image generating unit 227 generates a display image by changing the display mode to superimpose an image showing the object at a position where the error has been corrected.

[0062] When changing the display mode of the object in step S30, the sound generation unit 229 generates a sound for making the user 230 aware of the detected object, and outputs the sound from the display devices 170 and 240. Specifically, when highlighting an object, the sound generation unit 229 generates a sound or a warning sound for calling the attention of the user 230, and outputs the sound or the warning sound from the display devices 170 and 240.

[0063] In step S40, the display image generating unit 227 determines whether or not it is no longer necessary to change the display mode. Specifically, when the user 230 notices the object, the display image generating unit 227 determines that it is no longer necessary to change the display mode. The method of determining whether or not the object has been noticed is to obtain the head direction and line of sight direction of the user 230 from the display device 240, and determine whether or not the line of sight of the user 230 is directed toward the object. The display image generating unit 227 also determines that it is no longer necessary to change the display mode when the object no longer exists within the range of the display image.

[0064] If the change in the display mode is not no longer necessary, the display image generating unit 227 determines whether it continues to be unnecessary, and if the change in the display mode is no longer necessary, the process proceeds to step S50.

[0065] In step S50, the display image generating unit 227 generates a display image in a normal display mode. In particular, if it is determined in step S40 that the change in the display mode is no longer necessary, the display image generating unit 227 ends the change in the display mode, returns to the normal display mode, and generates a display image. Specifically, the display image generating unit 227 generates a display image by erasing highlighting and icon overlay.

[0066] In step S60, the display image generation unit 227 determines whether or not the user 230 has finished using the virtual space display system 1, and if not, returns to step S10, and if so, ends the object display mode change process related to this embodiment.

[0067] [User Attention Processing] Next, a user attention calling process by the image processing device 220 according to this embodiment will be described. Fig. 8 is a flowchart showing the processing procedure of the user attention calling process by the image processing device 220 according to this embodiment. The user attention calling process shown in Fig. 8 starts when the user 230 starts using the virtual space display system 1.

[0068] 8, in step S110, the voice generation unit 229 determines whether or not the occupant 160 is speaking to the user 230 in the virtual space 300. The voice generation unit 229 acquires the voice of the occupant 160 collected by the microphone of the display device 170, and determines whether or not the occupant 160 is speaking to the user 230. Then, if the occupant 160 is speaking, the process proceeds to step S120, and if not, the process proceeds to step S160.

[0069] In step S120, the voice generation unit 229 determines whether or not the user 230 is aware that the occupant 160 is speaking to him / her. The voice generation unit 229 acquires the voice of the user 230 collected by the microphone of the display device 240, and determines whether or not the user 230 is replying. If the user 230 has not replied, it is determined that the user is unaware and the process proceeds to step S130, whereas if the user has replied, it is determined that the user is aware and the process proceeds to step S160.

[0070] In step S130, the voice generating unit 229 and the display image generating unit 227 call the attention of the user 230. Specifically, the voice generating unit 229 calls the attention of the user 230 by playing back a virtual voice added from the direction of the speaking occupant 160. Furthermore, if the user 230 does not reply, the display determining unit 225 determines that the user 230 is unaware that he or she is being spoken to, that is, determines that the image of the avatar 310 representing the occupant 160 is unclear. Therefore, the display image generating unit 227 displays the image of the avatar 310 representing the occupant 160 on the display image and changes the display mode to highlighted display.

[0071] In step S140, the voice generating unit 229 judges whether or not a call for attention is no longer necessary. The voice generating unit 229 acquires the voice of the user 230 collected by the microphone of the display device 240, and when the user 230 is replying, it judges that a call for attention is no longer necessary and proceeds to step S150. On the other hand, when the user 230 is not replying, it judges that a call for attention is necessary and returns to step S130. In addition, the voice generating unit 229 acquires the voice of the occupant 160 collected by the microphone of the display device 170, and when the occupant 160 has stopped speaking, it judges that a call for attention is no longer necessary and proceeds to step S150. On the other hand, when the occupant 160 has not stopped speaking, it judges that a call for attention is necessary and returns to step S130.

[0072] In step S150, when it is determined in step S140 that the attention call is no longer necessary, the voice generation unit 229 ends the attention call. Specifically, the voice generation unit 229 ends the output of the voice that was generated for the attention call. In addition, the display image generation unit 227 erases the highlighting of the avatar 310 of the occupant 160 that was displayed for the attention call, ends the change of the display mode, returns to the normal display mode, and generates a display image.

[0073] In step S160, the audio generation unit 229 determines whether or not the user 230 has finished using the virtual space display system 1, and if not, returns to step S110, and if so, terminates the user attention alert processing of this embodiment.

[0074] [Effects of the embodiment] As described above in detail, the image processing device 220 according to this embodiment acquires an image captured by an imaging device, and detects an object that the user of the display device needs to view on the acquired captured image. Then, it is determined whether the detected object is unclear, and if it is determined that the object is unclear, the display mode of the object is changed to generate a display image. In particular, it is determined that the object is unclear when at least one of the following occurs: the size of the object is equal to or smaller than a predetermined threshold, the time during which the object is displayed is equal to or smaller than a predetermined threshold, and the rate of change in the size of the object is equal to or larger than a predetermined threshold. This allows the display mode of an object that is difficult to view or a moving object to be changed even when a moving or moving object is displayed at a remote location, so that the user can clearly understand the object.

[0075] Furthermore, in the image processing device 220 according to the present embodiment, when the difference in color between the object and its surroundings is equal to or less than a predetermined threshold, the object is determined to be unclear. This allows the user to clearly understand the object even if the object has a color that is difficult to distinguish from its surroundings.

[0076] Furthermore, in the image processing device 220 according to this embodiment, when an object is determined to be unclear, the display mode of the object is changed to a highlighted display and a display image is generated. This allows the user to clearly understand even an object that is displayed unclearly by highlighting it.

[0077] Furthermore, in the image processing device 220 according to this embodiment, when an object is determined to be unclear, the display mode of the object is changed to a superimposed display of an image showing the object to generate a display image. This allows the user to clearly understand even an object that is displayed unclearly by superimposing an icon or the like.

[0078] Furthermore, in the image processing device 220 according to the present embodiment, when there is an object captured in the captured image but not displayed in the display image, the object is determined to be unclear. This allows the user to clearly understand the object even if there is an object not displayed in the display image.

[0079] Furthermore, in the image processing device 220 according to this embodiment, when there is an object captured in the captured image but not displayed in the display image, the display mode is changed to one in which an image showing the direction of the object is superimposed to generate a display image. This allows the user to clearly understand that there is an object not displayed in the display image by superimposing the image showing the direction of the object.

[0080] Furthermore, in the image processing device 220 according to the present embodiment, when there is an error in the display position of an object, the object is determined to be unclear. This allows the user to clearly understand the object even if there is an error in the display position of the object.

[0081] Furthermore, in the image processing device 220 according to this embodiment, when there is an error in the display position of an object, the display mode is changed to superimpose an image showing the object at a position where the error has been corrected, and a display image is generated. This allows the user to clearly understand even an object with an error in its display position by correcting the error.

[0082] Furthermore, in the image processing device 220 according to this embodiment, the size of the display image is changed based on the distance between the object and the imaging device. This allows the display image to be displayed at a size that matches the distance between the object and the imaging device, allowing the user to clearly understand the object.

[0083] Furthermore, the image processing device 220 according to this embodiment generates a sound to make the user recognize the detected object, and if the user does not respond to the sound, it determines that the object is unclear, changes the display mode of the object, and generates a display image. This allows the user to clearly understand the object by using not only the display but also the sound.

[0084] Furthermore, in the image processing device 220 according to the present embodiment, if the user responds to the voice after changing the display mode of the object, the change in the display mode is terminated. This makes it possible to prevent the change in the display mode from being continued unnecessarily.

[0085] The above-described embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiment, and various modifications can be made to other forms according to the design, etc., as long as they do not deviate from the technical idea of ​​the present invention. In particular, in the present embodiment, the surroundings of a vehicle and the interior of a vehicle are given as an example of a virtual space, but the present invention is not limited to this. For example, the present invention can be applied to trains, airplanes, concert halls, movie theaters, theaters, etc. [Explanation of symbols]

[0086] 1. Virtual space display system 100 Vehicle space 110 Exterior Camera 120 In-car camera 130 Controller 140, 210 Communication equipment 150 vehicles 160 crew members 170, 240 display device 200 Remote Space 220 Image Processing Device 221 Image acquisition unit 223 Object Detection Unit 225 Display judgment section 227 Display image generation unit 229 Voice Generation Unit 230 users 300 Virtual Space 310, 320 Avatar

Claims

1. 1. An image processing device that generates a display image to be displayed on a display device located at a remote location from an imaging device, based on an image captured by the imaging device, a captured image acquisition unit that acquires a captured image captured by the imaging device; an object detection unit that detects an object that a user of the display device needs to view on the captured image; a display determination unit that determines whether the detected object is unclear; a display image generating unit that generates the display image by changing a display mode of the object when the object is determined to be unclear; a voice generating unit that generates a voice for causing the user to recognize the object detected by the object detection unit, the display determination unit determines that the object is unclear when there is an object that is captured in the captured image and not displayed on the display image, and when the user does not reply to the voice; the display image generating unit, when there is an object that is captured on the captured image and not displayed on the display image, changes the display mode to one in which an image indicating a direction of the object is superimposed and displayed, to generate the display image.

13. An image processing device comprising:

2. 1. An image processing device that generates a display image to be displayed on a display device located at a remote location from an imaging device, based on an image captured by the imaging device, a captured image acquisition unit that acquires a captured image captured by the imaging device; an object detection unit that detects an object that a user of the display device needs to view on the captured image; a display determination unit that determines whether the detected object is unclear; a display image generating unit that generates the display image by changing a display mode of the object when the object is determined to be unclear; a voice generating unit that generates a voice for causing the user to recognize the object detected by the object detection unit, the display determination unit determines that the object is unclear when there is an error in the display position of the object and when the user does not reply to the voice.

13. An image processing device comprising:

3. The image processing device according to claim 2, characterized in that, when there is an error in the display position of the object, the display image generating unit generates the display image by changing the display mode to superimpose an image showing the object at a position where the error is corrected.

4. The image processing device according to claim 3 , wherein the display image generating section changes a size of the display image based on a distance between the object and the imaging device.

5. 3 . The image processing device according to claim 1 , wherein the display image generating unit terminates the change in the display mode when the user responds to the voice after changing the display mode of the object. 4 .

6. 1. An image processing method for generating a display image to be displayed on a display device located at a remote location from an imaging device, based on an image captured by the imaging device, comprising: Acquire an image captured by the imaging device; Detecting an object that needs to be visually recognized by a user of the display device on the captured image; determining whether the detected object is obscured; if the object is determined to be unclear, modifying a display manner of the object to generate the display image; generating a sound for making the user aware of the detected object; determining that the object is unclear when an object is captured on the captured image but not displayed on the display image and when the user does not respond to the voice; and when there is an object captured on the captured image but not displayed on the display image, the display mode is changed to one in which an image showing the direction of the object is superimposed, and the display image is generated.

13. An image processing method comprising:

7. 1. An image processing method for generating a display image to be displayed on a display device located at a remote location from an imaging device, based on an image captured by the imaging device, comprising: Acquire an image captured by the imaging device; Detecting an object that needs to be visually recognized by a user of the display device on the captured image; determining whether the detected object is obscured; if the object is determined to be unclear, modifying a display manner of the object to generate the display image; generating a sound for making the user aware of the detected object; If there is an error in the display position of the object and if the user does not reply to the voice, it is determined that the object is unclear.

13. An image processing method comprising:

Images