Image processing device and image processing method

Abstract

This image processing device comprises: an image acquisition unit 111 that acquires a captured image captured by a vehicle interior camera 221 and / or a vehicle exterior camera 222, which are fixed to a vehicle 200; a behavior data acquisition unit 112 that acquires behavior data indicating behavior of the vehicle 200 during movement; a determination unit 113 that determines whether the captured image includes scenery outside of the vehicle 200; and an image processing unit 114 that, if it has been determined that the captured image includes scenery outside of the vehicle 200, subjects the captured image to blur correction processing on the basis of the behavior data acquired by the behavior data acquisition unit 112, so as to counteract blur in the captured image that is due to the behavior.

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] There is known an imaging device that includes a driving means for changing the imaging direction of the imaging means by rotating the imaging means that includes an imaging optical system and an imaging element, an image correction means for correcting the image captured by the imaging means through image processing, and a control means for acquiring a detection signal from a detection means for detecting vibration of the imaging means. In this imaging device, the control means executes a first control for controlling image blur correction of the captured image by controlling the driving means, and a second control for controlling image blur correction of the captured image by controlling the image correction means against vibrations that remain in the image blur correction by the first control (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-129019 A Summary of the Invention [Problem to be solved by the invention]

[0004] The imaging device described in Patent Document 1 has a problem in that when a captured image contains a part that becomes still due to shaking together with the vibration of the imaging means, if image stabilization is performed on the entire captured image, the still parts that do not require image stabilization are corrected, causing the still parts to move unnaturally.

[0005] An object of the present invention is to provide an image processing device and an image processing method capable of performing blur correction processing on an image that is shaken due to the behavior of a moving object while it is moving. [Means for solving the problem]

[0006] The present invention solves the above problem by determining whether or not a captured image includes a view outside the moving body, and if it is determined that the captured image includes a view outside the moving body, performing a blur correction process on the captured image based on behavior data that indicates the behavior of the moving body while it is moving, so as to cancel out shaking of the captured image caused by the behavior. Effect of the Invention

[0007] According to the present invention, it is possible to suppress image shaking caused by the behavior of a moving object during movement. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of an image processing system according to this embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an image captured by an in-vehicle camera. [Diagram 3] FIG. 3 is a diagram showing an example of an image captured by the in-car camera. [Figure 4] FIG. 4 is a flowchart showing an example of a sequence chart in the image processing system of this embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of the procedure of the image processing method in the image processing device of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment An embodiment of an image processing device according to the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of an image processing system 10 including an image processing device 100 according to the present embodiment. As shown in FIG. 1, in the present embodiment, the image processing system 10 includes an image processing device 100, a vehicle 200, and a terminal device 300. The image processing device 100 is a server that communicates with the vehicle 200 and the terminal device 300 and can exchange information. The image processing system 10 performs task processing with a high computation load via the network by constructing a network between the image processing device 100, the vehicle 200, and the terminal device 300, and can provide a high-speed and versatile service because it is easy to change algorithms. The image processing device 100 acquires images of the interior and / or exterior of the vehicle 200 from the vehicle 200, performs image processing on the acquired images as necessary, and transmits the images to the terminal device 300 to display the images of the interior and / or exterior of the vehicle 200 to a second user. The terminal device 300 is located in a remote space away from the vehicle 200. The remote space is a place away from the location where the image processing device 100 is installed and the vehicle 200, and is, for example, the residence of the second user. In the following description of the image processing system 10, the vehicle 200 is taken as an example of a moving body, but the moving body is not limited to the vehicle 200 and may be, for example, a moving body such as a ship or an airplane.

[0010] The first user is a passenger in the vehicle 200. In the example of FIG. 1, the first user is the driver of the vehicle 200, but may be a passenger in the passenger seat or the back seat. The second user wears the display device 340 of the terminal device 300 on his / her eyes, looks at the display screen of the display device 340, and can enjoy a drive in a virtual space with the first user in the vehicle 200. The second user can also have a conversation with the first user using the terminal device 300.

[0011] The image processing device 100 includes a controller 110 and a communication device 120. The controller 110 includes a computer having hardware and software. This computer includes a ROM storing a program, a CPU that executes the program stored in the ROM, and a RAM that functions as an accessible storage device. Note that, as the operating circuit, an MPU, a DSP, an ASIC, an FPGA, or the like can be used instead of or in addition to the CPU.

[0012] The controller 110 includes at least an image acquisition unit 111, a behavior data acquisition unit 112, a determination unit 113, an image processing unit 114, a transmission unit 115, and a reception unit 116 as functional blocks. Each functional block executes each function by cooperation between software for implementing each function or executing each process and hardware. In this embodiment, the functions of the controller 110 are divided into six blocks and the functions of each functional block are described. However, the functions of the controller 110 do not necessarily have to be divided into six blocks, and may be divided into five or less functional blocks, or seven or more functional blocks. The controller 110 acquires captured images captured by the in-vehicle camera 221 and / or the outside-vehicle camera 222 from the vehicle 200 while it is traveling, and acquires behavior data indicating the behavior of the vehicle 200 while it is traveling. The controller 110 performs a blur correction process on the captured images so as to cancel out the shaking of the captured images caused by the behavior of the vehicle 200 based on the behavior data, and transmits the captured images after the blur correction process to the terminal device 300. Each functional block included in the controller 110 will be described below.

[0013] The image acquisition unit 111 acquires captured images captured by the in-vehicle camera 211 and / or the exterior camera 222. The captured images include vehicle interior images (mobile body interior images) showing the interior of the vehicle such as the dashboard and pillars, and / or scenery images showing scenery including buildings and nature outside the vehicle. For example, when the in-vehicle camera 211 captures scenery through a vehicle window and includes the pillar and dashboard in the frame, the captured images of the in-vehicle camera 211 include scenery images and vehicle interior images. FIG. 2(a) is an example of an image captured by the in-vehicle camera 211. In the example of FIG. 2(a), an image in an area A capturing the scenery is the scenery image, and an image in an area B capturing the pillar and dashboard is the vehicle interior image.

[0014] The behavior data acquisition unit 112 acquires behavior data indicating the behavior of the vehicle when it is running (moving). The behavior is vibration of the vehicle 200 while it is running. The behavior data is data detected by the behavior detection sensor 240 in the vehicle 200. The behavior of the vehicle 200 is represented by the vertical movement (vibration) of the vehicle 200, the rotation of the vehicle 200 around an axis in the vertical direction of the vehicle 200, the inclination of the vehicle 200, and the like. The vertical direction of the vehicle 200 is a normal direction to the running surface of the vehicle 200. The data detected by the behavior detection sensor 240 is vibration wave data indicating the vertical movement of the vehicle 200 and the rotation of the vehicle 200 around an axis in the vertical direction. For example, when the data detected by the behavior detection sensor 240 includes data indicating the vertical vibration of the vehicle 200, the behavior data acquisition unit 112 identifies the vertical movement of the vehicle 200. Furthermore, when the data detected by the behavior detection sensor 240 includes data indicating a rotation of the vehicle 200 about an axis in the vertical direction, the behavior data acquisition unit 112 identifies the rotation of the vehicle 200 about an axis in the vertical direction. The behavior data acquisition unit 112 may identify the vertical movement of the vehicle 200 and the rotation about an axis in the vertical direction. Note that the behavior data acquisition unit 112 may acquire images captured by the in-vehicle camera 221 and / or the outside-vehicle camera 222 as behavior data, instead of the data detected by the behavior detection sensor 240.

[0015] The determination unit 113 determines whether or not there is a vehicle behavior based on the behavior data acquired by the behavior data acquisition unit 112. Specifically, the determination unit 113 identifies a vibration wave having a specific frequency from the behavior data, and compares the amplitude value (magnitude of amplitude) of the identified vibration wave with a predetermined amplitude threshold. Then, when the amplitude value of the vibration wave is equal to or greater than the amplitude threshold, the determination unit 113 determines that there is a vehicle behavior. The specific frequency of the vibration wave is a frequency in a specific frequency band set in advance, and the specific frequency band is a range of vibration frequencies that affect human sickness due to image shaking. The amplitude threshold indicates a lower limit value of the vibration amplitude that causes VR sickness, and is determined by an experimental value, an empirical value, or the like. Note that "VR sickness" will be described later. The determination unit 113 also determines whether or not the captured image acquired by the image acquisition unit 111 includes a view outside the vehicle 200. Since the in-vehicle camera 221 and the outside camera 222 are fixed to the vehicle 200, when the vehicle behaves, the captured images of the in-vehicle camera 221 and the outside camera 222 shake in accordance with the behavior of the vehicle 200. In addition, since the vehicle interior image is an image of a part of the vehicle such as a pillar, it vibrates in accordance with the behavior of the vehicle 200. On the other hand, the scenery image does not vibrate in accordance with the behavior of the vehicle 200. And, since the entire captured images of the in-vehicle camera 221 and the outside camera 222 shake in accordance with the behavior of the vehicle 200, and the vehicle interior image shakes in synchronization with the behavior of the vehicle 200, the vehicle interior image does not appear to vibrate. On the other hand, since the scenery image does not shake in accordance with the behavior of the vehicle 200, it appears to vibrate. That is, when the vehicle 200 behaves, the scenery image appears to shake and the vehicle interior image does not shake in the captured images. In the example of FIG. 2, the image in area A shakes, and the image in area B does not shake. In addition, when the vehicle behaves, the image in area B shakes. 2 The entire captured image shakes.

[0016] When the captured image that is shaking due to the behavior of the vehicle 200 is shown to the second user, the second user may experience "VR sickness." The second user is in a remote location and does not directly feel the shaking of the vehicle 200. When the second user views the captured image that is shaking due to the behavior of the vehicle 200, the second user views the shaking image without feeling any vibration. Therefore, the second user experiences a state similar to video-induced sickness, as the sense of sight does not match the sense of sensation felt by the semicircular canals due to the shaking of the image. This state is called "VR sickness."

[0017] When the behavior data acquisition unit 112 acquires captured images of the in-vehicle camera 221 and / or the outside camera 222 as behavior data, the determination unit 113 may determine whether or not there is a behavior of the vehicle based on the captured images. The determination unit 113 identifies the movement of the vehicle 200 using information on the movement of the image, such as optical flow, that can be acquired from the captured images of the in-vehicle camera and / or the outside camera 222, and determines whether or not the movement of the vehicle 200 is due to a behavior. For example, when the movement of the vehicle 200 is a periodic movement, it can be estimated that the movement is not a movement associated with the traveling of the vehicle 200, but a vibration of the vehicle 200 received from the road surface or a vibration due to the engine of the vehicle 200. The determination unit 113 determines whether or not there is a behavior of the vehicle 200 by identifying a periodic movement of the vehicle 200 due to the behavior of the vehicle 200 from the captured images.

[0018] The image processing unit 114 performs a blur correction process on the captured image based on the determination result of the determination unit 113 and the behavior data acquired by the behavior data acquisition unit 112 so as to cancel out the shaking of the captured image caused by the behavior of the vehicle 200. When the captured image includes a scenery image, the image processing unit 114 performs a blur correction process on the captured image based on the determination result of the determination unit 113 and the behavior data acquired by the behavior data acquisition unit 112. sceneryA blur correction process is performed on the captured image so as to cancel out the shaking of the image. Furthermore, when the entire captured image is a scenery image, the image processing unit 114 performs a blur correction process on the entire captured image. The blur correction process is a process of processing the scenery image so as to cancel out the shaking of the scenery image. For the blur correction process, a known image processing method for correcting blur may be used. For example, when the scenery image shakes at a predetermined cycle, the image processing unit 114 performs a blur correction process by adding a displacement of the opposite phase to the scenery image. This makes it possible to reduce the shaking of the image caused by the behavior of the vehicle 200.

[0019] For example, the image processing unit 114 calculates, from the amplitude and frequency of the vibration of the vehicle 200 specified by the behavior data acquisition unit 112, scenery The amount of image blur and the period of the blur caused by the shaking of the image are calculated, and the position of the scenery image is corrected so as to return the shift of the amount of blur according to the calculated period of the blur. FIG. 2(b) is an example of an image captured by the in-vehicle camera 211 when the vehicle 200 is moving. Note that FIG. 2(a) and FIG. 2(b) are images of the vehicle 200 looking at the same scenery from the windshield at the same position, and FIG. 2(a) shows a captured image in a state where the scenery image is not shaken by the behavior of the vehicle 200, and FIG. 2(b) shows a captured image in a state where the scenery image is shaken by the behavior of the vehicle 200. As shown by the arrow P in FIG. 2(b), the scenery image is shifted clockwise due to the shaking. Note that in FIG. 2(b), the image of the part of the arrow P is omitted in order to explain the shaking of the scenery image due to the shaking, but the scenery image is actually captured. In such a case, the image processing unit 114 corrects the captured image by rotating the scenery image counterclockwise by the amount of the shift represented by the arrow P. In the example of FIG. 2(b), the blur correction process is performed by rotating the scenery image, but the image processing unit 114 may also correct the position of the scenery image in the horizontal and / or vertical directions as the blur correction process.

[0020] The image processing unit 114 may correct the misalignment between the scenery image after the shake correction process and the vehicle interior image by performing image processing on the boundary between the scenery image after the shake correction process and the vehicle interior image. FIG. 3 is an example of an image captured by the in-vehicle camera 211. FIG. 3(a) is a captured image in a state where the misalignment between the scenery image after the shake correction process and the vehicle interior image is not corrected. FIG. 3(b) is a captured image in a state where the misalignment between the scenery image after the shake correction process and the vehicle interior image is corrected. As shown in FIG. 3(a), the scenery image shakes due to the behavior of the vehicle 200, and the shake correction process is performed on the scenery image to cancel the shaking. First, the image processing unit 114 identifies the vehicle interior image and the scenery image from the captured image. The image processing unit 114 performs the shake correction process so that the scenery image moves diagonally in the upper right direction on the paper surface of FIG. 3(a). A misalignment (gap: see arrow Q in FIG. 3(a)) occurs between the scenery image after the shake correction process and the pillar (window frame) included in the vehicle interior image due to the movement of the scenery image. In the example of Fig. 3, the image processing unit 114 identifies a pillar included in the vehicle interior image as the boundary. The boundary portion between the vehicle interior image and the scenery image is divided into an image that has not been subjected to blur correction processing and an image that has been subjected to blur correction processing. Ta When the images are synthesized, a gap may occur at the boundary between the pillar and the image. In such a case, the image processing unit 114 performs image processing on the pillar to fill the gap. As shown in FIG. 3(b), the image processing unit 114 thickens the pillar so as to fill the gap. The image processing unit 114 may perform image processing not only on the pillar but also on the boundary part of the vehicle interior image so as to eliminate the gap at the boundary part between the image that has been subjected to the shake correction process and the image that has not been subjected to the shake correction process among the captured images by enlarging the image of the boundary part. The image of the shake correction boundary part is the boundary between the scenery image and the vehicle interior image. The image processing unit 114 may also perform image processing on the boundary part of the scenery image so as to fill the gap at the boundary part between the image that has been subjected to the shake correction process and the image that has not been subjected to the shake correction process among the captured images by enlarging the image of the boundary part. The image processing unit 114 may also perform image processing not only on the enlargement of the image but also on the color of the boundary part so that the gap between the image that has been subjected to the shake correction process and the image that has not been subjected to the shake correction process among the captured images becomes inconspicuous.

[0021] When the captured image includes a vehicle interior image and a scenery image, and the image processing unit 114 performs a shake correction process on the scenery image, the image processing unit 114 synthesizes the scenery image that has been image-processed by the shake correction process with the vehicle interior image that has not been image-processed by the shake correction process. Note that, when the entire captured image is a scenery image and the image processing unit 114 performs a shake correction process on the scenery image, the image processing unit 114 does not need to synthesize the images. Also, when the image processing unit 114 performs image processing on the vehicle interior image so as to fill in the gap at the boundary with the image that has not been subjected to the shake correction process, it is sufficient to synthesize the scenery image after the shake correction process with the vehicle interior image after the image processing.

[0022] When the image processing unit 114 performs the shake correction process on the captured image, the image data after the shake correction process is stored in the memory. When the captured image includes a vehicle interior image and a scenery image, the image data after the shake correction process is data of the captured image that has been subjected to the synthesis process. When the vehicle behavior that may cause VR sickness is not occurring, the image processing unit 114 does not perform the shake correction process on the captured image, and stores the captured image data that has not been subjected to the image processing by the shake correction process in the memory.

[0023] The transmitting unit 115 transmits a control signal for displaying the image processed by the image processing unit 114 on the terminal device 300 via the communication device 120. The receiving unit 116 receives operation information input by the second user to the input device 320. The operation information is an operation command for the vehicle 200 to acquire a VR image, etc.

[0024] Next, the vehicle 200 will be described. The vehicle 200 includes an in-vehicle controller 210, an in-vehicle camera 221, an outside-vehicle camera 222, an in-vehicle communication device 230, and a behavior detection sensor 240. The in-vehicle controller 210 controls the functions of each device and sensor of the vehicle 200. The in-vehicle controller 210 outputs captured images captured by the in-vehicle camera 221 and / or the outside-vehicle camera 222, and detection data detected by the behavior detection sensor 240 to the in-vehicle communication device 230.

[0025] The in-vehicle camera 221 is fixed inside the vehicle cabin, and captures images of the vehicle cabin and the surrounding area from a fixed position inside the vehicle cabin. / Or, the camera 221 captures an image of the outside of the vehicle cabin and outputs the captured image data to the in-vehicle controller 210. The in-vehicle camera 221 captures an image of the scenery outside the vehicle cabin through a window. The in-vehicle camera 221 may be a camera that can be removed from a fixed position in the vehicle cabin. For example, a mobile terminal such as a smartphone may be brought into the vehicle cabin and fixed to a folder or the like to be used as the in-vehicle camera 221. The outside-vehicle camera 222 is fixed to the body of the vehicle 200 or the like, captures an image of the scenery outside the vehicle cabin and the surroundings of the vehicle 200, and outputs the captured image data to the in-vehicle controller 210. The in-vehicle camera 221 and the outside-vehicle camera 222 do not necessarily have to be one camera, and may be multiple cameras. When multiple cameras are used, multiple images may be synthesized to create an image of the inside of the vehicle and an image of the scenery. This can increase the resolution of the captured image.

[0026] The in-vehicle communication device 230 communicates with the communication device 120 to exchange data. Specifically, the in-vehicle communication device 230 receives a signal including an operation command for acquiring a VR image of the vehicle 200 from the communication device 120, and transmits a signal including a captured image and behavior data to the communication device 120.

[0027] The behavior detection sensor 240 is a sensor that detects the behavior of the vehicle 200, and is provided in the vehicle 200. A G sensor or the like is used as the behavior detection sensor 240. Note that the behavior detection sensor 240 is not limited to a sensor provided in the vehicle 200, and a portable terminal with a built-in acceleration sensor may be brought into the vehicle cabin and used as the behavior detection sensor 240. The behavior detection sensor 240, which is a portable terminal, is connected to the in-vehicle controller 210 via an in-vehicle network, and transmits data indicating the behavior to the in-vehicle controller 210.

[0028] The terminal device 300 is a device used by the second user in the remote space, and outputs a virtual image as viewed from a predetermined position in the vehicle 200, such as the passenger seat. The terminal device 300 may be installed in the remote space, for example. The terminal device 300 includes a terminal controller 310, an input device 320, a terminal communication device 330, and a display device 340. The terminal controller 310 controls the functions of each device of the terminal device 300. Specifically, the terminal controller 310 outputs a control signal for causing the display device 340 to output a captured image represented by image data received by the terminal communication device 330 to the second user.

[0029] The input device 320 is a device operated by the second user, and is used by the second user to input operation information for acquiring a VR image viewed from a predetermined position of the vehicle 200 to the image processing device 100.

[0030] The terminal communication device 330 communicates with the image processing device 100 and the in-vehicle communication device 230 of the vehicle 200 to exchange information. Specifically, the terminal communication device 330 receives image data from the image processing device 100 and transmits operation information to the image processing device 100.

[0031] The display device 340 is a device that outputs the environment inside and outside the vehicle at a predetermined position inside or outside the vehicle to the second user. The display device 340 is, for example, a glasses-type or goggles-type VR head-mounted display, and is worn on the head of the second user. The second user can view a virtual reality (VR) in which the environment inside and outside the vehicle is displayed via the terminal device 300. The display device 340 outputs virtual images representing the environment inside and outside the vehicle, and virtual audio information representing audio heard from a predetermined position inside or outside the vehicle. This allows the second user to see the scenery and hear the audio as if he or she were riding in the vehicle 200. The second user can also see the scenery outside the vehicle 200 from the image captured by the exterior camera 222. The display device 340 has, for example, a non-transparent display, a speaker, etc.

[0032] Next, data transmission and reception between the image processing device 100, the vehicle 200, and the terminal device 300, and data processing of each device will be described with reference to Fig. 4. Fig. 4 is a sequence chart for explaining a data flow in the image processing system 10.

[0033] In step S1, the second user operates the input device 320 to switch the VR system from off to on. The VR system is a system for realizing a virtual reality of the environment inside and outside the vehicle by displaying images on the display device 340, and providing a space in which the second user feels as if he or she is riding in the vehicle 200. When the VR system is switched from off to on, the terminal controller 310 transmits an operation signal to the image processing device 100 to acquire a VR image.

[0034] In step S2, when the controller 110 of the image processing device 100 receives a VR request signal by the communication device 120, the controller 110 starts VR request processing. First, the controller 110 transmits a VR request signal to the in-vehicle communication device 230. In step S3, the in-vehicle controller 210 judges whether or not to accept the VR request based on the VR request signal received by the in-vehicle communication device 230. For example, when the in-vehicle controller 210 receives a VR request signal, the in-vehicle controller 210 displays a selection screen for whether or not to accept the VR request on a display in the vehicle. Then, when an operation is performed from the first user to accept the VR request, the in-vehicle controller 210 judges that the VR request is acceptable. On the other hand, when an operation is performed from the first user to not accept the VR request, the in-vehicle controller 210 judges that the VR request is not acceptable. When it is judged that the VR request is acceptable, the in-vehicle controller 210 transmits a VR permission signal to the image processing device 100 using the in-vehicle communication device 230.

[0035] In step S4, the controller 110 of the image processing device 100 receives a VR permission signal from the vehicle 200, starts the VR system, and transmits an image request signal to the vehicle 200 by the communication device 120. In step S5, the in-vehicle controller 210 acquires VR data from the in-vehicle camera 221, the outside-vehicle camera 222, and the behavior detection sensor 240. The VR data is images captured by the in-vehicle camera 221 and / or the outside-vehicle camera 222, and data (behavior data) detected by the behavior detection sensor 240. The controller 110 transmits the acquired VR data to the image processing device 100.

[0036] In step S6, the controller 110 of the image processing device 100 determines whether or not blur correction processing is required for the captured image based on the captured image and behavior data included in the VR data. If the captured image includes a scenery image and includes behavior of the vehicle 200, the controller 110 determines that blur correction processing is required. If the captured image does not include a scenery image or does not include behavior of the vehicle 200, the controller 110 determines that blur correction processing is not required.

[0037] If it is determined that the blur correction process is necessary, in step S7, the controller 110 performs the blur correction process on the captured image. In step S8, the controller 110 generates an image for VR display based on the image after the blur correction process. For example, if the captured image includes a scenery image and an image inside the vehicle, and the blur correction process is performed on the scenery image, the image processing unit 114 of the controller 110 synthesizes the scenery image after the blur correction process and the image inside the vehicle, and generates a screen for VR display based on the synthesized image. For example, in the case where a character representing the first user is displayed in addition to the image inside the vehicle in the virtual reality space, the image processing unit 114 processes the image so that the image representing the character is included in the synthesized image. In this way, the image processing unit 114 generates a captured image for VR display. Then, the controller 110 transmits data including the captured image for VR display to the terminal device 300. Note that the character display is performed to protect the privacy of the first user. Instead of the character, only the face area of ​​the first user may be masked.

[0038] In step S9, the terminal controller 310 outputs the captured image for VR display received by the terminal communication device 330 to the display device 340, and the display device 340 displays the VR image.

[0039] Next, a detailed control flow of the controller 110 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the control flow of the controller 110.

[0040] In step S11, the image acquisition unit 111 of the controller 110 acquires captured images captured by the in-vehicle camera 221 and / or the exterior camera 222. In step S12, the behavior data acquisition unit 112 acquires behavior data indicating the behavior of the vehicle 200 while it is moving. In step S13, the determination unit 113 determines whether or not there is a behavior of the vehicle 200 based on the behavior data. If it is determined that there is no vehicle behavior, the controller 110 executes the control process of step S22 described later.

[0041] In step S14, the determination unit 113 determines whether or not the captured image includes a view outside the vehicle 200. Whether or not the captured image includes a view is determined based on whether or not the captured image includes a view image. If the captured image does not include a view outside the vehicle 200, the controller 110 executes the control process of step S22, which will be described later.

[0042] If the captured image includes a view outside the vehicle 200, in step S15, the determination unit 113 determines whether or not the captured image includes an internal image (vehicle interior image) of the vehicle 200. If the captured image does not include a vehicle interior image, the controller 110 executes a control process of step S20, which will be described later.

[0043] If the captured image includes an image of the inside of the vehicle, in step S16, the image processing unit 114 extracts from the captured image View A color image and an image inside the vehicle are identified. In step S17, the image processing unit 114 performs blur correction processing on the scenery image based on the behavior data so as to cancel out shaking of the captured image caused by the behavior. On the other hand, the image processing unit 114 does not perform blur correction processing on the image inside the vehicle. In other words, the image processing unit 114 does not perform blur correction processing on images that are not affected by the behavior of the vehicle 200, and performs blur correction processing on images that are affected by the behavior of the vehicle 200. This makes it possible to separate image processing according to image shaking for images inside and outside the vehicle 200.

[0044] In step S18, the image processing unit 114 combines the blur-corrected scenery image with the vehicle interior image. In step S19, the image processing unit 114 performs image processing so that the combined captured image becomes an image for VR display, and stores the processed image data in memory.

[0045] If it is determined in the control flow of step S15 that the captured image does not include an image of the inside of the vehicle, then in step S20, the image processing unit 114 performs a blur correction process on the entire captured image. In step S21, the image processing unit 114 performs image processing so that the image after the blur correction process becomes an image for VR display, and stores the data of the processed image in memory.

[0046] If it is determined in the control flow of step S13 that there is no vehicle behavior, in step S22, the image processing unit 114 performs image processing on the captured image acquired by the image acquisition unit 111 so that it becomes an image for VR display, and stores the data of the processed image in memory.

[0047] In step S23, the transmission unit 115 transmits the captured image stored in the memory to the terminal device 300 via the communication device 120.

[0048] As described above, in this embodiment, the image processing device acquires a captured image captured by the in-vehicle camera 221 and / or the outside camera 222 fixed to the vehicle 200, acquires behavior data indicating the behavior of the vehicle 200 when moving, determines whether the captured image includes a view outside the vehicle 200, and performs a blur correction process on the captured image so as to cancel the shaking of the captured image caused by the behavior based on the behavior data acquired by the behavior data acquisition unit 112 when it is determined that the captured image includes a view outside the vehicle 200. This makes it possible to suppress the shaking of the image caused by the behavior of the vehicle 200 when moving. For example, when a second user in a remote location views an image that is shaking due to the behavior of the vehicle 200, the second user is not actually riding in the vehicle 200 and does not feel the vibration of the vehicle 200, so there is a risk of experiencing VR sickness. In this embodiment, the shaking of the image caused by the behavior of the vehicle 200 when moving is suppressed, so that it is possible to provide the second user with an experience as if he or she had boarded the vehicle 200 while reducing VR sickness.

[0049] In this embodiment, the image processor 114 identifies a scenery image and an image inside the vehicle from the captured image, performs blur correction processing on the scenery image, and synthesizes the scenery image processed by the blur correction processing with the image inside the vehicle that is not processed by the blur correction processing. This allows the second user to feel as if he or she is actually riding in the vehicle when viewing the synthesized image, and an image with reduced VR sickness can be provided to the second user.

[0050] In the present embodiment, the image processing device 100 includes a transmission unit 115 that transmits a signal including the captured image synthesized by the image processing unit 114 to a display device 340 located outside the vehicle 200. This allows the second user to feel as if he or she is actually riding in the vehicle, and an image with reduced VR sickness can be provided to the second user.

[0051] Furthermore, in this embodiment, the behavior data acquisition unit 112 identifies the vertical movement of the vehicle 200 and / or the rotation of the vehicle 200 about an axis of the vertical direction based on the behavior data, and performs a blur correction process on the captured image so as to cancel out shaking of the captured image caused by the movement and / or rotation of the vehicle 200. This makes it possible to identify the behavior of the vehicle 200 with high accuracy.

[0052] In this embodiment, the behavior data acquisition unit 112 identifies the magnitude and frequency of the behavior based on the behavior data, and the image processing unit 114 performs a blur correction process on the captured image when the magnitude of the behavior having a specific frequency is equal to or greater than a threshold value. This makes it possible to perform a blur correction process on the captured image when a behavior that may cause VR sickness occurs.

[0053] In this embodiment, the image processor 114 thickens the window frames included in the vehicle interior image. This makes it possible to distinguish between the scenery image and the vehicle interior image after the blur correction process. image It can eliminate the discrepancy between

[0054] In this embodiment, the image processor 114 corrects the captured image by enlarging either the scenery image or the vehicle interior image, thereby eliminating the misalignment between the image after the shake correction process and the image without the shake correction process.

[0055] In this embodiment, the image processor 114 performs color processing on the boundary between the scenery image after the blur correction processing and the vehicle interior image. cormorant. This makes it possible to provide the user with images in which there is no unnatural transition between images that have been subjected to shake correction processing and images that have not been subjected to shake correction processing.

[0056] In this embodiment, the image processor 114 does not perform blur correction processing on the captured image when the captured image is an image of the inside of the vehicle, and performs blur correction processing on the captured image when the captured image is an image of a view. This makes it possible to prevent shaking of the image that causes VR sickness by performing blur correction processing in accordance with the timing of displaying the image that causes VR sickness.

[0057] In the modified example of this embodiment, the image processing unit 114 has a correction mode in which the image is subjected to a shake correction process and a non-correction mode in which the image is not subjected to a shake correction process, and the correction mode and the non-correction mode can be selected according to the user's selection. This allows the correction mode and the non-correction mode to be selected according to the user's command.

[0058] In a modified example of this embodiment, the image processing device 100 may distribute the image for VR display to a network by the image processing unit 114. This allows the second user to view the VR image online or offline.

[0059] In this embodiment, the image processing device 100 may be provided in the vehicle 200. That is, the in-vehicle controller 210 has a functional block such as the image acquisition unit 111 included in the controller 110, and the VR system may be constructed by communication between the vehicle 200 and the terminal device 300 without going through a server. In addition, when the in-vehicle controller 210 has a functional block such as the image acquisition unit 111 included in the controller 110, the shake correction process of the image processing unit 114 may be executed by a portable terminal equipped with a camera and capable of being carried in the vehicle.

[0060] In the image processing system 10 according to the modified example of this embodiment, when the in-vehicle camera 211 is fixed inside the vehicle via a vibration-absorbing head (camera mounting part), the image processing part 114 performs blur correction processing on the vehicle interior image and does not perform blur correction on the scenery image. Righteous The head has a mechanism for moving the base on which the camera is fixed in accordance with the behavior of the vehicle 200, for example. The head may have a mechanism with a support member such as a tripod. When the vehicle behavior occurs while the in-vehicle camera 211 is fixed inside the vehicle cabin via the head, the head absorbs the vibrations and the vehicle behavior is not transmitted to the in-vehicle camera 211. On the other hand, the behavior is transmitted to the internal equipment of the vehicle such as the dashboard and pillars. Therefore, when the captured image of the in-vehicle camera 211 includes a scenery image, the scenery image does not shake and the image inside the vehicle shakes. In such a case, the image processing unit 114 vehicle The image processing unit 114 performs blur correction processing on the internal image and does not perform blur correction processing on the scenery image. The image processing unit 114 synthesizes the vehicle internal image that has been image-processed by blur correction processing with the scenery image that has not been image-processed by blur correction processing. The image processing unit 114 then stores the data of the captured image that has been image-processed by blur correction processing in memory.

[0061] In a modified example of this embodiment, the image processing unit 114 identifies a scenery image and an image inside the vehicle from the captured image, performs a blur correction process on the image inside the vehicle, and synthesizes the image inside the vehicle that has been image-processed by the blur correction process with the scenery image that has not been image-processed by the blur correction process. That is, in this embodiment, the image processing unit 114 identifies a scenery image and an image inside the vehicle from the captured image, performs a blur correction process on either the scenery image or the image inside the vehicle, and synthesizes one image that has been image-processed by the blur correction process and the other image that has not been image-processed by the blur correction process among the scenery image and the image inside the vehicle. As a result, when the second user views the synthesized image, the second user can feel as if he or she is actually riding in the vehicle, and an image with reduced VR sickness can be provided to the second user.

[0062] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0063] 10. Image processing system 100...Image processing device 111…Image acquisition unit 112...Behavior data acquisition unit 113…Judgment section 114...Image processing unit 115...Transmitter 116…Receiver 120...Communication equipment 200…Vehicle 300...Terminal device

Claims

1. an image acquisition unit that acquires an image captured by an imaging device fixed to a moving body; A behavior data acquisition unit that acquires behavior data indicating a behavior of the moving object during movement; a determination unit that determines whether the captured image includes a view outside the moving object; an image processing unit that performs a blur correction process on the captured image based on the behavior data acquired by the behavior data acquisition unit when it is determined that the captured image includes a view outside the moving object, so as to cancel out shaking of the captured image caused by the behavior, The image processing unit includes: A scenery image representing a scenery outside the moving body and a moving body internal image representing an interior of the moving body are identified from the captured images; performing the blur correction process on the scenery image of the captured image; An image processing device that performs image processing by enlarging either the boundary portion of the moving body internal image or the boundary portion of the scenery image so that there is no gap at the boundary portion between the scenery image that has been subjected to the blur correction processing and the moving body internal image.

2. 2. The image processing device according to claim 1, an image processing device that synthesizes one of the scenery image and the moving object internal image, which has been image-processed by the shake correction processing, with the other image, which has not been image-processed by the shake correction processing;

3. 3. The image processing apparatus according to claim 2, an image processing device comprising: a transmission unit that transmits a signal including the captured image combined by the image processing unit to a display device located outside the moving body.

4. 4. The image processing device according to claim 1, The behavior data acquisition unit identifies a vertical movement of the moving object and / or a rotation of the moving object around an axis of the vertical direction of the moving object based on the behavior data, the vertical direction of the moving body is a normal direction to a running surface of the moving body, The image processing unit is an image processing device that performs the shake correction process on the captured image so as to cancel out shaking of the captured image caused by the movement and / or the rotation.

5. 5. The image processing device according to claim 1, The behavior data acquisition unit identifies a magnitude of the behavior and a frequency of the behavior based on the behavior data, The image processing unit is an image processing device that performs a blur correction process on the captured image when the magnitude of the behavior having a specific frequency is equal to or greater than a threshold value.

6. 6. The image processing device according to claim 1, The image processing unit is an image processing device that thickens a window frame of the moving object included in the moving object internal image.

7. 2. The image processing device according to claim 1, The image processing unit is an image processing device that enlarges either the scenery image or the moving object internal image.

8. The image processing device according to any one of claims 1 to 7, The image processing unit is an image processing device that performs color processing on a boundary portion between the scenery image after the blur correction processing and the image inside the moving object.

9. In the image processing device according to any one of claims 1 to 8, The image processing unit includes: When the captured image is a moving object internal image showing the inside of the moving object, the blur correction process is not performed on the captured image; When the captured image is a scenery image showing a scene outside the moving body, the image processing device performs the blur correction process on the captured image.

10. The image processing device according to any one of claims 1 to 7, The image processing device is configured to select a mode in which the image processing unit performs the image blur correction process on the captured image and a mode in which the image blur correction process is not performed on the captured image.

11. 1. An image processing method for performing image processing by a processor, comprising: Acquire an image captured by an imaging device fixed to the moving body; Obtaining behavior data indicating a behavior of the moving object during movement; determining whether the captured image includes a view outside the moving object; A scenery image representing a scenery outside the moving body and a moving body internal image representing an interior of the moving body are identified from the captured images; when it is determined that the captured image includes a view outside the moving object, a blur correction process is performed on the captured image based on the behavior data so as to cancel out shaking of the captured image caused by the behavior, the blur correction process being performed on the view image of the captured image; An image processing method that performs image processing by enlarging either the boundary portion of the image of the interior of the moving body or the boundary portion of the image of the scenery so that there is no gap at the boundary portion between the image of the interior of the moving body that has been subjected to the blur correction processing.

Images