Image processing device, image processing system, and image processing method

Abstract

An image processing device (10) comprises: an image processing unit (11) that acquires, frame by frame, temporally continuous real-world images; a position information holding unit (12) that acquires, frame by frame, position information indicating the position at which the real-world images were captured; a virtual image memory (14) that acquires a virtual image created using a past real-world image N (N is an integer of 1 or more) frames prior to a current real-world image; a correction amount calculation unit (13) that calculates a correction amount for a positional deviation of the virtual image from the current real-world image on the basis of, among the position information acquired frame by frame, current position information indicating the position at which the current real-world image was captured and past position information indicating the position at which the past real-world image was captured; a correction unit (15) that corrects the positional deviation on the basis of the correction amount; and an output unit (16) that outputs the virtual image in which the positional deviation has been corrected.

Description

Image processing device, image processing system, and image processing method

[0001] The present disclosure relates to an image processing device for displaying a virtual image.

[0002] Japanese Patent Laid-Open No. 2003-124222 discloses a technique for superimposing and displaying an image of a virtual space (called a virtual image) such as a CG (Computer Graphics) on an image of a real space (called a real image).

[0003] JP 2023-183684 A

[0004] However, in the technology for displaying virtual images, a real image may be input into an image processing device, and then a virtual image may be created using the real image. In this case, a processing delay occurs between the input of the real image and the display of the virtual image, which may cause motion sickness.

[0005] Therefore, the present disclosure provides an image processing device and the like that can reduce the delay between the input of a real image and the display of a virtual image.

[0006] The image processing device according to the present disclosure includes a real image acquisition unit that acquires temporally consecutive real images for each frame; a position information acquisition unit that acquires position information for each frame indicating the position at which the real image was taken; a virtual image acquisition unit that acquires a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation unit that calculates a correction amount for a positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction unit that corrects the positional deviation based on the correction amount; and an output unit that outputs the virtual image with the positional deviation corrected.

[0007] An image processing system according to the present disclosure includes the image processing device described above and an external device that creates the virtual image.

[0008] The image processing method according to the present disclosure includes a real image acquisition step of acquiring temporally consecutive real images for each frame; a position information acquisition step of acquiring, for each frame, position information indicating the position at which the real images were taken; a virtual image acquisition step of acquiring a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation step of calculating a correction amount for positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction step of correcting the positional deviation based on the correction amount; and an output step of outputting the virtual image with the positional deviation corrected.

[0009] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to an image processing device according to an aspect of the present disclosure, it is possible to reduce the delay between the input of a real image and the display of a virtual image.

[0011] Fig. 1 is a block diagram showing an example of an image processing system according to embodiment 1. Fig. 2 is a diagram for explaining a specific example of the processing flow of the image processing system according to embodiment 1. Fig. 3 is a block diagram showing an example of an image processing system according to embodiment 2. Fig. 4 is a diagram for explaining the processing flow of the image processing system according to embodiment 2. Fig. 5 is a flowchart showing an example of an image processing method according to another embodiment.

[0012] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0013] The embodiments described below are all comprehensive or specific examples, and the numerical values, shapes, materials, components, arrangement and connection of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

[0014] In this specification, "present" refers to the timing at which a virtual image is output to a display, and "past" refers to a timing prior to that timing.

[0015] (First Embodiment) Hereinafter, an image processing device and an image processing system according to a first embodiment will be described.

[0016] FIG. 1 is a block diagram showing an example of an image processing system 1 according to the first embodiment.

[0017] The image processing system 1 is a system used in fields such as MR (Mixed Reality), AR (Augmented Reality), or VR (Virtual Reality), and is a system for displaying a virtual image superimposed on a real image. The image processing system 1 also uses a pass-through method, which is one of the video see-through methods. The image processing system 1 may also use an optical see-through method, and the virtual image may be displayed on a transparent display or the like. In other words, the virtual image does not have to be superimposed on the real image.

[0018] The image processing system 1 includes an image processing device 10 , an external device 20 , an image sensor 100 , a sensor 200 , and a display unit 300 .

[0019] For example, the image processing system 1 may include AR glasses or a VR headset, and the image processing device 10, the external device 20, the image sensor 100, the sensor 200, and the display unit 300 may be mounted on the AR glasses or the VR headset. Note that the external device 20 does not have to be mounted on the AR glasses or the VR headset, and may be a computer such as a server that communicates wirelessly with the AR glasses or the VR headset.

[0020] Image sensor 100 is a sensor that captures an image of the surroundings of AR glasses, a VR headset, or the like, and is attached to AR glasses, a VR headset, or the like so that the captured image corresponds to the field of view of a user who uses image processing system 1, specifically, a user wearing the AR glasses, the VR headset, or the like. The image captured by image sensor 100 is called a real image.

[0021] Sensor 200 is a position sensor that detects the position of a user, specifically, the position of AR glasses or a VR headset equipped with sensor 200. Sensor 200 is not particularly limited as long as it is a sensor that can detect a position, and may be, for example, an acceleration sensor or a GPS (Global Positioning System) sensor.

[0022] The display unit 300 is a display that displays a virtual image or a real image on which a virtual image is superimposed (also called a superimposed image). For example, the display unit 300 is an OLED microdisplay of AR glasses or a VR headset.

[0023] The external device 20 includes a virtual image creation unit 21 that creates a virtual image such as a CG image.

[0024] The image processing device 10 includes an image processing unit 11, a position information holding unit 12, a correction amount calculation unit 13, a virtual image memory 14, a correction unit 15, and an output unit 16. The image processing device 10 is a computer including a processor (microprocessor) and a memory. The memory may be a read-only memory (ROM) or a random access memory (RAM), and can store a program executed by the processor. The image processing unit 11, the position information holding unit 12, the correction amount calculation unit 13, the correction unit 15, and the output unit 16 are realized by a processor that executes a program stored in the memory. The virtual image memory 14 may be the same memory as the memory in which the program is stored, or may be a different memory.

[0025] The image processing unit 11 acquires temporally consecutive real images for each frame from the image sensor 100. That is, the image processing unit 11 acquires video from the image sensor 100 and processes the images for each frame that make up the video. The image processing unit 11 is an example of a real image acquisition unit. The image processing unit 11 transmits the real image to the external device 20. Note that, when a virtual image is superimposed on a real image, as shown in FIG. 1 , the image processing unit 11 transmits the real image to the output unit 16 and the external device 20.

[0026] The position information holding unit 12 acquires position information indicating the position where the real image was captured (specifically, the position of the AR glasses or VR headset, etc.) for each frame (in other words, each time the image processing unit 11 acquires a real image). That is, the position information holding unit 12 holds the position information for each frame. The position information holding unit 12 is an example of a position information acquisition unit. For example, the position information for each frame includes the position where the real image was captured and the frame number of the real image.

[0027] The virtual image creation unit 21 creates a virtual image using the real image acquired from the image processing unit 11. Note that, because it takes some time to create a virtual image, it is difficult to output the virtual image created using the real image to the display unit 300 (display) at approximately the same time as the real image is input from the image sensor 100. Therefore, if the timing at which the virtual image is output to the display unit 300 after the virtual image creation unit 21 creates the virtual image is considered to be the present, and the virtual image creation unit 21 creates the virtual image using a past real image N frames before the current real image (N is an integer greater than or equal to 1). In other words, the real image used in the virtual image created by the virtual image creation unit 21 is a past real image when compared to the real image N frames after the timing at which the virtual image is output to the display unit 300. The virtual image creation unit 21 transmits the created virtual image and the frame number of the virtual image, specifically, the frame number of the real image (past real image) used to create the virtual image, to the image processing device 10.

[0028] The virtual image memory 14 acquires a virtual image created using a past real image N frames before the current real image. The virtual image memory 14 is an example of a virtual image acquisition unit. The virtual image memory 14 is, for example, a frame memory, and holds the virtual image during the waiting time from when the virtual image is created until when it is output to the display unit 300.

[0029] Since the virtual image is created using a past real image, if the position at which the past real image was taken is shifted from the position at which the current real image was taken, the virtual image will also be shifted from the current real image.

[0030] The correction amount calculation unit 13 calculates a correction amount for a positional deviation of the virtual image relative to the current real image based on current position information indicating a position where the current real image was captured and past position information indicating a position where a past real image was captured, among the position information acquired for each frame. The correction amount calculation unit 13 is an example of a calculation unit. For example, when the virtual image is created using a real image that is two frames before the current real image, the correction amount calculation unit 13 uses the current position information of the current real image and past position information of the real image with a frame number two frames before the current real image to calculate the correction amount.

[0031] For example, a virtual image may be created using a real image that is two frames before the current real image, or a real image that is three frames before the current real image. In other words, a virtual image is not always created using a real image that is a fixed number of frames before the current real image. Therefore, there is a risk that the past position information of the real image that was used to create the virtual image cannot be identified.

[0032] Therefore, for example, the correction amount calculation unit 13 acquires the frame number of the past real image used to create the virtual image from the external device 20. This makes it possible to identify the past real image used to create the virtual image, and ultimately to identify the past position information of the past real image used to create the virtual image.

[0033] For example, the current position information includes current coordinates, which are the coordinates of the position where the current real image was captured, and the past position information includes past coordinates, which are the coordinates of the position where the past real image was captured. The correction amount calculation unit 13 calculates the positional deviation of the virtual image with respect to the current real image based on the amount of change in coordinates from the past coordinates to the current coordinates. For example, if the amount of positional deviation is +a in the x direction and +b in the y direction, the correction amount can be calculated as -a in the x direction and -b in the y direction. In this way, the positional deviation of the past real image with respect to the current real image, i.e., the positional deviation of the virtual image created using the past real image with respect to the current real image, can be calculated from the amount of change in coordinates from the past coordinates to the current coordinates.

[0034] The correction unit 15 corrects the positional deviation of the virtual image with respect to the current real image based on the calculated correction amount. For example, the correction unit 15 corrects the positional deviation by shifting the position where the virtual image is displayed on the display unit 300 or the position where the virtual image is superimposed on the current real image in the up, down, left, or right directions based on the calculated correction amount. For example, if the positional deviation amount is +a in the x direction and +b in the y direction, the correction unit 15 can correct the positional deviation of the virtual image with respect to the current real image by shifting the position where the virtual image is displayed on the display unit 300 or the position where the virtual image is superimposed on the current real image by -a in the x direction and -b in the y direction.

[0035] The output unit 16 outputs the virtual image whose positional deviation has been corrected to the display unit 300. For example, the output unit 16 superimposes the virtual image whose positional deviation has been corrected on the current real image and outputs the superimposed image to the display unit 300. For example, the virtual image may be displayed on the display unit 300 as is, or an image in which the virtual image is superimposed on the real image may be displayed on the display unit 300.

[0036] Next, a specific example of the processing flow of the image processing system 1 will be described with reference to FIG.

[0037] 2 is a diagram illustrating a specific example of the processing flow of the image processing system 1 according to the first embodiment. For example, the real image "2" of frame 2 is the latest real image, i.e., the current real image, and the real image "0" of frame 0 and the real image "1" of frame 1 are past real images. In the example of FIG. 2, the virtual image creation unit 21 creates a virtual image using the past real image that is two frames before the current real image.

[0038] The image sensor 100 captures a real image "0" of frame 0, and the image processing unit 11 performs image processing on the real image "0" and transmits the real image "0" to the external device 20. The virtual image creation unit 21 creates a virtual image "c0" using the real image "0". Time passes while the virtual image "c0" is being created, and a real image "1" of frame 1 and a real image "2" of frame 2 are captured. The position information holding unit 12 holds the position information of the real images "0", "1", and "2". The correction amount calculation unit 13 calculates a correction amount for the positional deviation of the virtual image "c0" with respect to the current real image "2" based on, among the position information of the real images "0", "1", and "2", current position information indicating the position where the current real image "2" was captured and past position information indicating the position where the past real image "0" used to create the virtual image "c0" was captured. The correction unit 15 corrects the positional deviation of the virtual image "c0" based on the calculated correction amount, and the output unit 16 superimposes the virtual image "c0" with the positional deviation corrected on the current real image "2". Then, the display unit 300 displays a superimposed image "2+c0" in which the virtual image "c0" with the positional deviation corrected is superimposed. Note that the output unit 16 may output the virtual image "c0" with the positional deviation corrected to the display unit 300, and the display unit 300 may display the virtual image "c0" with the positional deviation corrected.

[0039] As described above, when a real image is input and then a virtual image is created using the input real image and then displayed, a processing delay occurs between the input of the real image and the display of the virtual image, which may cause visually induced motion sickness. In contrast, a virtual image created using a past real image rather than a current real image (in other words, the most recent real image) is displayed, thereby reducing the delay between the input of the real image and the display of the virtual image (see FIG. 2 ). However, because the virtual image is created using a past real image, there is a risk of misalignment with the current real image. Therefore, by checking the degree of misalignment based on the position information of the current real image and the position information of the past real image, a correction amount for eliminating the misalignment can be calculated. Therefore, by correcting the misalignment of the virtual image based on the calculated correction amount, a virtual image with reduced misalignment can be displayed.

[0040] Second Embodiment Next, an image processing device and an image processing system according to a second embodiment will be described.

[0041] FIG. 3 is a block diagram showing an example of an image processing system 2 according to the second embodiment.

[0042] Image processing system 2 differs from image processing system 1 according to embodiment 1 in that image processing system 2 includes image processing device 10a instead of image processing device 10. Image processing device 10a also differs from image processing device 10 according to embodiment 1 in that it includes a synchronization control unit 17. Other points are basically the same as those in embodiment 1, so the following description will focus on the differences.

[0043] The synchronization control unit 17 synchronizes the timing at which the virtual image is acquired with the timing at which the current real image is acquired. For example, the synchronization control unit 17 acquires a vertical synchronization signal (Vsync information) of the virtual image from the external device 20, and synchronizes the timing at which the virtual image is acquired with the timing at which the current real image is acquired using the vertical synchronization signal of the virtual image.

[0044] A specific example of the processing flow of the image processing system 2 will now be described with reference to FIG.

[0045] 4 is a diagram for explaining a specific example of the processing flow of the image processing system 2 according to the second embodiment. The left side of FIG. 4 shows the processing flow when control (Vsync control) is performed to synchronize the timing at which a virtual image is acquired with the timing at which a current real image is acquired. The right side of FIG. 4 shows the processing flow when Vsync control is not performed. A superimposed image "2+c0" in which virtual image "c0" is superimposed on real image "2" in frame 2 is displayed, and a superimposed image "3+c1" in which virtual image "c1" is superimposed on real image "3" in frame 3 is displayed. The following description focuses on real image "3" and virtual image "c1".

[0046] If Vsync control is not performed, it is difficult to acquire and superimpose the virtual image "c1" simultaneously with the timing at which the current real image "3" is acquired. Therefore, after the virtual image "c1" is created, a virtual image memory 14 with a relatively large capacity is required as a buffer to store the virtual image "c1" for the waiting time until the virtual image "c1" is superimposed on the current real image "3." For example, a frame memory is required as the virtual image memory 14.

[0047] On the other hand, when Vsync control is performed, the virtual image "c1" and the current real image "3" can be acquired almost simultaneously, which reduces the waiting time after the virtual image "c1" is created until it is superimposed on the current real image "3", thereby reducing the capacity of the buffer that stores the virtual image "c1". For example, a line buffer can be used as the virtual image memory 14.

[0048] As described above, without Vsync control, it is difficult to acquire and superimpose a virtual image created using a past real image simultaneously with the acquisition of a current real image. Therefore, a virtual image memory 14 with a relatively large capacity is required to store the virtual image from the time the virtual image is created until the time the virtual image is superimposed on the current real image. Since the larger the capacity of the virtual image memory 14, the higher the cost of the image processing device 10a, it is desirable to reduce the capacity of the virtual image memory 14. Therefore, by performing Vsync control, the virtual image is superimposed on the current real image immediately after it is created, thereby reducing the capacity of the virtual image memory 14. For example, while without Vsync control, the virtual image memory 14 needs to be a frame buffer, with Vsync control, the virtual image memory 14 can be a line buffer with a smaller capacity than a frame buffer.

[0049] (Other Embodiments) As described above, the embodiments have been described as examples of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. For example, the following modifications are also included in one embodiment of the present disclosure.

[0050] For example, the present disclosure can be realized not only as an image processing device, but also as an image processing method including steps (processing) performed by components that make up the image processing device.

[0051] FIG. 5 is a flowchart showing an example of an image processing method according to another embodiment.

[0052] As shown in FIG. 5 , the image processing method includes a real image acquisition step (step S11) of acquiring temporally consecutive real images for each frame, a position information acquisition step (step S12) of acquiring, for each frame, position information indicating the position at which the real image was captured, a virtual image acquisition step (step S13) of acquiring a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1), a calculation step (step S14) of calculating a correction amount for the positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was captured and past position information indicating the position at which the past real image was captured, among the position information acquired for each frame, a correction step (step S15) of correcting the positional deviation based on the correction amount, and an output step (step S16) of outputting the virtual image with the positional deviation corrected.

[0053] For example, the present disclosure can be realized as a program for causing a computer (processor) to execute steps included in an image processing method. Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

[0054] For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU, memory, input / output circuits, etc. In other words, each step is performed by the CPU acquiring data from memory or input / output circuits, etc., performing calculations, and outputting the calculation results to memory or input / output circuits, etc.

[0055] In the above-described embodiment, each component included in the image processing system may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0056] Some or all of the functions of the image processing system according to the above-described embodiments are typically realized as an LSI, which is an integrated circuit. These functions may be individually integrated into single chips, or may be integrated into a single chip that includes some or all of the functions. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which allows the connections and settings of circuit cells within the LSI to be reconfigured.

[0057] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, that technology may naturally be used to integrate each component included in the image processing system.

[0058] In addition, this disclosure also includes forms obtained by making various modifications to the embodiments that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions in each embodiment within the scope of the present disclosure.

[0059] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0060] (Technology 1) An image processing device comprising: a real image acquisition unit that acquires temporally consecutive real images for each frame; a position information acquisition unit that acquires, for each frame, position information indicating the position at which the real images were taken; a virtual image acquisition unit that acquires a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation unit that calculates a correction amount for positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction unit that corrects the positional deviation based on the correction amount; and an output unit that superimposes the virtual image with the positional deviation corrected on the current real image.

[0061] When a real image is input and then a virtual image is created using the input real image and displayed, a processing delay occurs between the input of the real image and the display of the virtual image, which may cause visually induced motion sickness. In contrast, a virtual image created using a past real image rather than a current real image (in other words, the most recent real image) is displayed, thereby reducing the delay between the input of the real image and the display of the virtual image. However, since the virtual image is created using a past real image, there is a risk of misalignment with the current real image. Therefore, by checking the degree of misalignment from the position information of the current real image and the position information of the past real image, a correction amount for eliminating the misalignment can be calculated. Therefore, by correcting the misalignment of the virtual image based on the calculated correction amount, a virtual image with reduced misalignment with respect to the real image can be displayed.

[0062] (Technology 2) The image processing device described in Technology 1, wherein the current position information includes current coordinates, which are the coordinates of the position where the current real image was taken, and the past position information includes past coordinates, which are the coordinates of the position where the past real image was taken, and the calculation unit calculates the positional deviation based on the amount of change in coordinates from the past coordinates to the current coordinates.

[0063] In this way, the positional deviation of the past real image from the current real image, i.e., the positional deviation of the virtual image created using the past real image from the current real image, can be calculated from the amount of change in coordinates from the past coordinates to the current coordinates.

[0064] (Technology 3) The image processing device according to Technology 1 or 2, wherein the calculation unit acquires a frame number of the past real image used to create the virtual image.

[0065] For example, a virtual image may be created using a past real image that is two frames before the current real image, or a virtual image may be created using a past real image that is three frames before the current real image. In other words, a virtual image is not always created using a real image that is a fixed number of frames before the current real image. Therefore, there is a risk that it may not be possible to identify the past location information of the past real image used to create the virtual image. Therefore, by acquiring the frame number of the past real image used to create the virtual image, it is possible to identify the past real image used to create the virtual image, and thus to identify the past location information of the past real image used to create the virtual image.

[0066] (Technology 4) The image processing device according to any one of technologies 1 to 3, wherein the output unit outputs the virtual image in which the positional deviation has been corrected by superimposing it on the current real image.

[0067] When a real image is input and then a virtual image is created using the input real image and then superimposed on the real image for display, a processing delay occurs between the input of the real image and the display of the image on which the virtual image is superimposed, which may cause visually induced motion sickness. In contrast, a virtual image created using a past real image, rather than a current real image (in other words, the most recent real image), is superimposed on the current real image for display, thereby reducing the delay between the input of the real image and the display of the image on which the virtual image is superimposed. However, since the virtual image is created using a past real image, there is a risk of misalignment with the current real image. Therefore, by checking the degree of misalignment from the position information of the current real image and the position information of the past real image, a correction amount for eliminating the misalignment can be calculated. Therefore, by superimposing a virtual image on the current real image in which the misalignment of the virtual image has been corrected based on the calculated correction amount, an image in which a virtual image with reduced misalignment with respect to the real image is superimposed can be displayed.

[0068] (Technology 5) The image processing device according to any one of technologies 1 to 4, further comprising a synchronization control unit that synchronizes the timing at which the virtual image is acquired with the timing at which the current real image is acquired.

[0069] Without control to synchronize the timing at which a virtual image is acquired with the timing at which a current real image is acquired, it is difficult to simultaneously acquire and superimpose a virtual image created using a past real image with the timing at which a current real image is acquired. Therefore, a virtual image memory with a certain capacity is required to store the virtual image from the time the virtual image is created until the time the virtual image is superimposed on the current real image. Since the larger the capacity of the virtual image memory, the higher the cost of the image processing device, the smaller the capacity of the virtual image memory is desired. Therefore, by performing synchronization control to synchronize the timing at which a virtual image is acquired with the timing at which a current real image is acquired, the virtual image is superimposed on the current real image immediately after it is created, thereby reducing the capacity of the virtual image memory. For example, without the synchronization control, the virtual image memory needs to be a frame buffer, whereas with the synchronization control, the virtual image memory can be a line buffer with a smaller capacity than the frame buffer.

[0070] (Technology 6) The image processing device described in Technology 5, wherein the synchronization control unit uses a vertical synchronization signal of the virtual image to synchronize the timing at which the virtual image is acquired with the timing at which the current real image is acquired.

[0071] In this way, by using the vertical synchronization signal of the virtual image, synchronization control can be easily performed.

[0072] (Technology 7) An image processing system comprising the image processing device according to any one of technologies 1 to 6 and an external device that creates the virtual image.

[0073] This makes it possible to provide an image processing system that can reduce the delay between the input of a real image and the display of a virtual image.

[0074] (Technology 7) An image processing method including: a real image acquisition step of acquiring temporally consecutive real images for each frame; a position information acquisition step of acquiring for each frame position information indicating the position at which the real images were taken; a virtual image acquisition step of acquiring a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation step of calculating a correction amount for positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction step of correcting the positional deviation based on the correction amount; and an output step of outputting the virtual image with the positional deviation corrected.

[0075] This makes it possible to provide an image processing method that can reduce the delay between the input of a real image and the display of an image on which a virtual image is superimposed.

[0076] The present disclosure is applicable to systems that display virtual images, such as AR glasses or VR headsets.

[0077] REFERENCE SIGNS LIST 1, 2 Image processing system 10, 10a Image processing device 11 Image processing unit 12 Position information holding unit 13 Correction amount calculation unit 14 Virtual image memory 15 Correction unit 16 Output unit 17 Synchronization control unit 20 External device 21 Virtual image creation unit 100 Image sensor 200 Sensor 300 Display unit

Claims

1. An image processing device comprising: a real image acquisition unit that acquires temporally consecutive real images for each frame; a position information acquisition unit that acquires, for each frame, position information indicating the position at which the real image was taken; a virtual image acquisition unit that acquires a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation unit that calculates a correction amount for positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction unit that corrects the positional deviation based on the correction amount; and an output unit that outputs the virtual image with the positional deviation corrected.

2. The image processing device described in claim 1, wherein the current position information includes current coordinates, which are the coordinates of the position where the current real image was taken, and the past position information includes past coordinates, which are the coordinates of the position where the past real image was taken, and the calculation unit calculates the positional deviation based on the amount of change in coordinates from the past coordinates to the current coordinates.

3. The image processing device according to claim 1 or 2, wherein the calculation unit acquires the frame number of the past real image used to create the virtual image.

4. The image processing device according to any one of claims 1 to 3, wherein the output unit outputs the virtual image, the positional deviation of which has been corrected, superimposed on the current real image.

5. The image processing device according to claim 4, further comprising a synchronization control unit that synchronizes the timing at which the virtual image is acquired with the timing at which the current real image is acquired.

6. The image processing device according to claim 5, wherein the synchronization control unit uses a vertical synchronization signal of the virtual image to synchronize the timing at which the virtual image is acquired with the timing at which the current real image is acquired.

7. An image processing system comprising: an image processing device according to any one of claims 1 to 6; and an external device that creates the virtual image.

8. An image processing method comprising: a real image acquisition step of acquiring temporally consecutive real images for each frame; a position information acquisition step of acquiring, for each frame, position information indicating the position at which the real image was taken; a virtual image acquisition step of acquiring a virtual image created using a past real image N frames before the current real image (N is an integer greater than or equal to 1); a calculation step of calculating a correction amount for positional deviation of the virtual image relative to the current real image based on current position information indicating the position at which the current real image was taken and past position information indicating the position at which the past real image was taken, among the position information acquired for each frame; a correction step of correcting the positional deviation based on the correction amount; and an output step of outputting the virtual image with the positional deviation corrected.

Images