Image processing apparatus, image processing system, and image processing method

Abstract

An image processing apparatus (10) includes: an image processing section (11) that acquires a real image that is continuous in time for each frame; a position information holding section (12) that acquires position information indicating a position at which the real image is captured for each frame; a virtual image storage (14) that acquires a virtual image that is created using a past real image that is N frames (N is an integer of 1 or more) before a current real image; a correction amount calculation section (13) that calculates a correction amount of a positional deviation of the virtual image with respect to the current real image based on current position information and past position information among the position information acquired for each frame, the current position information indicating a position at which the current real image is captured, and the past position information indicating a position at which the past real image is captured; a correction section (15) that corrects the positional deviation based on the correction amount; and an output section (16) that outputs the virtual image after the positional deviation is corrected.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus for displaying virtual images, etc. Background Technology

[0002] Patent document 1 discloses a technique for overlaying images of virtual space, such as CG (Computer Graphics), onto images of real space (referred to as real images).

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2023-183684 Summary of the Invention

[0004] The problem that the invention aims to solve However, in technologies for displaying virtual images, sometimes a real image is input into an image processing device and then used to create a virtual image. In this case, a processing delay occurs from the input of the real image to the display of the virtual image, which may cause visual dizziness.

[0005] Therefore, this disclosure provides an image processing apparatus, etc., capable of suppressing the delay from the input of a real image to the display of a virtual image.

[0006] Methods for solving problems The image processing apparatus disclosed herein includes: a real image acquisition unit that acquires real images that are sequentially continuous in time, frame by frame; a position information acquisition unit that acquires position information representing the position where the real image was captured, frame by frame; a virtual image acquisition unit that acquires a virtual image created using past real images (N is an integer greater than or equal to 1) N frames prior to the current real image; a calculation unit that calculates a correction amount for the positional deviation of the virtual image relative to the current real image based on the current position information and the past position information acquired in each frame, wherein the current position information represents the position where the current real image was captured and the past position information represents the position where the past real image was captured; a correction unit that corrects the positional deviation based on the correction amount; and an output unit that outputs the virtual image after the positional deviation has been corrected.

[0007] The image processing system disclosed herein includes the image processing apparatus described above and an external device for creating the virtual image.

[0008] The image processing method disclosed herein includes: a real image acquisition step, acquiring real images that are sequentially continuous in time, frame by frame; a location information acquisition step, acquiring location information representing the location where the real image was captured, frame by frame; a virtual image acquisition step, acquiring a virtual image created using past real images (N is an integer greater than or equal to 1) N frames prior to the current real image; a calculation step, calculating a correction amount for the positional deviation of the virtual image relative to the current real image based on the current location information and the past location information acquired, where the current location information represents the location where the current real image was captured and the past location information represents the location where the past real image was captured; a correction step, correcting the positional deviation based on the correction amount; and an output step, outputting the virtual image after correcting the positional deviation.

[0009] Furthermore, these general or specific methods can be implemented either through systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0010] Invention Effects An image processing apparatus or the like according to one aspect of the present disclosure can suppress the delay from the input of a real image to the display of a virtual image. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example of the image processing system of Embodiment 1.

[0012] Figure 2 This is a diagram illustrating a specific example of the processing flow of the image processing system in Embodiment 1.

[0013] Figure 3 This is a block diagram illustrating an example of the image processing system of Embodiment 2.

[0014] Figure 4 This is a diagram illustrating the processing flow of the image processing system in Embodiment 2.

[0015] Figure 5 This is a flowchart illustrating an example of an image processing method in another implementation. Detailed Implementation

[0016] The embodiments will now be described in detail with reference to the accompanying drawings.

[0017] Furthermore, the embodiments described below are general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement of constituent elements, connection methods, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit this disclosure.

[0018] In addition, in this specification, "current" refers to the timing when the virtual image is output to the display, and "past" refers to the timing before that.

[0019] (Implementation Method 1) The image processing apparatus and image processing system of Embodiment 1 will be described below.

[0020] Figure 1 This is a block diagram illustrating an example of the image processing system 1 according to Embodiment 1.

[0021] Image processing system 1 is, for example, a system used in fields such as MR (Mixed Reality), AR (Augmented Reality), or VR (Virtual Reality), and is used to overlay virtual images onto real images. Furthermore, image processing system 1 uses a pass-through method as one of the video perspective methods. Additionally, image processing system 1 can use optical perspective methods, and virtual images can be displayed on transparent displays, etc. That is, virtual images can also be displayed without overlaying onto real images.

[0022] The image processing system 1 includes an image processing device 10, an external device 20, an image sensor 100, a sensor 200, and a display unit 300.

[0023] For example, the image processing system 1 may include AR glasses or a VR headset, with the image processing device 10, external device 20, image sensor 100, sensor 200, and display unit 300 mounted on the AR glasses or VR headset. Alternatively, the external device 20 may not be mounted on the AR glasses or VR headset, and may be a computer such as a server that wirelessly communicates with the AR glasses or VR headset.

[0024] Image sensor 100 is a sensor that captures images of the surroundings of AR glasses or VR headsets, etc., and is installed on the AR glasses or VR headset, etc., in such a way that the captured images correspond to the field of view of the user using image processing system 1, specifically the user wearing AR glasses or VR headsets, etc. The images captured by image sensor 100 are referred to as real-world images.

[0025] Sensor 200 is a position sensor that detects the user's position, specifically the position of AR glasses or VR headsets equipped with sensor 200. Sensor 200 is not particularly limited as long as it is a sensor capable of detecting position; for example, it can be an accelerometer or a GPS (Global Positioning System) sensor.

[0026] Display unit 300 is a display that shows virtual images or real images overlaid with virtual images (also called overlaid images). For example, display unit 300 is an OLED microdisplay for AR glasses or VR headsets.

[0027] External device 20 includes a virtual image production unit 21 for creating virtual images such as CG.

[0028] The image processing apparatus 10 includes an image processing unit 11, a position information holding unit 12, a correction amount calculation unit 13, a virtual image memory 14, a correction unit 15, and an output unit 16. The image processing apparatus 10 is a computer including a processor (microprocessor) and memory. The memory is the same as ROM (Read Only Memory) or RAM (Random Access Memory), and is capable of storing programs executed by the processor. The image processing unit 11, the position information holding unit 12, the correction amount calculation unit 13, the correction unit 15, and the output unit 16 are implemented by a processor that executes the program stored in the memory. The virtual image memory 14 can be the same memory as the memory storing the program, or it can be a different memory.

[0029] The image processing unit 11 acquires real-world images that are sequentially acquired in time from the image sensor 100, frame by frame. That is, the image processing unit 11 acquires images from the image sensor 100 and processes the image of each frame constituting the image. The image processing unit 11 is an example of a real-world image acquisition unit. The image processing unit 11 sends the real-world images to the external device 20. Furthermore, in the case where a virtual image is superimposed on a real-world image, such as... Figure 1 As shown, the image processing unit 11 sends the real image to the output unit 16 and the external device 20.

[0030] The position information holding unit 12 acquires position information representing the location of the captured real-world image (specifically, the location of AR glasses or VR headsets, etc.) for each frame (in other words, whenever the image processing unit 11 acquires a real-world image). That is, the position information holding unit 12 holds the position information for each frame. The position information holding unit 12 is an example of a position information acquisition unit. For example, the position information for each frame includes the location of the captured real-world image and the frame number of that real-world image.

[0031] The virtual image creation unit 21 creates a virtual image using a real image acquired from the image processing unit 11. However, since creating a virtual image takes some time, it is difficult to output the virtual image created using that real image to the display unit 300 (display) at a timing that is almost simultaneous with the timing of the input of the real image from the image sensor 100. Therefore, after the virtual image creation unit 21 creates the virtual image, if the timing for outputting the virtual image to the display unit 300 is set to the current time, the virtual image creation unit 21 uses a past real image (N is an integer greater than or equal to 1) from N frames before the current real image to create the virtual image. That is, the real image used by the virtual image creation unit 21 to create the virtual image is a past real image relative to the real image N frames after the timing when the virtual image is output to the display unit 300. The virtual image creation unit 21 sends the created virtual image and its frame number, specifically the frame number of the real image used in the creation of the virtual image (the past real image), to the image processing device 10.

[0032] The virtual image memory 14 acquires a virtual image created using past real images N frames prior to the current real image. The virtual image memory 14 is an example of a virtual image acquisition unit. The virtual image memory 14 is, for example, a frame memory, and holds the virtual image during the waiting period from creation to output to the display unit 300.

[0033] In addition, virtual images are created using past real-world images. Therefore, if the location of the past real-world image is different from the location of the current real-world image, the virtual image will also be different from the current real-world image.

[0034] The correction amount calculation unit 13 calculates the correction amount for the positional deviation of the virtual image relative to the current real image based on the current positional information and past positional information acquired in each frame. The current positional information represents the position of the current real image captured, and the past positional information represents the position of a past real image captured. The correction amount calculation unit 13 is one example of a calculation unit. For example, when creating a virtual image using a real image two frames prior to the current real image, the correction amount calculation unit 13 uses the current positional information of the current real image and the past positional information of the real image with frame numbers two frames prior to the current real image to calculate the correction amount.

[0035] Furthermore, for example, sometimes a virtual image is created using a past real-world image two frames ago from the current real-world image, and sometimes it is created using a past real-world image three frames ago from the current real-world image. That is, a virtual image is not always created using a fixed number of past real-world images relative to the current real-world image. Therefore, it may be impossible to determine the past position information of the past real-world images used in the creation of the virtual image.

[0036] Therefore, for example, the correction calculation unit 13 obtains the frame number of the past real image used in the production of the virtual image from the external device 20. This allows the determination of the past real image used in the production of the virtual image, and further, the determination of the past position information of the past real image used in the production of the virtual image.

[0037] For example, the current location information includes the coordinates of the location where the current real-world image was captured, i.e., the current coordinates, and the past location information includes the coordinates of the location where a past real-world image was captured, i.e., the past coordinates. The correction calculation unit 13 calculates the positional deviation of the virtual image relative to the current real-world image based on the change in coordinates from the past coordinates to the current coordinates. For example, if the positional deviation is +a in the x-direction and +b in the y-direction, the correction amount can be calculated as -a in the x-direction and -b in the y-direction. In this way, based on the change in coordinates from the past coordinates to the current coordinates, the positional deviation of the past real-world image relative to the current real-world image can be calculated, i.e., the positional deviation of the virtual image created using the past real-world image relative to the current real-world image.

[0038] The correction unit 15 corrects the positional deviation of the virtual image relative to the current real image based on a calculated correction amount. For example, based on the calculated correction amount, the correction unit 15 offsets the position where the virtual image is displayed on the display unit 300 or where the virtual image is superimposed on the current real image in the up, down, left, and right directions, thereby correcting the positional deviation. For example, if the positional deviation is +a in the x-direction and +b in the y-direction, the correction unit 15 can correct the positional deviation of the virtual image relative to the current real image by offsetting the position where the virtual image is displayed on the display unit 300 or where the virtual image is superimposed on the current real image by -a in the x-direction and -b in the y-direction.

[0039] The output unit 16 outputs the virtual image after correcting for positional deviation to the display unit 300. For example, the output unit 16 overlays the virtual image after correcting for positional deviation onto the current real image and outputs it to the display unit 300. For example, the virtual image can be displayed on the display unit 300 as is, or the image formed by overlaying the virtual image onto the real image can be displayed on the display unit 300.

[0040] Next, use Figure 2 A specific example of the processing flow of image processing system 1 is explained.

[0041] Figure 2 This is a diagram illustrating a specific example of the processing flow of the image processing system 1 in Embodiment 1. For example, the real image "2" in frame 2 is the latest real image, i.e., the current real image, while the real image "0" in frame 0 and the real image "1" in frame 1 are past real images. Furthermore, in Figure 2 In the example, the virtual image production unit 21 uses a past real image two frames before the current real image to create a virtual image.

[0042] Image sensor 100 captures a real image "0" of frame 0. Image processing unit 11 performs image processing on real image "0" and sends real image "0" to external device 20. Virtual image creation unit 21 uses real image "0" to create virtual image "c0". During the creation of virtual image "c0", real image "1" of frame 1 and real image "2" of frame 2 are captured. Position information holding unit 12 holds the position information of real images "0", "1", and "2". Correction amount calculation unit 13 calculates a correction amount for the positional deviation of virtual image "c0" relative to the current real image "2" based on the current position information indicating the position of the current real image "2" and the past position information indicating the position of the past real image "0" used in the creation of virtual image "c0". Correction unit 15 corrects the positional deviation of virtual image "c0" based on the calculated correction amount, and output unit 16 overlays the corrected virtual image "c0" onto the current real image "2". Then, the display unit 300 displays an overlay image "2+c0" that overlaps the virtual image "c0" after the positional deviation correction. Alternatively, the output unit 16 may output the virtual image "c0" after the positional deviation correction to the display unit 300, and the display unit 300 may display the virtual image "c0" after the positional deviation correction.

[0043] As explained above, when a virtual image is created and displayed using a real-world image after it has been input, a processing delay occurs from the input of the real-world image to the display of the virtual image, potentially causing visual dizziness. In contrast, by displaying a virtual image created using a past real-world image instead of the current one (in other words, the latest real-world image), the delay from the input of the real-world image to the display of the virtual image can be suppressed (see [reference]). Figure 2However, since virtual images are created using past real-world images, they may have positional deviations relative to the current real-world image. Therefore, by determining the degree of positional deviation based on the positional information of the current real-world image and the positional information of the past real-world image, it is possible to calculate a correction amount to eliminate this positional deviation. Thus, by correcting the positional deviation of the virtual image based on the calculated correction amount, it is possible to display a virtual image with the positional deviation suppressed.

[0044] (Implementation Method 2) Next, the image processing apparatus and image processing system of Embodiment 2 will be described.

[0045] Figure 3 This is a block diagram illustrating an example of the image processing system 2 according to Embodiment 2.

[0046] Image processing system 2 differs from image processing system 1 of embodiment 1 in that it has image processing device 10a instead of image processing device 10. Additionally, image processing device 10a also includes a synchronization control unit 17, unlike image processing device 10 of embodiment 1. Other aspects are basically the same as the structure in embodiment 1, therefore, the following description focuses on the differences.

[0047] The synchronization control unit 17 synchronizes the timing of acquiring the virtual image with the timing of acquiring the current real image. For example, the synchronization control unit 17 acquires the vertical synchronization signal (Vsync information) of the virtual image from the external device 20, and uses the vertical synchronization signal of the virtual image to synchronize the timing of acquiring the virtual image with the timing of acquiring the current real image.

[0048] Here, use Figure 4 A specific example of the processing flow of image processing system 2 is explained.

[0049] Figure 4 This is a diagram illustrating a specific example of the processing flow of the image processing system 2 in Embodiment 2. Figure 4 The left side shows the processing flow when the timing for acquiring the virtual image is synchronized with the timing for acquiring the current real image (Vsync control). Figure 4 The right side shows the processing flow without Vsync control. It shows the overlaid image "2+c0" obtained by overlaying the virtual image "c0" onto the real image "2" in frame 2, and the overlaid image "3+c1" obtained by overlaying the virtual image "c1" onto the real image "3" in frame 3. The following explanation focuses on the real image "3" and the virtual image "c1".

[0050] Without Vsync control, it is difficult to acquire and overlay the virtual image "c1" simultaneously with the timing of acquiring the current real image "3". Therefore, a virtual image memory 14 with a certain capacity is needed as a buffer for holding the virtual image "c1" for a period of time equivalent to the waiting time until the overlay timing with the current real image "3" is reached after the virtual image "c1" is created. For example, a frame memory is needed as the virtual image memory 14.

[0051] On the other hand, when Vsync control is performed, the virtual image "c1" and the current real image "3" can be acquired almost simultaneously. Therefore, after the virtual image "c1" is created, the waiting time until the overlap timing of the virtual image "3" is reduced, and the capacity of the buffer holding the virtual image "c1" can be suppressed. For example, a row buffer can be used as the virtual image memory 14.

[0052] As explained above, without Vsync control, it is difficult to acquire and overlay virtual images created from past real images simultaneously with the timing of acquiring the current real image. Therefore, the virtual image memory 14, which holds the virtual image after its creation until the overlay process with the current real image is performed, needs to have a sufficiently large capacity. The larger the capacity of the virtual image memory 14, the higher the cost of the image processing apparatus 10a, so it is desirable to reduce the capacity of the virtual image memory 14. Therefore, by performing Vsync control, the overlay process with the current real image is performed immediately after the virtual image is generated, thus reducing the capacity of the virtual image memory 14. For example, without Vsync control, the virtual image memory 14 needs to be configured as a frame buffer; in contrast, with Vsync control, the virtual image memory 14 can be configured as a line buffer with a smaller capacity than the frame buffer.

[0053] (Other implementation methods) As described above, embodiments have been illustrated as examples of the technology disclosed herein. However, the technology disclosed herein is not limited thereto and can be applied to embodiments with appropriate modifications, substitutions, additions, omissions, etc. For example, the following variations are also included in one embodiment of this disclosure.

[0054] For example, this disclosure can be implemented not only as an image processing apparatus, but also as an image processing method including steps (processing) performed on the constituent elements constituting the image processing apparatus.

[0055] Figure 5 This is a flowchart illustrating an example of an image processing method in another implementation.

[0056] like Figure 5 As shown, the image processing method includes: a real image acquisition step (step S11), acquiring real images that are sequentially continuous in time, frame by frame; a position information acquisition step (step S12), acquiring position information representing the position of the captured real image, frame by frame; a virtual image acquisition step (step S13), acquiring a virtual image created using past real images N frames ago (N is an integer greater than or equal to 1) of the current real image; a calculation step (step S14), calculating a correction amount for the positional deviation of the virtual image relative to the current real image based on the current position information and past position information acquired in each frame, where the current position information represents the position of the captured real image and the past position information represents the position of the captured past real images; a correction step (step S15), correcting the positional deviation based on the correction amount; and an output step (step S16), outputting the virtual image after correcting the positional deviation.

[0057] For example, this disclosure can be implemented as a program for causing a computer (processor) to execute the steps included in an image processing method. Furthermore, this disclosure can be implemented as a non-transitory computer-readable recording medium such as a CD-ROM containing the program.

[0058] For example, when this disclosure is implemented by a program (software), the steps are executed by utilizing the computer's hardware resources such as the CPU, memory, and input / output circuits. That is, the CPU retrieves data from the memory or input / output circuits and performs calculations, or outputs the calculation results to the memory or input / output circuits, thereby executing the steps.

[0059] Furthermore, in the above embodiments, each component of the image processing system can be constructed using dedicated hardware, or implemented by executing software programs suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on a recording medium such as a hard disk or semiconductor memory.

[0060] The image processing systems described above typically implement some or all of their functionality as integrated circuits, i.e., LSIs. They can be implemented as a single chip, or as a single chip including some or all of the components. Furthermore, integrated circuit implementation is not limited to LSIs; it can also be implemented using dedicated circuits or general-purpose processors. Alternatively, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI fabrication, or reconfigurable processors that can reconfigure the connections and settings of the circuit units within the LSI, can be utilized.

[0061] Furthermore, if an integrated circuit technology emerges to replace LSI as semiconductor technology advances or other derived technologies emerge, then this technology can certainly be used to integrate the various components contained in an image processing system.

[0062] Furthermore, this disclosure also includes various modifications to the embodiments that would be conceived by those skilled in the art, and methods implemented by arbitrarily combining the constituent elements and functions of each embodiment without departing from the spirit of this disclosure.

[0063] (Postscript) The following technology has been disclosed through the above description of the embodiments.

[0064] (Technology 1) An image processing apparatus comprising: a real image acquisition unit that acquires real images that are sequentially continuous in time, frame by frame; a position information acquisition unit that acquires position information representing the position where the real images were captured, frame by frame; a virtual image acquisition unit that acquires a virtual image created using past real images (N is an integer greater than or equal to 1) N frames prior to the current real image; a calculation unit that calculates a correction amount for a positional deviation of the virtual image relative to the current real image based on the current position information and the past position information acquired in each frame, the current position information representing the position where the current real image was captured, and the past position information representing the position where the past real images were captured; a correction unit that corrects the positional deviation based on the correction amount; and an output unit that overlays the virtual image after the positional deviation has been corrected onto the current real image.

[0065] When a virtual image is created and displayed using a real-world image after input, a processing delay occurs from the input of the real-world image to its display, potentially causing visual dizziness. In contrast, if a virtual image is created using a past real-world image instead of the current one (in other words, the latest real-world image), this delay can be suppressed. However, since the virtual image is created using a past real-world image, it may have a positional deviation relative to the current real-world image. Therefore, by determining the degree of positional deviation based on the positional information of the current and past real-world images, a correction amount can be calculated to eliminate this deviation. Thus, by correcting the positional deviation of the virtual image based on the calculated correction amount, a virtual image with suppressed positional deviation relative to the real-world image can be displayed.

[0066] (Technology 2) The image processing apparatus according to Technology 1, wherein the current position information includes the coordinates of the position where the current real image was captured, i.e., the current coordinates, the past position information includes the coordinates of the position where the past real image was captured, i.e., the past coordinates, and the calculation unit calculates the position deviation based on the amount of coordinate change from the past coordinates to the current coordinates.

[0067] In this way, the positional deviation of the past real image relative to the current real image can be calculated based on the change in coordinates from the past coordinates to the current coordinates; that is, the positional deviation of the virtual image created using the past real image relative to the current real image.

[0068] (Technology 3) The image processing apparatus according to Technology 1 or 2, wherein the computing unit obtains the frame number of the past real image used in the creation of the virtual image.

[0069] For example, sometimes a virtual image is created using a past real-world image two frames ago, and sometimes it's created using a past real-world image three frames ago. That is, a virtual image is not always created using a fixed number of past real-world images relative to the current real-world image. Therefore, it may be impossible to determine the past position information of the past real-world images used in the creation of the virtual image. Therefore, by obtaining the frame number of the past real-world images used in the creation of the virtual image, it is possible to determine the past real-world images used in the creation of the virtual image, and thus, to determine their past position information.

[0070] (Technology 4) The image processing apparatus according to any one of Technologies 1 to 3, wherein the output unit outputs the virtual image after correcting the positional deviation by superimposing it on the current real image.

[0071] When a virtual image is created from a real-world image after it has been input, and then overlaid on the real-world image for display, a processing delay occurs from the input of the real-world image to the display of the overlaid virtual image, potentially causing image dizziness. In contrast, by overlaying a virtual image created from a past real-world image onto the current real-world image instead of the current real-world image (in other words, the latest real-world image), the delay from the input of the real-world image to the display of the overlaid virtual image can be suppressed. However, since the virtual image is created using a past real-world image, a positional deviation may occur relative to the current real-world image. Therefore, by determining the degree of positional deviation based on the positional information of the current real-world image and the past real-world image, a correction amount for eliminating this positional deviation can be calculated. Thus, by overlaying a virtual image, after correcting for the positional deviation based on the calculated correction amount, onto the current real-world image, an image can be displayed that has suppressed the positional deviation relative to the real-world image.

[0072] (Technology 5) The image processing apparatus according to any one of Technologies 1 to 4, further comprising a synchronization control unit that makes the timing of acquiring the virtual image consistent with the timing of acquiring the current real image.

[0073] Without control to synchronize the timing of acquiring the virtual image with the timing of acquiring the current real image, it is difficult to acquire and overlay a virtual image created using a past real image simultaneously with the timing of acquiring the current real image. Therefore, a virtual image memory, which holds the virtual image after its creation and until the overlay process with the current real image is performed, requires a memory with a sufficiently large capacity. The larger the capacity of the virtual image memory, the higher the cost of the image processing device; therefore, it is desirable to reduce the capacity of the virtual image memory. Therefore, by performing synchronization control to synchronize the timing of acquiring the virtual image with the timing of acquiring the current real image, the overlay process with the current real image is performed immediately after the virtual image is created, thus reducing the capacity of the virtual image memory. For example, without the aforementioned synchronization control, the virtual image memory would need to be a frame buffer; conversely, with the aforementioned synchronization control, the virtual image memory can be a line buffer with a smaller capacity than a frame buffer.

[0074] (Technology 6) The image processing apparatus according to Technology 5, wherein the synchronization control unit uses the vertical synchronization signal of the virtual image to make the timing of acquiring the virtual image consistent with the timing of acquiring the current real image.

[0075] In this way, synchronization control can be easily achieved by using the vertical synchronization signal of the virtual image.

[0076] (Technology 7) An image processing system comprising: an image processing apparatus as described in any one of Technologies 1 to 6; and an external device for creating the virtual image.

[0077] Therefore, it is possible to provide an image processing system that can suppress the delay from the input of a real image to the display of a virtual image.

[0078] (Technology 7) An image processing method, comprising: a real image acquisition step, acquiring real images that are sequentially continuous in time, frame by frame; a location information acquisition step, acquiring location information representing the location where the real image was captured, frame by frame; a virtual image acquisition step, acquiring a virtual image created using past real images (N is an integer greater than or equal to 1) N frames prior to the current real image; a calculation step, calculating a correction amount for the positional deviation of the virtual image relative to the current real image based on the current location information and the past location information acquired, frame by frame, wherein the current location information represents the location where the current real image was captured, and the past location information represents the location where the past real image was captured; a correction step, correcting the positional deviation based on the correction amount; and an output step, outputting the virtual image after correcting the positional deviation.

[0079] Therefore, an image processing method is provided that can suppress the delay from the input of a real image to the display of an image after an overlaid virtual image.

[0080] Industrial applicability This disclosure can be applied to systems that display virtual images, such as AR glasses or VR headsets.

[0081] Explanation of reference numerals in the attached figures 1.2 Image Processing System 10, 10a Image processing apparatus 11 Image Processing Department 12. Position Information Holding Department 13. Calibration Calculation Department 14 Virtual Image Storage 15 Correction Department 16 Output Section 17 Synchronization Control Unit 20 External devices 21 Virtual Image Production Department 100 Image Sensor 200 sensors 300 Display Unit

Claims

1. An image processing apparatus, wherein, have: The real-world image acquisition unit acquires real-world images that are sequential in time, frame by frame; The location information acquisition unit acquires location information representing the location of the captured real-world image for each frame. The virtual image acquisition unit acquires a virtual image created using past real images N frames prior to the current real image, where N is an integer greater than or equal to 1. The computing unit calculates a correction amount for the positional deviation of the virtual image relative to the current real image based on the current positional information and past positional information obtained in each frame. The current positional information represents the position where the current real image was captured, and the past positional information represents the position where the past real image was captured. The correction unit corrects the position deviation based on the correction amount; as well as The output unit outputs the virtual image after correcting the positional deviation.

2. The image processing apparatus according to claim 1, wherein, The current location information includes the coordinates of the location where the current real-world image was captured, i.e., the current coordinates; the past location information includes the coordinates of the location where the past real-world image was captured, i.e., the past coordinates. The calculation unit calculates the position deviation based on the amount of coordinate change from the past coordinates to the current coordinates.

3. The image processing apparatus according to claim 1 or 2, wherein, The computing unit obtains the frame number of the past real-world image used in the creation of the virtual image.

4. The image processing apparatus according to any one of claims 1 to 3, wherein, The output unit outputs the virtual image, after correcting the positional deviation, superimposed on the current real image.

5. The image processing apparatus according to claim 4, wherein, It also includes a synchronization control unit that makes the timing of acquiring the virtual image consistent with the timing of acquiring the current real image.

6. The image processing apparatus according to claim 5, wherein, The synchronization control unit uses the vertical synchronization signal of the virtual image to make the timing of acquiring the virtual image consistent with the timing of acquiring the current real image.

7. An image processing system, wherein, have: The image processing apparatus according to any one of claims 1 to 6; and External devices are used to create the virtual image.

8. An image processing method, wherein, include: The real-world image acquisition step involves acquiring real-world images that are sequential in time, frame by frame. The location information acquisition step involves acquiring location information representing the location of the captured real-world image for each frame. The virtual image acquisition step involves acquiring a virtual image created using past real-world images N frames prior to the current real-world image, where N is an integer greater than 1. The calculation step involves calculating the correction amount for the positional deviation of the virtual image relative to the current real image based on the current positional information and past positional information obtained in each frame. The current positional information represents the position where the current real image was captured, and the past positional information represents the position where the past real image was captured. The correction step involves correcting the positional deviation based on the correction amount; and The output step outputs the virtual image after correcting the positional deviation.

Images