Image correction device and image correction method

Abstract

To reduce a circuit scale in a device for matching coordinates of an image with coordinates of a spherical display.SOLUTION: The present disclosure relates to an image correction device which comprises a coordinate conversion unit configured to convert pixel coordinates of an image into pixel coordinates of a spherical display; and a data buffer unit configured to store pixel values of the image, in which a pixel value stored in the data buffer unit is output as a pixel value of converted coordinates converted by the coordinate conversion unit. The image correction device is capable of reducing a circuit size of at least one of the coordinate conversion unit and the data buffer unit based on a fact that distortion in the image is different between a vicinity of a center of a horizontal coordinate of the image and a vicinity of a periphery of the horizontal coordinate of the image.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a technique for displaying images on a spherical display. [Background technology]

[0002] To provide a display that gives the user an immersive feeling, a technology has been proposed that uses a fisheye lens or multiple cameras to capture images of the user's surroundings in all directions and provide the user with omnidirectional images (see, for example, Patent Document 1). However, Patent Document 1 requires the user to wear a head-mounted display.

[0003] One way to provide an immersive experience without wearing a head-mounted display is to output images to a spherical display. However, when images are output to a spherical display, the images are distorted, so coordinate transformation is required to match the image coordinates with the spherical display coordinates. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-026670 Summary of the Invention [Problem to be solved by the invention]

[0005] The coordinate transformation required to match the coordinates of an image with the coordinates of a spherical display is complex, which increases the circuit size and causes delays. Therefore, the present disclosure aims to reduce the circuit size in a device that matches the coordinates of an image with the coordinates of a spherical display. [Means for solving the problem]

[0006] An image display system according to the present disclosure includes an image correction device according to the present disclosure and a spherical display that displays an image corrected by the image correction device according to the present disclosure.

[0007] The image correction device disclosed herein comprises a coordinate conversion unit that converts pixel coordinates of an image into pixel coordinates of a spherical display, and a data buffer unit that stores pixel values ​​of the image, and outputs the pixel values ​​stored in the data buffer unit as pixel values ​​of converted coordinates after conversion by the coordinate conversion unit.

[0008] The image correction device of the present disclosure executes the image correction method of the present disclosure, which enables the circuit scale of at least one of the coordinate conversion unit and the data buffer unit to be reduced based on the fact that distortion in the image differs between near the center and near the periphery of the horizontal coordinate of the image.

[0009] The coordinate conversion unit corrects pixel coordinates of the image so that a horizontal line of the image coincides with a horizontal line of the spherical display. For example, the coordinate conversion unit may include a conversion map that defines a vertical coordinate change amount from the horizontal line of the image to the horizontal line of the spherical display for each horizontal coordinate included in the horizontal line. The conversion map may store horizontal coordinates corresponding to the vertical coordinate change amount.

[0010] The data buffer unit may include a plurality of buffers each having a storage capacity corresponding to a horizontal coordinate of the video. Addresses of the plurality of buffers may be associated with the horizontal coordinate of the video. In this case, the data buffer unit may have a terminal capable of simultaneously writing and reading pixel values ​​to and from addresses corresponding to the horizontal coordinate of the video.

[0011] The above disclosures can be combined as much as possible. [Effects of the Invention]

[0012] The present disclosure enables a device that matches the coordinates of an image with the coordinates of a spherical display to have a reduced circuit scale, thereby shortening the time required to display the image on the spherical display and reducing delays. [Brief explanation of the drawings]

[0013] [Figure 1] 1 shows an example of a video display system. [Figure 2] An example of a spherical display is shown. [Figure 3] 1 shows an example configuration of an image correction device according to the present disclosure. [Figure 4] 1 shows an example of creating a transformation map, where (a) shows a correction input map, and (b) shows an example of displaying the correction input map on a spherical display. [Figure 5] FIG. 10 is an explanatory diagram showing one mode of projecting a horizon onto a spherical display. [Figure 6] 1 shows an example of a horizon line projected onto a spherical display. [Figure 7] 1A and 1B are examples of input and output images of the image correction device of the present disclosure, where (a) shows the input image and (b) shows the output image. [Figure 8] 1 shows an example of the configuration of a data buffer unit. [Figure 9] 1 shows an example configuration of an image correction device according to the present disclosure. [Figure 10] 1 shows an example of a coordinate conversion unit. [Figure 11] 1 shows an example configuration of an image correction device according to the present disclosure. [Figure 12] 1 shows an example of a data buffer unit. [Figure 13] These are examples of display on a spherical display, where (a) shows the image before correction and (b) shows the image after correction. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0015] An example of a video display system is shown in Figure 1. A spherical display 91 projects an image onto the inner wall of a spherical dome, and an observer inside the dome watches the image projected onto the inner wall. A projector 92 that projects the image onto the inner wall of the dome can be placed, for example, above the observer's head.

[0016] 2 shows an example of a display on the spherical display 91. When projecting onto the inner wall surface of a dome, the following distortion occurs. (1) Spherical distortion: This distortion is caused by the dome's spherical shape. (2) Fisheye distortion: This is distortion caused by taking pictures with a fisheye camera. (3) Oblique projection distortion: This distortion is caused by projecting the image from above the observer's head.

[0017] (Image correction method) As can be seen from the display example in Fig. 2, the image needs to be corrected on the spherical display 91. As shown in Fig. 1, the image display system of the present disclosure includes a spherical display 91 and an image correction device 93 of the present disclosure. The image correction device 93 of the present disclosure executes the image correction method of the present disclosure and converts pixel coordinates of the image into pixel coordinates of the spherical display 91. The spherical display 91 displays the image corrected by the image correction device 93.

[0018] (Image correction method) 3 shows an example of the configuration of an image correction device according to the present disclosure. The image correction device according to this embodiment includes a coordinate conversion unit 84 and a data buffer unit 82. External memories 81 and 83 are connected to the image correction device according to this embodiment.

[0019] Coordinate conversion unit 84 converts pixel coordinates of the image into pixel coordinates of the spherical display. Data buffer unit 82 stores pixel values ​​of the image. The image correction device of this embodiment outputs the pixel values ​​stored in data buffer unit 82 as pixel values ​​of converted coordinates after conversion by coordinate conversion unit 84. Specifically, the image correction device of this embodiment executes the following procedure.

[0020] Step S111: Video data is stored in the external memory 81. At this time, the external memory 81 stores pixel values ​​at pixel coordinates included in one frame so that the pixel values ​​for each pixel can be read out. Step S112: The data buffer unit 82 stores, for each pixel, the pixel value stored in the external memory 81. The pixel value is an arbitrary value that indicates the color or brightness of a pixel, and can be, for example, RGB data that indicates the gradations of red, green, and blue. Step S113: The coordinate conversion unit 84 converts the pixel coordinates of each pixel into pixel coordinates for the spherical display. Here, in the present disclosure, the pixel coordinates converted by the coordinate conversion unit 84 may be referred to as converted coordinates. The converted coordinates are stored in the external memory 83. At this time, the external memory 83 stores video data that is compatible with the data format of the projector 92. Step S114: The projector 92 projects the video data stored in the external memory 83 onto the spherical display 91.

[0021] In step S113, coordinate conversion is performed for each pixel, which increases the processing time in proportion to the number of pixels on the display. Furthermore, because video data is read frame by frame, the processing in steps S112 to S114 becomes very complicated, resulting in a problem of long delays before output. Furthermore, for video having a frame of 1920 x 1080 pixels, the amount of data is large, so external memories 81 and 83 are used. When these external memories 81 and 83 are used, data is written to and read from the coordinate conversion unit 84, data buffer unit 82, and external memory 83, which places a heavy load on the write and read processing, thereby also resulting in a problem of long delays before output.

[0022] 2, the distortion in the image differs between the center and periphery of the horizontal coordinate of the image. Therefore, in the present disclosure, the circuit scale of at least one of the coordinate conversion unit 84 and the data buffer unit 82 can be reduced based on the difference in the magnitude of this distortion.

[0023] (First embodiment) In this embodiment, an example will be described in which the circuit scale of the coordinate conversion unit 84 can be reduced based on the fact that distortion in an image differs between near the center and near the periphery of the horizontal coordinate of the image.

[0024] As can be seen from the display example in Fig. 2, the distortion occurs symmetrically. Therefore, in this embodiment, pixel coordinate conversion is performed for each horizontal line, thereby making it possible to reduce the circuit scale of the coordinate conversion unit 84.

[0025] Specifically, in step S113, the pixel coordinates of the image are corrected so that the horizontal line in the image coincides with the horizontal line of the spherical display 91. This allows processing according to the number of pixels included in the horizontal line to be performed in a single process. The conversion map referenced by the coordinate conversion unit 84 at this time can be created, for example, by the following procedure.

[0026] Step S211: A correction input map indicating the coordinates of the spherical display 91 is displayed on the spherical display 91. The correction input map is a map indicating coordinates in an image, and as shown in FIG. 4(a), a map in which the horizontal coordinates are finer than the vertical coordinates can be used. The vertical coordinates may be one by one, but can be spaced, for example, at intervals of 12 pixels (hereinafter, pixels may be referred to as pixels [px]). The correction input map is displayed on the spherical display 91. As a result, an image distorted in the horizontal and vertical directions, as shown in FIG. 4(b), is displayed on the spherical display 91.

[0027] Step S212: With the correction input map displayed on the spherical display 91, a horizon is projected onto the spherical display 91 as shown in FIG. C The laser pointer is illuminated at the position of vertical coordinate Y from the point A, and the laser pointer is rotated horizontally so as to maintain the position of vertical coordinate Y.

[0028] Step S213: An example of the horizontal line projected in step S212 is shown as the dashed line L in FIG. HThe X coordinate that is shifted one line down on the displayed image based on the line the laser pointer passes through is recorded. Figure 6 shows an example where the Y coordinate shifts by 1px at 12px, by 3px at 24px, and by 7px at 36px. In this case, the distortion occurs symmetrically, so it is sufficient to record the distortion on either the left or right side relative to the front of the observer. This allows the transformation map size of the horizontal coordinate X to be reduced from 1920 to 960.

[0029] Step S214: Based on the obtained observation data, a vertical coordinate variation α is calculated for all pixel coordinates (X, Y) using an interpolation function. Here, the vertical coordinate variation α can be any value that can change the vertical coordinate Y, for example, 0≦α≦300.

[0030] By executing steps S211 to S214, a conversion map can be created that defines the vertical coordinate change amount α from the horizontal line of the video to the horizontal line of the spherical display for each horizontal coordinate included in the horizontal line. In this embodiment, the size of the conversion map is halved for the horizontal coordinate X. This reduces the load on the coordinate conversion unit 84 when it reads the data in step S113.

[0031] Here, as shown in FIG. 4(b), horizontal lines with similar vertical coordinates Y have approximately the same curve. Therefore, the conversion of pixel coordinates for each horizontal line in step S113 may be performed for multiple horizontal lines. For example, the conversion of pixel coordinates may be performed for every three horizontal lines. In this case, the average value of the three lines may be used as the converted pixel coordinate. In this way, by reducing the number of horizontal lines converted by the coordinate conversion unit 84, the conversion map size for the vertical coordinate Y can be reduced from 1080 to 360. Note that the average error from the original conversion map obtained by averaging the vertical coordinate change amount α for the three horizontal lines was approximately 0.4 pixels.

[0032] Furthermore, the vertical coordinate change amount α is larger at the periphery and smaller toward the center. Therefore, the coordinate conversion unit 84 can also read out X such that it becomes (α, Y). For example, the conversion map may store the horizontal coordinate corresponding to the vertical coordinate change amount α. By employing a coordinate conversion unit 84 that reads in this way, the conversion map size, which has already been reduced to 960 x 360, can be further reduced to 300 x 360. In other words, it can be reduced by 95% compared to the original 1920 x 1080 pixels.

[0033] (Second embodiment) In this embodiment, an example will be described in which the circuit scale of the data buffer unit 82 can be reduced based on the fact that distortion in an image differs between near the center and near the periphery of the horizontal coordinate of the image.

[0034] The input image shown in FIG. 7(a) is converted into the output image shown in FIG. 7(b) by the coordinate conversion unit 84. The input image is read out in accordance with the readout rules for video data, and the pixel P b1 After pixel P b2 is read out from the external memory 81 and stored in the data buffer unit 82. On the other hand, in the output image, pixel P a1 More pixels than a2 is output first. In this way, the time from when data is written to the data buffer unit 82 until it is read out is shorter in the center of the horizontal coordinate X than in the peripheral area.

[0035] That is, the pixel value of the center portion at horizontal coordinate X is read out immediately after being written to data buffer unit 82. For this reason, the storage capacity of the buffer that stores the pixel values ​​of the center portion at horizontal coordinate X may be small. On the other hand, the buffer that stores the pixel values ​​of the peripheral portion at horizontal coordinate X is read out after a time corresponding to α has elapsed since the pixel values ​​were written to data buffer unit 82. Therefore, data buffer unit 82 of this embodiment has a buffer size corresponding to α.

[0036] FIG. 8 shows an example of the configuration of the data buffer unit 82. The data buffer unit 82 includes N buffers 54_0 to 54_N, each having a storage capacity corresponding to the horizontal coordinate X of the video. For ease of understanding, the diagram shows an example where N=9. α is smallest at the center of the horizontal coordinate X and increases toward the periphery. Therefore, buffers 54_4 and 54_5 corresponding to the center of the horizontal coordinate X have the smallest storage capacity, and buffers 54_0 and 54_9 corresponding to the periphery of the horizontal coordinate X have larger storage capacities toward the periphery. By adopting this structure, it is possible to reduce the buffer size of the data buffer unit 82 by approximately 36%. According to this embodiment, the buffer size of the data buffer unit 82 can be reduced, eliminating the need for an external memory 81 for temporarily storing video data.

[0037] (Third embodiment) The data buffer unit 82 described in the second embodiment includes a plurality of buffers 54_0 to 54_N having different storage capacities. The address of each of the buffers 54_0 to 54_N corresponds to the horizontal coordinate of the video, and data is written to and read from each of the buffers 54_0 to 54_N by performing address control.

[0038] 9 shows an example of the configuration of an image correction device according to this embodiment. In this embodiment, image data is directly stored in a data buffer unit 82. At this time, the image data is stored together with pixel coordinates in buffers 54_0 to 54_N at addresses corresponding to the horizontal coordinates of the image data. When pixel coordinates (X, Y) are input, a coordinate conversion unit 84 outputs the input pixel coordinates (X, Y) and their vertical coordinate change amount α as converted coordinates.

[0039] When the external memory 83 acquires the pixel coordinates (X, Y) and the vertical coordinate change amount α from the coordinate conversion unit 84, it reads out the pixel values ​​stored in the buffers 54_0 to 54_N at the addresses corresponding to the pixel coordinates (X, Y) and stores them as pixel values ​​of the converted coordinates. This allows the external memory 83 to store the corrected pixel coordinates and pixel values ​​corresponding to the spherical display 91.

[0040] 10 shows an example of the configuration of the coordinate conversion unit 84. The coordinate conversion unit 84 includes an address calculation unit 41, a ROM (Read Only Memory) 42, a comparison unit 43, and a converted coordinate output unit 44.

[0041] ROM 42 stores a transformation map. Addresses in ROM 42 correspond to horizontal coordinates. Address calculation unit 41 calculates an address for accessing ROM 42 based on pixel coordinates (X, Y). When an address is specified by address calculation unit 41, ROM 42 outputs the transformation coordinates stored at the specified address. Comparison unit 43 compares the transformation coordinates output from ROM 42 with the input pixel coordinates. This obtains a vertical coordinate change amount α. Transformation coordinate output unit 44 outputs the input pixel coordinates (X, Y) and the vertical coordinate change amount α.

[0042] In this embodiment, the data buffer unit 82 includes a plurality of buffers corresponding to the horizontal coordinate X, and the external memory 83 reads pixel values ​​from the buffers in the data buffer unit 82 corresponding to the horizontal coordinate X. Therefore, the pixel values ​​of each pixel coordinate (X, Y) can be directly transferred from the data buffer unit 82 to the external memory 83.

[0043] (Fourth embodiment) An example of the configuration of the image correction device of this embodiment is shown in Fig. 11. In this embodiment, a data buffer unit 85 is provided instead of the data buffer unit 82 and external memory 83 of the third embodiment.

[0044] 12 shows an example of the configuration of the data buffer unit 85. The data buffer unit 85 includes a write address calculation unit 51, a read address calculation unit 52, a selection unit 53, and a buffer 54. The addresses of the buffers 54_0 to 54_N correspond to the horizontal coordinates of the video, as in the second embodiment.

[0045] The data buffer unit 85 has terminals 55I and 55O. The terminal 55I is an input terminal to which pixel values ​​of video data are input, and the terminal 55O is an output terminal that outputs pixel values ​​of transformed coordinates. The data buffer unit 85 includes the terminals 55I and 55O, and is capable of simultaneously writing to and reading from the buffers 54_0 to 54_N.

[0046] (When writing) When the data buffer unit 85 acquires the pixel values ​​of the video data, it executes the following process. The write address calculation unit 51 calculates a write address in the buffer 54 based on the pixel coordinates (X, Y) of the video data. The selector 53 selects a buffer from among the buffers 54_0 to 54_N according to the write address. The buffer 54 stores the pixel values ​​in the buffer selected by the selector 53 .

[0047] (when reading) When the data buffer unit 85 acquires the converted coordinates from the coordinate conversion unit 84, it executes the following process. The read address calculation unit 52 calculates a read address in the buffer 54 using the vertical coordinate change amount α from the coordinate conversion unit 84 . The selection unit 53 selects a buffer from among the buffers 54_0 to 54_N according to the read address. The buffer 54 reads out pixel values ​​from the buffer selected by the selector 53 .

[0048] (When reset) When the data buffer unit 85 receives the reset signal, it executes the following process. The write address calculation unit 51 calculates a reset address in the buffer 54 based on the pixel coordinates (X, Y) specified by the reset signal. The selection unit 53 selects a buffer from among the buffers 54_0 to 54_N according to the reset address. The buffer 54 stores the reset data in the buffer selected by the selector 53. As a result, the addresses that have already been read are reset.

[0049] (Other embodiments) The above-described embodiments can be combined. For example, the transformation map provided in the first embodiment may be used as the transformation map used in the second to fourth embodiments. The image correction device of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0050] As described above, the present disclosure employs a horizontal line transformation map and further makes the transformation map symmetrical, thereby reducing the circuit size and processing delay of the coordinate transformation unit. For example, if the transformation map size is 1920 x 1080, the transformation map size can be reduced to 300 x 360, a reduction of 95% compared to the original size. Furthermore, by adopting the configuration of the data buffer unit 82 shown in FIG. 8, the storage capacity of the data buffer unit 82 can be reduced by 36%. The memory usage of the entire circuit is reduced to 57%, making it possible to implement the circuit on a field programmable gate array (FPGA).

[0051] (Example) Verification was carried out using an actual system. The equipment used is as follows: FPGA evaluation board: Genesys2 (Digilent) Projector 92: CineBeam Laser 4K (LG) Fisheye lens: DCR-CF187PRO (manufactured by RAYNOX) Spherical display 91: inner diameter 800mm

[0052] Figure 13(a) shows the image projected onto the spherical display before correction, and Figure 13(b) shows the image projected onto the spherical display after correction. Comparing Figure 13(a) and Figure 13(b) shows that the distortion has been corrected. Furthermore, by adopting a horizontal line transformation map, the present disclosure can reduce the number of output buffers and input buffers, thereby reducing processing delays. [Explanation of symbols]

[0053] 41: Address calculation unit 42:ROM 43: Comparison section 44: Transformation coordinate output section 51: Write address calculation unit 52: Read address calculation unit 53: Selection section 54: Buffer 55I, 55O: Terminals 81, 83: External memory 82, 85: Data buffer section 84: Coordinate conversion section 91: Spherical display 92: Projector 93: Image correction device

Claims

1. a coordinate conversion unit that converts pixel coordinates of the image into pixel coordinates of the spherical display; a data buffer unit that stores pixel values ​​of the image for each pixel coordinate; Equipped with an image correction device that outputs pixel values ​​stored in the data buffer unit as pixel values ​​of transformed coordinates after transformation by the coordinate transformation unit, and reducing the circuit scale of at least one of the coordinate conversion unit and the data buffer unit based on the fact that the distortion in the image differs between near the center and near the periphery of the horizontal coordinate of the image. Image correction device.

2. the coordinate conversion unit corrects pixel coordinates of the image so that a horizontal line of the image coincides with a horizontal line of the spherical display. The image correction device according to claim 1 .

3. the coordinate conversion unit includes a conversion map that defines a vertical coordinate change amount from a horizontal line of an image to a horizontal line of a spherical display for each horizontal coordinate included in the horizontal line; The image correction device according to claim 2 .

4. the transformation map stores horizontal coordinates corresponding to the vertical coordinate changes; 4. The image correction device according to claim 3.

5. the data buffer unit includes a plurality of buffers each having a storage capacity according to a horizontal coordinate of the video; The image correction device according to claim 1 .

6. the addresses of the plurality of buffers are associated with horizontal coordinates of the video; the data buffer unit has a terminal capable of simultaneously writing and reading pixel values ​​at addresses corresponding to horizontal coordinates of the video; 6. The image correction device according to claim 5.

7. An image correction device according to any one of claims 1 to 6; a spherical display that displays the image corrected by the image correction device; A video display system comprising:

8. a coordinate conversion unit that converts pixel coordinates of the image into pixel coordinates of the spherical display; a data buffer unit for storing pixel values ​​of the image; Equipped with an image correction method for outputting pixel values ​​stored in the data buffer unit as pixel values ​​of transformed coordinates after transformation by the coordinate transformation unit, and reducing the circuit scale of at least one of the coordinate conversion unit and the data buffer unit based on the fact that the distortion in the image differs between near the center and near the periphery of the horizontal coordinate of the image. Image correction method.

Images