Image-processing method, image-processing device, and recording medium

Abstract

An image-processing method according to one embodiment of the present technology includes controlling a graphics-processing device to obtain position information that is in an orthogonal coordinate system and represents a user viewpoint with respect to at least one display panel, the user viewpoint being a prescribed distance away from a prescribed pixel of the display panel in a normal direction of the prescribed pixel. A luminance correction texture that includes a luminance value associated with the user viewpoint is obtained. A color conversion matrix is generated on the basis of the user viewpoint and the obtained luminance correction texture. The luminance and chromaticity of a rendered image are corrected on the basis of the generated color conversion matrix.

Description

Image processing method, image processing apparatus, and recording medium

[0001] This technology relates to an image processing method, an image processing apparatus, and a recording medium applicable to image correction.

[0002] Patent Document 1 discloses a playback device that performs image correction. This playback device detects the viewer's viewing angle and performs color correction, brightness correction, and motion correction of the image according to the detected viewing angle. As a result, the image is corrected to achieve optimal image quality according to the viewer's viewing angle, making it possible to display high-quality images.

[0003] Patent Document 2 discloses a liquid crystal display device that performs image correction. In this liquid crystal display device, the chromaticity point of the image is corrected based on the user's viewing angle and color change amount information. The color change amount information is a correspondence between the viewing angle and the corresponding amount of color change in the display light, expressed as a color difference. This makes it possible to suppress color changes caused by changes in the viewing angle.

[0004] Japanese Patent Publication No. 2006-276692 Japanese Patent Publication No. 2010-169868

[0005] In image correction technology, there is room for improvement in the calculation process of the user's field of view, and there is a need for technology that enables efficient calculation.

[0006] In light of the above circumstances, the objective of this technology is to provide an image processing method, an image processing apparatus, and a recording medium that enable efficient computational processing related to image correction.

[0007] To achieve the above objective, an image processing method according to one embodiment of the present technology includes controlling a graphics processing device to acquire position information of the user's viewpoint of a display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of at least one pixel of the display panel. A correction luminance texture including luminance values ​​associated with the user's viewpoint is acquired. A color transformation matrix is ​​generated based on the user's viewpoint and the acquired correction luminance texture. The luminance and chromaticity of the rendered image are corrected based on the generated color transformation matrix.

[0008] An image processing device according to one embodiment of this technology comprises a storage unit and a graphics processing device. The storage unit stores a correction luminance texture including luminance values ​​associated with the position information of the user's viewpoint of the display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of at least one pixel of the display panel. The graphics processing device is configured to generate a color conversion matrix based on the user's viewpoint and the correction luminance texture stored by the storage unit, and to correct the luminance and chromaticity of the rendered image based on the color conversion matrix.

[0009] A recording medium containing a program relating to one embodiment of this technology causes a computer system to execute a predetermined instruction. The predetermined instruction is to control a graphics processing unit to obtain position information of the user's viewpoint of a display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of at least one pixel of the display panel, to obtain a correction luminance texture including luminance values ​​associated with the user's viewpoint, to generate a color conversion matrix based on the user's viewpoint and the obtained correction luminance texture, and to correct the luminance and chromaticity of a rendered image based on the generated color conversion matrix.

[0010] It is a schematic diagram showing a configuration example of a display system according to an embodiment of the present technology. It is a block diagram showing a functional configuration example of the display system. It is a flowchart showing an example of a process for generating a texture. It is a schematic diagram showing the content of texture generation. It is a schematic diagram showing an example of defining the angle of the back panel. It is a diagram showing an example of a set of colorimetric luminance values in a table format. It is a plot diagram of luminance values. It is a schematic diagram showing an example of a texture is. It is a flowchart showing an example of a process for correcting an image. It is a flowchart showing an example of a process for determining a luminance reference. It is a schematic diagram showing an example of defining a range of viewing angles. It is a flowchart showing an example of a process for generating a color conversion matrix. It is a schematic diagram showing a display system of a comparative example. It is a schematic diagram showing a state in which pixel blur occurs. It is a schematic diagram showing an example of the light distribution characteristics of the display panel. It is a schematic diagram showing an example of a rectangular coordinate system is. It is a schematic diagram showing an example in which a panel is arranged at an inclination. It is a schematic diagram showing a state of conversion of a rectangular coordinate system. It is a schematic diagram showing a configuration example related to virtual production. It is a schematic diagram showing a processing order related to image integration. It is a block diagram showing an example of the hardware configuration of a computer capable of realizing an image processing apparatus.

[0011] Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[0012] [Display System] FIG. 1 is a schematic diagram showing a configuration example of a display system 1 according to an embodiment of the present technology. FIG. 1A shows a view of the display system 1 seen obliquely. FIG. 1B shows a view of the display system 1 seen from the front. The display system 1 includes a back panel 2 and a floor panel 3.

[0013] The back panel 2 is a display panel such as a liquid crystal or EL (Electro-Luminescence). The back panel 2 has a rectangular shape and is arranged, for example, parallel to the vertical direction on a wall surface or the like. It is not limited to this, and the back panel 2 may be arranged slightly obliquely with respect to the vertical direction. Other specific arrangements and shapes of the back panel 2 are not limited.

[0014] In this example, an image of a landscape such as a tree, clouds, or a road is displayed on the rear panel 2. The specific content of the image displayed on the rear panel 2 is not limited, and for example, an object such as a staircase may be displayed. Alternatively, a video may be displayed on the rear panel 2.

[0015] Similarly, the floor panel 3 is also a display panel such as a liquid crystal or EL. The floor panel 3 has a rectangular shape and is arranged, for example, on the floor of the space where the display system 1 exists. In this example, the lower side of the rear panel 2 and the back side of the floor panel 3 are in contact, and the two panels are arranged so as to form an angle of approximately 90°. Further, an image of a road is displayed so as to be continuous by the rear panel 2 and the floor panel 3. However, it is not limited to this, and an image expressing the color or pattern of the floor may be displayed by the floor panel 3.

[0016] The image displayed by the display system 1 is visually recognized by the user. For example, the user stands in front of the display system 1 and visually recognizes the images displayed on the rear panel 2 and the floor panel 3. Also, it is possible that the user visually recognizes the image from the diagonal side or the side.

[0017] It may be possible to simultaneously view the image and the sound by outputting the sound in accordance with the display of the image. Hereinafter, when the user is said to "view" the image, of course, the present technology is also applicable when no sound is output and only the visual recognition of the image is performed.

[0018] In the present embodiment, an image capable of stereoscopic vision (hereinafter, a stereoscopic image) is displayed on the rear panel 2 and the floor panel 3. In this example, an image of a person 4 (hereinafter, referred to as a person image 4) is displayed as the stereoscopic image. The user can view the person image 4 three-dimensionally, for example, by wearing special glasses and viewing the rear panel 2 and the floor panel 3.

[0019] As shown in FIGS. 1A and B, for the user, the person image 4 appears as if it exists in front of the rear panel 2.

[0020] The display of stereoscopic images allows users to view the images with a high degree of immersion. In addition, any image capable of stereoscopic viewing may be displayed, not limited to the human image 4. For example, objects such as a ball or an apple may be displayed. Alternatively, naked-eye stereoscopic displays may be used as the back panel 2 and the floor panel 3, allowing users to view stereoscopic images without wearing any special equipment.

[0021] The number of display panels included in display system 1 is not limited. For example, display system 1 may be composed of three or more display panels. In this case, stereoscopic images may be displayed by all of the display panels, or only some of the display panels may be used to display stereoscopic images. The arrangement of the display panels is also not limited; for example, display panels may be arranged on the ceiling.

[0022] Figure 2 is a block diagram showing an example of the functional configuration of the display system 1. The display system 1 further includes a speaker 7, a colorimeter 8, a camera 9, a distance sensor 10, and an image processing device 11.

[0023] Speaker 7 can output various types of sound. For example, in conjunction with the video displayed on the rear panel 2 and floor panel 3, speaker 7 can output background music, sound effects, and other audio. This allows users to view the video with a greater sense of immersion. The specific configuration of speaker 7 is not limited.

[0024] The colorimeter 8 is a device that measures the color of images displayed on the rear panel 2 and the floor panel 3. For example, a luminance meter is used as the colorimeter 8 to perform color measurement. The specific details of color measurement by the colorimeter 8 will be described later.

[0025] Camera 9 captures images of the user and the colorimeter 8. In this embodiment, for example, a stereo camera is used as camera 9, and images and depth information of the user and the colorimeter 8 are acquired. Note that when the user is being captured, the colorimeter 8 may be used instead of camera 9.

[0026] The distance measuring sensor 10 acquires depth information from the user and the colorimeter 8. As the distance measuring sensor 10, for example, a ToF (Time of Flight) sensor or a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) sensor may be used.

[0027] In this embodiment, the positions of the user and the colorimeter 8 are calculated based on images and depth information acquired by the camera 9 and the distance sensor 10. Alternatively, depth information may be acquired by only one of the cameras (camera 9 or distance sensor 10). Specific details regarding position calculation will be described later.

[0028] The image processing device 11 is implemented by any computer, such as a PC (Personal Computer). Alternatively, the image processing device 11 may be built into the rear panel 2 or the floor panel 3, and these may be configured as a single unit. The image processing device 11 has a display unit 12, an operation unit 13, a communication unit 14, a storage unit 15, and a controller 16. These blocks are interconnected via a bus 17. Instead of the bus 17, each block may be connected using a communication network or a non-standardized proprietary communication method.

[0029] The display unit 12 is a display device that uses, for example, liquid crystal or electroluminescent material, and displays various images indicating the status of the image processing device 11, a GUI (Graphical User Interface) for settings, etc.

[0030] The operation unit 13 may be, for example, a keyboard, a pointing device, a touch panel, or other operating device. If the operation unit 13 includes a touch panel, the touch panel may be integrated with the display unit 12.

[0031] The communication unit 14 is a communication module for communicating with other devices via a network such as a LAN (Local Area Network) or WAN (Wide Area Network). It may be equipped with a wireless LAN module such as Wi-Fi or a communication module for short-range wireless communication such as Bluetooth®. Communication equipment such as a modem or router may also be used.

[0032] The storage unit 15 is a storage device such as non-volatile memory, and for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive) may be used. In addition, any non-transient storage medium that can be read by a computer may be used. The storage unit 15 stores a control program for controlling the overall operation of the image processing device 1. The method of installing the control program in the image processing device 1 is not limited. For example, installation may be performed via various recording media, or the program may be installed via the internet or the like.

[0033] The controller 16 has a GPU (Graphics Processing Unit). In this embodiment, for example, the image processing method according to this technology is executed by the GPU loading a program according to this technology, which is pre-recorded in ROM or the like, into RAM and executing it. In addition, the controller 16 has hardware necessary for the configuration of a computer, such as a processor such as a CPU or DSP, memory such as ROM or RAM, and storage devices such as an HDD. As the controller 16, for example, a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or other devices such as an ASIC (Application Specific Integrated Circuit) may be used.

[0034] In this embodiment, the CPU of the controller 16 executes a program related to this technology (for example, an application program), thereby realizing the following functional blocks: an image generation unit 20, a display control unit 21, a colorimeter control unit 22, a viewing position detection unit 23, a pixel position calculation unit 24, a field of view angle calculation unit 25, a texture generation unit 26, a texture acquisition unit 27, a brightness reference determination unit 28, and a brightness color correction unit 29. Dedicated hardware such as ICs (integrated circuits) may be used as appropriate to realize each functional block.

[0035] The image generation unit 20 generates images to be displayed on the rear panel 2. In this embodiment, the image generation unit 20 generates images of a landscape, a road, and a person 4. The image generation unit 20 also generates images that are entirely red, green, blue, or white, which are used in the color measurement described later. The specific method of image generation is not limited; for example, the image generation unit 20 can generate an image by reading an image previously stored in the storage unit 15. The specific content of the generated images is also not limited.

[0036] Similarly, the images displayed by the floor panel 3 are also generated by the image generation unit 20. The same processes as those performed on the rear panel 2 are also performed on the floor panel 3, but the explanation of the floor panel 3 may be omitted in some cases.

[0037] In this embodiment, the image generated by the image generation unit 20 is corrected by the brightness and color correction unit 29, and the corrected image is displayed on the rear panel 2.

[0038] The display control unit 21 controls the display of images on the rear panel 2. Specifically, the display control unit 21 acquires an image from the image generation unit 20 and controls the display so that the image is displayed on the rear panel 2 at a predetermined timing. The display control unit 21 can also display stereoscopic images on the rear panel 2. When displaying stereoscopic images, time-division display control may be used.

[0039] The colorimeter control unit 22 controls the image acquisition by the colorimeter 8. The colorimeter control unit 22 can also control the position and imaging direction of the colorimeter 8. For example, a mechanism for changing the position and orientation of the colorimeter 8 can be appropriately provided, and the operation of this mechanism can be controlled by the colorimeter control unit 22, thereby indirectly controlling the position and imaging direction of the colorimeter 8. As a result, it becomes possible to capture the image displayed on the rear panel 2 from various angles. In addition, any other operation of the colorimeter 8 may be controlled.

[0040] The viewing position detection unit 23 acquires the viewing position of the user viewing the display panel. Specifically, the viewing position detection unit 23 acquires the position of the user's eyes as the viewing position. In this embodiment, a three-dimensional world coordinate system (X, Y, Z) is defined in real space, and the three-dimensional coordinates of the user's eyes in this coordinate system are acquired. In the world coordinate system, the X, Y, and Z axes are orthogonal to each other, but their specific orientations are not limited. The position of the origin of the world coordinate system is also not limited. The world coordinate system is sometimes referred to as the global coordinate system.

[0041] In this example, the camera 9 and distance sensor 10 acquire the user's image and depth information, and the viewing position detection unit 23 calculates the coordinates of the user's eyes based on the acquired image and depth information. Similarly, the viewing position detection unit 23 can also acquire the coordinates of the colorimeter 8. The method of acquiring the viewing position by the viewing position detection unit 23 is not limited. In addition, other information such as the direction of the gaze may also be acquired.

[0042] The pixel position calculation unit 24 calculates the position of a pixel on the back panel 2. Specifically, the pixel position calculation unit 24 calculates the three-dimensional coordinates of a predetermined pixel on the back panel 2. These three-dimensional coordinates are also calculated in the world coordinate system.

[0043] For example, information such as the coordinates of a pixel in the 100th row and 200th column of the rear panel 2 being (10, 20, 30) is pre-stored in the storage unit 15, and the pixel position is calculated by reading this information from the storage unit 15. Of course, these values ​​are just examples, and the specific values ​​are not limited to what they may be.

[0044] The field of view calculation unit 25 calculates the user's viewing angle based on the viewing position acquired by the viewing position detection unit 23. Specifically, it calculates the user's viewing angle for a predetermined pixel on the back panel 2. First, the field of view calculation unit 25 acquires the coordinates of the user's eyes, calculated by the viewing position detection unit 23, and the coordinates of a predetermined pixel, calculated by the pixel position calculation unit 24. Furthermore, based on the two acquired coordinates, the user's viewing angle for the predetermined pixel is calculated. The viewing angle can be calculated, for example, by calculating the difference between the two coordinates, or by using trigonometric ratios. Of course, the specific method for calculating the viewing angle is not limited.

[0045] Furthermore, the viewing angle calculation unit 25 can also calculate the angle of the colorimeter 8 relative to the rear panel 2 (color measurement angle) when the colorimeter 8 measures the color of the rear panel 2. The calculation of the color measurement angle is performed in the same way as the calculation of the user's viewing angle. Note that since the operation of the colorimeter 8 is controlled by the colorimeter control unit 22, the coordinates and color measurement angle of the colorimeter 8 may be known, and it may be unnecessary to perform processes such as acquiring the image and depth information of the colorimeter 8, calculating the coordinates, and calculating the color measurement angle. In such cases, the known color measurement angle may be used as is.

[0046] The texture generation unit 26 generates textures related to the correction of the image displayed by the rear panel 2. The specific details of texture generation will be described later.

[0047] The texture acquisition unit 27 acquires the texture. For example, a texture generated by the texture generation unit 26 is stored in the storage unit 15, and the texture acquisition unit 27 acquires the texture from the storage unit 15.

[0048] The luminance reference determination unit 28 determines a luminance reference that serves as the basis for correcting the luminance values ​​of the rendered image (image before correction). The luminance color correction unit 29 corrects the rendered image and generates a corrected image. The specific details of determining the luminance reference and generating the corrected image will be described later.

[0049] [Texture Generation] Figure 3 is a flowchart showing an example of the texture generation process. In this embodiment, when the display system 1 is used by a user, the corrected image corrected by the brightness and color correction unit 29 is displayed on the back panel 2. Then, as a pre-processing step before the display system 1 is used, a process is performed to generate the texture used to generate the corrected image in advance. Figure 3 is a flowchart related to this pre-processing.

[0050] The tilt of the rear panel 2 is set to a predetermined angle (step 101). Figure 4 is a schematic diagram showing the process of texture generation. As shown in Figure 4A, the angle of the rear panel 2 is set so that the color measurement angle of the rear panel 2 with respect to the colorimeter 8 is a predetermined angle. For example, the angle may be set by tilting the rear panel 2 manually, or a mechanism for adjusting the tilt of the rear panel 2 may be provided in advance. In addition, the coordinates of the colorimeter 8 are acquired by the viewing position detection unit 23 and the color measurement angle is calculated by the viewing angle calculation unit 25 as needed.

[0051] Figure 5 is a schematic diagram showing an example of the definition of the angle of the back panel 2. The back panel 2 has an array of elements in which a red LED (Light Emitting Diode) (R), a green LED (G), and a blue LED (B) are arranged in that order within a single pixel. As shown in Figure 5, in this example, the tilt of the back panel 2 when it is tilted around the axis of the direction in which the RGB LEDs are arranged (the up and down direction in Figure 5) is defined as the tilt in the θ direction. Also, when the colorimeter 8 photographs the back panel 2 from the front (from the front side in Figure 5), the tilt when R is tilted towards the back and B is tilted towards the front is defined as the tilt on the positive side of the φ direction. Figure 4A schematically illustrates a state in which the back panel 2 is tilted in only one direction, but in reality the back panel 2 can be tilted in both the θ and φ directions.

[0052] For example, θ ranges from 0° to 80°, and φ ranges from -80° to 80°, with angles set in 10° increments. In this case, there are nine angle settings in the θ direction: 0°, 10°, 20°, ..., 80°. Similarly, there are 17 angle settings in the φ direction: -80°, -70°, ..., 80°. In other words, combining the two directions, there are 153 different angle settings.

[0053] In this example, symmetry in the θ direction is assumed, and the angle is set in the range of 0° to 80°, rather than in the range of -80° to 80°. This reduces the number of color measurements, enabling more efficient color measurement. Of course, angle settings can be made for all possible angles without assuming symmetry. Furthermore, the angle ranges of θ and φ, as well as the grid (width of the angle setting), are not limited. Also, the method of defining the tilt of the back panel 2 is not limited to the above. For example, θ and φ can be defined after considering the specifications of the back panel 2, such as the arrangement of LED elements.

[0054] As shown in Figure 4B, the angle setting may be performed by changing the imaging direction of the colorimeter 8 while the rear panel 2 is fixed. In this case, the position and orientation of the colorimeter 8 are controlled by the colorimeter control unit 22. Alternatively, the angles of both the rear panel 2 and the colorimeter 8 may be changed. Below, an example of processing when the angle is set to (θ = 0°, φ = -80°) in step 101 will be described.

[0055] A color image is displayed on the rear panel 2 (step 102). In this embodiment, a red image, a green image, a blue image, and a white image are displayed across the entire surface of the rear panel 2. Specifically, the image is displayed with R, G, B, and W at their maximum emission levels, i.e., for an 8-bit display panel, the image is displayed with the maximum value of the 8-bit value, "255". For example, for red, the image is displayed with (R, G, B) = (255, 0, 0). In this example, it is assumed that there are no differences in light distribution characteristics within the rear panel 2, and that there are no differences due to color unevenness, etc. (uniformity is sufficiently good). The following describes the process when a red image is displayed. In this case, a red image is generated by the image generation unit 20, and the display control unit 21 controls the display of the red image on the rear panel 2.

[0056] Furthermore, images of other colors, such as brown images, may be displayed as colorimetric images. In addition, any other colorimetric image may be displayed to the extent that this technology can be implemented.

[0057] Image color measurement is performed (step 103). In this embodiment, the colorimeter 8 measures the color of the red image displayed on the rear panel 2. After processing up to step 102, the rear panel 2 is tilted to (θ = 0°, φ = -80°) and the red image is displayed across the entire surface. In this state, the colorimeter control unit 22 controls the imaging by the colorimeter 8 and measures the color of the red image.

[0058] In this embodiment, the texture generation unit 26 generates colorimetric luminance values, which are the luminance values ​​of the colorimetric image for each colorimetric angle relative to the rear panel 2. Specifically, when color measurement is performed by the colorimetric device 8, the texture generation unit 26 obtains luminance values ​​from the colorimetric device 8. In addition, the colorimetric angle "θ = 0°, φ = -80°" is obtained from the viewing angle calculation unit 25. Furthermore, information that the colorimetric image was a "red image" is obtained from the image generation unit 20.

[0059] The texture generation unit 26 generates a colorimetric luminance value by associating the three acquired pieces of information (colorimetric angle, colorimetric image, and luminance value). In this example, the colorimetric luminance value is generated by associating the colorimetric angle "θ = 0°, φ = -80°", the colorimetric image "red image", and the luminance value.

[0060] The colorimetric luminance values ​​obtained by color measurement are stored (step 104). In this example, the colorimetric luminance values ​​are stored in the storage unit 15.

[0061] It is determined whether or not color measurements have been taken for all measurement angles and all colors (step 105). If color measurements have not yet been taken for any of the measurement angles (θ, φ) and colors (red, green, blue, white) (No. in step 105), the angle setting of the rear panel 2 is performed again in step 101.

[0062] For example, the colorimetric angles and colorimetric images are set in a lexicographical order, such as (θ=0°, φ=-80°), green, (θ=0°, φ=-80°), blue, (θ=0°, φ=-80°), white, (θ=0°, φ=-70°), red, ..., (θ=0°, φ=80°), white, (θ=10°, φ=-80°), red, ..., (θ=80°, φ=80°), white, and the colorimetric measurement is performed. Of course, this order is just an example, and the colorimetric measurement may be performed in any order. For example, the colorimetric measurement may be performed in the order of red at all colorimetric angles, then blue at all colorimetric angles, and so on.

[0063] The colorimetric luminance values ​​obtained from each color measurement are stored in the storage unit 15 each time in step 104. These colorimetric luminance values ​​are stored in the storage unit 15 in a table format, for example. Figure 6 shows an example of a table format for the colorimetric luminance values.

[0064] As shown in Figure 6, a table-formatted set of colorimetric luminance values ​​is stored in the storage unit 15, with the colorimetric image (display color) in the left column, the colorimetric angle in the middle column, and the luminance value in the right column. The colorimetric image is, for example, information where the display color is represented by three values ​​(R, G, B). For example, a red image is stored as three values ​​such as (1.0, 0.0, 0.0). The colorimetric angle is, for example, (θ ０= 0°, φ ０ It is stored as binary information such as (= -80°). The luminance value is, for example, "LumR （０，０） The information is saved as "=500". Note that there are no restrictions on the units used for the brightness value.

[0065] In this way, a table-formatted set of colorimetric luminance values ​​is stored, with one colorimetric luminance value corresponding to each row. In this example, there are four colorimetric images, nine colorimetric angles θ, and seventeen colorimetric angles φ, so 612 colorimetric luminance values ​​are stored, which is the product of these. In other words, the number of rows in the table is also 612.

[0066] The format of the information stored in the table is not limited. Furthermore, the format of the stored information is not limited to a table format; any format of information, including colorimetric luminance values, may be stored.

[0067] Furthermore, interpolation may be used to interpolate the measured colorimetric luminance values. For example, if two measured colorimetric luminance values ​​are obtained, "θ = 0°, φ = -80°", "red", "luminance value 500" and "θ = 0°, φ = -70°", "red", "luminance value 700", interpolation will generate a new measured colorimetric luminance value, "θ = 0°, φ = -75°", "red", "luminance value 600". This generates measured colorimetric luminance values ​​at 5° intervals, making it possible to obtain many measured colorimetric luminance values ​​with fewer measurements. Note that although the method of taking the average of two values ​​was used above, interpolation may be performed using any other interpolation method.

[0068] Figure 7 is a plot of luminance values. Figure 7A shows a three-dimensional plot of luminance values. In Figure 7A, the θ axis is taken in the depth direction, the φ axis is taken in the left-right direction, and the luminance values ​​at the colorimetric angle (θ, φ) are taken in the up-down direction. In this example, the luminance values ​​are distributed in a double-humped shape, with the luminance values ​​being high in parts of the positive and negative sides of φ (right and left sides of Figure 7A) and low around φ = 0°.

[0069] Figure 7B shows a two-dimensional contour line representation of luminance values. The upper left figure shows the distribution of luminance values ​​(red luminance values) in a red image, with θ on the vertical axis and φ on the horizontal axis. The luminance values ​​at the colorimetric angle (θ, φ) are shown in shades of gray, with higher luminance values ​​being closer to black and lower luminance values ​​being closer to white. In this example, the distribution of red luminance values ​​is such that the red luminance values ​​are generally high on the positive and negative sides of φ, and low around φ = 0°. Similarly, the upper right of Figure 7B shows the distribution of green luminance, the lower left shows blue luminance, and the lower right shows white luminance.

[0070] The luminance value distribution shown in Figure 7 is merely an example, and various distributions are possible depending on the light distribution characteristics of the display panel. The application of this technology is not limited to cases where the luminance value distribution is as shown in Figure 7.

[0071] If color measurement is completed for all measurement angles and colors (Yes in step 105), the maximum value of the brightness value for each color is extracted (step 106). Specifically, the texture generation unit 26 first extracts the highest brightness value from the 153 brightness values ​​(9 types of θ, 17 types of φ) in the red image as the maximum brightness value. Similarly, the maximum brightness values ​​for green brightness, blue brightness, and white brightness are also extracted.

[0072] The calculation is performed to divide each brightness value by the maximum brightness value (step 107). For example, "red image" "θ ０ = 0°, φ ０ The luminance value at "-90°" is "LumR （０，０） If the value is 500, and the maximum red luminance value calculated in step 106 is 1000, the texture generation unit 26 performs an operation to divide the luminance value by the maximum luminance value, generating a normalized luminance value of "0.5". The normalized luminance value will be a value in the range of 0 to 1. Note that the above luminance value and maximum luminance value are merely examples, and in reality, they can take on various values.

[0073] Similarly, calculations are performed on other colorimetric angles for the red image, and on the luminance values ​​in images of other colors, generating normalized luminance values ​​for each color and each colorimetric angle. Hereafter, the normalized luminance values ​​may be simply referred to as luminance values ​​without distinction.

[0074] A texture is generated (step 108). In this embodiment, the texture generation unit 26 generates a texture 32 that includes each of the measured colorimetric luminance values. Figure 8 is a schematic diagram showing an example of the texture 32.

[0075] A texture is a two-dimensional arrangement of pixels, i.e., an image. A typical texture consists of up to four channels, for example, R (red), G (green), B (blue), and A (opacity). In this embodiment, as shown in Figure 8A, the red luminance is stored in the R channel. Similarly, the green luminance is stored in the G channel, and the blue luminance is stored in the B channel. The white luminance is stored in the A (opacity) channel.

[0076] Furthermore, the luminance value is stored in each pixel of the texture 32 such that the colorimetric angle θ corresponds to the vertical direction of the texture 32 and φ corresponds to the horizontal direction of the texture 32. For example, the luminance value for "θ = 0°, φ = -80°" is stored in the bottom left pixel of the texture 32. Also, the luminance value for "θ = 40°, φ = 0°" is stored in the center pixel of the texture 32. In this way, the texture 32 becomes an image that contains all the information of the colorimetric luminance values ​​on the back panel 2, with the colorimetric angle corresponding to the pixel position, the color of the colorimetric image corresponding to the channel type, and the luminance value stored for each channel of each pixel.

[0077] Figure 8B shows an example of an actual image of texture 32. In this figure, texture 32 is shown in grayscale, but in reality, texture 32 is a color image. Specifically, the top edge of texture 32 is dark pink, the left edge is red, the bottom edge is light pink, and the right edge is blue.

[0078] Thus, the color of each pixel in texture 32 corresponds to the luminance values ​​held in each channel of that pixel. For example, if the red luminance held in a pixel's channel is relatively high, and the green and blue luminances are relatively low, that pixel will be a pixel with a color close to red. Also, the opacity of a pixel corresponds to the white luminance value held in the A channel of that pixel.

[0079] In this example, symmetry in the θ direction is assumed, so the texture 32 is an image that is shorter in the vertical direction (θ axis) and longer in the horizontal direction (φ axis). However, if symmetry is not assumed, the texture 32 could be a square image, etc. Also, the texture 32 shown in Figure 8B is just one example, and it can take on various color distributions depending on the brightness value.

[0080] The generated texture 32 is stored (step 109). Specifically, the GPU of the controller 16 is controlled and the texture 32 is stored in the storage unit 15.

[0081] [Image Correction] Figure 9 is a flowchart showing an example of the image correction process. The image generation unit 20 generates the pre-correction image that will be used as the basis for correction (step 201). For example, the pre-correction image stored in the storage unit 15 is acquired by the image generation unit 20.

[0082] The user's viewing position is acquired (step 202). The viewing position detection unit 23 acquires the three-dimensional coordinates of the user's eyes.

[0083] The brightness standard is determined (step 203). The following series of processes in steps 203 to 205 are performed for each pixel that makes up the uncorrected image. For example, the series of processes is performed for the pixels in the first row and first column of the uncorrected image, then for the pixels in the first row and second column, and so on, until all pixels are processed in order. The order of processing is not limited and the processes may be performed in any order.

[0084] The following describes an example where processing is performed on pixels in a 1x1 grid. Figure 10 is a flowchart showing an example of the process for determining the brightness standard. In step 203, the series of processes shown in Figure 10 are executed.

[0085] The range of the user's viewing angle is calculated (step 301). Figure 11 is a schematic diagram showing an example of the definition of the viewing angle range. Figure 11A schematically shows a state in which the user is standing in front of the center of the back panel 2 and viewing the back panel 2. Figure 11B schematically shows a state in which the user is standing to the right of the back panel 2 and viewing it.

[0086] The viewing angle range refers to the range of viewing angles that each pixel on the back panel 2 can take. For example, in the example in Figure 11A, the viewing angle of the leftmost pixel is 45°. The viewing angle of the rightmost pixel is -45°. Also, for example, the viewing angle of the central pixel is 0°. The viewing angles of the other pixels also depend on the position of the pixel, but the viewing angle of each pixel will take a value in the range of -45° to 45°. Conversely, no matter which pixel on the back panel 2 the user views at the position in Figure 11A, the viewing angle will never be outside the range of -45° to 45° (for example, 80°). Therefore, in the example in Figure 11A, the viewing angle range is -45° to 45°. Note that in this example, symmetry in the θ direction is assumed, so the viewing angle range is calculated to be 0° to 45°.

[0087] Similarly, in the example shown in Figure 11B, the viewing angle range is 10° to 70°. Since the user can view the rear panel 2 while moving, the viewing angle can constantly change, but the viewing angle range is calculated as an angle range dependent on the viewing angle at that time.

[0088] In Figures 11A and 11B, the range of the viewing angle is explained in one dimension for simplicity, but in reality, the angle range is calculated for a two-dimensional viewing angle (θ, φ), as shown in Figure 11C. The range of the viewing angle can also be called the field of view.

[0089] In step 301, the pixel position calculation unit 24 first calculates the coordinates of all pixels. For example, the memory unit 15 has the coordinates of each pixel stored in advance, and the coordinate calculation is performed by reading out the stored coordinates.

[0090] Next, the field of view angle calculation unit 25 calculates the viewing angle for each pixel based on the coordinates of each pixel calculated by the pixel position calculation unit 24 and the coordinates of the user's eyes obtained by the viewing position detection unit 23 in step 202. For example, the viewing angle (θ = 40°, 40°) is calculated based on a pixel "1 row, 1 column, coordinate (10, 10, 10)" and the eye coordinate (30, 30, 30), the viewing angle (θ = 40°, 41°) is calculated based on a pixel "1 row, 2 columns, coordinate (10, 20, 30)" and the eye coordinate (30, 30, 30), and so on, and so on, are executed in order to calculate the viewing angle for all pixels. Note that the above values ​​are merely examples, and the specific processing order is not limited. Furthermore, the field of view angle calculation unit 25 calculates the range of the viewing angle based on the viewing angles for all pixels.

[0091] The minimum white luminance is obtained (step 302). First, the texture acquisition unit 27 acquires the texture 32 stored in the storage unit 15. Next, the luminance reference determination unit 28 acquires each white luminance within the viewing angle range calculated by the viewing angle calculation unit 25. Since the texture 32 holds the white luminance for each viewing angle, the luminance reference determination unit 28 can acquire each white luminance within the viewing angle range from the texture 32. Furthermore, the lowest white luminance value among them is calculated as the minimum white luminance.

[0092] Front white luminance is acquired (step 303). The luminance reference determination unit 28 acquires the white luminance at an angle (θ = 0°, φ = 0°) from the texture 32 as the front white luminance.

[0093] The color transformation matrix for the front view is calculated (step 304). Specifically, the luminance reference determination unit 28 calculates the color transformation matrix at an angle (θ = 0°, φ = 0°). The method for calculating the color transformation matrix will be explained below without limiting the angle. Calculating the color transformation matrix for the front view corresponds to performing the following calculation at an angle (θ = 0°, φ = 0°).

[0094] First, the tristimulus values ​​shown in red are calculated using the following formula. X Ｒ , Y Ｒ ZＲ : Trichromatic value of red, x Ｒ , y Ｒ : Chromaticity of red, R max : Maximum value of red luminance, r: Red luminance value

[0095] When the back panel 2 is an LED, since the changes in chromaticity x Ｒ , y Ｒ with respect to the viewing angle are relatively small, in this embodiment, chromaticity x Ｒ , y Ｒ are treated as constants. Also, R max is the maximum red luminance value calculated for normalization in step 106 of the preprocessing. r is the red luminance value at a predetermined angle where the calculation is performed and is obtained from the texture 32.

[0096] Similarly, the trichromatic values of green and blue are calculated. Next, the color conversion matrix is defined by the following formula. [RGB2XYZ]: Color conversion matrix, X Ｒ , Y Ｒ , Z Ｒ : Trichromatic value of red, X Ｇ , Y Ｇ , Z Ｇ : Trichromatic value of green, X Ｂ , Y Ｂ , Z Ｂ : Trichromatic value of blue

[0097] The color conversion matrix [RGB2XYZ] calculated in this way depends on the angles (θ, φ). In step 304, the above calculation is executed at the angles (θ = 0°, φ = 0°), and the color conversion matrix at (θ = 0°, φ = 0°), that is, the front color conversion matrix, is calculated. Since the front white luminance and the front color conversion matrix do not depend on the viewing position of the user, the acquisition of the front white luminance and the calculation of the front color conversion matrix may be performed in advance at a stage before step 202 (acquisition of the viewing position).

[0098] Next, the luminance reference matrix is calculated by the luminance reference determination unit 28 using the following formula (step 305). [RGB2XYZ base : Luminance reference matrix, Lum base : Minimum white luminance, Lum front:Front white brightness [RGB2XYZ front ]: Front color transformation matrix

[0099] Thus, the luminance reference matrix is ​​obtained by scaling the front color transformation matrix by multiplying it by the ratio of the minimum white luminance obtained in steps 302 and 303 to the front white luminance.

[0100] A color transformation matrix is ​​generated (step 204). Figure 12 is a flowchart showing an example of the process for generating a color transformation matrix. In step 204, the series of processes shown in Figure 12 are executed.

[0101] The user's viewing angle is calculated (step 401). The viewing angle for a pixel in a 1x1 grid is calculated using the same process as for calculating the range of the viewing angle in step 301.

[0102] A color transformation matrix is ​​calculated (step 402). Based on the viewing angle for a 1x1 pixel, the color transformation matrix for a 1x1 pixel is calculated using the same process as for calculating the front color transformation matrix in step 304.

[0103] In other words, in steps 401 and 402, the luminance values ​​of each color corresponding to the viewing angle are obtained from the texture 32, and a color conversion matrix is ​​generated based on the obtained luminance values. Thus, the texture 32 becomes an image containing luminance values ​​associated with the user's viewing angle. The color conversion matrix is ​​generated based on the viewing angle and the stored texture 32.

[0104] In this example, step 203 (determining the luminance criterion) is performed before step 204 (generating the color transformation matrix), but the generation of the color transformation matrix may be performed before determining the luminance criterion.

[0105] Pixel correction is performed (step 205). In this embodiment, the brightness and chromaticity of the uncorrected image are corrected based on the color conversion matrix. Specifically, the brightness and color correction unit 29 performs the following calculations to first generate corrected pixels. R', G', B': RGB values ​​of the corrected pixels [RGB2XYZ]: Color transformation matrix [RGB2XYZ] base]: Luminance reference matrix R', G', B': RGB values ​​of pixels before correction

[0106] If no correction has been performed on any pixel (No. in step 206), the processes in steps 203 to 205 are repeated on the uncorrected pixels. For example, once the correction of the pixels in row 1, column 1 is complete, the correction is then performed in the order of row 1, column 2, row 1, column 3, ..., row 2, column 1, ... and so on. The order of correction is not limited.

[0107] If correction is performed on all pixels (Yes in step 206), a corrected image is generated (step 207). Specifically, the brightness and color correction unit 29 generates a corrected image by integrating each corrected pixel. The corrected image may be stored in the storage unit 15.

[0108] The corrected image is displayed (step 208). Specifically, the display control unit 21 controls the display, and the corrected image is displayed on the rear panel 2. It is determined whether or not to terminate the display by the display system 1 (step 209). For example, the display is terminated when the final frame of the video is reached or when the user performs an action to end viewing (such as pressing the end button) (Yes in step 209). If the display is not terminated (No in step 209), the process returns to step 201, and the image for the next frame is generated, and the image correction and display are performed in the same manner. In this way, the series of processes in steps 201 to 209 are performed for each frame in which an image is displayed, but the specific frame rate is not limited.

[0109] In this embodiment, the storage of the texture 32 by the storage unit 15 (processing in step 109) and the generation of the color conversion matrix (processing in steps 203 and 204) are performed under the control of the GPU. That is, processing that handles the texture 32, such as obtaining luminance values ​​from the texture 32 based on the viewing angle and generating a color conversion matrix based on the obtained luminance values, is performed by the GPU. Note that the texture acquisition process in step 302 may be performed by the GPU, or it may be performed by the CPU and then transferred to the GPU.

[0110] Furthermore, the pixel correction processing in steps 203 to 205 may be performed in parallel rather than sequentially for each pixel. Since GPUs have the advantage of being able to perform such parallel processing at high speed, this makes it possible to achieve even faster processing.

[0111] The color measurement process (steps 101-105) is performed, for example, by the CPU, and the pixel correction process (steps 205-208) is performed, for example, by the GPU. Of course, the color measurement process may also be performed by the GPU.

[0112] In this embodiment, the range of possible values ​​for the viewing angle is calculated. Specifically, in step 301, angle ranges such as 0° to 45° or 10° to 70° are calculated. Furthermore, from the texture 32, a colorimetric luminance value is obtained at any colorimetric angle within the same colorimetric angle range as the calculated viewing angle range. Specifically, in step 302, the lowest white luminance is obtained within the angle ranges such as 0° to 45° or 10° to 70°.

[0113] Furthermore, based on the acquired colorimetric luminance values, a luminance standard is determined that serves as the basis for correcting the luminance values ​​of the uncorrected image. Specifically, a luminance standard matrix is ​​calculated in step 305. The calculation of the luminance standard matrix corresponds to the determination of the luminance standard in this technology. In this example, the minimum white luminance was acquired for the calculation of the luminance standard matrix, but luminance values ​​may be acquired using other criteria.

[0114] The range of possible viewing angles may not be calculated, and the lowest white brightness may be obtained from all angle ranges (0° to 80°, -80° to 80°).

[0115] In cases where the rear panel 2 is a simple rectangle, as in this example, the maximum and minimum viewing angles may always be the viewing angles of pixels located on the four sides or corners, regardless of the user's viewing position. In such cases, the viewing angles may be calculated only for pixels located on the four sides or corners, and the angle range may be calculated based solely on these viewing angles. Furthermore, depending on the light distribution characteristics, the maximum and minimum white luminance may always be obtained at the viewing angles of pixels located on the four sides or corners. In such cases, the angle range may not be calculated, and the minimum white luminance may be obtained by calculating the viewing angles only for pixels located on the four sides or corners and comparing the white luminance at these viewing angles.

[0116] Alternatively, when display panels are arranged spherically or when large LED panels are used, information on the direction of the user's gaze may also be used. For example, it is possible to reduce the amount of computation by processing only the screen in front of the user. The FOV (Field of View) may be specified as a parameter to narrow the processing range. In particular, if it is possible to obtain the direction of the user's gaze, the brightness value may be searched only for pixels in the central field of view. Alternatively, the screen may be divided into patches of a certain size, and the brightness value search may be performed only for representative points. By using such methods, the amount of computation can be reduced and processing can be sped up.

[0117] Of course, if the display panel has a complex shape, the viewing angle may be calculated for all pixels as usual. Furthermore, the shape of the display panel and other factors may be taken into consideration, and the angle range calculation and brightness value acquisition may be performed using any method.

[0118] By specifying the luminance criterion in advance, the image correction process may be performed without including dynamic determination. In this case, the process in step 203 (calculation of the luminance criterion matrix) is performed only once, omitting steps 301 and 302, and the same luminance criterion matrix [RGB2XYZ] is always used in (Equation 4) in step 205 (calculation of the RGB values ​​of the corrected pixels). base It is calculated using [ ].

[0119] When the corrected image is displayed, banding (the striped pattern that appears in the image) may become noticeable. To mitigate this, dithering (a process that adds random noise to transform the banding into a form that is imperceptible to humans) may be performed separately.

[0120] When color measurement is performed from an oblique angle, the display panel's light distribution characteristics may result in the acquisition of brightness values ​​that include brightness shifts, potentially leading to inaccurate color measurement. To prevent this, processing may be implemented to create a comparison format that is less prone to overflow. Specifically, for example, by preparing data for the maximum brightness that can be output for each viewing angle, using the chromaticity of white defined as the chromaticity standard, comparison under a unified scale becomes possible.

[0121] In this embodiment, the texture 32 is stored under the control of the GPU. A color transformation matrix is ​​generated based on the user's viewing angle relative to the rear panel 2. Furthermore, the brightness and chromaticity of the image are corrected based on the generated color transformation matrix. This makes it possible to efficiently correct the image according to the user's viewing angle using the GPU.

[0122] Figure 13 is a schematic diagram showing a comparative example display system 35. The comparative example display system 35 has a back panel 36 and a floor panel 37 similar to the display system 1 according to the present technology shown in Figure 1, and a human image 38 is displayed using these.

[0123] In the display system 35, no image correction is performed. That is, unlike the display system 1 related to this technology, no processing such as generating a color conversion matrix according to the viewing angle and correcting the image using the color conversion matrix is ​​performed, and the image generated by the image generation unit is displayed as is. In other words, a predetermined image that is independent of the viewing angle is displayed as is.

[0124] As shown in Figure 13A, the floor panel 37 may appear discolored in the display system 35. Specifically, when the display system 35 is viewed from an oblique angle, the area near the boundary may appear discolored to yellow, cyan, or other colors. Also, as shown in Figure 13B, the boundary between the two display panels, and the portion of the human image 38 located in front of the boundary between the two display panels (near the knees) may appear discolored.

[0125] Figure 14 is a schematic diagram showing a state in which pixel vignetting occurs. As shown in Figure 14, the pixels 41 of a display panel, including LEDs, are composed of RGB units and resin 42 that houses them. Since the upper edge of the resin 42 is higher than the upper edge of the RGB units, when viewing the B unit from the upper right, a part of the B unit is obscured by the resin 42 and becomes invisible. In other words, the R and G units are fully visible, while only a part of the B unit is visible. As a result, to the user, red and green appear relatively stronger than blue, and the pixels appear yellowish.

[0126] This phenomenon is sometimes called pixel vignetting, and it is known to be one of the causes of discoloration in display panels, as shown in Figure 13. Similarly, when viewing a display panel from a different angle, it may appear to change to a different color. In this way, panels, including LEDs, have light distribution characteristics, and their brightness and color change depending on the viewing angle. In addition, display panels using LEDs are sometimes equipped with a mechanism to diffuse the light from the elements, and the presence of this mechanism is also one of the causes of light distribution characteristics.

[0127] Figure 15 is a schematic diagram showing an example of the light distribution characteristics of a display panel. Figure 15A shows the luminance value distribution of a display panel without light distribution characteristics. Figure 15B shows the luminance value distribution of a display panel with light distribution characteristics. As shown in Figure 15A, in an ideal display panel, the luminance value is constant regardless of the viewing angle, and discoloration of the display panel does not occur without special processing. However, in reality, due to the reasons mentioned above, display panels have light distribution characteristics, and as shown in Figure 15B, the luminance value varies depending on the viewing angle.

[0128] Such problems are widely recognized, and it is common to improve the hardware, such as by designing the surface of the display panel to diffuse light isotropically. However, even with such measures, it is not possible to completely eliminate brightness and color changes according to the viewing angle, and in viewing modes such as 6DoF (6 Degrees of Freedom), where people move freely while viewing the display, there are cases where the viewing experience is disrupted.

[0129] In this technology, the image is corrected according to the user's viewing position, and an image with a brightness value corresponding to the viewing position is displayed on the display panel. Furthermore, by generating a color transformation matrix using (Equation 1) to (Equation 4) and correcting the image using the color transformation matrix, a corrected image is generated in which the brightness value perceived by the user does not change depending on the viewing angle. As a result, the shift in the brightness value of the image displayed on the display panel is suppressed, as shown in Figure 1, making it possible to provide a high-quality viewing experience.

[0130] Furthermore, in this technology, information used for image correction (luminance values) is stored in a texture, and processing of this texture is performed by the GPU. This makes it possible to achieve high-speed processing.

[0131] A GPU is a computing device specialized for screen display and image processing, and has become indispensable for displaying images on high-resolution panels, which are becoming increasingly common. Because GPUs excel at performing the same operations in parallel, they can process each pixel at high speed. Processing each pixel includes texture mapping, and the processing system is often optimized for this. For example, when you want to obtain a value at an arbitrary coordinate, if you use a regular array, you first need to obtain information from discrete indices and then explicitly interpolate it. On the other hand, if you use a texture, you can obtain a value at an arbitrary coordinate without explicit calculations.

[0132] As an example of conventional technology, there is a known technique in which information used for image correction is stored in a matrix format. The correction matrix used in such techniques generally contains 3x3 or more elements. On the other hand, since typical textures consist of a maximum of 4 channels, textures cannot be used in such techniques.

[0133] Therefore, if the correction matrix is ​​processed using a GPU, it is likely that it will be processed as regular array data. In this case, when obtaining coordinate values ​​corresponding to an arbitrary field of view, it is necessary to first obtain information from discrete indices and then explicitly interpolate it. Here, the coordinates corresponding to the field of view represent coordinates as angles, and mean texture coordinates or angle points that do not hold the correction matrix. Obtaining information using such regular array data is less efficient than using textures with a corresponding architecture.

[0134] This technology utilizes array data necessary for color correction as a texture, enabling efficient calculations that maximize the benefits of the GPU. Instead of storing the necessary information as a matrix, this technology stores it as normalized RGBW luminance data, and performs matrix calculations within the GPU, thereby enabling the use of a computational processing flow via textures.

[0135] Furthermore, this technology obtains luminance values ​​associated with the viewing angle from the texture. Specifically, it obtains colorimetric luminance values ​​from the texture at the same colorimetric angle as the viewing angle. In addition, a color conversion matrix is ​​generated based on the obtained colorimetric luminance values. This allows for accurate image correction.

[0136] This technology performs color measurement on a colorimetric image and generates colorimetric luminance values ​​for each colorimetric angle. A texture containing each of these colorimetric luminance values ​​is also generated. This allows for the accurate generation of a color transformation matrix.

[0137] In this technology, the colorimetric luminance value is obtained from the texture at any colorimetric angle, and the luminance standard for image correction is determined based on the colorimetric luminance value. This makes it possible to avoid corrections that excessively reduce luminance and RGB value overflows caused by corrections.

[0138] Alternatively, the range of possible viewing angles may be calculated, and the colorimetric luminance value at any of the colorimetric angles within that angle range may be obtained. This minimizes the reduction in luminance. Furthermore, the amount of computation is reduced, enabling high-speed processing.

[0139] This technology acquires the viewing position and calculates the viewing angle and the range of the viewing angle based on that position. This allows for accurate calculation of the viewing angle and the range of the viewing angle.

[0140] <Other Embodiments> This technology is not limited to the embodiments described above, and various other embodiments can be realized. [Application to Multiple Panels] This technology is applicable when the display system 1 includes multiple display panels. For example, as shown in Figure 1, it can include cases where there are two display panels, or where a large number of display panels are installed surrounding the user.

[0141] In this embodiment, when multiple display panels exist, a texture 32 is generated for each of the multiple display panels. For example, when there are two display panels A and B, the texture generation unit 26 generates texture A corresponding to display panel A and texture B corresponding to display panel B. The image displayed on display panel A is then corrected by a color conversion matrix generated based on texture A. On the other hand, the image displayed on display panel B is corrected by a color conversion matrix generated based on texture B.

[0142] By generating a texture 32 individually for each display panel in this way, it becomes possible to perform appropriate image correction according to the arrangement of the display panels. Furthermore, when two display panels are connected at an angle, the sudden difference in angle may cause a shift in brightness and color at the boundary between the display panels. If color correction is forcibly performed to match such a condition, the RGB values ​​may overflow after conversion, making it highly likely that the colors will not match. However, with this technology, the color conversion standard is dynamically determined to prevent RGB values ​​from overflowing, thus enabling stable color correction.

[0143] Furthermore, when generating the reference luminance matrix, the search for the lowest luminance value may be performed at the boundary where the display panels meet, and the luminance value search may be performed on the display panel side with the lower luminance value. In addition, in order to suppress luminance color differences at the boundaries of the display panels, a method may be used in which luminance values ​​are obtained with emphasis on the area near the boundary where the panels meet.

[0144] Similarly, even if there are three or more display panels, a separate texture 23 may be generated for each panel. Furthermore, if the light distribution characteristics of multiple display panels are identical, a common texture 23 may be generated.

[0145] [Measures to improve brightness] The conversion using this technology's algorithm may result in a decrease in image brightness. However, in configurations that include multiple display panels, there is a need to match the colors only at the boundaries, and to display with increased brightness at positions away from the boundaries.

[0146] In this embodiment, the brightness and chromaticity of the uncorrected image are corrected according to the distance between the pixels displayed by the display panel and the boundary of the image displayed by the display panel. Specifically, in step 205, the pixels are corrected according to the following formula. R', G', B': RGB values ​​of the corrected pixels [RGB2XYZ]: Color conversion matrix Lumcoef: Coefficient of the brightness correction value [RGB2XYZ base ]: Luminance reference matrix R', G', B': RGB values ​​of pixels before correction

[0147] The luminance correction coefficient, Lumcoef, is determined according to the physical distance from the boundary of the image displayed by the display panel. For example, if the distance between the pixel to be corrected and the boundary is large, Lumcoef is determined so that the luminance value of the corrected pixel is high. On the other hand, if the distance is small, Lumcoef is determined so that the luminance value of the corrected pixel is low. The physical distance between the pixel and the boundary is calculated, for example, by the pixel position calculation unit 24.

[0148] The image corrected in this way will have higher brightness values ​​at locations far from the boundary and lower brightness values ​​near the boundary. This makes it possible to increase the brightness values ​​in areas where color matching is not necessary, such as the center of the image, and to display an even higher quality image.

[0149] [Correction using Cartesian coordinates] Figure 16 is a schematic diagram showing an example of a Cartesian coordinate system. In this embodiment, image correction is performed using Cartesian coordinates (u, v), which are the position in the Cartesian coordinate system of the user's viewpoint, instead of the user's viewing angle (θ, φ). The Cartesian coordinates of the user's viewpoint correspond to one embodiment of the user's viewpoint position information related to this technology.

[0150] Figure 16 illustrates the world coordinate system (X, Y, Z). In this example, the world coordinate system is left-handed, but it may also be right-handed. In this embodiment, a flat panel 45 is arranged parallel to the YZ plane. The Z axis is vertical (direction of gravity), and Figure 16 shows the panel 45 viewed from vertically above. The panel 45 corresponds to one embodiment of the display panel according to this technology.

[0151] A Cartesian coordinate system is a two-dimensional coordinate system consisting of mutually orthogonal u-axis and v-axis. In this example, the u-axis is parallel to the Y-axis, and the v-axis is parallel to the z-axis. A Cartesian coordinate system is a different coordinate system from the world coordinate system. In Figure 16, the v-axis is omitted from the illustration for clarity, and only the u-axis is shown.

[0152] The Cartesian coordinate system is a coordinate system defined for each predetermined pixel on panel 45. The following explanation will describe how the Cartesian coordinate system is defined when the pixel located in the center of panel 45 is designated as the predetermined pixel; however, the same method is used to define the Cartesian coordinate system even when the predetermined pixel is located at a different position.

[0153] The origin of the Cartesian coordinate system, i.e., the point where u = 0, is defined as being located at a predetermined distance in the X direction relative to a predetermined pixel. In this embodiment, since the panel 45 is arranged parallel to the YZ plane, the X direction coincides with the normal direction of the panel 45. The relative distance, when this predetermined distance is set to 1, is defined as the coordinate on the u-axis. The real-space value of the predetermined distance is not limited. Furthermore, the u-axis is defined to be perpendicular to the normal direction of the panel 45.

[0154] In this embodiment, a process is performed to project the actual user viewpoint (the position of the user's eyes) onto the u-axis defined in this way. Hereinafter, the actual user viewpoint will be referred to as the real user viewpoint, and the projected user viewpoint will be referred to as the projected user viewpoint 46. The real user viewpoint originally exists in the world coordinate system, but in the projection process onto the u-axis, the Z coordinate is ignored, and only the XY coordinates are considered. That is, it is assumed to exist on the XY plane.

[0155] The figure schematically shows, as an example, a situation where the projected user viewpoint 46 is located at positions u = -5, 0, and 5 on the u-axis. However, the u-coordinate of the projected user viewpoint 46 can take any value. The projected user viewpoint 46 corresponds to one embodiment of the user viewpoint according to this technology.

[0156] The actual user's viewpoint may be closer to panel 45 than the u-axis, or conversely, it may be further away. The projection process broadly involves defining the projected user viewpoint 46 as the intersection point of a line connecting the actual user's viewpoint and a predetermined pixel with the u-axis.

[0157] Specifically, the X difference is calculated by subtracting the X coordinate of a predetermined pixel from the X coordinate of the actual user's viewpoint, assuming a predetermined distance of 1. In other words, the X difference can be said to be the distance in the X direction between the actual user's viewpoint and the predetermined pixel. Similarly, the Y difference is calculated. Then, the value obtained by dividing the Y difference by the X difference becomes the u coordinate of the projected user viewpoint 46.

[0158] For example, if the X difference is 3 and the Y difference is 6, the projected user viewpoint 46 will be located at u=2 on the u-axis. Also, if the X difference is 0.5 and the Y difference is -1.5, the projected user viewpoint 46 will be located at u=-3 on the u-axis.

[0159] Similarly, projection processing is also performed on the v-axis. That is, the process of dividing the Z difference by the X difference and calculating the v-coordinate of the projected user viewpoint 46 is executed. The order of projection processing on the u-axis and projection processing on the v-axis is not limited, and they may be performed simultaneously. As a result, the actual user viewpoint is projected onto a point in the Cartesian coordinate system. The plane formed by the Cartesian coordinate system can also be said to be tangent at the origin to a sphere of radius 1 centered at a predetermined pixel.

[0160] In this embodiment, the viewing position detection unit 23 in Figure 2 detects the world coordinates of the actual user's viewpoint. The pixel position calculation unit 24 calculates the world coordinates of a predetermined pixel. Based on these, the projection processing unit 48 (shown in Figure 16), which is configured in place of the field of view calculation unit 25, performs the above projection processing and obtains the orthogonal coordinates of the projected user viewpoint 46.

[0161] Then, correction processing is performed based on the orthogonal coordinates (u, v) of the projected user viewpoint 46. The differences from processing using the user's viewing angle (θ, φ) will be explained below. In the pre-processing shown in Figure 3, instead of rotating the panel 45, the colorimeter 8 is moved in the u and v directions. Alternatively, the colorimeter 8 may be fixed and the panel 45 moved in the u and v directions. Then, the luminance value of each color is calculated for each orthogonal coordinate of the projected user viewpoint 46 in the orthogonal coordinate system of the central pixel of the panel 45.

[0162] The texture is then generated such that, for example, the horizontal position corresponds to the value of u and the vertical position corresponds to the value of v. This texture corresponds to one embodiment of a corrective luminance texture that includes luminance values ​​associated with the projected user viewpoint 46.

[0163] When using viewing angles, the range of possible values ​​for θ and φ is finite (for example, -90° < φ < 90°). However, when using Cartesian coordinates, the range of values ​​for u and v is not finite and can take on any value. Therefore, in order to make the texture size finite, it is necessary to restrict the range of values.

[0164] Therefore, in this embodiment, the value of u is restricted (clamped) to a partial range from -5 to 5, as shown by the solid line portion in Figure 16. Similarly, the value of v is also restricted to a predetermined range, such as from -5 to 5. This makes it possible to restrict the texture to a predetermined size. Note that there are no specific limits on the range to which the values ​​are restricted. For example, symmetry in the values ​​may be assumed, such as setting the value of u from 0 to 5.

[0165] On the other hand, limiting the range of values ​​has the disadvantage that correction cannot be performed if the Cartesian coordinate values ​​of the projected user viewpoint 46 actually fall outside the range. However, even when using the viewing angle, correction at steep angles is not performed properly due to individual differences in panels, etc., and there is a problem that it is prone to overcorrection. Moreover, it is rare for the user viewpoint to be at a steep angle such as 80°. Therefore, the disadvantages in practical use are considered to be minor.

[0166] In the correction process shown in Figures 9, 10, and 12, the Cartesian coordinates of the projected user viewpoint 46 of the pixels to be processed are obtained in step 202. In step 301, the Cartesian coordinates of the projected user viewpoint 46 are obtained for all pixels of the panel 45, and then the position range is calculated. In step 302, the minimum white luminance within that range is obtained from the texture. In step 303, the front white luminance, i.e., the white luminance at (u, v) = (0, 0), is obtained from the texture.

[0167] The subsequent processing is almost the same as when using the viewing angle. Specifically, in step 204, a color transformation matrix is ​​generated based on the projected user viewpoint 46 and the acquired texture. Then, in step 205, the brightness and chromaticity of the image are corrected based on the color transformation matrix.

[0168] As shown in Figure 16, if θ is the angle between the normal direction of panel 45 and the line connecting a predetermined pixel and the projected user viewpoint 46, then θ corresponds to the viewing angle. Furthermore, the relationship u = arctanθ holds between u and θ. For example, when -5 ≤ u ≤ 5, it is approximately -79° ≤ θ ≤ 79°. Similarly, the relationship v = arctanφ holds.

[0169] Furthermore, when considering a straight line connecting a given pixel and the actual user's viewpoint, the angles formed are similarly θ and φ. Therefore, when performing processing using the viewing angle, it was necessary to calculate the viewing angle based on the coordinates of the actual user's viewpoint using the above-mentioned relation. Such trigonometric calculations are relatively computationally intensive.

[0170] On the other hand, since orthogonal coordinates are used in this embodiment, the calculation process described above is not required, making it possible to reduce the processing load and speed up processing.

[0171] [Transformation of Cartesian Coordinate System] Figure 17 is a schematic diagram showing an example where the panels 47 are arranged at an angle. In this example, flat panels 47 (47a to 47c) are connected in this order. Panel 47b is arranged parallel to the YZ plane. Panel 47a is parallel to the Z axis, but is tilted 30° counterclockwise from its parallel position to the YZ plane when viewed from the positive side of the Z axis. Conversely, panel 47c is tilted 30° clockwise. These tilts are merely examples, and specific values ​​are not limited.

[0172] Figure 18 is a schematic diagram showing the state of the transformation to a Cartesian coordinate system. Because panel 47a is tilted, when a Cartesian coordinate system is defined for the central pixel (a predetermined pixel) of panel 47a using the method described above, the state shown in Figure 18A is obtained. In this case, the normal of the Cartesian coordinate system and the normal of panel 47a do not coincide, and the Cartesian coordinate system is tilted 30° clockwise with respect to panel 47a.

[0173] However, the texture is generated assuming that the Cartesian coordinate system is parallel to panel 45 and separated by a predetermined distance in the normal direction, as shown in Figure 16. Therefore, when using a single texture common to panels 47a to 47c, for example, defining the Cartesian coordinate system as shown in Figure 18A will cause problems with correction.

[0174] Therefore, in this embodiment, as shown in Figure 18B, a Cartesian coordinate system is considered such that the positional relationship with a predetermined pixel is the same as in Figure 16, and a process is performed to project the viewpoint of the actual user onto the Cartesian coordinate system of Figure 18B. Hereinafter, the Cartesian coordinate system of Figure 18A may be referred to as the original Cartesian coordinate system, and the Cartesian coordinate system of Figure 18B may be referred to as the transformed Cartesian coordinate system. The original Cartesian coordinate system and the transformed Cartesian coordinate system in this disclosure are examples of local coordinate systems. Figure 18B shows the u' axis of the transformed Cartesian coordinate system.

[0175] The origin of the u' axis is located at a distance of 1 from a predetermined pixel in the direction of the normal to panel 47a. Furthermore, the inclination of the u' axis is the same as the inclination of panel 47a. Therefore, by using the u' coordinates obtained by projecting the actual user's viewpoint onto the u' axis, it becomes possible to achieve appropriate correction.

[0176] First, as shown in Figure 18A, the diff is a direction vector on the XY plane that points from a given pixel towards the actual user's viewpoint before projection. world This is defined. world This is calculated by taking the difference between the actual user's viewpoint and a predetermined pixel.

[0177] Next, the direction vector diff world The process involves rotating the image by the rotation angle of panel 47a around a predetermined pixel. In other words, in this example, diff worldIt is rotated 30° clockwise. This rotation process is performed on the local coordinate system, which is rotated 30° counterclockwise from the world coordinate system. world This corresponds to the process of determining the value of . Figure 18B illustrates the local coordinate system (X', Y', Z').

[0178] Specifically, diff world Rot is a rotation matrix of 30° １ The process of multiplying by is performed. In other words, the coordinates of the rotated direction vector in the local coordinate system are diff panel1 = Rot １ diff world This is expressed as follows. Figure 18B shows the direction vector diff. panel1 This is illustrated.

[0179] And then, diff panel1 The process involves projecting the two-dimensional point represented by onto the u' axis of the transformed Cartesian coordinate system. In the local coordinate system, the u' axis is located at a distance of 1 in the X' direction from a given pixel and is parallel to the Y'Z' plane. Similarly, in the world coordinate system, the u axis is located at a distance of 1 in the X direction from a given pixel and is parallel to the YZ plane. In other words, these positions are identical. Therefore, this projection process is achieved by using the same method in the local coordinate system as the projection process in the world coordinate system. As shown in Figures 18A and 18B, diff panel1 The u' coordinates projected by diff are world This will be different from the u-coordinate obtained by projecting it.

[0180] Similarly, rotation and projection are performed with respect to the v' axis. The order of processing with respect to the u' axis and the v' axis is not limited. Furthermore, these processing may be performed simultaneously. For example, consider the 3D vector diff. world The rotation process can be performed in a single step by multiplying it by a three-dimensional rotation matrix.

[0181] The specific processes for rotation and projection begin with the acquisition of the world coordinates from the user's viewpoint by the projection processing unit 48 mentioned above. The pixel position calculation unit 24 then calculates the world coordinates of a predetermined pixel. These predetermined pixel world coordinates correspond to one embodiment of the display panel's position information. Finally, the projection processing unit 48 calculates the difference between these coordinates as the direction vector diff. world The following is calculated: world The above rotation and projection processes are performed on the object, and the u' coordinate and v' coordinate are calculated.

[0182] As shown in Figure 18A, the direction vector from a predetermined pixel to the projected point is called dir world Therefore, dir world The world coordinates are (1, u). Also, as shown in Figure 18B, the direction vector from a given pixel to the projected point is dir panel1 Therefore, dir panel1 The local coordinates of dir are (1, u'). world Geometric calculations using diff world diff panel1 ,dir panel1 The u' coordinates and other values ​​may be calculated.

[0183] When considered in the world coordinate system, the u' axis is rotated 30° counterclockwise relative to the original u axis. Furthermore, the origin of the u' axis is shifted slightly to the lower left relative to the origin of the u axis. In other words, the u' axis is generated by performing a coordinate transformation (rotation and translation) on the u axis. Therefore, it can be said that this process performs a coordinate transformation of the original Cartesian coordinate system.

[0184] Furthermore, the v' axis is generated by a similar transformation. In this example, panel 47a is not tilted with respect to the v axis, but translation is necessary to make the origin of the v' axis coincide with the origin of the u' axis.

[0185] The tilt of the transformed Cartesian coordinate system coincides with the tilt of panel 47a. Therefore, the tilt of panel 47a corresponds to one embodiment of the reference orientation that serves as the basis for rotation. Furthermore, the position of the origin of the transformed Cartesian coordinate system is at a distance of 1 from a predetermined pixel in the normal direction of panel 47a. Therefore, the position at a distance of 1 from a predetermined pixel in the normal direction of panel 47a corresponds to one embodiment of the reference position that serves as the basis for translation.

[0186] A similar process is performed on panel 47c, which has a different tilt. That is, if there are multiple panels 47 with tilts, a different coordinate transformation is performed individually for each panel 47. Also, if panel 47 is curved, a different coordinate transformation is performed for each pixel.

[0187] This makes it possible to perform image correction processing without any problems, even when a single common texture is used. Furthermore, if panel 47 is rotated during processing using the viewing angle, the same rotation processing should be performed for each of (θ, φ).

[0188] [Use in Virtual Production] This technology can also be used in virtual production. Virtual production is a technique in which a background image (including video) is displayed on a display panel, and an actor performs in front of it, allowing the background and actor to be filmed together. Virtual production is sometimes referred to as in-camera VFX.

[0189] One method for generating images that include both the background and the performers is to have the performers act in front of a green screen, but this method requires the background image to be composited later. Virtual production, on the other hand, does not require the background image compositing process, making it possible to shoot more efficiently. In addition, since the background image is actually displayed during shooting, it has the advantage that performers can more easily imagine the situation of the scene, improving the quality of their acting, and the cameraman can check in real time whether the acting matches the background.

[0190] Figure 19 is a schematic diagram showing an example configuration for virtual production. In this example, panels 49a, 49b, and 49c are connected and arranged in the same way as in Figure 17, with panels 49a and 49c slightly tilted relative to panel 49b. Together, these panels display a single background image 50.

[0191] The display on panel 49a is controlled by an image processing device 57a having the same configuration as the image processing device 11 in Figure 2. The display on panel 49b is controlled by image processing device 57b, and the display on panel 49c is controlled by image processing device 57c. In other words, in this embodiment, the display on each panel 49 is individually controlled by different image processing devices 57.

[0192] The user 51, who is the performer, is located in front of panel 49. Camera 52 captures images of the background image 50 and the user 51. Figure 19 schematically shows the field of view 53 of camera 52 as a rectangle between camera 52 and user 51. On panel 49, an area 56 is shown as an extension of the field of view 53, as indicated by the dashed line in the figure. Area 56 spans across panels 49a and 49b.

[0193] When imaging is performed by camera 52, the background image 50 inside region 56 and the user 51 in front of it will be captured. In this embodiment, a different image from the outside of region 56 is displayed inside region 56. The image displayed inside region 56 is sometimes called the inner frustum 54, and the image displayed outside (the region excluding region 56) is sometimes called the outer frustum 55, and these terms will be used in the following explanation.

[0194] The inner frustum 54 corresponds to one embodiment of the first rendering image. The outer frustum 55 corresponds to one embodiment of the second rendering image. The background image 50 corresponds to one embodiment of the third rendering image.

[0195] For example, the inner frustum 54 is an image with an effect added to the outer frustum 55. Alternatively, it may be a high-resolution version of the outer frustum 55. In addition, the inner frustum 54 may be any other image different from the outer frustum 55.

[0196] In other words, since the outer frustum 55 is an image that is not actually captured, it does not need to be of the same high quality as the inner frustum 54; it only needs to be of sufficient quality to, for example, make it easier for performers to imagine the situation in the scene. Therefore, in this technology, the processing load related to display is reduced by lowering the quality of the outer frustum 55 to a certain extent.

[0197] Furthermore, the inner frustum 54 may be displayed in an area slightly wider than the area 56. This makes it possible to suppress problems such as the outer frustum 55 being captured due to rendering delay when the area 56 changes due to the movement or rotation of the camera 52, and problems such as diffracted light from the display of the outer frustum 55 entering the area 56.

[0198] Here, the image processing device 57 needs to perform a process to identify the region 56 in which the inner frustum 54 will be displayed. An example of this identification process will be described below. In this embodiment, the region calculation unit 58 of the image processing device 57 shown in the figure calculates the intersection points 60 of the camera 52 and the direction vectors 59 that go from the viewpoint of the camera 52 to each position within the field of view 53 and the panel 49. Furthermore, the region on the panel 49 consisting of each of the intersection points 60 is calculated as region 56. Region 56 corresponds to one embodiment of the intersection region.

[0199] The region calculation unit 58 first calculates direction vectors 59 directed to each position within the field of view 53. This calculation is performed using the internal parameters of the camera 52 (position, orientation, focal length, etc.). In the figure, only the direction vector 59 directed to the center of the field of view 53 is shown with an arrow as an example, but of course, direction vectors 59 directed to other positions are also calculated simultaneously.

[0200] Next, the intersection point 60 between the straight line extending the direction vector 59 and the virtual panel 49 held by the image processing device 57 is calculated. In other words, the image processing device 57 virtually holds the position information of the panel 49, and uses this to calculate the intersection point 60 internally.

[0201] The figure only shows the intersection point 60 corresponding to the direction vector 59 pointing towards the center of the field of view 53. This intersection point 60 is also located in the center of the region 56. Similarly, intersection points 60 corresponding to other direction vectors 59 are also calculated. For example, the intersection point 60 corresponding to the direction vector 59 pointing towards the upper right of the field of view 53 is located in the upper right of the region 56.

[0202] The region consisting of all the intersection points 60 is defined as region 56. The region 56 calculated in this way corresponds to the area that would be captured if the camera 52 were actually taking images.

[0203] Figure 20 is a schematic diagram showing the processing order related to image integration. In this embodiment, for example, the processing is performed in the order shown in Figure 20A. First, the outer frustum 55 is drawn (step 501). Next, the inner frustum 54 within region 53 is drawn (step 502).

[0204] Specifically, the image processing device 57 virtually holds the 3D model to be drawn, and in step 502, the image obtained by capturing the virtual 3D model with the camera 52 is drawn as the inner frustum 54.

[0205] Even if the image processing device 57 does not have information about region 56 at the time of step 502, the processing in step 502 is performed using the position and orientation and field of view of the camera 52, as well as coordinate information of a virtual 3D model, and information about region 56 is not required, so there is no problem.

[0206] Next, the correction process related to this technology is performed on the inner frustum 54 (step 503). At this point, each intersection 60 is calculated using the method described above. Then, the orthogonal coordinates of the camera 52 are calculated at each intersection 60, and a correction process is performed using the orthogonal coordinates. That is, a correction process is performed on each intersection within the region 56.

[0207] Here, since panel 49a has a tilt, the pixels of the inner frustum 54 that are located within panel 49a undergo the Cartesian coordinate system transformation process described above. In other words, although Figures 16 to 18 explained the process using Cartesian coordinates assuming that the user is viewing panel 45, etc., the user in this case may be camera 52.

[0208] On the other hand, the correction process may be performed using the viewing angle. In this case, the viewing angle conversion process is performed on the pixels within panel 49a. The correction process only needs to be performed on the inner frustum 54 of the inner frustum 54 and outer frustum 55. That is, the correction process may or may not be performed on the outer frustum 55.

[0209] In step 504, the process of identifying region 56 is performed first. Since the intersection points 60 are calculated in step 503, it could be said that region 56 has already been identified at this point. However, the processing content, i.e., the method of identification, in steps 503 and 504 may differ. For example, in step 504, the intersection points 60 may be calculated only for direction vectors pointing to the four vertices of the field of view 53.

[0210] Next, the image integration unit 61 shown in Figure 19 performs a process to integrate the outer frustum 55 and the inner frustum 54 based on the region 56 identified in step 504. This generates a background image 50. The generated background image 50 is displayed on the panel 49 (step 505).

[0211] The processing below the dashed line in the figure (from step 504 onwards) is internal processing of the game engine that performs the rendering, and may be difficult to modify in terms of implementation, or it may be confidential. In such cases, even if correction processing is desired, it is not possible to add that processing from step 504 onwards. Therefore, as shown in Figure 20A, the correction processing is added before step 504. However, in order to perform the correction processing, information on where within region 56 the image will be projected is required, so the process of calculating the intersection point 60 is performed at this point. This method has the advantage that correction can be performed on the screen process of the camera 52, and therefore the processing can be performed without being subject to implementation constraints on the game engine.

[0212] In the process shown in Figure 20B, the correction process (step 604) is performed after the integration process (step 603). That is, the correction process is performed on the portion of the background image 50 that is displayed in region 56. This processing order may also be used.

[0213] Furthermore, each image processing device 57a to 57c may render only the outer frustum 55, while the inner frustum 54 is rendered by a separate, dedicated image processing device. In this case as well, either the method of performing color correction before integration as shown in Figure 20A, or the method of performing color correction after integration as shown in Figure 20B, may be used. In addition, the specific configuration of which image each image processing device 57 renders is not limited.

[0214] If a real object such as a person or other object is placed in front of the display panel and obscures a pixel, the correction for that pixel may be omitted. This reduces the amount of computation and speeds up processing.

[0215] Furthermore, when display panels are installed on the ceiling, the brightness reference may change abruptly at the moment the boundary of the ceiling display panel is reflected. For this reason, a separate stabilization process in the time direction may be introduced. Specifically, the brightness and chromaticity of the uncorrected image are corrected so that the amount of change in the brightness value of the corrected image over a predetermined time interval does not exceed a predetermined value (threshold).

[0216] Specifically, the stabilization process for the luminance reference value is performed using the following formula. Lum: Lum value (standard brightness after stabilization) base : Brightness reference value ΔLum before stabilization processing limit : The limiting value for the amount of change. Note that absmin compares the absolute values ​​of the arguments and assigns the sign of the second argument to the smaller value.

[0217] Δt corresponds to a predetermined time width; for example, the time width of one frame is Δt. Also, ΔLum limit This corresponds to a predetermined value. For example, if you want to keep the change in brightness value in one frame below 100, then ΔLum limit It is set to =100.

[0218] For example, if the luminance value of the previous frame (t - Δt) is 200 and the luminance value of the current frame (t) is 250, then ΔLum limit = 100, Lum base (t)-Lum base (t - Δt) = 250 - 200 = 50, so the value of absmin is 50. Therefore, Lum(t) = 250, which is Lum base (t) is equal to (t). In other words, if the change in brightness value between the previous frame and the current frame is 100 or less, the brightness value of the current frame is used as is.

[0219] On the other hand, if, for example, the brightness value of the previous frame was 200 and the brightness value of the current frame is 350, then the value of absmin will be 100. Therefore, Lum(t) = 300. In this way, when the change in brightness value is greater than 100, the amount of change is limited to the regulated value of +100.

[0220] Similarly, if the brightness value changes in the negative direction, the brightness value of the current frame is used as is if the change is between 0 and -100, and if the change is less than -100, it is limited to -100.

[0221] This suppresses abrupt changes in brightness standards, enabling more precise image correction. In virtual production, captured footage is often edited manually afterward to correct inconsistencies. However, by using this technology, high-quality footage is captured in advance, reducing the effort required for post-processing.

[0222] Figure 21 is a block diagram showing an example of the hardware configuration of a computer 500 capable of realizing the image processing devices 11 and 57. The computer 500 includes a CPU 501, ROM 502, RAM 503, an input / output interface 505, and a bus 504 connecting these to each other. A display unit 506, an input unit 507, a storage unit 508, a communication unit 509, and a drive unit 510 are connected to the input / output interface 505.

[0223] The display unit 506 is a display device using, for example, liquid crystal, EL, etc. The input unit 507 is, for example, a keyboard, pointing device, touch panel, or other operating device. If the input unit 507 includes a touch panel, the touch panel may be integrated with the display unit 506. The storage unit 508 is a non-volatile storage device, for example, an HDD, flash memory, or other solid memory. The drive unit 510 is a device capable of driving the removable recording medium 511, for example, an optical recording medium or magnetic recording tape. The communication unit 509 is a modem, router, or other communication device for communicating with other devices, which can be connected to a LAN, WAN, etc. The communication unit 509 may communicate using either wired or wireless methods. The communication unit 509 is often used separately from the computer 500.

[0224] Image processing by the computer 500 having the hardware configuration described above is realized through the cooperation of software stored in the memory unit 508 or ROM 502, etc., and the hardware resources of the computer 500. Specifically, the image processing method according to this technology is realized by loading the programs constituting the software, which are stored in the ROM 502, etc., into the RAM 503 and executing them.

[0225] The program is installed on the computer 500, for example, via a removable recording medium 511. Alternatively, the program may be installed on the computer 500 via a global network or the like. In addition, any non-transient storage medium that the computer 500 can read may be used.

[0226] In this disclosure, "system" means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0227] The execution of the image processing method related to this technology by a computer system includes both cases where, for example, texture storage, texture acquisition, brightness value acquisition, color conversion matrix generation, brightness reference determination, image correction, image display, orthogonal coordinate generation, and orthogonal coordinate transformation are performed by a single computer, and cases where each process is performed by different computers. Furthermore, the execution of each process by a predetermined computer includes having another computer perform part or all of the process and obtaining the results. In other words, the image processing method related to this technology can also be applied to cloud computing configurations in which a single function is shared and processed collaboratively by multiple devices via a network.

[0228] The image display system, image processing device, display panel, texture configurations, and processing flows described with reference to each drawing are merely embodiments and can be modified as needed without departing from the spirit of this technology. In other words, other arbitrary configurations and algorithms may be adopted to implement this technology.

[0229] It is also possible to combine at least two of the feature features of the present technology described above. In other words, the various feature features described in each embodiment may be combined arbitrarily, regardless of the specific embodiment. Furthermore, the various effects described above are merely examples and not limiting, and other effects may also be exhibited.

[0230] Furthermore, this technology can also be configured as follows: (1) An image processing method comprising: controlling a graphics processing device to acquire position information of the user viewpoint of a display panel in an orthogonal coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of a predetermined pixel of at least one display panel; acquiring a correction luminance texture including a luminance value associated with the user viewpoint; generating a color transformation matrix based on the user viewpoint and the acquired correction luminance texture; and correcting the luminance and chromaticity of a rendered image based on the generated color transformation matrix. (2) An image processing method according to (1), wherein the range of acquisition of the user viewpoint position information is limited to a partial range. (3) An image processing method according to (1) or (2), further comprising: acquiring position information of the display panel in a world coordinate system in real space different from the orthogonal coordinate system; and rotating the orthogonal coordinate system in the world coordinate system based on the user viewpoint position information and the display panel position information, wherein the user viewpoint position information is acquired based on the rotated orthogonal coordinate system. (4) An image processing method according to (3), wherein the Cartesian coordinate system is rotated to have a predetermined reference orientation. (5) An image processing method according to (3) or (4), wherein the at least one display panel is a plurality of display panels, and for each of the plurality of display panels, the image processing method is performed to acquire position information of the display panel, rotate the Cartesian coordinate system, and acquire position information of the user's viewpoint. (6) An image processing method according to any one of (1) to (5), further comprising acquiring position information of the display panel in a world coordinate system in real space different from the Cartesian coordinate system, and translating the Cartesian coordinate system in the world coordinate system based on the position information of the user's viewpoint and the position information of the display panel, wherein the position information of the user's viewpoint is acquired based on the translated Cartesian coordinate system. (7) An image processing method according to (6), wherein the translation is performed so that the position of the Cartesian coordinate system becomes a predetermined reference position.(8) An image processing method according to (6) or (7), wherein the at least one display panel is a plurality of display panels, and for each of the plurality of display panels, the image processing method is used to acquire positional information of the display panel, translate the Cartesian coordinate system, and acquire positional information of the user's viewpoint. (9) An image processing method according to any one of (1) to (8), wherein the user is a camera having a predetermined field of view, and the image processing method further includes calculating the intersection points of each direction vector from the viewpoint of the camera toward each position within the field of view and the display panel, and calculating an intersection region which is a region on the display panel consisting of each of the intersection points. (10) An image processing method according to (9), further including generating a third rendering image by integrating a first rendering image displayed in the intersection region and a second rendering image displayed in the region on the display panel excluding the intersection region, wherein the brightness and chromaticity correction is performed on at least the first rendering image, of the first rendering image or the second rendering image. (11) An image processing method according to (10), wherein the brightness and chromaticity correction is performed only on the first rendering image. (12) An image processing method according to any one of (9) to (11), further comprising generating a third rendering image by integrating the first rendering image displayed in the intersection region and the second rendering image displayed in the area of ​​the display panel excluding the intersection region, wherein the brightness and chromaticity correction is performed on the third rendering image. (13) An image processing method according to (12), wherein the brightness and chromaticity correction is performed only on the portion of the third rendering image displayed in the intersection region. (14) An image processing method according to any one of (1) to (13), wherein the user is a camera.(15) An image processing method according to any one of (1) to (14), further comprising performing a virtual production by capturing the rendered image displayed on the display panel with the camera. (16) An image processing apparatus comprising: a storage unit that stores a correction luminance texture including luminance values ​​associated with the position information of the user viewpoint of the display panel in a Cartesian coordinate system located at a predetermined distance from the predetermined pixel in the normal direction of at least one of the pixels of the display panel; and a graphics processing apparatus configured to generate a color conversion matrix based on the user viewpoint and the correction luminance texture stored by the storage unit, and to correct the luminance and chromaticity of the rendered image based on the color conversion matrix. (17) A storage medium containing a program that controls a graphics processing unit to obtain position information of the user viewpoint of at least one display panel in a Cartesian coordinate system located at a predetermined distance from the predetermined pixel in the normal direction of the predetermined pixel of the display panel; obtain a correction luminance texture including luminance values ​​associated with the user viewpoint; generate a color conversion matrix based on the user viewpoint and the obtained correction luminance texture; and instruct a computer system to perform an instruction to correct the luminance and chromaticity of a rendered image based on the generated color conversion matrix.

[0231] 1…Display system 2…Rear panel 3…Floor panel 4…Human image 8…Colorimeter 11, 57…Image processing unit 15…Storage unit 20…Image generation unit 21…Display control unit 22…Colorimeter control unit 23…Viewing position detection unit 24…Pixel position calculation unit 25…Viewing angle calculation unit 26…Texture generation unit 27…Texture acquisition unit 28…Brightness reference determination unit 29…Brightness color correction unit 32…Texture 45, 47, 49…Panel 46…Projection user viewpoint 48…Projection processing unit 50…Background image 52…Camera 53…Field of view 54…Inner frustum 55…Outer frustum 56…Region 58…Region calculation unit 59…Direction vector 60…Intersection 61…Image integration unit

Claims

1. An image processing method comprising: controlling a graphics processing unit to obtain position information of the user's viewpoint of at least one display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of the predetermined pixel of the display panel; obtaining a correction luminance texture including luminance values ​​associated with the user's viewpoint; generating a color transformation matrix based on the user's viewpoint and the obtained correction luminance texture; and correcting the luminance and chromaticity of a rendered image based on the generated color transformation matrix.

2. The image processing method according to claim 1, wherein the acquisition range of the user's viewpoint position information is limited to a partial range.

3. The image processing method according to claim 1, further comprising: acquiring position information of the display panel in a world coordinate system in real space different from the Cartesian coordinate system; and rotating the Cartesian coordinate system in the world coordinate system based on the position information of the user viewpoint and the position information of the display panel, wherein the position information of the user viewpoint is acquired based on the Cartesian coordinate system after the rotation.

4. The image processing method according to claim 3, wherein the Cartesian coordinate system is rotated to have a predetermined reference orientation.

5. The image processing method according to claim 3, wherein the at least one display panel is a plurality of display panels, and for each of the plurality of display panels, the method is used to acquire the position information of the display panel, rotate the Cartesian coordinate system, and acquire the position information of the user's viewpoint.

6. The image processing method according to claim 1, further comprising: acquiring position information of the display panel in a world coordinate system in real space different from the Cartesian coordinate system; and translating the Cartesian coordinate system in the world coordinate system based on the position information of the user viewpoint and the position information of the display panel, wherein the position information of the user viewpoint is acquired based on the translated Cartesian coordinate system.

7. The image processing method according to claim 6, wherein the translation is performed so that the position in the Cartesian coordinate system becomes a predetermined reference position.

8. The image processing method according to claim 6, wherein the at least one display panel is a plurality of display panels, and for each of the plurality of display panels, the method is used to acquire the position information of the display panel, translate the Cartesian coordinate system, and acquire the position information of the user's viewpoint.

9. The image processing method according to claim 1, further comprising: the user being a camera having a predetermined field of view; calculating the intersection points of each direction vector from the viewpoint of the camera toward each position within the field of view and the display panel; and calculating an intersection region which is an area on the display panel consisting of each of the aforementioned intersection points.

10. The image processing method according to claim 9, further comprising generating a third rendering image by integrating a first rendering image displayed in the intersection region and a second rendering image displayed in the region of the display panel excluding the intersection region, wherein the luminance and chromaticity correction is performed on at least the first rendering image among the first rendering image and the second rendering image.

11. The image processing method according to claim 10, wherein the brightness and chromaticity correction is performed only on the first rendered image.

12. The image processing method according to claim 9, further comprising generating a third rendering image by integrating a first rendering image displayed in the intersection region and a second rendering image displayed in the region of the display panel excluding the intersection region, wherein the brightness and chromaticity correction is performed on the third rendering image.

13. The image processing method according to claim 12, wherein the luminance and chromaticity correction is performed only on the portion of the third rendered image that is displayed in the intersection region.

14. The image processing method according to claim 1, wherein the user is a camera.

15. The image processing method according to claim 14, further comprising performing a virtual production by capturing the rendered image displayed on the display panel using the camera.

16. An image processing apparatus comprising: a storage unit that stores a correction luminance texture including a luminance value associated with the position information of the user's viewpoint of the display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of at least one pixel of the display panel; and a graphics processing apparatus configured to generate a color conversion matrix based on the user's viewpoint and the correction luminance texture stored by the storage unit, and to correct the luminance and chromaticity of a rendered image based on the color conversion matrix.

17. A storage medium containing a program that controls a graphics processing unit to obtain position information of the user's viewpoint of at least one display panel in a Cartesian coordinate system located at a predetermined distance from a predetermined pixel in the normal direction of the predetermined pixel of the display panel; obtains a correction luminance texture including luminance values ​​associated with the user's viewpoint; generates a color transformation matrix based on the user's viewpoint and the obtained correction luminance texture; and instructs a computer system to correct the luminance and chromaticity of a rendered image based on the generated color transformation matrix.

Images