Image processing device, image processing method, and program

Abstract

Provided is an image processing device that acquires a frame image constituting a video to be presented to a user each time the frame image is drawn, and outputs a converted frame image obtained by converting the acquired frame image using a machine learning model prepared in advance as the frame image to be presented to the user.

Description

Image processing device, image processing method, and program

[0001] The present invention relates to an image processing device, an image processing method, and a program for generating an image to be displayed on a display device.

[0002] For example, in a video game, an application program such as a game program repeatedly executes a process of drawing frame images in real time, outputting the drawn frame images as video signals, and displaying them on the screen of a display device, thereby presenting images to the user.

[0003] In the conventional techniques described above, the frame images drawn by the application program may not match the user's viewing environment or preferences in terms of resolution, color tone, etc. For this reason, techniques have been known in the past that convert the resolution, etc., of frame images using hardware logic, etc., before outputting them. However, such techniques perform fixed conversions, and therefore do not necessarily perform conversions that are appropriate for the content of the images.

[0004] The present invention has been made in consideration of the above-mentioned situation, and one of its objectives is to provide an image processing device, an image processing method, and a program that can present to the user the results of a desired conversion performed according to the content of a frame image.

[0005] An image processing device according to one aspect of the present invention is an image processing device that includes a frame image acquisition unit that acquires a frame image constituting an image to be presented to a user each time the frame image is drawn, and a frame image conversion unit that converts the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputs the converted frame image as the frame image to be presented to the user.

[0006] An image processing method according to one aspect of the present invention includes the steps of acquiring a frame image constituting a video to be presented to a user each time the frame image is drawn, and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user.

[0007] A program according to one aspect of the present invention is a program for causing a computer to execute the following steps: acquiring a frame image constituting a video to be presented to a user each time the frame image is drawn, and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user. This program may be provided by being stored in a computer-readable, non-transitory information storage medium.

[0008] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments and is not to be limited to the disclosed exemplary embodiments.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] 1 is a block diagram showing the configuration of an image processing device 10 according to one embodiment of the present invention. The image processing device 10 is, for example, a home game console or a personal computer, and as shown in the figure, is configured to include a control unit 11, a storage unit 12, and an interface unit 13. The image processing device 10 is also connected to a display device 14 and an operation device 15.

[0011] The control unit 11 includes at least one processor such as a CPU, and performs various information processing by executing programs stored in the storage unit 12. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores the programs executed by the control unit 11 and data processed by the programs.

[0012] The interface unit 13 is an interface for data communication with the display device 14 and the operation device 15. The image processing device 10 is connected to each of the display device 14 and the operation device 15 via the interface unit 13 either wired or wirelessly. Specifically, the interface unit 13 includes a multimedia interface for transmitting a video signal supplied by the image processing device 10 to the display device 14. It also includes a data communication interface for receiving a signal indicating the content of an operation performed by a user on the operation device 15. Furthermore, the interface unit 13 may include a communication interface for transmitting and receiving data to and from other communication devices via a communication network such as the Internet.

[0013] The display device 14 displays on a screen an image corresponding to the video signal supplied from the image processing device 10. The display device 14 may be a stationary display device such as a home television receiver, or may be a portable display device. The display device 14 may also be a head-mounted display device capable of presenting a three-dimensional image by presenting separate images to the left and right eyes of the user.

[0014] The operation device 15 is, for example, a controller for a home game console, and receives operation input from a user. The operation device 15 is connected to the image processing device 10 by wire or wirelessly, and transmits an operation signal indicating the content of the operation input received from the user to the image processing device 10. The operation device 15 may be of various shapes, such as a device that the user holds in their hand or a device that the user wears on their hand.

[0015] The functions realized by the image processing device 10 will be described below with reference to the functional block diagram of Fig. 2. As shown in Fig. 2, the image processing device 10 is functionally configured to include an application execution unit 21, a frame image acquisition unit 22, and a frame image conversion unit 23. These functions are realized by the control unit 11 operating in accordance with one or more programs stored in the storage unit 12. These programs may be provided to the image processing device 10 via a communication network such as the Internet, or may be provided by being stored on a computer-readable information storage medium such as an optical disk.

[0016] The application execution unit 21 executes an application program and repeatedly executes the process of drawing frame images that show the results of the processing. These frame images are images that constitute the video to be presented to the user, and these frame images are displayed on the display device 14 at predetermined intervals, allowing the user to view the video that is the result of processing by the application execution unit 21. Note that the drawing of the frame images may be executed by a processor such as a GPU based on drawing commands from the application program. The drawn frame images are written to a predetermined frame buffer memory allocated in the storage unit 12.

[0017] The frame image acquisition unit 22 reads and acquires the drawn frame image from the frame buffer memory each time a frame image is drawn by the application execution unit 21. In other words, in this embodiment, the frame image drawn by the application execution unit 21 is not transmitted to the display device 14 as is.

[0018] Each time the frame image acquisition unit 22 acquires a newly drawn frame image, the frame image conversion unit 23 performs a given conversion process on the acquired frame image. This conversion process is a process for adjusting the content of the frame image in accordance with the user's viewing environment, etc. For ease of explanation, the frame image obtained by the conversion process by the frame image conversion unit 23 will be referred to as a "converted frame image," and the frame image acquired by the frame image acquisition unit 22 before the conversion process is performed will be referred to as a "pre-converted frame image." In this embodiment, this converted frame image is output as the frame image to be presented to the user. The converted frame image output by the frame image conversion unit 23 is transmitted to the display device 14 via the interface unit 13 and displayed on the screen of the display device 14.

[0019] Here, the frame image conversion unit 23 converts the frame images using a pre-prepared machine learning model, which allows conversion to be performed according to the content of each frame image, compared to conversion using hardware logic, fixed filters, etc.

[0020] In this embodiment, it is assumed that a plurality of machine learning models are prepared in advance depending on the target conversion content, etc. The frame image conversion unit 23 performs the conversion process for the frame image using a machine learning model selected from the plurality of machine learning models based on given conditions such as a user selection. The types of machine learning models prepared, selection criteria for selecting the machine learning model to be used, and specific examples of methods for generating machine learning models will be described later.

[0021] An example of the flow of processing executed by the image processing device 10 according to this embodiment will now be described with reference to the timing diagram of Fig. 3. In the diagram, (N-1), (N), and (N+1) denote numbers conveniently assigned to the frame images that are drawn in sequence.

[0022] As shown in the figure, the application execution unit 21 begins drawing a new pre-conversion frame image every time a predetermined time interval Tf has elapsed. This time interval Tf is determined according to the frame rate; for example, when the frame rate is 60 fps, the time interval Tf is 1 / 60 seconds. In this embodiment, it is assumed that the drawing process of the pre-conversion frame images is completed in a time shorter than the time interval Tf. In the figure, the drawing process of the pre-conversion frame image (N) begins at time t0, and the drawing process of the pre-conversion frame image (N+1) begins at time t1, which is the time interval Tf after time t0.

[0023] When the rendering process of the pre-conversion frame image is completed, the frame image conversion unit 23 performs a conversion process on the pre-conversion frame image and outputs a converted frame image. This conversion process, together with the rendering process of the pre-conversion frame image, is expected to be completed within the time interval Tf.

[0024] The converted frame image output by the frame image conversion unit 23 is transmitted to the display device 14 in the next cycle. That is, when a new cycle begins and the drawing process for the next frame image to be displayed begins, the converted frame image output in the previous cycle is transmitted to the display device 14. In the figure, the converted frame image (N-1) whose conversion was completed by time t0 is transmitted to the display device 14 between time t0 and time t1, the converted frame image (N) whose conversion was completed by time t1 is transmitted to the display device 14 between time t1 and time t2, and the converted frame image (N+1) whose conversion was completed by time t2 is transmitted after time t2. By repeating this cycle, the image processing device 10 can display on the display device 14 the converted frame images obtained by performing a given conversion process on all pre-conversion frame images in real time.

[0025] Below, several specific examples of the frame image conversion process performed by the frame image conversion section 23 will be described.

[0026] As a first example, an example of a conversion process for increasing the resolution of a frame image will be described. In recent years, display devices capable of displaying at extremely high resolutions, such as 4K, have appeared, but older application programs, for example, may not be compatible with such high-resolution displays. In such cases, by converting pre-conversion frame images drawn by the older application program into high-resolution post-conversion frame images, it is possible to present the user with high-quality video that can be displayed by the display device 14.

[0027] Specifically, the frame image conversion unit 23 converts the pre-conversion frame image into a post-conversion frame image with a higher, predetermined resolution. By utilizing a machine learning model, the frame image conversion unit 23 not only increases the number of pixels in the post-conversion frame image, but also increases the resolution of each pixel in the pre-conversion frame image while taking into account the content of the surrounding pixels.

[0028] In this example, multiple machine learning models with different output resolutions may be prepared in advance to support multiple types of resolution. The frame image conversion unit 23 performs the frame image conversion process using one of the multiple machine learning models prepared in advance, which converts the frame image to one with a resolution that can be displayed by the display device 14. More specifically, for example, the frame image conversion unit 23 selects, as the machine learning model to use, a machine learning model that converts the frame image to a resolution that the user has specified in advance on a settings screen or the like. Alternatively, the frame image conversion unit 23 may acquire information regarding the display performance of the display device 14 and, based on the acquired information, select, for example, the maximum resolution that the display device 14 can display as the resolution of the converted frame image. This allows display at various resolutions.

[0029] As a second example, an example of adjusting the brightness and color gamut of pixels contained in a frame image will be described. As with the resolution in the first example, the color gamut that a display device can display varies depending on the type of display device. Therefore, by converting a pre-conversion frame image drawn by an application program that does not support display in a wide color gamut into a post-conversion frame image with a wider color gamut, it is possible to present the user with a high-quality image that can be displayed on the display device 14.

[0030] Specifically, the frame image conversion unit 23 converts the colors of the pixels included in the frame image so that the color gamut used in the converted frame image is wider than the color gamut used in the pre-conversion frame image. For example, the frame image conversion unit 23 converts a pre-conversion frame image with a dynamic range of SDR (Standard Dynamic Range) into a post-conversion frame image with a dynamic range of HDR (High Dynamic Range).

[0031] In this example as well, by using a machine learning model, the color of each pixel in the converted frame image is determined taking into account the colors of surrounding pixels, etc. This makes it possible to convert areas that cannot be fully expressed in SDR and are blurred into colors with a gradation that appears natural to the human eye. Also in this example as well, multiple machine learning models may be prepared in advance to correspond to various color gamuts. The frame image conversion unit 23 may then select a machine learning model to use based on user specifications, information regarding the display performance of the display device 14, etc.

[0032] Furthermore, regardless of the color gamut used, depending on the rendering conditions, such as the color space used when rendering the pre-conversion frame image, subtle color variations within the frame image may not be expressed, resulting in washed-out colors. Therefore, the frame image conversion unit 23 may perform a conversion in which the luminance of each pixel in the pre-conversion frame image is changed in accordance with the luminance of the surrounding pixels, while maintaining the range of possible luminance values ​​before and after conversion. By increasing or decreasing the luminance of some pixels, this type of conversion can ensure that the luminance of the converted frame image as a whole changes in a gradation that appears natural to the human eye.

[0033] As a third example, we will explain an example of conversion processing that converts the colors of a frame image to a color tone that matches the viewer's color vision characteristics. Color vision diversity is a known characteristic in which different people see colors differently. This characteristic can make it difficult for some people to distinguish between certain colors. Therefore, in this example, the colors of pixels contained in the pre-conversion frame image are converted to colors that are easy to see for viewers with specific color vision characteristics.

[0034] In this example, simply converting a specific color into another color mechanically will not make the image easier for viewers to see. For example, for people with color vision deficiencies that make it difficult to distinguish between two specific colors, when those two colors are adjacent, the colors must be converted so that they are easier to distinguish. Therefore, it is desirable to determine the color to which each pixel should be converted, taking into account the color composition and distribution of the image itself, the color arrangement of pixels surrounding each pixel, etc.

[0035] Therefore, in this example, the frame image conversion unit 23 uses a machine learning model that has been prepared in advance according to the type of color vision characteristics of the viewer to convert the colors of the pixels included in the pre-conversion frame image into color tones that correspond to the color vision characteristics of the viewer. This makes it possible to perform color conversion that takes into account the content of the frame image.

[0036] In this example, it is assumed that multiple machine learning models are prepared in advance according to various types of color vision characteristics. A user of the image processing device 10 selects one machine learning model in advance, according to their own color vision characteristics, on a settings screen or the like. The frame image conversion unit 23 converts the frame image using the machine learning model selected by the user. This allows the user to always view images with color tones that are easy to see according to their own color vision characteristics, regardless of the type of application program that draws the frame image or the content of the image to be displayed.

[0037] The frame image conversion unit 23 may perform a conversion that combines two or more of the examples described above. As a specific example, the frame image conversion unit 23 may first perform a conversion that changes the color tone of the pre-conversion frame image, and then perform a conversion that further improves the resolution of the frame image whose color tone has been changed, thereby generating a post-conversion frame image.

[0038] Alternatively, machine learning models for performing multiple types of conversions at once may be prepared in advance. In this case, machine learning models generated by performing machine learning individually for each combination of conversions expected to be required are prepared in advance. As an example, if high-resolution conversion corresponding to two types of output resolution and color tone conversion corresponding to two types of color vision characteristics are required, a total of nine types of conversion combinations are expected (three types x three types), taking into account the possibility that only one of the conversions is required. For one of these combinations, it is sufficient to output the pre-conversion frame image as is without performing either conversion. Therefore, machine learning models are prepared for each of the remaining eight conversion combinations. The user selects one of these eight machine learning models to use based on their own color vision characteristics and the supported resolution of the display device 14. By converting the frame image using the selected machine learning model, a converted frame image with the user's desired color tone and resolution can be obtained in a single conversion.

[0039] Furthermore, the frame image conversion unit 23 may select a machine learning model to use depending on the type of application program that draws the pre-conversion frame images, and convert the frame images using the selected machine learning model. For example, if the application program is a game program, different machine learning models may be prepared in advance for each game genre or title. By converting the frame images using a machine learning model selected depending on the type of application program in this way, it is possible to perform conversion that corresponds to the tendencies of the frame images to be drawn, or to make the color tone of the converted frame images suitable for the content of the application program.

[0040] The multiple machine learning models corresponding to the multiple types of conversion described above may be stored in advance in the storage unit 12 of the image processing device 10, or may be provided to the image processing device 10 as needed via a communication network, etc. For example, a predetermined server device connected to the image processing device 10 via the Internet may store multiple machine learning models corresponding to various conversions in advance and transmit model data necessary for executing the machine learning models in response to a request from the image processing device 10. In this case, when a user of the image processing device 10 selects a conversion content on a settings screen, etc., the image processing device 10 requests model data of the machine learning model corresponding to the selected conversion content from the server device. The machine learning model provided by the server device in response to this request is then stored in the storage unit 12, and the frame image is converted using the model. This makes it easy to update the machine learning model over time and improve the quality of the conversion process.

[0041] Below, a specific example of a method for generating a machine learning model used for the conversion of frame images described above will be described.

[0042] The machine learning model used by the frame image conversion unit 23 may be a model obtained by machine learning using one of two frame images obtained by drawing the same content under different drawing conditions as input data and the other as training data.

[0043] As a specific example, when the frame image conversion unit 23 converts frame images to higher resolution, it first causes an application program that supports both low and high resolutions to execute the same process multiple times while changing the output resolution. This results in high-resolution and low-resolution frame images representing the same content being rendered. Of these, machine learning is performed using the low-resolution frame images as input data and the high-resolution frame images as training data to generate a machine learning model that can convert low-resolution frame images into high-resolution frame images. In this way, it is possible to efficiently generate a machine learning model that increases the resolution of frame images without having to prepare images that have been manually adjusted.

[0044] Furthermore, when the frame image conversion unit 23 converts the color tone of a frame image, it causes an application program equipped with an output mode that accommodates color vision diversity to execute the same processing multiple times in different output modes. This makes it possible to render frame images with the same content but with unadjusted color tones and frame images with color tones adjusted for people with specific color vision characteristics. By performing machine learning using the unadjusted frame images as input data and the adjusted frame images as training data, it is possible to efficiently generate a machine learning model that can convert color tones that are easier to view for people with specific color vision characteristics.

[0045] Furthermore, the machine learning model used by the frame image conversion unit 23 may be a model obtained by machine learning using, as input data, images obtained by performing a given conversion on images used as training data.

[0046] As a specific example, when a frame image rendered by an application program compatible with high resolution is used as training data, machine learning is performed using the image obtained by converting this frame image to a lower resolution as input data. This allows for the inverse conversion, i.e., high resolution, to be performed to obtain a machine learning model capable of restoring the original image. Similarly, when a conversion to expand the color gamut is desired, machine learning is performed using, as training data, a frame image rendered with a wide luminance range, such as HDR, and as input data, an image obtained by converting this frame image to a narrower luminance range, such as SDR. This allows for the generation of a machine learning model capable of conversion to a wide color gamut.

[0047] Although in the above description a single frame image is used as input to the machine learning model, the frame image conversion unit 23 may convert frame images using a machine learning model that accepts multiple frame images as input. In particular, by using a machine learning model that accepts multiple frame images drawn consecutively in chronological order as input, it is possible to convert the latest pre-conversion frame image into a post-conversion frame image, taking into account changes over time in the content of the frame images.

[0048] As described above, the image processing device 10 according to this embodiment can present the converted frame image obtained by effectively converting the content of the frame image to the user in real time.

[0049] It should be noted that the embodiments of the present invention are not limited to those described above. For example, the content of the conversion process performed by the frame image conversion unit 23 is not limited to the above examples, and various other conversions may be performed on the frame images.

[0050] In the above description, the image processing device 10 is an information processing device that is located relatively close to the user and is directly connected to the display device 14 and the operation device 15. However, this is not limited to this. For example, in services such as cloud gaming services, a server device connected to the client device via a communications network may render frame images to be displayed on the screen of the display device 14, rather than a client device directly connected to the display device 14 and operation device 15 used by the user. In such cases, the server device connected to the client device used by the user via a communications network may function as the image processing device 10 of the present invention. In this case, the server device functioning as the image processing device 10 executes an application program to convert the rendered pre-conversion frame images to generate converted frame images, and transmits the generated converted frame images to the client device.

[0051] Furthermore, in the above explanation, the image processing device 10 also executes an application program that draws pre-conversion frame images, but this is not limiting, and the image processing device 10 according to the embodiment of the present invention may acquire pre-conversion frame images drawn by another information processing device, convert them into post-conversion frame images, and send them to the display device 14 where they are viewed by the user. For example, in the above-mentioned cloud gaming service or the like, a server device may draw pre-conversion frame images and send the drawn pre-conversion frame images to a client device functioning as the image processing device 10, and the pre-conversion frame images received by the image processing device 10 may be converted into post-conversion frame images and presented to the user.

[0052] It should be noted that the functions provided by the components described herein may be implemented by any circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, configured or programmed to provide the described functions. A processor includes transistors and other circuits and is considered to be a circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0053] In this specification, a circuit, unit, or means is hardware that is programmed to realize a described function or that performs that function. The hardware may be any hardware disclosed in this specification or any hardware that is programmed to realize or known to perform the described function. If the hardware is a processor, which is considered to be a type of circuit, the circuit, means, or unit is a combination of hardware and software used to operate the hardware and / or processor.

[0054] The present disclosure may include the following aspects. [Item 1] An image processing device comprising a circuit configured to acquire a frame image constituting a video to be presented to a user each time the frame image is drawn, and to convert the acquired frame image using a previously prepared machine learning model to obtain a converted frame image, and output the converted frame image as the frame image to be presented to the user. [Item 2] The image processing device according to item 1, wherein the circuit performs a conversion to increase the resolution of the acquired frame image. [Item 3] The image processing device according to item 1, wherein the circuit performs a conversion to change the luminance of pixels included in the acquired frame image depending on the luminance of surrounding pixels. [Item 4] The image processing device according to item 1, wherein the circuit converts the colors of pixels included in the acquired frame image so that the color gamut used in the converted frame image is wider than the color gamut used in the acquired frame image. [Item 5] The image processing device according to item 1, wherein the circuit converts the colors of pixels included in the acquired frame image to have a color tone that corresponds to the color vision characteristics of a viewer. [Item 6] The image processing device according to item 1, wherein the circuit performs the conversion using a machine learning model selected based on given conditions from a plurality of machine learning models prepared in advance. [Item 7] The image processing device according to item 6, wherein the circuit performs the conversion using a machine learning model selected according to the type of application program that rendered the acquired frame image. [Item 8] The image processing device according to item 1, wherein the machine learning model is a model obtained by machine learning using, as input data and training data, one of two frame images obtained by rendering the same content under different rendering conditions.[Item 9] An image processing method, comprising: acquiring a frame image each time a frame image constituting a video to be presented to a user is drawn, and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user. [Item 10] A computer-readable, non-transitory information storage medium that stores a program for causing a computer to execute a process, comprising: acquiring a frame image each time a frame image constituting a video to be presented to a user is drawn, and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user.

[0055] REFERENCE SIGNS LIST 10 image processing device, 11 control unit, 12 storage unit, 13 interface unit, 14 display device, 15 operation device, 21 application execution unit, 22 frame image acquisition unit, 23 frame image conversion unit

Claims

1. An image processing device including: a frame image acquisition unit that acquires a frame image each time the frame image constituting an image to be presented to a user is drawn; and a frame image conversion unit that converts the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputs the converted frame image as the frame image to be presented to the user.

2. An image processing device according to claim 1, wherein the frame image conversion section converts the acquired frame images to have a higher resolution.

3. An image processing device according to claim 1, wherein the frame image conversion section performs conversion to change the brightness of pixels included in the acquired frame image in accordance with the brightness of surrounding pixels.

4. An image processing device according to claim 1, wherein the frame image conversion unit converts the colors of the pixels contained in the acquired frame image so that the color gamut used in the converted frame image is wider than the color gamut used in the acquired frame image.

5. An image processing device according to claim 1, wherein the frame image conversion unit converts the colors of pixels contained in the acquired frame image into colors that correspond to the color vision characteristics of the viewer.

6. An image processing device according to claim 1, wherein the frame image conversion unit performs the conversion using a machine learning model selected based on given conditions from a plurality of pre-prepared machine learning models.

7. An image processing device according to claim 6, wherein the frame image conversion unit performs the conversion using a machine learning model selected according to the type of application program that has drawn the acquired frame image.

8. An image processing device according to claim 1, wherein the machine learning model is a model obtained by machine learning using one of two frame images obtained by drawing the same content under different drawing conditions as input data and the other as training data.

9. An image processing method comprising: a step of acquiring a frame image each time the frame image constituting an image to be presented to a user is drawn; and a step of converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user.

10. A program for causing a computer to execute the steps of: acquiring a frame image each time the frame image constituting a video to be presented to a user is drawn; and converting the acquired frame image using a pre-prepared machine learning model to obtain a converted frame image, and outputting the converted frame image as the frame image to be presented to the user.

Images