Style transfer program and style transfer method

Abstract

To increase expressiveness of an image provided to a user.SOLUTION: A style transfer program causes a server to achieve an acquisition function that acquires buffer data from a buffer used in rendering, a style transfer function that applies a style transfer based on one or more style image to the buffer data, and an output function that outputs data after applying the style transfer.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] At least one of the embodiments of the present invention relates to a style transfer program and a style transfer method.

Background Art

[0002] There is known a style transfer technology for converting a photographic image into an image according to a predetermined style such as the style of Gogh or Monet.

[0003] Patent Document 1 describes style transfer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in video games and the like, there are a plurality of buffers used for rendering images. If style conversion can be performed in units of these buffers, the expressiveness of the images provided to the user will increase.

[0006] An object of at least one embodiment of the present invention is to solve the above problems and increase the expressiveness of the images provided to the user.

Means for Solving the Problems

[0007] From a non-limiting perspective, a style transfer program according to an embodiment of the present invention enables a server to have an acquisition function for acquiring buffer data from a buffer used for rendering, a style transfer function for applying a style transfer based on one or more style images to the buffer data, and an output function for outputting the data after the style transfer is applied.

[0008] From a non-limiting perspective, a style transfer method according to an embodiment of the present invention is a style transfer method by a computer, including an acquisition process for acquiring buffer data from a buffer used for rendering, a style transfer process for applying a style transfer based on one or more style images to the buffer data, and an output process for outputting the data after the style transfer is applied.

Advantages of the Invention

[0009] Each embodiment of the present application solves one or two or more deficiencies.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0011] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that various components in the examples of each embodiment described below can be appropriately combined as long as there is no contradiction. In addition, the content described as an example of a certain embodiment may be omitted in other embodiments. Also, operations and processes not related to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes constituting the various flows and sequences described below can be changed as long as there is no contradiction in the processing content.

[0012] [First Embodiment] The outline of the first embodiment of the present invention will be described. Hereinafter, as the first embodiment, a style transfer program executed on a server will be exemplified and described.

[0013] FIG. 1 is a block diagram showing an example of the configuration of a video game processing system corresponding to at least one of the embodiments of the present invention. The video game processing system 100 includes a video game processing server 10 (server 10) and a user terminal 20 used by a user (such as a game player) of the video game processing system 100. User terminals 20A, 20B, and 20C are each an example of the user terminal 20. The configuration of the video game processing system 100 is not limited to this. For example, the video game processing system 100 may be configured such that a single user terminal is used by a plurality of users. The video game processing system 100 may include a plurality of servers.

[0014] The server 10 and the user terminal 20 are each communicably connected to a communication network 30 such as the Internet. The connection between the communication network 30 and the server 10 and the connection between the communication network 30 and the user terminal 20 may be a wired connection or a wireless connection. For example, the user terminal 20 may be connected to the communication network 30 by performing data communication with a base station managed by a communication carrier via a wireless communication line.

[0015] The video game processing system 100 realizes various functions for executing various processes according to user operations by including a server 10 and user terminals 20.

[0016] The server 10 controls the progress of the video game. The server 10 is managed by the administrator of the video game processing system 100 and has various functions for providing information regarding various processes to a plurality of user terminals 20.

[0017] The server 10 includes a processor 11, a memory 12, and a storage device 13. The processor 11 is a central processing unit such as a CPU (Central Processing Unit) that performs various operations and controls, for example. Also, when the server 10 includes a GPU (Graphics Processing Unit), a part of various operations and controls may be performed by the GPU. The server 10 executes various information processes by the processor 11 using the data read into the memory 12, and stores the obtained processing results in the storage device 13 as necessary.

[0018] The storage device 13 has a function as a storage medium for storing various information. The configuration of the storage device 13 is not particularly limited, but from the viewpoint of reducing the processing load on the user terminal 20, it is preferably configured to be able to store all the various information necessary for the control performed in the video game processing system 100. Examples of such include HDDs and SSDs. However, the storage device for storing various information only needs to have a storage area in a state accessible by the server 10, and for example, it may be configured to have a dedicated storage area outside the server 10. The server 10 is managed by the administrator of the video game processing system 100 or the like and has various functions for providing information regarding various processes to a plurality of user terminals 20. The server 10 is configured by an information processing device such as a game server capable of rendering game images.

[0019] The user terminal 20 is managed by the user and is composed of a communication terminal capable of playing network-distributed games. Examples of communication terminals capable of playing network-distributed games include, for example, mobile phone terminals, PDAs (Personal Digital Assistants), portable game devices, VR goggles, AR glasses, smart glasses, so-called wearable devices, and the like. The configuration of the user terminal that the video game processing system 100 may include is not limited to these, and any configuration in which the user can recognize a composite image is acceptable. Other examples of the configuration of the user terminal include combinations of various communication terminals, personal computers, and stationary game devices.

[0020] The user terminal 20 is connected to the communication network 30 and is equipped with hardware (such as a display device for displaying a browser screen or a game screen according to coordinates) and software for executing various processes by communicating with the server 10. Note that each of the plurality of user terminals 20 may be configured to be able to communicate directly with each other without going through the server 10.

[0021] The user terminal 20 may have a built-in display device. Also, a display device may be wirelessly or wiredly connected to the user terminal 20. Since the display device has a very common configuration, its illustration is omitted here. The game screen is displayed by the display device, for example, as the aforementioned composite image, and the user recognizes this composite image. The game screen is displayed, for example, on a display which is an example of the display device included in the user terminal or on a display which is an example of the display device connected to the user terminal. The display device includes, for example, a hologram display device capable of hologram display and a projection device for projecting an image (including the game screen) onto a screen or the like.

[0022] The user terminal 20 includes a processor 21, a memory 22, and a storage device 23. The processor 21 is a central processing unit such as a CPU (Central Processing Unit) that performs various operations and controls. Also, when the user terminal 20 includes a GPU (Graphics Processing Unit), part of the various operations and controls may be performed by the GPU. The user terminal 20 uses the data read into the memory 22 to execute various information processes by the processor 21, and stores the obtained processing results in the storage device 23 as necessary. The storage device 23 has a function as a storage medium for storing various information.

[0023] The user terminal 20 may have a built-in input device. Also, an input device may be wirelessly or wiredly connected to the user terminal 20. The input device receives operation inputs from the user. In response to the operation inputs from the user, the processor included in the server 10 or the processor included in the user terminal 20 executes various control processes. Examples of the input device include a touch panel screen provided in a mobile phone terminal, a controller wirelessly or wiredly connected to AR glasses, etc. Also, the camera included in the user terminal 20 may correspond to an input device. The user performs an operation input by gestures such as moving a hand in front of the camera (gesture input).

[0024] In addition, the user terminal 20 may include other output devices such as a speaker. The other output devices output voice and other various information to the user.

[0025] Figure 2 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10A, which is an example of the configuration of the server 10, includes at least an acquisition unit 101, a style transfer unit 102, and an output unit 103. The processor included in the server 10A refers to the style transfer program held in the storage device and functionally realizes the acquisition unit 101, the style transfer unit 102, and the output unit 103 by executing the program.

[0026] The acquisition unit 101 has a function of acquiring buffer data from a buffer used for rendering. The style transfer unit 102 has a function of applying style transfer based on one or more style images to the buffer data. The output unit 103 has a function of outputting the data after the style transfer is applied.

[0027] Next, the program execution process in the first embodiment of the present invention will be described. FIG. 3 is a flowchart showing an example of a style transfer program process corresponding to at least one of the embodiments of the present invention.

[0028] The acquisition unit 101 acquires buffer data from a buffer used for rendering (St11). The style transfer unit 102 applies style transfer based on one or more style images to the acquired buffer data (St12). The output unit 103 outputs the data after the style transfer is applied (St13).

[0029] Style means, for example, a style or type in architecture, art, music, etc. Style may mean, for example, a painting style such as Van Gogh style or Picasso style. Style may also mean the format of an image (e.g., color, a predetermined pattern, or a pattern, etc.).

[0030] The buffer used for rendering means, for example, a buffer used by a rendering engine having a function of rendering a three-dimensional CG image.

[0031] The style transfer unit 102 may use a neural network for style transfer. Related technologies include, for example, Vincent Dumoulin, et.al. 「A LEARNED REPRESENTATION FOR ARTISTIC STYLE」. By inputting an input image of a predetermined size into the neural network by the style transfer unit 102, an output image to which style transfer is applied can be obtained.

[0032] The output destination of the data after applying the style transfer by the output unit 103 may be a buffer different from the buffer from which the acquisition unit 101 acquired the buffer data. More specifically, when the buffer from which the acquisition unit 101 acquired the buffer data is regarded as the first buffer, the output destination of the data after applying the style transfer may be a second buffer different from the first buffer. The second buffer may be a buffer used after the first buffer in the rendering process.

[0033] In addition, the output destination of the data after applying the style transfer by the output unit 103 may be an output device provided in the server 10A, or may be an external device as viewed from the server 10A.

[0034] As one aspect of the first embodiment, style transfer can be applied in units of buffers used for rendering. Thereby, the expressiveness of the image provided to the user can be increased.

[0035] As one aspect of the first embodiment, by incorporating the acquisition unit 101 and the style transfer unit 102, etc. into a game engine capable of operating the above-described buffer, style transfer can be performed based on richer information than data such as photographs used conventionally. For example, the game engine can recognize a specific object to be displayed on the game screen, and style transfer can be applied only to the buffer corresponding to the specific object.

[0036] As one aspect of the first embodiment, if the style transfer is applied to the buffer, the type of style transfer applied to each buffer can be appropriately selected, so that various CG expressions become possible. For example, it becomes newly possible to apply a style transfer that converts the style to a Gogh style to the first buffer and a style transfer that converts the style to a Gauguin style to the second buffer, and so on.

[0037] As one aspect of the first embodiment, by applying a style transfer to the buffer used for rendering, the style of the game image can be changed in real time.

[0038] [Second Embodiment] The outline of the second embodiment of the present invention will be described. Hereinafter, as the second embodiment, a style transfer program executed on a server will be exemplified and described. The server may be the server 10 included in the video game processing system 100 described in FIG. 1.

[0039] FIG. 4 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10B, which is an example of the configuration of the server 10, includes at least an acquisition unit 101B, a style transfer unit 102, and an output unit 103. The processor included in the server 10B refers to the style transfer program held in the storage device and functionally realizes the acquisition unit 101B, the style transfer unit 102, and the output unit 103 by executing the program.

[0040] The acquisition unit 101B has a function of acquiring buffer data from a 3D buffer used for rendering. The style transfer unit 102 has a function of applying a style transfer based on one or more style images to the buffer data. The output unit 103 has a function of outputting the data after the style transfer is applied.

[0041] Next, the program execution process in the second embodiment of the present invention will be described. FIG. 5 is a flowchart showing an example of a style transfer program process corresponding to at least one of the embodiments of the present invention.

[0042] The acquisition unit 101B acquires buffer data from the 3D buffer used for rendering (St21). The style transfer unit 102 applies a style transfer based on one or more style images to the acquired buffer data (St22). The output unit 103 outputs the data after the style transfer is applied (St23).

[0043] Style means, for example, a style or type in architecture, art, music, etc. The style may mean, for example, a painting style such as the style of Gogh or Picasso. The style may also mean the format of an image (for example, color, a predetermined pattern, or a pattern, etc.).

[0044] The 3D buffer used for rendering means, for example, a buffer that stores data capable of representing a three-dimensional space.

[0045] Since the style transfer unit 102 and the output unit 103 are the same as those in the first embodiment, detailed descriptions thereof are omitted.

[0046] As one aspect of the second embodiment, a style transfer can be applied to the 3D buffer used for rendering. Thereby, the style of the entire 3DCG map displayed to the user can be converted based on various conditions in the game and the like.

[0047] [Third Embodiment] The outline of the third embodiment of the present invention will be described. Hereinafter, as the third embodiment, a style transfer program executed on a server will be exemplified and described.

[0048] FIG. 6 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10C, which is an example of the configuration of the server 10, includes at least an acquisition unit 101C, a style transfer unit 102, and an output unit 103. The processor included in the server 10C functionally realizes the acquisition unit 101C, the style transfer unit 102, and the output unit 103 by referring to a style transfer program held in a storage device and executing the program.

[0049] The acquisition unit 101C has a function of acquiring buffer data from an intermediate buffer used for rendering. The style transfer unit 102 has a function of applying a style transfer based on one or more style images to the acquired buffer data. The output unit 103 has a function of outputting the data after the style transfer is applied.

[0050] Next, program execution processing in the third embodiment of the present invention will be described. FIG. 7 is a flowchart showing an example of style transfer program processing corresponding to at least one of the embodiments of the present invention.

[0051] The acquisition unit 101C acquires buffer data from an intermediate buffer used for rendering (St31). The style transfer unit 102 applies a style transfer based on one or more style images to the acquired buffer data (St32). The output unit 103 outputs the data after the style transfer is applied (St33).

[0052] Style means, for example, a style or type in architecture, art, music, etc. Style may mean, for example, a painting style such as the style of Gogh or Picasso. Style may mean the format of an image (e.g., color, a predetermined pattern, or a pattern, etc.).

[0053] The intermediate buffer used for rendering is a buffer used during the rendering process. Examples of intermediate buffers include an RGB buffer, a BaseColor buffer, a Metallic buffer, a Specular buffer, a Roughness buffer, a Normal buffer, and the like. These buffers are buffers arranged before the final buffer in which the finally output CG image is stored, and are different from the final buffer. The intermediate buffer used for rendering is not limited to the aforementioned enumerated buffers.

[0054] Since the style transfer unit 102 and the output unit 103 are the same as those in the first embodiment, detailed description thereof will be omitted.

[0055] As one aspect of the third embodiment, by applying a style transfer to the data stored in the intermediate buffer used for rendering, the data to which the style transfer has been applied can be obtained as the output of the intermediate buffer. Since a style transfer can be applied to some data in the middle stage of rendering, which is cut out from various viewpoints such as color information and light reflection information, it becomes possible to perform delicate style conversion and style control. Further, by generating a final image using the data to which the style transfer has been applied, the expressiveness of the final image provided to the user can be further increased.

[0056] [Fourth Embodiment] An overview of the fourth embodiment of the present invention will be described. Hereinafter, as the fourth embodiment, a style transfer program executed on a server will be exemplified and described.

[0057] FIG. 8 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10D, which is an example of the configuration of the server 10, includes at least an acquisition unit 101, a style transfer unit 102D, and an output unit 103. The processor included in the server 10D functionally realizes the acquisition unit 101, the style transfer unit 102D, and the output unit 103 by referring to a style transfer program held in a storage device and executing the program.

[0058] The acquisition unit 101 has a function of acquiring buffer data from a buffer used for rendering. The style transfer unit 102D has a function of applying a style transfer based on a plurality of style images to the acquired buffer data. The application of the style transfer is performed by inputting the buffer data to a learned neural network obtained by mixing parameters based on a plurality of style images in a predetermined layer of the neural network and performing an optimization process based on an optimization function defined based on the plurality of style images. The output unit 103 has a function of outputting the data after the style transfer is applied.

[0059] Next, the program execution process in the fourth embodiment of the present invention will be described. FIG. 9 is a flowchart showing an example of a style transfer program process corresponding to at least one of the embodiments of the present invention.

[0060] The acquisition unit 101 acquires buffer data from a buffer used for rendering (St41). The style transfer unit 102D applies a style transfer based on a plurality of style images to the acquired buffer data (St42). The output unit 103 outputs the data after the style transfer is applied (St43).

[0061] Style means a pattern or type in, for example, architecture, art, music, etc. Style may mean, for example, a painting style such as the style of Van Gogh or Picasso. Style may also mean the format of an image (e.g., color, a predetermined pattern, or a design, etc.).

[0062] The buffer used for rendering may be any one of the buffers described in the first to third embodiments.

[0063] The style transfer unit 102D has a function of applying a style transfer based on a plurality of style images to the acquired buffer data. The style transfer unit 102D performs a style transfer that blends a plurality of styles.

[0064] The style transfer unit 102D may use a neural network for style transfer. As related technologies, for example, there are Vincent Dumoulin, et.al. 「A LEARNED REPRESENTATION FOR ARTISTIC STYLE」, etc.

[0065] In the fourth embodiment, a plurality of styles are blended and applied. For this purpose, parameters based on a plurality of style images are mixed in a predetermined layer of the neural network during learning. The predetermined layer may be, for example, an affine layer (fully connected layer).

[0066] In the fourth embodiment, further, optimization processing is performed on the neural network based on an optimization function defined based on a plurality of style images. By this optimization processing, a learned neural network that is more suitable for a plurality of style images is obtained.

[0067] The style transfer unit 102D inputs an input image of a predetermined size into a learned neural network, thereby obtaining an output image to which style transfer is applied. In the fourth embodiment, the input image corresponds to the above-described buffer data.

[0068] Since the output unit 103 is the same as that in the first embodiment, a detailed description thereof will be omitted.

[0069] As one aspect of the fourth embodiment, not only parameters based on a plurality of style images are mixed in a predetermined layer of the neural network, but also optimization processing is performed based on an optimization function defined based on the plurality of style images. Thereby, using the learned neural network obtained by this optimization processing, it is possible to realize a style transfer in which a plurality of style images are beautifully blended.

[0070] [Fifth Embodiment] An overview of the fifth embodiment of the present invention will be described. Hereinafter, as the fifth embodiment, a style transfer program executed on a server will be exemplified and described.

[0071] FIG. 10 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10Y, which is an example of the configuration of the server 10, includes at least an acquisition unit 101Y, a style transfer unit 102Y, an output unit 103Y, and a style image selection unit 104Y. The processor included in the server 10Y refers to a style transfer program held in a storage device and executes the program to functionally realize the acquisition unit 101Y, the style transfer unit 102Y, the output unit 103Y, and the style image selection unit 104Y.

[0072] The acquisition unit 101Y has a function of acquiring buffer data from a buffer used for rendering. The style transfer unit 102Y has a function of applying style transfer based on one or more style images to the acquired buffer data. The output unit 103Y has a function of outputting the data after the style transfer is applied. The style image selection unit 104Y has a function of selecting the one or more style images based on a predetermined condition.

[0073] Next, the program execution process in the fifth embodiment of the present invention will be described. FIG. 11 is a flowchart showing an example of style transfer program processing corresponding to at least one of the embodiments of the present invention.

[0074] The style image selection unit 104Y selects one or more style images based on a predetermined condition (St51). The acquisition unit 101Y acquires buffer data from a buffer used for rendering (St52). The style transfer unit 102Y applies style transfer based on the one or more style images selected in step St51 to the acquired buffer data (St53). The output unit 103Y outputs the data after the style transfer is applied (St54).

[0075] The predetermined condition may be various conditions in a video game controlled by the server 10Y. Examples of the predetermined condition include, for example, the emotion of a character appearing in the ongoing video game, the type of technique or magic used by the character, buffs or debuffs, and state changes of the character such as level up. Note that the character may be a player character (PC) or a non-player character (NPC). The character may be a friendly character or an enemy character.

[0076] The predetermined conditions may be conditions related to objects other than characters in an ongoing video game. Examples of objects include buildings, creatures placed on the map of the video game, items that appear during the progress of the video game, and the like. The predetermined conditions can be various, such as whether an object has appeared, how many times an object has appeared, whether an object has disappeared from the map due to being destroyed, and so on.

[0077] The predetermined conditions may be conditions related to meta information or system information in an ongoing video game. An example of a condition related to meta information in an ongoing video game is a change in the in-game stage where the player character is located (for example, a stage move to a future city). An example of a condition related to system information in an ongoing video game is the play time of the video game by the user.

[0078] The predetermined conditions are not limited to the above, and may be various other conditions.

[0079] Style means, for example, a style or type in architecture, art, music, etc. Style may, for example, mean a painting style such as the style of Van Gogh or Picasso. Style may also mean the format of an image (for example, color, a predetermined pattern, or a pattern, etc.).

[0080] The buffer used for rendering means, for example, a buffer used by a rendering engine having a function of rendering a three-dimensional CG image.

[0081] The buffer used for rendering may be a 3D buffer. The 3D buffer used for rendering means, for example, a buffer that stores data capable of representing a three-dimensional space.

[0082] The buffer used for rendering may be an intermediate buffer. The intermediate buffer used for rendering is a buffer used during the rendering process. Examples of intermediate buffers include an RGB buffer, a BaseColor buffer, a Metallic buffer, a Specular buffer, a Roughness buffer, a Normal buffer, and the like. These buffers are buffers arranged before the final buffer in which the finally output CG image is stored, and are different from the final buffer. The intermediate buffer used for rendering is not limited to the buffers listed above.

[0083] The style transfer unit 102Y may use a neural network for style transfer. Related technologies include, for example, Vincent Dumoulin, et.al. 「A LEARNED REPRESENTATION FOR ARTISTIC STYLE」. By inputting an input image of a predetermined size into the neural network by the style transfer unit 102Y, an output image to which style transfer is applied can be obtained.

[0084] FIG. 12 is a conceptual diagram showing a structural example of a neural network N1 used for style transfer corresponding to at least one of the embodiments of the present invention. The neural network N1 includes a first conversion layer that converts a pixel group based on an input image into latent parameters, one or more layers that perform downsampling by convolution or the like, a plurality of residual block layers, a layer that performs upsampling, and a second conversion layer that converts latent parameters into a pixel group. Note that an output image is obtained based on the pixel group that is the output of the second conversion layer.

[0085] A fully connected layer is arranged between the first conversion layer of the neural network N1 and the layer that performs downsampling, or between a plurality of convolutional layers included in the layer that performs downsampling. The fully connected layer is also called an Affine layer.

[0086] The style transfer unit 102Y inputs the buffer data acquired by the acquisition unit 101Y into the first conversion layer of the neural network N1. As a result, the data after style transfer application is output from the second conversion layer of the neural network N1.

[0087] The output destination of the data after style transfer application by the output unit 103Y may be a buffer different from the buffer from which the acquisition unit 101Y acquired the buffer data. More specifically, when the buffer from which the acquisition unit 101Y acquired the buffer data is regarded as the first buffer, the output destination of the data after style transfer application may be a second buffer different from the first buffer. The second buffer may be a buffer used after the first buffer in the rendering process.

[0088] In addition, the output destination of the data after style transfer application by the output unit 103Y may be an output device provided in the server 10Y, or may be an external device as viewed from the server 10Y.

[0089] As one aspect of the fifth embodiment, style transfer can be applied in units of buffers used for rendering. Thereby, the expressiveness of the image provided to the user can be increased.

[0090] As one aspect of the fifth embodiment, by incorporating the acquisition unit 101Y, the style transfer unit 102Y, etc. into a game engine capable of operating the above-described buffer, style transfer can be performed based on richer information than data such as photos used conventionally. For example, the game engine can recognize a specific object displayed on the game screen and apply style transfer only to the buffer corresponding to the specific object.

[0091] As one aspect of the fifth embodiment, if the configuration is such that style transfer is applied to the buffer, the type of style transfer applied to each buffer can be appropriately selected, enabling a variety of CG expressions. For example, it becomes newly possible to apply a style transfer that converts the style to a Gogh style to the first buffer and a style transfer that converts the style to a Gauguin style to the second buffer, etc.

[0092] As one aspect of the fifth embodiment, by applying style transfer to the buffer used for rendering, the style of the game image can be changed in real time.

[0093] As one aspect of the fifth embodiment, style transfer can be applied to the 3D buffer used for rendering. Thereby, the style of the entire 3D CG map displayed to the user can be converted based on various conditions in the game, etc.

[0094] As one aspect of the fifth embodiment, by applying style transfer to the data stored in the intermediate buffer used for rendering, the data to which the style transfer has been applied can be obtained as the output of the intermediate buffer. Since style transfer can be applied to some of the data in the middle stage of rendering, which is cut out from various viewpoints such as color information and light reflection information, fine style conversion and style control can be performed. In addition, by generating the final image using the data to which the style transfer has been applied, the expressiveness of the final image provided to the user can be further increased.

[0095] As one aspect of the fifth embodiment, different style transfers can be applied to the above-mentioned buffer according to various conditions related to an ongoing video game or the like. For example, when the buffer is a buffer corresponding to an object appearing in a video game, different style transfers can be applied for each object. When the buffer is a 3D buffer, for example, based on the emotions (laughing, crying, anger, etc.) of the characters appearing in the game, the touch of the entire map can be changed. The touch of the entire map can also be changed based on the types of skills and magic used by the characters during the game. The touch of the entire map may also be changed based on the state changes of the characters. When the buffer is an intermediate buffer, based on the emotions of the characters, the types of magic, state changes, etc., the color (BaseColor buffer) controlled by the intermediate buffer, whether to make the surface metallic (Metallic buffer), the degree of light reflection (Specular buffer), roughness (Roughness buffer), etc. can be individually converted based on the target style image.

[0096] [Sixth Embodiment] The outline of the sixth embodiment of the present invention will be described. Hereinafter, as the sixth embodiment, a style transfer program executed on a server will be exemplified and described.

[0097] FIG. 13 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. A server 10Z, which is an example of the configuration of the server 10, includes at least an acquisition unit 101Z, a style transfer unit 102Z, and an output unit 103Z. The processor included in the server 10Z functionally realizes the acquisition unit 101Z, the style transfer unit 102Z, and the output unit 103Z by referring to a style transfer program held in a storage device and executing the program.

[0098] The acquisition unit 101Z has a function of acquiring buffer data from a buffer used for rendering. The style transfer unit 102Z has a function of applying a style transfer based on a plurality of style images to the buffer data. The output unit 103Z has a function of outputting the data after the style transfer is applied.

[0099] Next, the program execution process in the sixth embodiment of the present invention will be described. FIG. 14 is a flowchart showing an example of a style transfer program process corresponding to at least one of the embodiments of the present invention.

[0100] The acquisition unit 101Z acquires buffer data from a buffer used for rendering (St61). The style transfer unit 102Z applies a style transfer based on a plurality of style images to the acquired buffer data (St62). The output unit 103Z outputs the data after the style transfer is applied (St63).

[0101] Style means, for example, a style or type in architecture, art, music, etc. Style may mean, for example, a painting style such as the style of Gogh or Picasso. Style may also mean the format of an image (e.g., color, a predetermined pattern, or a pattern, etc.).

[0102] The buffer used for rendering means, for example, a buffer used by a rendering engine having a function of rendering a three-dimensional CG image.

[0103] The buffer used for rendering may be a 3D buffer. The 3D buffer used for rendering means, for example, a buffer that stores data capable of representing a three-dimensional space.

[0104] The buffer used for rendering may be an intermediate buffer. The intermediate buffer used for rendering is a buffer used during the rendering process. Examples of intermediate buffers include an RGB buffer, a BaseColor buffer, a Metallic buffer, a Specular buffer, a Roughness buffer, a Normal buffer, and the like. These buffers are buffers arranged before the final buffer in which the finally output CG image is stored, and are different from the final buffer. The intermediate buffer used for rendering is not limited to the aforementioned enumerated buffers.

[0105] Note that the acquisition unit 101Z may acquire data to which the style transfer is to be applied from a location other than the buffer used for rendering. For example, the acquisition unit 101Z may acquire data to which the style transfer is to be applied from the memory 12 or an external device as viewed from the server 10Z. The acquired data is typically image data, but may be other types of data (for example, audio data, etc.).

[0106] The application of the style transfer by the style transfer unit 102Z (step St62) is performed by inputting buffer data into a learned neural network obtained by mixing parameters based on a plurality of style images in a predetermined layer of the neural network and performing an optimization process based on an optimization function defined based on the plurality of style images. Hereinafter, the learned neural network will be described.

[0107] FIG. 15 is a conceptual diagram showing an example of the structure of a neural network N2 used for style transfer, corresponding to at least one of the embodiments of the present invention. The neural network N2 includes a first conversion layer that converts a pixel group based on an input image into latent parameters, one or more layers that perform downsampling by convolution or the like, a plurality of residual block layers, a layer that performs upsampling, and a second conversion layer that converts latent parameters into a pixel group. Note that an output image is obtained based on the pixel group that is the output of the second conversion layer.

[0108] A fully connected layer is arranged between the first conversion layer of the neural network N2 and the layer that performs downsampling, or between a plurality of convolutional layers included in the layer that performs downsampling. The fully connected layer is also called an affine layer.

[0109] Parameters based on a plurality of style images are mixed into the affine layer A1 of the neural network N2. More specifically, it is as follows.

[0110] The affine layer A1 of the neural network N2 is a layer that performs a process of converting the latent variable x of the output of the convolutional layer into x*a + b, where a and b are the parameters of the affine transformation and x is the latent variable of the pixel of the image.

[0111] Here, when blending any style 1 and style 2, under the control of the style transfer unit 102Z, the process performed by the affine layer A1 is as follows. Let the affine transformation parameters derived from the style image related to style 1 be a 1 and b 1 Let the affine transformation parameters derived from the style image related to style 2 be a 2 and b 2Let's assume. At this time, when blending Style 1 and Style 2, the affine transformation parameters are \(a=(a 1 +a 2 ) / 2\) and \(b=(b 1 +b 2 ) / 2\). Then, by calculating \(x*a + b\) in the affine layer A1, the blending of Style 1 and Style 2 can be performed. Note that the above shows the calculation formula for the case of blending Style 1 and Style 2 evenly (50% each). Based on the ordinary knowledge of those skilled in the art, it may also be blended after weighting so that the influence degrees based on each style are different ratios, such as 80% for Style 1 and 20% for Style 2.

[0112] The number of styles to be blended may be 3 or more. When \(n\) is a natural number of 3 or more, the affine transformation parameters for blending \(n\) styles are, for example, \(a=(a 1 +a 2 ……+a n ) / n\) and \(b=(b 1 +b 2 ……+b n ) / n\). Regarding the point that it may also be blended after weighting so that the influence degrees based on each style are different ratios, it is the same as the case where the number of styles is 2 described above.

[0113] In the memory 12 etc. of the server 10Z, the conversion parameters \(a k and \(b k (k is an arbitrary natural number between 1 and n) may be stored. Also, the conversion parameters for a plurality of styles may be stored in the memory 12, the storage device 13, etc. in vector form, such as \((a 1 ,a 2 ,……,a n )\) and \((b 1 ,b 2 ,……,b n ). When weighting is performed so that the influence degrees based on each style are different ratios, the values indicating the weights corresponding to each style may also be stored in the memory 12, the storage device 13, etc.

[0114] Next, an optimization function for performing machine learning on the neural network N2 will be described. The optimization function is sometimes also referred to as a loss function. For the neural network N2, by performing an optimization process based on an optimization function defined based on a plurality of style images, a learned neural network N2 is obtained. For convenience of explanation, the same reference numeral N2 is used for each neural network before and after learning.

[0115] For example, in the above-described related art, an optimization function defined as follows is used.

[0116] Style optimization function:

Equation

[0117] Content optimization function:

Equation

[0118] In the above optimization function, p represents the generated image. The generated image corresponds to the output image of the neural network used for machine learning. s (lowercase s) represents a style image such as an abstract painting. U i represents the total number of units in the i-th layer. U j represents the total number of units in the j-th layer. G represents the Gram matrix. φ i represents the output of the i-th activation function of the VGG-16 architecture. S (uppercase S) represents the group of layers of VGG-16 for calculating the style optimization. c (lowercase c) represents the content image. C (uppercase C) is the group of layers of VGG-16 for calculating the content optimization function, and j is the index of the layer included in the layer group. F with the absolute value symbol means the Frobenius norm.

[0119] Machine learning is performed on the neural network to minimize the value of the optimization function defined by the above style optimization function and content optimization function, and by inputting the input image into the learned neural network, an output image is output from the neural network, which is transformed to approach the style shown in the style image.

[0120] Here, in the optimization process using the optimization function as described above, when performing style transfer by blending a plurality of styles, the result of the blend has room for improvement.

[0121] Therefore, in the sixth embodiment of the present invention, an optimization process is performed based on an optimization function defined based on a plurality of style images. Thereby, optimization based on a plurality of style images can be performed. As a result, an output image in which a plurality of styles are beautifully blended with respect to the input image can be obtained.

[0122] More specifically, the optimization process may include a first optimization process of performing an optimization process using a first optimization function defined based on any two style images selected from a plurality of style images, and a second optimization process of performing an optimization process using a second optimization function defined based on one style image among the plurality of style images. Thereby, when the number of styles to be blended is 3 or more, suitable optimization can be performed. As a result, an output image in which a plurality of styles are more beautifully blended with respect to the input image can be obtained.

[0123] Next, the first optimization function and the second optimization function will be described. As one aspect of the sixth embodiment, the first optimization function may be defined by the following formula (1).

[0124]

Equation

[0125] As one aspect of the sixth embodiment, the second optimization function may be defined by the following formula (2).

[0126]

Number

[0127] In the above formula,

[0128]

Number

[0129] is a style image group consisting of a plurality of style images, and q and r represent any style images included in the style image group. However, q and r are different style images from each other. N i,r is φ i is the number of rows of the feature map. N i,c is φ i is the number of columns of the feature map. p, s (lowercase s), G, φ i , S, c (lowercase c), and F are the same as those in the related technologies described above.

[0130] When the above first optimization function uses the generated image as p and any two style images selected from a plurality of style images as q and r, it is a function that sums the norms between the value obtained by performing a predetermined operation on the image p and the average value of the values obtained by performing the predetermined operation on the style images q and r respectively. The above formula (1) shows the case where the predetermined operation is

[0131]

Number

[0132] The predetermined operation may be an operation other than the above.

[0133] When the above-mentioned second optimization function takes the generated image as p and the style image as s, it is a function that sums the norms between the value obtained by performing a predetermined operation on the image p and the value obtained by performing the predetermined operation on the style image s. The above formula (2) shows the case where the predetermined operation is

[0134]

Number

[0135] This is shown. The predetermined operation may be an operation other than the above.

[0136] Next, an example of the optimization process using the above-mentioned first optimization function and second optimization function will be described.

[0137] FIG. 16 is a flowchart showing a processing example of the optimization process corresponding to at least one of the embodiments of the present invention. Here, a processing example will be described in the case where the first optimization function is the function defined by the above formula (1) and the second optimization function is the function defined by the above formula (2).

[0138] The processing entity of the optimization process is the processor provided in the device. The device provided with the processor (hereinafter, device A) may be the above-mentioned server 10Z. In this case, the processor 11 shown in FIG. 1 becomes the processing entity. The device A provided with the processor may be another device other than the server 10Z (for example, the user terminal 20 or another server, etc.).

[0139] Let the number of styles to be blended be n. The processor selects any two style images q and r from the n style images included in the style image group (St71).

[0140] The processor performs optimization to minimize the value of the first optimization function for the selected style images q and r (St72). For the generated image p, the processor obtains the output image of the neural network as image p. The neural network may be implemented in device A or in another device other than device A.

[0141] The processor n C 2 determines whether optimization has been performed for all patterns as shown (St73). That is, the processor determines whether all patterns have been processed for selecting any two style images q and r from among the n style images. n C 2 If optimization has been performed for all patterns as shown (St73: YES), the process transitions to step St74. n C 2 If optimization has not been performed for all patterns as shown (St73: NO), the process returns to step St71, and the processor selects the next combination of two style images q and r.

[0142] The processor selects one style image s from among the n style images included in the style image group (St74).

[0143] The processor performs optimization to minimize the value of the second optimization function for the selected style image s (St75). For the generated image p, the processor obtains the output image of the neural network as image p. The neural network may be implemented in device A or in another device other than device A.

[0144] The processor n C 1 determines whether optimization has been performed for all patterns as shown (St76). That is, the processor determines whether all patterns have been processed for selecting any style image s from among the n style images.n C 1 When optimization is performed for all patterns of the street (St76: YES), the optimization process shown in FIG. 16 ends. n C 1 When optimization has not been performed for all patterns of the street (St76: NO), the process returns to step St74, and the processor selects the next single style image s.

[0145] The style transfer unit 102Z inputs the buffer data acquired by the acquisition unit 101Z to the first conversion layer of the learned neural network N2 optimized as described above. As a result, data after application of style transfer in which n style images are beautifully blended is output from the second conversion layer of the neural network N2.

[0146] In the case of image data output based on the optimization process in related technologies, the blending results of a plurality of styles were blurred and only the colors were averaged. On the other hand, in the case of the output image output based on the above-described optimization process according to the sixth embodiment of the present invention, the colors and patterns are beautifully blended.

[0147] The output destination of the data after application of style transfer by the output unit 103Z may be a buffer different from the buffer from which the acquisition unit 101Z acquired the buffer data. More specifically, when the buffer from which the acquisition unit 101 acquired the buffer data is the first buffer, the output destination of the data after application of style transfer may be a second buffer different from the first buffer. The second buffer may be a buffer used after the first buffer in the rendering process.

[0148] In addition, the output destination of the data after application of style transfer by the output unit 103Z may be an output device provided in the server 10Z, or may be an external device as viewed from the server 10Z.

[0149] As one aspect of the sixth embodiment, in style transfer based on a plurality of style images, by performing optimization processing based on an optimization function defined based on the plurality of style images, it is possible to improve the quality of style blending in the output image.

[0150] As one aspect of the sixth embodiment, after performing optimization for any two style images, by performing optimization for each style image, it is possible to further improve the quality of style blending in the output image.

[0151] As one aspect of the sixth embodiment, by taking the average of the respective values derived from two style images and then calculating the norm between the generated image, it is possible to perform balanced optimization for the two style images.

[0152] As one aspect of the sixth embodiment, since the first optimization function is a function defined by the above formula (1), it is possible to obtain an output image in which colors and patterns are beautifully blended according to a plurality of style images.

[0153] As one aspect of the sixth embodiment, after selecting and optimizing two style images, by also performing optimization for each style image, it is possible to perform balanced optimization for each of the respective style images.

[0154] As one aspect of the sixth embodiment, since the second optimization function is a function defined by the above formula (2), it is possible to obtain an output image in which colors and patterns are beautifully blended according to each style image.

[0155] As described above, one or more deficiencies are solved by each embodiment of the present application. Note that the effects of each embodiment are non-limiting effects or examples of effects.

[0156] In each of the above-described embodiments, the user terminal 20 and the server 10 execute the various processes described above in accordance with various control programs (e.g., style transfer programs) stored in the storage devices they each include. Also, other computers not limited to the user terminal 20 or the server 10 may execute the various processes described above in accordance with various control programs (e.g., style transfer programs) stored in the storage devices they each include.

[0157] Moreover, the configuration of the video game processing system 100 is not limited to the configurations described as examples in the above-described embodiments. For example, a configuration may be adopted in which the server 10 executes part or all of the processes described as being executed by the user terminal, or a configuration may be adopted in which the user terminal 20 executes part or all of the processes described as being executed by the server 10. Also, a configuration may be adopted in which part or all of the storage unit (storage device) included in the server 10 is included in the user terminal 20. That is, in the video game processing system 100, part or all of the functions provided by either the user terminal or the server may be provided by the other.

[0158] Also, the program may be configured to implement part or all of the functions described as examples in the above-described embodiments in a single device that does not include a communication network.

[0159] [Appendix] The description of the above-described embodiments has been provided so that at least the following invention can be practiced by a person having ordinary skill in the art to which the invention pertains. [1] In a server, an acquisition function for acquiring buffer data from a buffer used for rendering, a style transfer function for applying style transfer based on one or more style images to the buffer data, and an output function for outputting the data after the style transfer has been applied. A style transfer program for realizing [2] The buffer is a 3D buffer, The style transfer program according to [1]. [3] The buffer is an intermediate buffer, The style transfer program according to [1]. [4] On the server, A style image selection function for selecting the one or more style images based on a predetermined condition The style transfer program according to any one of [1] to [3] for realizing. [5] In the style transfer function, a function of applying style transfer based on a plurality of style images to the buffer data is realized, The application of the style transfer is performed by mixing parameters based on a plurality of style images in a predetermined layer of a neural network and inputting the buffer data into a learned neural network obtained by performing optimization processing based on an optimization function defined based on the plurality of style images. The style transfer program according to any one of [1] to [4]. [6] The optimization processing includes a first optimization processing of performing optimization processing using a first optimization function defined based on any two style images selected from the plurality of style images, A second optimization processing of performing optimization processing using a second optimization function defined based on one style image among the plurality of style images. The style transfer program according to [5]. [7] The first optimization function is A function that sums the norms between a value obtained by performing a predetermined operation on an image p, where p is the generated image, and the average value of values obtained by performing the predetermined operation on each of the style images q and r, where q and r are any two style images selected from the plurality of style images. The style transfer program according to [6]. [8] The first optimization function is

Number

[10] The second optimization function is

Number

[11] A server installed with the style transfer program according to any one of [1] to

[10] .

[12] A computer installed with the style transfer program according to any one of [1] to

[10] .

[13] A style transfer method by a computer, comprising: An acquisition process of acquiring buffer data from a buffer used for rendering, Apply style transfer based on one or more style images to the buffer data, a style transfer process, and an output process that outputs the data after the style transfer is applied. A style transfer method.

Industrial Applicability

[0160] According to one embodiment of the present invention, it is useful as a style transfer program and a style transfer method for increasing the expressiveness of an image provided to a user.

Explanation of Signs

[0161] 10, 10A, 10B, 10C, 10D, 10Y, 10Z Servers 11 Processor 12 Memory 13 Storage Device 20, 20A, 20B, 20C User Terminals 21 Processor 22 Memory 23 Storage Device 30 Communication Network 100 Video Game Processing System 101, 101B, 101C, 101Y, 101Z Acquisition Units 102, 102D, 102Y, 102Z Style Transfer Units 103, 103Y, 103Z Output Units 104Y Style Image Selection Unit N1, N2 Neural Networks

Claims

1. A style transfer program for causing a server to have an acquisition function of acquiring buffer data from a buffer used for rendering, a style transfer function of applying a style transfer based on one or more style images to the buffer data, and an output function of outputting data after the style transfer is applied, wherein in the style transfer function, a function of applying a style transfer based on a plurality of style images to the buffer data is realized, the application of the style transfer is performed by inputting the buffer data to a trained neural network obtained by mixing parameters based on a plurality of style images in a predetermined layer of the neural network and performing an optimization process based on an optimization function defined based on the plurality of style images, the optimization process includes a first optimization process of performing an optimization process using a first optimization function defined based on any two style images selected from the plurality of style images, and a second optimization process of performing an optimization process using a second optimization function defined based on one style image among the plurality of style images, the style transfer program.

2. The buffer is a 3D buffer, The style transfer program according to Claim 1.

3. The buffer is an intermediate buffer, The style transfer program according to Claim 1.

4. A style transfer program according to any one of Claims 1 to 3 for causing the server to have a style image selection function of selecting the one or more style images based on a predetermined condition.

5. A server installed with the style transfer program according to any one of Claims 1 to 4.

6. A style transfer method by a computer, the method including an acquisition process of acquiring buffer data from a buffer used for rendering, a style transfer process of applying a style transfer based on one or more style images to the buffer data, and an output process of outputting data after the style transfer is applied. In the style transfer process, a process of applying style transfer based on a plurality of style images to the buffer data is performed. The application of the style transfer is performed by inputting the buffer data into a learned neural network obtained by mixing parameters based on a plurality of style images in a predetermined layer of the neural network and performing an optimization process based on an optimization function defined based on the plurality of style images. The optimization process includes a first optimization process of performing an optimization process using a first optimization function defined based on any two style images selected from the plurality of style images, and a second optimization process of performing an optimization process using a second optimization function defined based on one style image among the plurality of style images. Style transfer method.

Images