How to learn

Abstract

To perform drawing with suppressed delay on a terminal side by efficiently utilizing a server-side rendering system.SOLUTION: A terminal 10 includes a second drawing unit 13 that obtains second drawing information at each time by performing drawing in a second drawing mode according to content drawing specification information at each time, and includes a generation unit 15 that receives first drawing information at each time at which the drawing was performed from a server 20 performing drawing in a first drawing mode with quality higher than in the second drawing mode according to the content drawing specification information at each time, and generates, as one obtained by imitating the first drawing information drawn by the server at the current time, generation information corresponding to a drawing result in the first drawing mode according to the content drawing specification information at the current time while using, as input, three pieces of drawing information composed of pieces of second drawing information at the current time and the past time and a piece of first drawing information at the past time, which has already been received at the current time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a terminal that performs drawing efficiently using a server-side rendering method, and a method and program for learning parameters of the terminal. [Background technology]

[0002] In an augmented reality system that estimates the relative position and orientation between an imaging target and an imaging unit and presents related information, if the information can be presented in real time and with high quality, user convenience can be improved. An example of a conventional technology for realizing this augmented reality is disclosed in Patent Document 1, which discloses the following method.

[0003] Patent Document 1 discloses a method in which an image of an object is captured by an imaging unit installed in a server, related information is drawn based on the result of recognizing the captured object in the image information, and the drawing result is transmitted to a terminal and presented on the terminal. At this time, the related information can be drawn with high quality by utilizing the high-performance computing resources of the server. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-44655 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, the server-side rendering method is adopted to utilize the abundant computing resources of the server for rendering, but since it takes time to transmit video, there is a problem that a delay occurs before related information is presented on the terminal.

[0006] In view of the problems with the conventional technology described above, the first object of the present invention is to provide a terminal that efficiently utilizes a server-side rendering method to generate server information corresponding to terminal information at the terminal, thereby reducing the effect of delay. The second object of the present invention is to provide a method for learning parameters of the terminal. The third object of the present invention is to provide a program corresponding to the terminal. [Means for solving the problem]

[0007] In order to achieve the above-mentioned first object, the present invention has a first feature in that it is a terminal comprising a second drawing unit that obtains second drawing information at each time by drawing in a second drawing mode in accordance with content drawing specification information at each time, and a generation unit that receives the first drawing information at each drawn time from a server that draws in a first drawing mode that is higher quality than the second drawing mode in accordance with the content drawing specification information at each time, and uses as input three pieces of drawing information, namely the second drawing information at the current time and a past time, and the first drawing information at the past time that has been received at the current time, and generates generation information equivalent to the drawing result in the first drawing mode in accordance with the content drawing specification information at the current time, as an imitation of the first drawing information drawn by the server at the current time.

[0008] In order to achieve the second object, the generation unit of the terminal uses a generator out of a generator and a classifier in a pre-trained generative adversarial network to generate the generated information using the three pieces of drawing information as input. The present invention is a method for learning parameters of the generator and the classifier in the generative adversarial network, comprising: second drawing information at each time that is prepared by performing drawing in a second drawing mode according to content drawing designation information at each time; and first drawing information at each time that is prepared by performing drawing in a first drawing mode that is higher in quality than the second drawing mode according to the content drawing designation information at each time. a second input for the generator to learn parameters so as to generate generated information simulating the first drawing information corresponding to the second time from the first input for learning, which is the first drawing information corresponding to the first time, the second input for learning, which is the second drawing information corresponding to the first time, and the third input for learning, which is the second drawing information corresponding to the second time; and a second input for the classifier to learn parameters so as to determine authenticity, assuming that the first drawing information and the second drawing information for learning are genuine and that the generated information generated by the generator is a counterfeit. Effect of the Invention

[0009] According to the first feature, by generating generated information from the three inputs as an imitation of the first drawing information at the current time, it is possible to efficiently utilize a server-side rendering method and reduce the effect of delay. According to the second feature, by learning parameters of a generator and a classifier in a generative adversarial network used in the terminal, it is possible to generate high-quality generated information as an imitation of the first drawing information. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of an information system according to an embodiment (including a schematic example of drawing information). [Diagram 2]FIG. 2 is a functional block diagram of a terminal and a server of an information system according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of an operation sequence at each discrete time t (t=1, 2, . . . ) in real time in an information system. [Figure 4] FIG. 2 is a functional block diagram of a generating unit according to an embodiment. [Diagram 5] 1 is a flowchart of an operation of an information system according to an embodiment. [Figure 6] FIG. 13 is a diagram showing examples of network configurations for determining correctness for training a classifier as configurations 6a to 6h. [Figure 7] FIG. 13 shows configurations 7a and 7b for evaluating the loss function Lpixel1. [Figure 8] FIG. 13 is a diagram showing configurations 8a and 8b for evaluating the loss function Lpixel2. [Figure 9] FIG. 13 shows configurations 9a and 9b for evaluating the loss function Lpixel3. [Figure 10] FIG. 1 shows configurations 10a and 10b for evaluating the loss function Llm. [Figure 11] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Fig. 1 is a configuration diagram (including a schematic example of drawing information) of an information system 100 according to an embodiment, and the information system 100 is composed of a terminal 10 and a server 20 that are capable of communicating with each other via a network NW such as the Internet. In addition to the system configuration, Fig. 1 also shows schematic examples of drawing information used by the terminal 10 and the server 20 in explanation columns CL1 and CL2, respectively, and these schematic examples will be referred to as appropriate in the following explanation.

[0012] The overall operation of the information system 100 is to capture an image of an object on the terminal 10 at each time t (t=1, 2, ...) in real time according to a predetermined display processing rate, such as 30 fps (frames per second) or 120 fps, to perform drawing processing according to the captured object on the terminal 10 and the server 20, and to present the drawing results in real time on the terminal 10, thereby providing an augmented reality display or virtual reality display according to the captured object to the user U using the terminal 10.

[0013] As an explanatory example, the following will explain a case where the subject to be imaged is the user U himself / herself who is using the terminal 10 and is present in a field F such as a room, and the drawing process according to the imaged subject is drawing process of the user U's avatar, that is, an avatar that moves in real time in conjunction with the facial expression and / or body pose of the user U (an avatar that reflects the facial expression and pose of the user U in real time). In the explanatory columns CL1 and CL2, schematic examples of the avatar are shown for the face only.

[0014] 2 is a functional block diagram of the terminal 10 and the server 20 of the information system 100 according to an embodiment. As shown in the functional block configuration, the terminal 10 includes an imaging unit 11, a recognition unit 12, a second drawing unit 13, a storage unit 14, a generation unit 15, and a presentation unit 16, and the server 20 includes a first drawing unit 21.

[0015] As shown in FIG. 2, at each time t in real time, the following processing is performed in each functional unit.

[0016] The imaging unit 11 outputs a captured image P(t) obtained by capturing an image to the recognition unit 12. The recognition unit 12 outputs recognition information r(t) obtained by recognizing the captured image P(t) to the second drawing unit 13 and the first drawing unit 21 (here, the output to the first drawing unit 21 is transmitted from the terminal 10 to the server 20 via the network NW). The second drawing unit 13 obtains second drawing information y(t) by drawing an avatar of the user U according to the recognition information r(t), and outputs the second drawing information y(t) to the storage unit 14.

[0017] The first drawing unit 21 obtains first drawing information x(t) by drawing an avatar of the user U according to the recognition information r(t), and outputs the first drawing information x(t) to the generation unit 15 (here, the output to the generation unit 15 is transmitted from the server 20 to the terminal 10 via the network NW). The generation unit 15 receives as input the second drawing information y(t) at the current time t and the second drawing information y(t-τ) at a past time t-τ (τ>0) stored in the storage unit 14, and the first drawing information x(t-τ) at a past time t-τ obtained from the first drawing unit 21, generates generated information x'(t) as an imitation of the first drawing information x(t) drawn by the first drawing unit 21 at the current time t, and outputs the generated information to the presentation unit 16. (Note that the technical significance of the processing in the generation unit 15 will be further explained in Figures 3 to 5 described later.) The presentation unit 16 displays this generation information x'(t) (which imitates the first drawing information x(t) and is the drawing result of the avatar of the user U).

[0018] FIG. 3 is a diagram showing an example of an operation sequence at each of the real-time discrete times t (t=1, 2, ...) in the information system 100. In FIG. 3, the processing steps in the information system 100 at any time t, which is a real-time processing timing, are shown together with the flow of information exchange occurring in each process of each functional unit shown in FIG. 2. In FIG. 3, the vertical lines attached to each functional unit of the terminal 10 and the server 20 shown in FIG. 2 are a timeline of the processing in each functional unit, with the downward direction representing a common time progression, and the bar area on the vertical line is a schematic representation of a time period during which the processing is performed in the functional unit, and in one embodiment, is a schematic representation of a continuous time period during which the processing is completed within the interval of each real-time discrete time t (t=1, 2, ...). In FIG. 3, the horizontal arrows represent the flow of information exchange between each functional unit.

[0019] In Fig. 3, the processing at past time t-τ (τ>0) and current time t used in the description of Fig. 2 are shown as a sequence. As shown in the schematic example of Fig. 3, due to the presence of processing delays (time required to complete processing) and transmission delays (particularly, transmission delays in information transmission via the network NW between the terminal 10 and the server 20) in each processing unit, when the generating unit 15 generates the generated information x'(t) as an avatar drawing result from the captured image P(t) captured at the current time t in the terminal 10, the latest first drawing information transmitted from the first drawing unit 21 of the server 20 and received by the terminal 10 is not the first drawing information x(t) at the current time t but the first drawing information x(t-τ) (τ>0) at the past time t-τ. In other words, the past time t-τ is a past time (time stamp) associated with the latest first drawing information received by the terminal 10 at the current time t.

[0020] Fig. 4 is a functional block diagram of the generation unit 15 according to an embodiment, and the generation unit 15 includes a generator G, a discriminator D, and a calculator V as an optional configuration. Fig. 5 is a flowchart of the operation of the information system 100 according to an embodiment.

[0021] The real-time avatar drawing in the information system 100 described above with reference to FIGS. 1 to 3 is performed in step S2 in FIG. 5, and step S1 exists as a preliminary step to make this possible.

[0022] In step S1, parameters of the generator G (and the discriminator D) of the generating unit 15 are trained in advance in accordance with the framework of GAN (generative adversarial network), which is an existing method in the field of deep learning. Using the parameters trained in step S1, inference processing by the generator G of the generating unit 15 becomes possible in step S2, and avatar drawing and presentation in real time in the information system 100 becomes possible.

[0023] As shown in Figure 4 (and the description in step S2 of Figure 5), during inference, the parameter-learned generator G of the generation unit 15 receives as input three pieces of drawing information, namely, the second drawing information y(t) and y(t-τ) at the current time t and past time t-τ, and the first drawing information x(t-τ) at past time t-τ, and generates generated information x'(t) at the current time t that imitates the first drawing information x(t) at the current time t.

[0024] As will be described in detail later, at each time t (including past time t-τ), the first drawing unit 21 and the second drawing unit 13 input the same recognition information r(t) (information representing the facial expression and pose of the user U) and draw the avatar of the user U in the same state (same state with respect to the facial expression, pose, and virtual camera position and orientation (position and orientation of the viewpoint serving as the drawing reference) used for drawing) to obtain the first drawing information x(t) and the second drawing information y(t), respectively, but the first drawing information x(t) and the second drawing information y(t) differ only in drawing quality. That is, the first drawing information x(t) is drawn with high quality by utilizing abundant computational resources of the server 20 realized as a business server machine or the like, whereas the second drawing information y(t) is drawn with standard quality (lower quality compared to the server 20 side) by utilizing limited computational resources of the terminal 10 realized as a general user's smartphone or the like.

[0025] Regarding this difference in drawing quality in the same state drawing (same content drawing), a schematic example is shown in the explanatory columns CL1 and CL2 in Fig. 1. The first drawing information x(t), x(t-τ) and the second drawing information y(t), y(t-τ) drawn for the avatar (face only) of the user U are the same in terms of the avatar's facial expression (the facial expression determined by reflecting the recognition information used for drawing) and the viewing angle (the angle determined by the virtual camera arrangement for drawing) at the respective times t and t-τ, but, for example, in terms of the light source model used, the first drawing information is drawn with a light source model that has a higher calculation load, and therefore the first drawing information is drawn with higher quality than the second drawing information.

[0026] Based on the above-mentioned quality difference between the first drawing information and the second drawing information, the generator G at the time of inference generates generated information x'(t) using the three pieces of drawing information y(t-τ), y(t), and x(t-τ) as input. The significance of using these three pieces of drawing information as input is as follows. (Note that the application relationship of the following processing information mod and high-quality information high is shown diagrammatically in the explanatory column CL1 in FIG. 1.)

[0027] (1) From two pieces of second drawing information y(t-τ) and y(t), which are standard-quality drawing results, it is possible to estimate information mod that processes the avatar drawing result by, for example, transforming it from a state y(t-τ) at past time t-τ to a state y(t) at the current time t. Similarly, from two pieces of standard-quality and high-quality drawing information y(t-τ) and x(t-τ), it is possible to estimate information high that improves the quality of the avatar drawing result from the standard-quality state y(t-τ) to the high-quality state x(t-τ).

[0028] (2) By processing a past piece of first drawing information x(t-τ), which is a high-quality drawing result, with this processing information mod, and by improving a current piece of second drawing information y(t), which is a standard-quality drawing result, with this high-quality information high, it is possible to generate generated information x'(t) that imitates the first drawing information x(t) at the current time t.

[0029] The processing information mod and the high-quality information high are not information that can be obtained explicitly (explicitly), but as a result of the generator G and the discriminator D being mutually adversarially trained in accordance with the GAN framework, processing equivalent to the estimation and application of the processing information mod and the high-quality information high is performed implicitly within the generation process of the generator G. (That is, the generator G of this embodiment is, so to speak, parameter-trained to precisely generate x'(t) that is a "fake" of x(t), which is an "authentic work" by the first drawing unit 21, using three inputs y(t-τ), x(t-τ), and y(t) as "models." Similarly, the discriminator D is qualitatively trained to provide x(t) and x'(t) (without distinguishing which is the "authentic work" x(t) and which is the "fake" x'(t)) and to accurately discern the "authenticity.")

[0030] Therefore, during parameter learning by the GAN of the generator G and the discriminator D in step S1, as shown in Fig. 4 (and the description in step S1 of Fig. 5), in addition to the three inputs y(t-τ), x(t-τ), and y(t) used during inference in step S2, the first drawing information x(t) at the current time t is also used as evaluation data (evaluation data for the generation accuracy of so-called "fake" generation in the generator G, and evaluation data for the discrimination accuracy of so-called "authenticity" in the discriminator D). The details of the learning of the generator G and the discriminator D in step S1 will be described later.

[0031] As described above, in this embodiment, the terminal 10 performs only standard quality drawing, which has a low computational load, in the second drawing unit 13, and does not perform high quality drawing, which has a high computational load, in the first drawing unit 21. Instead, the generator G (generation unit 15) performs inference processing (which has a low computational load) using three inputs y(t-τ), x(t-τ), and y(t), so that even if the high quality first drawing information x(t) has not been received, it is possible to generate generated information x'(t) that imitates the first drawing information x(t) with sufficient accuracy and display it in real time.

[0032] The details of the processing of each part of the information system 100 during the inference in step S2 will be described below.

[0033] <Imaging unit 11> The imaging unit 11 captures an image of the real world (field F in FIG. 1) in which an object to be recognized by the downstream recognition unit 12 exists, and outputs a captured image P(t) obtained at each time t in real time to the recognition unit 12. The imaging unit 11 can be configured as a camera in terms of hardware, and an image is captured by the imaging unit 11 when a user U or the like using the terminal 10 operates the camera. The object to be captured can be, for example, a marker, a plane, or a person. As described above, when the purpose is to draw an avatar of the user U in the information system 100, the imaging unit 11 can capture an image of the user U himself as the object to be captured.

[0034] <Recognition part 12> The recognition unit 12 recognizes the type of the object captured in the captured image P(t) obtained by the imaging unit 11, and obtains recognition information r(t) as a result of calculating the coordinates of the object. The recognition information r(t) is output to the second drawing unit 13 and transmitted to the first drawing unit 21 of the server 20 via the communication function. That is, the recognition information r(t) output to the first drawing unit 13 and the second drawing unit 21 is composed of information on the type of the object captured in the captured image P(t) and information on the coordinates of the object. (In the case where the image capture target is uniquely set in advance as the user U, such as in the case of drawing an avatar of the user U in the information system 100, the information on the type of the object may be omitted from the recognition information r(t).) Note that the recognition information r(t) is linked to information on the time t of the capture as information obtained from the captured image P(t) at time t.

[0035] Any existing method used in existing augmented reality display technology, etc., can be used for the process of recognizing the object type of the target captured from the captured image P(t) in the recognition unit 12 and the process of calculating the coordinates of the target. For example, feature points and feature amounts such as SIFT feature information are detected from the image, and a comparison is made with feature information registered in advance for one or more object types that serve as references, and the object type whose feature information is determined to be most consistent through the comparison is taken as the recognition result of the target, and the coordinates of the target may be obtained as a transformation relationship that gives the correspondence relationship between image coordinates of feature points that match during this comparison.

[0036] As is known in the field of three-dimensional computer graphics, the coordinates of the object thus obtained are in a three-dimensional camera coordinate system in the camera as the hardware constituting the imaging unit 11. Furthermore, a square marker or the like arranged to define a reference position in the field F may also be found in the three-dimensional camera coordinate system, and information corresponding to the external parameters of the camera may be obtained by determining the conversion relationship between the coordinates of the object in a predetermined three-dimensional world coordinate system of the field F and the coordinates of the object in the three-dimensional camera coordinate system.

[0037] Alternatively, in the case where the subject is a user U as a person and the information system 100 is intended to draw an avatar of the user U, the recognition unit 12 may recognize the facial expression, posture, etc. of the person captured in the captured image P(t) and use the recognition information r(t). For the facial expression, parameters of a blend shape expressed by weighting a plurality of mesh models may be used, and for the posture, parameters expressing the coordinates of each joint of the person may be used. These joint coordinates may be given in the above-mentioned three-dimensional camera coordinate system (or a three-dimensional world coordinate system based on a correspondence with a reference position marker, etc.), so that the avatar can be spatially arranged in a virtual three-dimensional space or an augmented reality space and drawn (drawn in the first drawing unit 21 or the second drawing unit 13).

[0038] <First drawing unit 21>...High-quality drawing in server 20 The first drawing unit 21 obtains first drawing information x(t) by drawing a three-dimensional model as a virtual object for augmented reality display or virtual reality display on a two-dimensional image plane based on the recognition information r(t) obtained from the recognition unit 12, and transmits this first drawing information x(t) to the generation unit 15 of the terminal 10 via the network NW. Here, based on the object type in the recognition information r(t), a predetermined three-dimensional model corresponding to this object type, which is prepared in advance by an administrator or the like as content corresponding to the use such as augmented reality display realized by the information system 100 and stored on the terminal 10 or the network NW, is read, and the three-dimensional model is arranged at the coordinates in the recognition information r(t) and drawn on the two-dimensional image plane, thereby obtaining the first drawing information x(t). As is known in the field of three-dimensional computer graphics as a relationship of a viewing pipeline, drawing on this two-dimensional image plane can be performed using known internal parameters for the camera constituting the imaging unit 11 as hardware.

[0039] <Second drawing unit 13>...Standard quality drawing in terminal 10 Similar to the processing in the first drawing unit 21, the second drawing unit 13 obtains second drawing information y(t) by drawing a three-dimensional model as a virtual object for augmented reality display or virtual reality display on a two-dimensional image plane based on the recognition information r(t) obtained from the recognition unit 12, and outputs this second drawing information y(t) to the memory unit 14.

[0040] As already explained in the explanation of the overall overview of the information system 100 with reference to Figs. 1 to 5, here, when drawing is performed in the second drawing unit 13, the predetermined three-dimensional model of the drawing target that is referred to according to the object type of the recognition information r(t) and the manner of reflecting the coordinates when drawing are the same as those in the first drawing unit 21. However, there is a difference in that the second drawing unit 13 draws the same three-dimensional model as that drawn by the first drawing unit 21 at the same coordinates, but draws in a drawing mode different from the quality of the first drawing unit 21. In other words, the first drawing information x(t) is drawn in a first drawing mode that gives the first drawing quality, and the second drawing information y(t) is drawn in a second drawing mode that gives the second drawing quality.

[0041] Various embodiments are possible for distinguishing between the first rendering mode that provides a first rendering quality in the first rendering unit 21 and the second rendering mode that provides a second rendering quality that is lower in the second rendering unit 13 (in exchange for which the calculation load for rendering is also low). For example, the first rendering mode and the second rendering mode can be distinguished and provided by using different light source models and reflection models.

[0042] Normally, the first rendering mode provides a higher quality rendering result than the second rendering mode, but since the server-side rendering method is adopted to obtain this high-quality rendering result as shown in the configuration example of the information system 100 in FIG. 1 and the like, it is assumed that it takes time to transmit the first rendering information x(t) from the server 20 to the terminal 10. In the schematic example of FIG. 3, as an example based on this assumption, it can be seen that the time required for the first rendering unit 21 to render and transmit the first rendering information x(t-τ) (the time from the reception of the recognition information r(t-τ) to the reception of the first rendering information x(t-τ) in the generation unit 15) is longer than the time required for the second rendering unit 13 to render the second rendering information y(t), y(t-τ) (the time from the reception of the recognition information r(t), r(t-τ) to the output of the second rendering information y(t), y(t-τ)).

[0043] <Storage section 14> The memory unit 14 is realized as hardware using a storage medium, and stores the second drawing information y(t) for each time t drawn by the second drawing unit 13. When the first drawing information x(t-τ) for a time t-τ that is earlier than the current time t drawn by the first drawing unit 21 is input (received) to the generation unit 15 at the current time t, the memory unit 14 outputs to the generation unit 15 the second drawing information y(t-τ) (or y(t-τ')) drawn with the same recognition information r(t-τ) (or similar recognition information r(t-τ')) as the recognition information r(t-τ) used to draw the first drawing information x(t-τ), and the latest second drawing information y(t).

[0044] The following description will be given on the assumption that the "second drawing information y(t-τ) drawn with the same recognition information r(t-τ)" is output from the storage unit 14 to the generation unit 15. The second drawing information y(t-τ) drawn with the same recognition information can be specified by the past time t-τ, which is the timestamp of the first drawing information x(t-τ) received by the generation unit 15. (When outputting the second drawing information y(t-τ') drawn with similar recognition information, it is sufficient to search for recognition information similar to the recognition information r(t-τ) corresponding to the past time t-τ, which is the timestamp of the first drawing information x(t-τ) received by the generation unit 15.)

[0045] In another embodiment, the first drawing information corresponding to a plurality of pieces of recognition information may be stored in advance or sequentially, thereby reducing or eliminating communication with the server 20 and reducing the amount of transmission. For example, by sequentially storing the received first drawing information x(t-τ') in association with the recognition information r(t-τ'), the recognition information r(t) at the current time t can reuse the first drawing information x(t-τ') similar to the recognition information r(t), and the generation unit 15 can omit the process of simulating and generating the original first drawing information x(t) as the generated information x'(t) (i.e., the first drawing information x(t-τ') that has been received and is determined to be sufficiently similar is treated as the generated information x'(t), thereby omitting the generation process). In addition, the drawing interval of the first drawing information x(t) in the first drawing unit 21 of the server 20 can be made more intermittent than the drawing interval of the second drawing information y(t) in the second drawing unit 13 of the terminal 10. The determination of whether or not to omit the processing in the generation unit 15 and the first drawing unit 21 for the current time t by using similar first drawing information x(t-τ') may be determined based on whether or not the similarity between the recognition information r(t) and r(t-τ') is high in a threshold determination. The threshold in this determination may be a strict value (a threshold value at which the similarity is sufficiently high) that it can be determined whether the recognition information r(t) and r(t-τ') are sufficiently similar to be considered to be almost the same.

[0046] <Generation part 15> The generating unit 15 generates generated information x'(t) from three inputs as explained in the outline description. That is, the generating unit 15 receives the first drawing information x(t-τ), the second drawing information y(t-τ) drawn by the same recognition information r(t-τ) as the recognition information r(t-τ) used to draw the first drawing information x(t-τ), and the second drawing information y(t) drawn by the latest recognition information r(t) at the current time t, generates information simulating the first drawing information x(t) corresponding to the latest recognition information r(t), and outputs the information to the presenting unit 16 as generated information x'(t).

[0047] As explained in the overall overview with reference to Figures 1 to 5, the generation unit 15 is capable of generating generated information x'(t) that imitates the first drawing information x(t) at the current time t from the above three inputs using a generator G that has been trained in advance in accordance with the GAN framework.

[0048] <Presentation part 16> The presentation unit 16 is configured as hardware including a display, and displays the generated information x'(t) obtained from the generation unit 15 (or, when the processing of the generation unit 15 is omitted, the past first drawing information x(t-τ') having similar recognition information obtained instead from the storage unit 14) at each time t, thereby realizing real-time virtual reality display or augmented reality display. In this case, when the hardware of the presentation unit 16 is configured as a normal display including a video see-through display, the generated information x'(t) may be displayed superimposed on the captured image P(t) obtained by the imaging unit 11 as a background for the generated information x'(t), or another image may be used as the background of the virtual reality space. In addition, when the hardware is configured as an optical see-through display, only the generated information x'(t) may be displayed superimposed on the background of the real space (field F in FIG. 1).

[0049] As already explained, in this embodiment, by generating generated information x'(t) that approximates the first drawing information x(t) corresponding to the latest recognition information r(t) from the past first drawing information x(t-τ) received by the terminal 10, the second drawing information y(t) corresponding to the latest recognition information r(t), and the second drawing information y(t-τ) in the past t-τ, it is possible to suppress transmission delays and processing delays and achieve smooth information presentation.

[0050] As described above, according to one embodiment of the present invention (an embodiment relating to the inference at step S2), high-quality first drawing information x(t) is rendered by server-side rendering in a server 20 with abundant computational resources, and in order to compensate for the delay when transmitting it via the network NW, the terminal 10 renders second drawing information y(t) of the quality possible with its computational resources, and then reflects the second drawing information y(t), y(t-τ) in the received first drawing information x(t-τ), thereby making it possible to generate generated information x'(t) that simulates the first drawing information x(t) without directly rendering the high-quality first drawing information x(t) in the terminal 10 itself.

[0051] In other words, according to one embodiment of the present invention, by utilizing the abundant computational resources of a server using a server-side rendering method, it is possible to achieve high-quality augmented reality display with low latency on a terminal with limited computational resources, without putting a strain on network bandwidth.

[0052] The details of the processing of the information system 100 during inference in step S2 have been described above. Below, details of the processing of each unit of the information system 100 during learning in step S1 (i.e., during parameter learning of the generator G and the discriminator D in the generation unit 15) will be described.

[0053] In step S1, the generator G and discriminator D parameters of the generation unit 15 are learned by GAN. At this time, the parameters of the generator G and the parameters of the discriminator D are alternately learned according to the framework of a generative model such as GAN, using a large amount of first drawing information x(t), x(t-τ) and second drawing information y(t), y(t-τ) at the present t and past t-τ as learning data.

[0054] Therefore, as learning data, recognition information time series data {r(t)|t=1,2,...,N} sufficiently covering the range in which an avatar or the like to be drawn can be drawn (in the case of an avatar, the range determined by the position, posture, facial expression, etc.) is prepared, and a high-quality first drawing information time series {x(t)|t=1,2,...,N} and a standard-quality second drawing information time series {y(t)|t=1,2,...,N} are prepared by applying the processes of the first drawing unit 21 and the second drawing unit 13 to the input data. From these two time series data {x(t)}, {y(t)}, the first drawing information x(t), x(t-τ) and the second drawing information y(t), y(t-τ) for many combinations of t and τ are picked up as learning data from these two time series data {x(t)}, {y(t)}, and the parameters of the generator G and the discriminator D are alternately learned according to the framework of a generative model such as GAN, and the parameters at the time when convergence is determined are regarded as the learning result. Convergence may be determined by whether the number of learning attempts has reached a threshold, whether the value of the evaluation function during learning has exceeded a threshold, or whether the change in the value of the evaluation function has become small.

[0055] In the following, for the sake of simplicity, a specific learning will be described with τ fixed at 1. For example, a GAN can be used as the generative model. When a GAN is used, a generator G and a classifier D constituting the GAN can be alternately trained using the loss function of the following equation (Equation 1).

[0056]

number

[0057] Here, L GAN (G,D) is the loss function of GAN, L pixel1 (G), L pixel2 (G), L pixel3 (G) is the pixel-level loss function, L lm (G) represents the feature-level loss function, γ pixel1 ,γ pixel2 ,γ pixel3 ,γ lmIn this embodiment, the loss function L GAN In addition to (G,D), various loss terms L pixel1 (G), L pixel2 (G), L pixel3 (G), L lm The addition of (G) is a notable feature.

[0058] When training the discriminator D, the parameters of the generator G are fixed, so one loss function L, which is a term that depends on D in Equation 1, is GAN Using only (G,D), maximize the value of this loss function (max D ), the parameters of the classifier D can be updated so as to improve the ability of the generator G to detect counterfeits.

[0059] When training the generator G, the parameters of the discriminator D are fixed, and the value of this loss function is minimized (min G ), the parameters of generator G can be trained so as to improve the generator G's ability to generate counterfeits.

[0060] L, which is part of the loss function GAN (G,D) can be defined by the following equation (Equation 2). E represents an expected value, G represents a generator under parameters as a function G of the first to third inputs (first input, second input, third input), D represents a discriminator under parameters as a function D of the first and second inputs (first input, second input), and w0 to w6 represent weighting coefficients. (Note that although the time notation of the drawing information is shown in the form of a function argument such as x(t) in the text and drawings, in the formulas, it is shown as x in order to simplify the display.) t In other words, "x in a formula" is written in subscript form. t " is the same as "x(t)" in the text and drawings.)

[0061]

number

[0062] During learning (when a negative example is the correct answer in the discriminator D shown in FIG. 6 described later, and in the cases of FIGS. 7 to 9), the generator G is given the first drawing information as the first input, the second drawing information having the same recognition information as the first drawing information as the second input, and the second drawing information as the third input, and is trained to generate generated information that imitates the first drawing information corresponding to the third input. (Accordingly, during inference, the generator G of the generation unit 15 is used with the first input being x(t-τ), the second input being y(t-τ), and the third input being y(t), and the generator G generates generated information x'(t) that imitates the first drawing information x(t) corresponding to the third input y(t).) Note that in FIGS. 6 to 10 showing the network configuration during learning described later, the first, second, and third inputs to the generator G are shown in order from the top as three input arrows to the left side (input stage side) of the block of the generator G.

[0063] Classifier D is given two pieces of drawing information (one is the first drawing information and the other is the second drawing information) as two inputs, and judges whether the correspondence between the two pieces of drawing information is correct or not based on whether the recognition information corresponding to the two pieces of drawing information is the same. (This judgment can be made based on whether the times associated with the two pieces of input drawing information are the same.)

[0064] FIG. 6 is a diagram showing examples of network configurations of correct / incorrect judgment for training the discriminator D (and the generator G) as configurations 6a to 6h. Of these multiple configurations 6a to 6h, the one corresponding to what is picked up as training data is selected and used to update the parameters of the discriminator D and the generator G. Note that the configurations 6a, 6b, 6c, and 6d are configurations in which only the discriminator D is trained by providing two inputs to the discriminator D and making it output a correct / incorrect judgment as shown in the figure, and the configurations 6e, 6f, 6g, and 6h are configurations in which the generator G is placed in front of the discriminator D, the first output simulated by providing three inputs to the generator G is used as the first input of the discriminator D, and the third input of the generator G is also used as the second input of the discriminator D, thereby simultaneously training the discriminator D and the generator G.

[0065] That is, in this embodiment, the learning of the discriminator D can use the following two types of evaluation (1) and (2) in order to improve the authenticity determination performance for the information generated by the generator G. Either one or both may be used. The evaluation (2) may be used for learning the generator G in combination with the configurations of FIGS. 7 to 10.

[0066] (1) In the configurations 6a, 6b, 6c, and 6d shown in FIG. 6, the first drawing information and the second drawing information are given to the classifier D as two inputs for learning, and the classifier D learns to output the correct answer as to whether or not the recognition information of the two pieces of drawing information is the same (whether or not it is at the same time) (if they are the same, the positive example (positive, real) is the correct answer, and if they are different, the negative example (no, fake) is the correct answer). (That is, if the classifier D outputs the correct answer for the two inputs, the value of the loss function becomes large, and if it is not the correct answer (the output is wrong), the value of the loss function becomes small. Note that, as shown in Equation 1, the loss function represents the loss from the perspective of the generator G, so for the classifier D, which is in an adversarial position to the generator G, the larger this loss is, the higher the classification performance becomes.)

[0067] (2) In the configurations shown by configurations 6e, 6f, 6g, and 6h in FIG. 6, the following first to third inputs (note that the same three inputs are used in FIGS. 7 to 10 as well) are given as three inputs for learning to a generator G arranged in the preceding stage, and the output is obtained as intermediate data (referred to as the fourth input). In a classifier D arranged in the subsequent stage, the fourth and third inputs are treated as two inputs, and the classifier D is trained to always output “no” as the correct answer regardless of whether these two inputs are the same recognition information (same time). First input: First drawing information Second input: Second drawing information with the same recognition information (same time) as the first input Third input: Second drawing information (first input and recognition information (time) are the same or different)

[0068] That is, in the above (2), unlike the above (1), even if the same recognition information (same time) is associated with the fourth input and the third input, which are two inputs to the discriminator D, the correct answer is "No." This is because the fourth input, which is the first input to the discriminator D, is the output of the generator G and corresponds to a counterfeit (while the second input, the third input, is an authentic work).

[0069] As shown in the figure, the components 6a to 6h are as follows. These eight components 6a to 6h are arranged in the order of the alphabets a to h suffixed to the reference numerals, GAN These correspond to the eight terms of the linear sum that appears in (G,D).

[0070] In the configurations 6a and 6b corresponding to the above (1), when the first drawing information and the second drawing information drawn with the same recognition information (i.e., the same time) are input (i.e., when two inputs of x(t) and y(t) or two inputs of x(t-1) and y(t-1) are input), the configurations learn to output as true (Real). On the other hand, in the configurations 6c and 6d, when the second drawing information with recognition information different from the first drawing information (i.e., different time) is input (i.e., when two inputs of x(t-1) and y(t) or two inputs of x(t) and y(t-1) are input), the configurations learn to output as false (Fake).

[0071] In the configurations 6e and 6f corresponding to the above (2), a generator G is used to generate generated information simulating the first drawing information corresponding to the different recognition information from the second drawing information having the same recognition information as the first drawing information (i.e., the same time) and the second drawing information corresponding to the different recognition information, and the generator G is trained to output the generated information as "Fake" even when the second drawing information having the same recognition information is input to the discriminator D. For example, in the configuration 6e, three inputs are given to the generator G in order as x(t-1), y(t-1), and y(t) to output the simulated generated information x'(t), and the output generated information x'(t) of the generator G is set as the first input of the discriminator D, and the third input y(t) of the generator G is set as the second input of the discriminator D, and the discriminator D is trained to output the generated information as "Fake" since the two inputs of the discriminator D are x'(t) and y(t).

[0072] Similarly, configurations 6g and 6h use a generator G to generate generated information that simulates the first drawing information corresponding to the second drawing information from the second drawing information having the same recognition information (same time) as the first drawing information, and learn to output as "fake" even when second drawing information having matching recognition information is input. In either case, the performance of the discriminator D is improved by training the discriminator D to reject the drawing information generated by the generator G.

[0073] As described above, in the configuration of Figure 6, the input of the discriminator D is learned to output a positive result only in the cases of configurations 6a and 6b (when both of the two inputs consisting of the first drawing information and the second drawing information are "genuine" (output of the first drawing unit 21 or the second drawing unit 13) and have the same time (same recognition information) (configurations 6a and 6b), and to output a negative result in all other cases (when both of the two inputs are "genuine" but have different times (configurations 6c and 6d), or when the two inputs have the same time but one is a "fake" (configurations 6e to 6h)). This can improve the performance of the discriminator D (and the generator G).

[0074] A configuration specialized for learning the generator G will be described below with reference to Figs. 7 to 10. Note that, with respect to the generator G, the first input x(t-1), the second input y(t-1), and the third input y(t) are input during inference, and a time constraint that the time of the first input and the second input is past and the time of the third input is present is automatically imposed for the purpose of inference described above (for the purpose of simulating and generating the first drawing information x(t) not yet received from the server 20 side in the terminal 10). On the other hand, during learning, it is not necessary to impose this time constraint on the three inputs to the generator G, and it is sufficient to use the first input and the second input as the first drawing information and the second drawing information given at a common time, and the third input as the second drawing information given at an arbitrary time, in order to enable learning of the generator G by efficiently utilizing the learning data. (In other words, the time of the third input may be future than the common time of the first and second inputs (as in the case of inference), or may be earlier than the first and second inputs, unlike the case of inference, or may be the same time as the first and second inputs.)

[0075] In addition, for the three inputs used for learning to the generator G, depending on their time relationship (the relationship between the times of the first and second inputs (common time) and the time of the third input), a corresponding configuration among the configurations in Figures 7 to 10 (and Figure 6) can be used as follows. (1) When the times of the first and second inputs are different from the time of the third input, all or part of the configurations in FIGS. 7, 9, and 10 (and configurations 6e and 6f in FIG. 6) may be used. (2) When the time of the first and second inputs is the same as the time of the third input, all or part of the configuration in FIG. 8 (and configurations 6g and 6h in FIG. 6) may be used.

[0076] In any of the configurations in Figures 7 to 9, the first drawing information, which is the first input of the three inputs to the generator G (the generator G on the front side in Figure 7), is output as generated information (output in the generator G on the rear side in Figure 7) as a simulation at the relevant time (the time of the first input or the time of the third input), and this output is compared with the initial first input to evaluate the generation accuracy (the more similar the output is to the first input, the lower the value of the loss function is evaluated). The configuration in Figure 10 performs evaluation along a similar line via an added calculator V.

[0077] L pixel1 can be defined by the following formula (Math. 3). Figure 7 shows configurations 7a and 7b that are evaluated by the following formula. (Note that Math. 3 shows configuration 7a, but by switching the letters, configuration 7b can be defined in exactly the same way.)

[0078]

number

[0079] In configuration 7a, first, a generator G on the front stage is used to generate generated information x'(t) simulating the first drawing information x(t) corresponding to the second drawing information y(t) at the current time (which are the first input x(t-1), the second input y(t-1), and the third input y(t) respectively) from the first drawing information x(t-1) and the second drawing information y(t-1) at the same time, and the generated information is used as the first input of the generator G on the rear stage. Next, a generator G on the rear stage is used to generate generated information x'(t-1) simulating the first drawing information x(t-1) corresponding to the second drawing information y(t-1) (which are the first input x'(t), the second input y(t), and the third input y(t-1) respectively) from the generated information x'(t) and the second drawing information y(t) at the same time. Finally, as shown by the dotted arrow in Fig. 7, in order to make the generated information x'(t-1) and the first drawing information x(t-1) match (the output x'(t-1) of the generator G on the latter stage matches the first input x(t-1) of the generator G on the former stage), the sum of squared errors between them at the pixel level is set as the loss function. The sum of squared errors is as shown in Equation 3.

[0080] The sum of squared errors when time t-1 and time t are swapped in configuration 7a can also be used as the loss function, as shown in configuration 7b. The sum of both configurations 7a and 7b can also be used as the loss function. It is expected that the performance of generator G can be improved by repeatedly generating generated information x'(t-1) from generated information x'(t) and calculating the loss.

[0081] L pixel2 can be defined by the following equation (Equation 4). Figure 8 shows configurations 8a and 8b that are evaluated by this equation. (Note that Equation 4 shows configuration 8a, but by switching the letters, configuration 8b can be defined in exactly the same way.)

[0082]

number

[0083] In configuration 8a, first, using a generator G, generated information x'(t-1) that simulates the first drawing information x(t-1) corresponding to the second drawing information y(t-1) (the first input x(t-1), the second input y(t-1), and the third input y(t-1)) is generated from the second drawing information y(t-1) at the same time as the first drawing information x(t-1). Next, as shown by the dotted arrow, in order to make the generated information x'(t-1) and the first drawing information x(t-1) match (the output x'(t) of the generator G matches the first input x(t)), the sum of squared errors between them at the pixel level is used as a loss function. The sum of squared errors is as shown in Equation 4.

[0084] The sum of squared errors when time t-1 and time t are swapped in configuration 8a can also be used as the loss function, as shown in configuration 8b. The sum of both configurations 8a and 8b can also be used as the loss function. By generating generated information x'(t-1) that matches the input x(t-1) and calculating the loss, it is expected that the performance of the generator G can be improved.

[0085] L pixel3 can be defined by the following equation (Equation 5). Figure 9 shows configurations 9a and 9b that are evaluated by this equation. (Note that Equation 5 shows configuration 9a, but by switching the letters, configuration 9b can be defined in exactly the same way.)

[0086]

number

[0087] In configuration 9a, first, using generator G, generated information x'(t) that simulates the first drawing information x(t) corresponding to the second drawing information y(t) at the current time (first input x(t-1), second input y(t-1), and third input y(t) respectively) is generated from the first drawing information x(t-1) and the second drawing information y(t-1) at the same time. Next, as shown by the dotted arrow, in order to make the generated information x'(t) match the first drawing information x(t) at the current time (the output x'(t) of generator G matches this "genuine work" x(t)), the sum of squared errors between them at the pixel level is used as the loss function. The sum of squared errors is as shown in Equation 5.

[0088] The sum of squared errors when time t-1 and time t are swapped in configuration 9a can also be used as the loss function, as shown in configuration 9b. The sum of both configurations 9a and 9b can also be used as the loss function. By generating generated information x'(t) that matches the system output x(t) and calculating the loss, it is expected that the performance of the generator can be improved.

[0089] L lm can be defined by the following equation (Equation 6). Figure 10 shows configurations 10a and 10b that are evaluated by this equation. (Note that Equation 6 is shown for configuration 10a, but by switching the letters, configuration 10b can be defined in exactly the same way.)

[0090]

number

[0091] Here, V represents feature calculation processing, and the calculator V shown as an optional configuration in FIG. 4 using the same symbol V is used during learning as a device that handles the feature calculation processing. (As described above, only the classifier D is used during inference, and this calculator V (and the classifier D) are not used.)

[0092] For the calculation of the feature amount by the calculator V, an existing deep learning encoder network (whose parameters are obtained by learning in advance) or the like can be used. In the case of a human model, the feature point coordinates of the face or body may be used. Although the drawing manner is different between the generated information simulating the first drawing information and the first drawing information, the coordinates of the feature points of the face are almost the same. Therefore, FIG. 10 (and Equation 6) evaluates the difference only in the feature point coordinates, instead of the difference in the pixel value (overall).

[0093] In the configuration 10a of FIG. 10, first, a generator G is used to generate generated information x'(t) simulating the first drawing information x(t) corresponding to the second drawing information y(t) at the current time (the first input x(t-1), the second input y(t-1), and the third input y(t) respectively) from the second drawing information y(t) at the same time as the first drawing information x(t-1) and the second drawing information y(t) at the current time. Next, a calculator V that calculates a feature amount from the generated information x'(t) (output of the generator G) calculates the coordinates of facial feature points and outputs them as V(x'(t)). Next, a calculator V that calculates a feature amount also calculates the coordinates of facial feature points from the second drawing information y(t) (third input of the generator G) and outputs them as V(y(t)). Finally, as shown by the dotted arrow, in order to make the feature quantities V(x'(t)) and V(y(t)) coincide with each other, the sum of squared errors between them at the pixel level is set as the loss function. The sum of squared errors is shown in Equation 6.

[0094] The sum of squared errors when time t-1 and time t are interchanged in the configuration 10a can also be used as the loss function, and this is shown as the configuration 10b. The linear sum of both the configurations 10a and 10b can also be used as the loss function. By calculating the same feature amount and calculating the loss even in the generated information simulating the first drawing information and the second drawing information which have different drawing forms, it is expected that the performance of the generator G can be improved.

[0095] The above has described the details of the learning performed by the generation unit 15 in step S1. Various supplementary examples, additional examples, alternative examples, etc. related to the present invention will now be described.

[0096] (1) According to an embodiment of the present invention, it is possible to realize realistic remote communication by drawing avatars. This makes it possible to hold remote conferences without necessarily requiring users to actually travel to remote locations, and it is possible to reduce carbon dioxide emissions by saving the energy resources required for user travel, which makes it possible to contribute to Goal 13 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "take urgent action to combat climate change and its impacts."

[0097] (2) The “recognition information r(t)” has been described as being used for drawing avatars and three-dimensional augmented reality displays. More generally, however, the present invention can be similarly applied to “content rendering specification information r(t)” that specifies the content when rendering any content (not necessarily three-dimensional) at each time t (using a server-side rendering method to render the same content at high quality on the server 20 and at standard quality on the terminal 10).

[0098] (3) The configuration for performing error backpropagation for updating the parameters of the generator G and the discriminator D during learning may be any one or a part of those shown in Figs. 6 to 10 that are used for loss evaluation. For example, when only the configurations 6a to 6d in Fig. 6 are used for loss evaluation regarding parameter update of the discriminator D, error backpropagation may be performed in this configuration (a network configuration in which only one discriminator D is arranged). When only the configurations 6e to 6h are used, error backpropagation may be performed after fixing the parameters of the generator G in this configuration (a configuration in which the discriminator D is in the front stage and the generator G is in the rear stage). The loss function may be evaluated using a combination (weighted sum) of the configurations 6a to 6d and the configurations 6e to 6h, and one or both may be used for error backpropagation, or different network configurations may be used for each batch for learning.

[0099] Similarly, for parameter update of the generator G, the loss function may be evaluated by using only one of the configurations of Fig. 7 to Fig. 10 for each batch for learning, and the error backpropagation may be performed by the single network configuration evaluated, or the configurations may be combined. (When using a single one, for example, the loss function may be evaluated using the learning batch data consisting of three inputs x(t-1), y(t-1), y(t) or x(t), y(t), y(t-1) in the configuration of Fig. 7, and the error backpropagation may be performed twice consecutively in the generator G arranged twice in the front and rear stages (the parameters of these two generators G are common) in the configuration of Fig. 7, or the loss function may be evaluated using the learning batch data consisting of the same three inputs in the configuration of Fig. 9, and the error backpropagation may be performed only once for the generator G consisting of only one stage in the configuration of Fig. 9.) Regarding the loss function evaluation and the error backpropagation, the configurations 6e to 6h in Fig. 6 may be further combined with the combination of the configurations of Fig. 7 to Fig. 10.

[0100] (4) FIG. 11 is a diagram showing an example of a hardware configuration of a general computer device 70. Each of the terminal 10 and the server 20 constituting the information system 100 can be realized as one or more computer devices 70 having such a configuration. When each of the terminal 10 and the server 20 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes part or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user inputs via a mouse, a keyboard, a touch panel, or the like, a camera 78, a microphone 79, and a speaker 80, and a bus BS for transmitting and receiving data between them.

[0101] Each functional unit of the information system 100 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the GPU 72. The avatar display function of the presentation unit 16 may be realized by the display 76, and the imaging unit 11 may be realized as a camera 78. Speech of a user U when performing avatar communication may be recorded by a microphone 79, and the voice of the other party may be output via a speaker 80. [Explanation of symbols]

[0102] 100...information system, 10...terminal, 20...server 11: imaging unit, 12: recognition unit, 13: second drawing unit, 14: storage unit, 15: generation unit, 16: presentation unit, 21: first drawing unit

Claims

1. a second drawing unit that performs drawing in a second drawing manner in accordance with content drawing specification information at each time to obtain second drawing information at each time; receiving first drawing information at each time from a server that performs drawing in a first drawing manner that is higher in quality than the second drawing manner in accordance with content drawing designation information at each time; a generation unit that receives as input three pieces of drawing information, namely, second drawing information at a current time and a past time, and first drawing information at the past time that has been received at the current time, and generates generation information that corresponds to a drawing result in the first drawing mode in accordance with the content drawing specification information at the current time, as an imitation of the first drawing information drawn by the server at the current time, The generation unit uses a generator out of a generator and a discriminator in a pre-trained generative adversarial network to generate the generated information using the three pieces of drawing information as input, for a terminal that generates the generated information: A method for learning parameters of a generator and a classifier in the generative adversarial network, comprising: second drawing information at each time that is prepared by performing drawing in the second drawing manner in accordance with the content drawing designation information at each time; selecting, as learning data, second drawing information and first drawing information corresponding to the first time and the second time from first drawing information at each time that is prepared by performing drawing in a first drawing mode that is higher in quality than the second drawing mode in accordance with the content drawing designation information at each time; learning parameters for the generator so as to generate generation information simulating the first drawing information corresponding to the second time for learning from first drawing information corresponding to the first time as a first input for learning, second drawing information corresponding to the first time as a second input, and second drawing information corresponding to the second time as a third input; learning parameters for the classifier so as to determine authenticity assuming that the first drawing information and the second drawing information for learning are genuine and that the generated information generated by the generator is a counterfeit; Parameter learning for determining authenticity of the classifier, A learning method characterized by including (6a, 6b, 6c, 6d) inputting first drawing information and second drawing information for learning, which have the same or different times, to the classifier, and learning parameters of the classifier so that when the times are the same, the information is judged to be positive, and when the times are different, the information is judged to be negative.

2. Parameter learning for determining authenticity of the classifier, selecting at least a portion of the learning data regardless of whether the first time and the second time are the same (6g, 6h) or different (6e, 6f); inputting, into the discriminator, two pieces of generation information generated by inputting the first input to the third input for learning into the generator, and a third input inputted into the generator; The learning method according to claim 1, further comprising: learning parameters of the classifier so that even if the times associated with the two inputs to the classifier are the same, one of the two inputs corresponds to generation information generated by the generation unit, and thus a negative determination is made in the same manner as when the times are different (6e, 6f, 6g, 6h).

3. a second drawing unit that performs drawing in a second drawing manner in accordance with content drawing specification information at each time to obtain second drawing information at each time; receiving first drawing information at each time from a server that performs drawing in a first drawing manner that is higher in quality than the second drawing manner in accordance with content drawing designation information at each time; a generation unit that receives as input three pieces of drawing information, namely, second drawing information at a current time and a past time, and first drawing information at the past time that has been received at the current time, and generates generation information that corresponds to a drawing result in the first drawing mode in accordance with the content drawing specification information at the current time, as an imitation of the first drawing information drawn by the server at the current time, The generation unit uses a generator out of a generator and a discriminator in a pre-trained generative adversarial network to generate the generated information using the three pieces of drawing information as input, for a terminal that generates the generated information: A method for learning parameters of a generator and a classifier in the generative adversarial network, comprising: second drawing information at each time that is prepared by performing drawing in the second drawing manner in accordance with the content drawing designation information at each time; selecting, as learning data, second drawing information and first drawing information corresponding to the first time and the second time from first drawing information at each time that is prepared by performing drawing in a first drawing mode that is higher in quality than the second drawing mode in accordance with the content drawing designation information at each time; learning parameters for the generator so as to generate generation information simulating the first drawing information corresponding to the second time for learning from first drawing information corresponding to the first time as a first input for learning, second drawing information corresponding to the first time as a second input, and second drawing information corresponding to the second time as a third input; learning parameters for the classifier so as to determine authenticity assuming that the first drawing information and the second drawing information for learning are genuine and that the generated information generated by the generator is a counterfeit; Parameter learning for the generator, Selecting at least a portion of data for learning, with the first time and the second time being different times; The generators are arranged in a two-stage configuration of a front-stage generator and a rear-stage generator, generating a first input for the latter-stage generator by inputting the first input through the third input for learning to the former-stage generator; using the third input and the second input inputted to the previous stage generator as the second input and the third input of the next stage generator, respectively, A learning method comprising: learning (7a, 7b) so that the generation information output by the latter-stage generator imitates a first input to the former-stage generator.

4. a second drawing unit that performs drawing in a second drawing manner in accordance with content drawing specification information at each time to obtain second drawing information at each time; receiving first drawing information at each time from a server that performs drawing in a first drawing manner that is higher in quality than the second drawing manner in accordance with content drawing designation information at each time; a generation unit that receives as input three pieces of drawing information, namely, second drawing information at a current time and a past time, and first drawing information at the past time that has been received at the current time, and generates generation information that corresponds to a drawing result in the first drawing mode in accordance with the content drawing specification information at the current time, as an imitation of the first drawing information drawn by the server at the current time, The generation unit uses a generator out of a generator and a discriminator in a pre-trained generative adversarial network to generate the generated information using the three pieces of drawing information as input, for a terminal that generates the generated information: A method for learning parameters of a generator and a classifier in the generative adversarial network, comprising: second drawing information at each time that is prepared by performing drawing in the second drawing manner in accordance with the content drawing designation information at each time; selecting, as learning data, second drawing information and first drawing information corresponding to the first time and the second time from first drawing information at each time that is prepared by performing drawing in a first drawing mode that is higher in quality than the second drawing mode in accordance with the content drawing designation information at each time; learning parameters for the generator so as to generate generation information simulating the first drawing information corresponding to the second time for learning from first drawing information corresponding to the first time as a first input for learning, second drawing information corresponding to the first time as a second input, and second drawing information corresponding to the second time as a third input; learning parameters for the classifier so as to determine authenticity assuming that the first drawing information and the second drawing information for learning are genuine and that the generated information generated by the generator is a counterfeit; Parameter learning for the generator, selecting at least a portion of data for learning with the first time and the second time being the same time; A learning method characterized by including (8a, 8b) learning that the generated information output by inputting the first input to the third input for learning into the generator is to mimic the first input for learning.

5. a second drawing unit that performs drawing in a second drawing manner in accordance with content drawing specification information at each time to obtain second drawing information at each time; receiving first drawing information at each time from a server that performs drawing in a first drawing manner that is higher in quality than the second drawing manner in accordance with content drawing designation information at each time; a generation unit that receives as input three pieces of drawing information, namely, second drawing information at a current time and a past time, and first drawing information at the past time that has been received at the current time, and generates generation information that corresponds to a drawing result in the first drawing mode in accordance with the content drawing specification information at the current time, as an imitation of the first drawing information drawn by the server at the current time, The generation unit uses a generator out of a generator and a discriminator in a pre-trained generative adversarial network to generate the generated information using the three pieces of drawing information as input, for a terminal that generates the generated information: A method for learning parameters of a generator and a classifier in the generative adversarial network, comprising: second drawing information at each time that is prepared by performing drawing in the second drawing manner in accordance with the content drawing designation information at each time; selecting, as learning data, second drawing information and first drawing information corresponding to the first time and the second time from first drawing information at each time that is prepared by performing drawing in a first drawing mode that is higher in quality than the second drawing mode in accordance with the content drawing designation information at each time; learning parameters for the generator so as to generate generation information simulating the first drawing information corresponding to the second time for learning from first drawing information corresponding to the first time as a first input for learning, second drawing information corresponding to the first time as a second input, and second drawing information corresponding to the second time as a third input; learning parameters for the classifier so as to determine authenticity assuming that the first drawing information and the second drawing information for learning are genuine and that the generated information generated by the generator is a counterfeit; Parameter learning for the generator, Selecting at least a portion of data for learning, with the first time and the second time being different times; inputting the first input to the third input for learning into the generator, and outputting generated information into a pre-trained feature calculator to output first feature information; The third input input to the generator is input to the feature calculator to output second feature information; A learning method comprising: learning (10a, 10b) such that the first feature information mimics the second feature information.

Images