Predicting corrections and comments in animation production

Abstract

A system configured to: obtain a frame created by an animator and supplemental information associated with an animation work; input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and output the frame, the correction layer, and the corresponding comments in distinct visual layers.

Description

PREDICTING CORRECTIONS AND COMMENTS IN ANIMATION PRODUCTION

[0001] The technology disclosed in the present description (hereinafter referred to as "the present disclosure") relates to an information processing system, an information processing method, and a computer program related to animation production.

[0002] Animation is made up of numerous frames, and hence an enormous amount of labor and cost is required for creating frames.　Digital technologies using a computer are thus being introduced in an animation production process.　For example, a method for creating a moving image has been proposed, the method including: a step of tracing a key frame by paths to create a traced moving image; and a step of creating an in-between moving image by automatic calculation on the basis of a pair of original moving images composed of paths (see PTL 1).

[0003] JP 2022-90830 A

[0004] Vision-Language Models for Vision Tasks: A Survey (arXiv:2304.00685v2 [cs.CV] 16 Feb 2024)Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020v1 [cs.CV] 26 Feb 2021)Summary

[0005] It is an object of the present disclosure to provide an information processing system, an information processing method, and a computer program configured to perform processing related to drawing work in an animation production process.

[0006] According to one exemplary embodiment, the disclosure is directed to an information processing system, comprising: circuitry configured to obtain a frame created by an animator and supplemental information associated with an animation work; input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and output the frame, the correction layer, and the corresponding comments in distinct visual layers.

[0007] The circuitry may be configured to receive a corrected version of the frame that selectively incorporates the proposed modifications.

[0008] The supplemental information may include at least one of a script, storyboard, layout, scenario, or setting, and may be selected based on cut metadata associated with the frame.

[0009] The correction layer may comprise a visual representation that highlights regions of the frame associated with the proposed modifications, and the visual representation may be presented with a bounding box around a part identified for correction.

[0010] The machine-learned model may comprise a vision-language model trained using annotated frames and associated comments.

[0011] According to another exemplary embodiment, the disclosure is directed to a method of processing animation data, the method comprising: obtaining a frame created by an animator and supplemental information associated with an animation work; inputting the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; generating, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and outputting the frame, the correction layer, and the corresponding comments in distinct visual layers.

[0012] According to another exemplary embodiment, the disclosure is directed to one or more non-transitory computer readable media including instructions, which when executed by circuitry, cause the circuitry to: obtain a frame created by an animator and supplemental information associated with an animation work; input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and output the frame, the correction layer, and the corresponding comments in distinct visual layers

[0013] Fig. 1 is a diagram illustrating a functional configuration of an information processing system 100.Fig. 2 is a diagram illustrating a flow for collecting new training data.Fig. 3 is a diagram illustrating a conversion flow in which existing data is reused for training data.Fig. 4 is a diagram exemplifying supplemental information (scenario information).Fig. 5 is a diagram exemplifying supplemental information (script information).Fig. 6 is a diagram exemplifying supplemental information (storyboard information).Fig. 7 is a diagram exemplifying supplemental information (layout information).Fig. 8 is a diagram illustrating an example of an annotation UI.Fig. 9 is a diagram illustrating a configuration of an animation director model.Fig. 10 is a flowchart illustrating a procedure of processing in which a key animator and an animation director perform work by utilizing the animation director model.Fig. 11 is a diagram illustrating a configuration example of a UI screen used in key frame creation, which takes the application of the animation director model into consideration.Fig. 12 is a diagram illustrating a hardware configuration example of an information processing device 2000.Fig. 13 is a diagram illustrating a configuration example of an animation work that is completed by superimposing a plurality of cels.

[0014] Referring to the drawings, embodiments of the present disclosure are described below in the following order.

[0015] A. Outline B. System configuration B-1. Data collection unit 　　B-1-1. New data collection 　　B-1-2. Existing data reuse 　　B-1-3. Annotation UI B-2. Model learning unit B-3. Model utilizing unit C. Modifications D. Configuration of information processing device

[0016] A. Outline An animation production process is divided into some steps, with each step involving work by a plurality of persons.　In particular, drawing work is often assigned with different persons for each scene, and depending on works, the total number of animators reaches several dozen.　Thus, there exists a role called an animation director who oversees all the animators.　The "drawing" as used herein refers to a process that starts from a storyboard, then moves to writing key frames, and finally to in-between animation that fills gaps between the key frames (a storyboard generally includes rough sketches indicating the story and directorial intentions along with instructional text).　The animation director broadly corrects fine details such as composition and expressions in key frames, and may also make corrections to the in-between animations, particularly in the part called "inbetweening" that adds movement.

[0017] The animation director has a role of unifying drawings of animation, and is indispensable for improving the quality of works.　The animation director may actually make corrections to the drawings by him / herself or communicate orders.　Depending on the scale of the work, there may be a plurality of animation directors, and in some works, there is a chief animation director who oversees the animation directors.　However, for the sake of simplicity, the present description is described based on a single-layered structure of a plurality of animators and an animation director who supervises the animators.

[0018] A step in which an animation director checks a key frame created by a key animator and adds corrections and comments is referred to as "animation director's correction".　In general, the animation director writes corrections and comments in a correction sheet by hand in the animation director's correction, and returns the correction sheet to the key animator.　The key animator corrects the key frame on the basis of the correction sheet from the animation director, and undergoes a check again by the animation director.　Such a flow of the creation of a key frame, the animation director's correction, and the correction of the key frame is repeatedly performed until the confirmation (OK) by the animation director is obtained.　In general, each scene in an animation work is made up by superimposing a plurality of cels.　Then, animation cels finished for each scene are disposed in a time-series manner, and an animation work is completed.　The animation director's correction is performed on key frames for each scene in each cel.

[0019] The animation director's correction is an important role for unifying the drawings by a large number of key animators and securing the quality of work.　However, the number of animation directors is small in general, and hence the animation director's correction is a bottleneck in the animation production process, and the number of reworks to the key animator is large.　Furthermore, the approach to animation director's correction varies among animation directors, which makes it challenging for both novices and experts to interpret and implement their intensions.　In a key frame, a storyboard is a rough sketch, and it is difficult to reflect higher-level context such as detailed expressions and directorial intentions.

[0020] The present disclosure is a technology related to drawing work in an animation production process, particularly focusing on key frame work, and mainly aims to make communications between animators and an animation director efficient.　Specifically, the present disclosure relates to a technology for automatically generating a correction sheet and a correction comment from the animation director for a work drawn from a key frame in drawing work.　Similarly, the present disclosure can be extended to moving images.

[0021] The present disclosure uses a learned model in which the patterns of past key frames and animation director's corrections from animation directors have been learned along with supplemental information such as scripts.　In the present description, the learned model is referred to as "animation director model".　The animation director model predicts or estimates animation director's correction by an animation director for a key frame created by a key animator, and automatically generates a correction sheet or a correction comment.　Furthermore, in the drawing work by the key animator, the animation director model automatically highlights a part of the key frame that is likely to be corrected by the animation director, and issues an alert, thereby reducing the number of reworks to the key animator due to animation director's correction.　In addition to issuing an alert, the animation director model depicts (generates) candidate correction matters, and presents what kind of correction is desirable to the key animator.　The animation director model predicts actual animation director's correction by the animation director.　The key animator can use the animation director model to correct the key frame in advance, thereby reducing labor of animation director's correction by the animation director, which has been a bottleneck.

[0022] The animation director model cannot be completely substituted by an animation director, and hence even if a key animator utilizes the animation director model, the occurrence of rework cannot be completely eliminated.　Thus, in the present disclosure, rework results are fed back in the form of Human-in-the-loop, so that the quality of the animation director model is improved to suppress the occurrence of rework.　Human-in-the-loop as used herein refers to the involvement of interactions with humans in a series of the machine learning process and life cycle (loop) for the animation director model including data collection and preprocessing, training of animation director models, operational monitoring of the trained animation director models, and constant updating, that is, the involvement of feedbacks from humans.

[0023] According to the present disclosure, the number of reworks in the key frame work can be reduced, with the result that the drawing work is made efficient and the quality of the animation work is secured.

[0024] The present disclosure can solve a problem in data / model development and problems in model operation.

[0025] Problem in data / model development: The mainstream of the current animation director's correction is to add corrections and comments directly to a key frame.　Thus, in order to collect training data for the animation director model, it is necessary to sort out which part of the key frame has been corrected, which part is pointed out by the comment, and which is unnecessary information.　The present disclosure solves this problem by the reuse of existing data and the designing of a UI (User Interface) for reducing cost of data creation.

[0026] Problems in model operation: There are two problems in model operation, a problem that the occurrence of animation director's correction cannot be completely eliminated even in the model operation and the rework itself occurs and a problem that the precision may decrease in the case of a new work.　The present disclosure solves those problems by constructing a system for improving the precision of the model by Human-in-the-loop during the operation.

[0027] Note that the effects disclosed in the present description are merely illustrative, and the effects obtained by the present disclosure are not limited thereto.　Furthermore, the present disclosure may exhibit additional effects other than the above-mentioned effects.　Other objects, features, and advantages of the present disclosure become apparent from the following more detailed descriptions based on the embodiments and the accompanying drawings.

[0028] B. System configuration Fig. 1 illustrates a functional configuration of an information processing system 100 according to the present disclosure.　The illustrated information processing system 100 includes a data collection unit 101, a model learning unit 102, and a model utilizing unit 103.

[0029] The data collection unit 101 is a functional module for collecting training data for learning an animation director model.　The data collection unit 101 mainly performs two types of data collection: "existing data reuse" where training data is collected from existing data accumulated in the animation production site (that is, data that has been subjected to animation director's correction in the past), and "new data collection" where training data is collected through animation director's correction of new key frames by an animation director.

[0030] As the existing data reuse, the data collection unit 101 reuses a "past work database" accumulated in the animation production site, and creates training data for learning an animation director model.　In the past work database, many pieces of data that can be utilized for learning of the animation director model, such as supplemental information on animation works in addition to key frames before and after animation director's correction, are accumulated, and are converted into training data in the data collection unit 101.　In the example illustrated in Fig. 1, "parts disassembly" for disassembling key frames before and after correction accumulated in the past work database into parts and "comment detection" for detecting comments from a correction sheet of each pre-correction key frame are performed, and further "parts / comments association" for associating each part and each comment by the animation director is performed, with the result that the data is formatted into training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame and saved in a correction database.

[0031] Furthermore, as the new data collection, the data collection unit 101 newly creates training data for learning an animation director model.　In the example illustrated in Fig. 1, training data is collected through key frame confirmation (animation director's correction) implemented by an animation director in the model utilizing unit 103.　Specifically, the animation director performs animation director's correction by using an "annotation UI" (described later) for associating a part to be corrected with a comment, thereby collecting training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in a key frame concurrently with the confirmation of the key frame, and saves the training data in the correction database.

[0032] The model learning unit 102 is a functional module for learning an animation director model on the basis of the training data (correction database) accumulated in the correction database.　The animation director model is a machine learning model configured to predict or estimate animation director's correction by an animation director on a key frame created by a key animator.　For example, the animation director model has an Encoder-Decoder configuration, and the model learning unit 102 performs supervised learning by using the training data accumulated in the correction database.　Alternatively, the model learning unit 102 may learn an animation director model by fine-tuning of a vision-language model.

[0033] The model utilizing unit 103 is a functional module for using the animation director model learned by the model learning unit 102 for drawing work in the actual animation production process.

[0034] The key animator creates key frames of an animation work.　The animation director model inputs the key frame created by the key animator and supplemental information on the animation work, and predicts or estimates animation director's correction by the animation director, thereby automatically generating a correction sheet and a correction comment.　Then, the key animator corrects the key frame input to the animation director model on the basis of the correction sheet and the correction comment for the key frame, which are output from the animation director model.

[0035] It is assumed that the key animator creates and corrects key frames by using a "key frame work UI" (described later).　The key frame work UI is provided with a function of displaying an original key frame initially created by the key animator and key frames that have been corrected in accordance with correction instructions output from the animation director model and animation director's correction by the real animation director in different layers, and a function of presenting comments by the animation director model and the real animation director for each correction layer.

[0036] The key frame that has been corrected by the key animator on the basis of corrections and comments by the animation director model subsequently undergoes confirmation of the key frame by the animation director, that is, animation director's correction.　It is assumed that the animation director's correction uses an "annotation UI" (described later).　By using the annotation UI, parts in the key frame that have been corrected by the animation director are associated with comments, and training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame is collected concurrently with the animation director's correction and saved in the correction database.

[0037] The key animator corrects the key frame again on the basis of the animation director's correction.　It is assumed that the key animator uses the "key frame work UI" (described later) for the correction.　Furthermore, the animation director performs animation director's correction again on the corrected key frame.　It is also assumed that the second and subsequent animation director's correction uses the "annotation UI" (described later), and training data for the animation director model is collected concurrently with the animation director's correction.　Then, the key frame for which the confirmation by the animation director has been finished is output to the subsequent step.

[0038] Even if the animation director model is used, rework by the animation director can occur.　In the information processing system 100, rework results are fed back in the form of Human-in-the-loop, and hence training data for the animation director model can be added to improve the quality of the animation director model.

[0039] The information processing system 100 illustrated in Fig. 1 can be configured by using an information device such as a computer system.　It should be understood that the information processing system 100 may be configured by using computer systems different for functional modules such as the data collection unit 101, the model learning unit 102, and the model utilizing unit 103.　Furthermore, a functional module for performing processing with high calculation load such as the model learning unit 102 may be configured by using a plurality of computer systems, and parallel processing may be used.

[0040] Note that the creation and correction work of key frames using the "key frame work UI" (described later) by the key animator may be implemented on a device physically separated from the information processing system 100, such as a tablet terminal or a personal computer (PC), and data on work results may be transferred to the information processing system 100.

[0041] Furthermore, the animation director's correction work using the "annotation UI" (described later) by the animation director may be implemented on a device physically separated from the information processing system 100, such as a tablet terminal or a personal computer (PC), and data on animation director's correction may be transferred to the information processing system 100.

[0042] The units in the information processing system 100 are described in detail below.

[0043] B-1. Data collection unit Training data is necessary for constructing an animation director model.　In the information processing system 100, the data collection unit 101 treats two flows: new data collection and existing data reuse.

[0044] B-1-1. New data collection In the new data collection flow, the data collection unit 101 newly collects training data for learning an animation director model.　The data collection unit 101 collects training data along with the flow implemented in the model utilizing unit 103 where the animation director adds animation director's correction on pre-correction key frames.　This is based on the fact that the collection of new training data is incorporated in an ongoing manner in the animation production process involving the ongoing drawing work.

[0045] In general animation director's correction, additional work is necessary for treating training data, and machine readability is low.　Thus, the present disclosure considers that, in order to increase machine readability, items that have been subjected to animation director's correction by the animation director are tagged.　It is conceivable that the tagging processing for each item is implemented on the annotation UI (described later).

[0046] Fig. 2 schematically illustrates the new data collection flow in which the data collection unit 101 collects new training data along with animation director's correction.　The data collection unit 101 includes a parts disassembly unit 201 and an annotation unit 202 as functional modules for the new data collection.

[0047] Parts disassembly unit: In animation director's correction, whether a character provides a feeling of strangeness as compared to other scenes and whether the composition of background is good are checked.　In the animation director's correction, the whole image may be rewritten, but the correction is mainly made on a part-by-part basis.　Thus, it is important to check whether corrections by the animation director will be made in units of parts (such as eyes, mouse, and hand in the case of a character).　Accordingly, the parts disassembly unit 201 detects each part from a pre-correction key frame, and prepares to associate each part with annotations such as correction content and comments by the animation director.　The parts disassembly unit 201 disassembles the key frame into items to be frequently checked by the animation director as parts.　The parts disassembly unit 201 may extract each part from the key frame by using an object recognition model, or items that have been frequently checked by the animation director in the past may be learned by an object recognition model.

[0048] Annotation unit: The annotation unit 202 is a functional module used for the animation director to add annotations such as corrections and comments for each part obtained by disassembling the key frame by the parts disassembly unit 201.　The annotation unit 202 includes a display device configured to present a subject key frame to the animation director and an input device configured to input annotations such as corrections and comments on the key frame by hand or other means.

[0049] The animation director adds correction content and correction comments for each part in the key frame by the annotation unit 202 as animation director's correction.　The annotation unit 202 outputs data consisting of a set of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part included in the key frame.　The data output from the annotation unit 202 is saved in the correction database 203 and utilized as training data for the animation director model.

[0050] Furthermore, the animation director may add metadata such as priority when adding corrections and comments for each part in the key frame.　The reason is that the key animator can be prompted to make corrections in the descending order of priority in the time-constrained key frame work to shorten the time required for the whole work.　Alternatively, the annotation unit 202 may automatically add priority to corrections and comments for each part in the key frame.　The metadata such as priority for each part in the key frame is also saved in the correction database 203.

[0051] As already described with reference to Fig. 1, the new data collection flow by the data collection unit 101 is incorporated in the animation director's correction implemented in the model utilizing unit 103.　The animation director's correction assumes the use of the "annotation UI" (described later).　By using the annotation UI, parts in the key frame that have been corrected by the animation director are associated with comments, and training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame is collected concurrently with the animation director's correction and saved in the correction database.

[0052] In the annotation unit 202, the animation director may point out the whole key frame instead of parts in some cases.　In other cases, in an animation work composed by superimposing a plurality of cels, the animation director may point out only one key frame among a plurality of successive scenes in a cel, or may point out a cel by referring to the points on other cels.　For example, in an animation work as illustrated in Fig. 13, which is completed by superimposing three-layered cels of a cel A where an object in the front is drawn, a cel B where an object in the back is drawn, and a cel C where background is drawn, the animation director may point out "the comment written in the key frame in the sixth scene on the cel A is applied to the subsequent predetermined scene sections" or "the eighth comment written on the cel A is applied to the same scene on the cel B" (in actual case, it is general that the sentence is simple expression that can be understood among animators).　It is preferred that the annotation unit 202 or the annotation UI applied to the annotation unit 202 be provided with a function of enabling the animation director to efficiently input not only points in units of parts but also points on the overall key frame, points over key frames in a plurality of successive scenes, and points over a plurality of cels and a function of appropriately collecting these points as new training data.

[0053] B-1-2. Existing data reuse As described above, animation director's correction is widely performed in the existing animation production, and data and comments (existing data) before and after animation director's correction are accumulated in the animation production site.　However, the pieces of existing data cannot be used directly for learning of the animation director model, and are thus need to be formatted into training data.　Accordingly, in the existing data reuse flow, the data collection unit 101 performs data conversion in order to reuse the existing data for training data for the animation director model.

[0054] Fig. 3 schematically illustrates an existing data reuse flow in which the data collection unit 101 reuses the "past work database" accumulated in the animation production site and creates training data for learning the animation director model.　The data collection unit 101 includes a parts disassembly unit 301, a comment detection unit 302, a parts / comment association unit 303, and a data storage unit 304 as functional modules for the existing data reuse.

[0055] Data input: The data collection unit 101 inputs, from the past work database 305, pre-correction key frames and corrected key frames as basic training data and other additional information if there is reference information.　In the past work database 305, the existing data occurring in the animation production site, such as pre-correction key frames and corrected key frames, and correction sheets and other supplemental information used in animation director's correction, are accumulated.　The pieces of existing data cannot be used directly for learning of the animation director model, and are thus need to be formatted into training data.

[0056] The supplemental information includes plots, screenplays, scripts, settings, scenarios, storyboards, and layouts of animation works.　The key animator selects which of those is used as supplemental information when drawing key frames.　In regard to the supplemental information, various aspects can be taken into consideration because various steps follow the key frame work in the animation production.　Supplemental information that has been used to create key frames can also be used to learn the animation director model.　Furthermore, a correction sheet on which the animation director has written corrections and comments by hand in animation director's correction can be used to learn the animation director model as supplemental information.

[0057] Fig. 4 to Fig. 7 exemplify screenplay information, script information, storyboard information, and layout information, respectively, as supplemental information.　The screenplay information (Fig. 4) includes scene-related simple information and scene description in addition to lines of characters.　The script information (Fig. 5) describes scene description and line parts for each cut.　The storyboard information (Fig. 6) describes information on the composition and the motion of performers for each cut as well as the content, sound effects, lines, and time for each cut.　The layout information (Fig. 7) indicates detailed information on the layout of the screen.　The supplemental information as illustrated in Fig. 4 to Fig. 7 can be materials for the supplemental information animation director to correct the key frames.

[0058] Parts disassembly unit: The parts disassembly unit 301 detects each part from the pre-correction key frame input from the past work database 305, and prepares to associate each part with annotations such as correction content and comments by the animation director.　The parts disassembly unit 301 may be the same as the parts disassembly unit 201 for the new data collection described above, and disassembles the key frame into items to be frequently checked by the animation director as parts.　The parts disassembly unit 301 may extract each part from the key frame by using an object recognition model, or items that have been frequently checked by the animation director in the past may be learned by an object recognition model.

[0059] Comment detection unit: The animation director's correction involves handwriting on a correction sheet in many cases.　Converting the handwritten characters into text data is necessary in the learning of the animation director model.　Thus, the comment detection unit 302 performs optical character recognition (OCR) on the handwritten correction sheet for the conversion from an image to text data.　In this case, a plurality of comments are converted into text individually.

[0060] Parts / comment association unit: Points may be written to a plurality of locations on a single correction sheet, and comments that are not directly related to correction (for example, greetings such as "Thank you") may be written.　Thus, the parts / comment association unit 303 associates each comment detected from the correction sheet by the comment detection unit 302 with a part obtained by disassembling the key frame by the parts disassembly unit 301, and determines whether the comment is not directly related to the drawing.　The correction comment includes a large number of points in terms of viewpoints particularly focused in animation, and various technical terms such as eye shade are used.　The parts / comment association unit 304 associates the terms with locations in the key frame.　The parts / comment association unit 303 associates each part with each comment by the animation director, thereby formatting the key frames before and after correction and the correction sheet accumulated in the animation production site into training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame.

[0061] The way of writing the correction sheet by the animation director varies.　There are correction comments on characters such as eyes, nose, hair, and mouse and correction comments on objects other than characters such as background.　Furthermore, there are a correction sheet on which a large number of comments are written for a single key frame and a correction sheet on which correction content are directly written for a key frame by hand with no comments.　The parts / comment association unit 303 may associate parts with comments on the basis of physical information such as a distance between a location where a correction comment is written and a part.　Furthermore, the parts / comment association unit 303 may associate parts with comments by using an image classification model (see NPL 2) that has learned linking between images and text such as CLIP (Contrastive Language-Image Pre-training).

[0062] Furthermore, the animation director may write a comprehensive correction comment on a key frame in a correction sheet in addition to correction comments for each part.　In other cases, the animation director may write a correction comment on a correction sheet for a single key frame as points over key frames in a plurality of successive scenes such as "the comment written in the key frame in the sixth scene on the cel A is applied to the subsequent predetermined scene sections", and may write a correction comment on a correction sheet for a single key frame as points for other cels in the same scene such as "the eighth comment written on the cel A is applied to the same scene on the cel B".　Also in those cases, the parts / comment association unit 303 associates comments written on a correction sheet for a single key frame with parts for all the corresponding key frames.

[0063] Data storage unit: The data storage unit 304 stores therein data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame, which has been formatted on the basis of the association between parts and comments by the preceding parts / comment association unit 303.　Each set of data stored in the data storage unit 305 is training data used to learn the animation director model.　The data storage unit 305 may be common to the correction database 203 used in the new data collection.

[0064] B-1-3. Annotation UI The animation director uses the annotation UI to perform animation director's correction on key frames.　The annotation UI has a function of associating a part to be corrected with a comment written by the animation director, and hence can correct training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame concurrently with the confirmation of the key frame.

[0065] Fig. 8 illustrates an example of the annotation UI.　The animation director can use the illustrated annotation UI to add annotations such as corrections and comments to each key frame.　In a case where the animation director's correction is implemented on a device physically separated from the information processing system 100, such as a tablet terminal or a PC, the annotation UI illustrated in Fig. 8 is displayed on a screen of the tablet terminal or the PC owned by the animation director.

[0066] The annotation UI illustrated in Fig. 8 includes a correction sheet display unit 801, an animation director's correction unit 802, and a confirmation unit 803.

[0067] The correction sheet display unit 801 occupies the left half of the annotation UI, and displays a correction sheet written by the animation director.　The correction sheet display unit 801 may display an image obtained by scanning a paper correction sheet written by the animation director by hand, or a stylus pen may be used to directly write correction content and comments on the screen of the correction sheet display unit 801.

[0068] The animation director's correction unit 802 and the confirmation unit 803 are displayed on the right half of the annotation UI in a superimposed manner.　When a "correction" tab indicated by reference numeral 804 is selected, the animation director's correction unit 802 appears on top.　Similarly, when a "confirmation" tab indicated by reference numeral 805 is selected, the confirmation unit 803 appears on top.　In other words, by selecting any one of the correction tab 804 and the confirmation tab 805, the display of the animation director's correction unit 802 and the confirmation unit 803 can be switched.　In Fig. 8, for the sake of description, the state in which the animation director's correction unit 802 appears on top and the state in which the confirmation unit 803 appears on top are illustrated side by side.

[0069] Animation director's correction: The animation director's correction unit 802 has a function of selecting a part to be corrected by the animation director and writing a comment on the selected part.　In the animation director's correction unit 802, parts to be annotated can be individually selected or a target range can be designated to add corrections and comments to the selected parts or the designated range.

[0070] The parts disassembly unit 201 / 301 disassembles a key frame into items to be frequently checked by the animation director as parts (described above).　On the animation director's correction unit 802, parts obtained by disassembling the key frame by the parts disassembly unit 201 / 301 are listed as "selected parts" indicated by reference numeral 806.　The "selected part" list may be a drop-down menu.

[0071] The animation director can individually select parts for which comments are to be written from the "selected part" list 806.　Then, in the correction sheet display unit 801 on the left half of the UI screen, the part selected in the "selected part" list 806 is indicated by a bounding box.　In the example illustrated in Fig. 8, "eyes" are selected and switched to highlight indication in the "selected part" list 806 in the animation director's correction unit 802, and a bounding box indicated by reference numeral 807 is displayed over a region of the eyes in the key frame in the correction sheet display unit 801.

[0072] Furthermore, a "designate range" button indicated by reference numeral 808 is provided on the right of the "selected part" list 806.　By pushing the "designate range" button 808 and dragging the "selected part" list 806, the animation director can designate a plurality of parts simultaneously in a range to add corrections and comments.　In this case, all parts in the designated range are highlighted in the "selected part" list 806, and are indicated by a bounding box surrounding all the parts in the designated range in the correction sheet display unit 801 (not shown).

[0073] After selecting a part to be corrected, when the animation director pushes a "correction work" button indicated by reference numeral 809 disposed under the drop-down menu, another layer ("correction layer") (not shown) appears in the correction sheet display unit 801, and the animation director can add corrections on the key frame by using a stylus pen on the layer.

[0074] Furthermore, the animation director can write annotations, such as correction and comments to be added to the individually selected parts or the parts in the designated range, in a comment input field indicated by reference numeral 810 provided under the "correction work" button 809.　The writing in the comment input field 810 may be performed in the form of text input using a keyboard, for example, or may be performed in the form of voice input and voice recognition.　In the example illustrated in Fig. 8, the comment "eye shade / see 3rd episode" for the key frame is written in the comment input field 810.　The message "see 3rd episode" in the written comment is correction content meaning that correction content for the key frame in another scene (3rd episode) are also applied to the key frame that is currently being corrected.

[0075] Confirmation: The confirmation unit 803 has a function of confirming the whole correction content written by the animation director's correction unit 802.　The confirmation unit 803 displays a list of correction content for each part in the key frame written by the animation director's correction unit 802.　In regard to parts that have not been corrected, "no corrections" is displayed on the list.　When all parts cannot be displayed as a list in the screen, a vertical scrollbar is used.　The animation director can see the list display in the confirmation unit 803 and confirm comments on parts that have been corrected and parts that have not been corrected, thereby preventing the overlooking of corrections on the key frame.

[0076] When the animation director performs the work of adding corrections and correction comments (that is, annotations) for each part in the key frame through the annotation UI illustrated in Fig. 8, a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame can be collected concurrently with the animation director's correction, and utilized as training data for the animation director model in the model learning unit 102.　In this manner, rework results are fed back in the form of Human-in-the-loop, and hence the quality of the animation director model can be improved to suppress the occurrence of rework.

[0077] B-2. Model learning unit The animation director model inputs a pre-correction key frame and supplemental information on the animation work (such as storyboard, layout, screenplay, script, setting, and correction sheet), and predicts or estimates animation director's correction by the real animation director to generate a corrected key frame (including difference of partial correction) and a correction comment.　In the drawing work by the key animator, the animation director model automatically highlights a part of the key frame that is likely to be corrected by the animation director, and issues an alert.　The key animator corrects the key frame before the real animation director's correction on the basis of the corrected key frame and the correction comment output from the animation director model, thereby reducing the number of reworks from the animation director to the key animator.　Animation director's correction is one of bottlenecks in the animation production process, but by using the animation director model to reduce the number of reworks, the efficient work can be implemented.

[0078] Fig. 9 illustrates a configuration example of the animation director model.　The illustrated animation director model 900 has a general Encoder-Decoder configuration.　Specifically, the animation director model includes an image encoder 901, a text encoder 902, an image decoder 903, and a text decoder 904.

[0079] The image encoder 901 inputs a pre-correction key frame to generate an intermediate representation (image feature amount) 911, and the text encoder 902 inputs supplemental information on the pre-correction key frame to generate an intermediate representation (text feature amount) 912.　The supplemental information is plots, setting, scenario, storyboard, layout, and a correction sheet created by animation director's correction in the corresponding work, and is supplied from, for example, a work database 905.　Then, the image decoder 903 and the text decoder 904 generate a corrected key frame and a correction comment from the intermediate representations 911 and 912 of the pre-correction key frame and the supplemental information, respectively.

[0080] The model learning unit 102 performs supervised learning of the animation director model by using training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame, which have been corrected by the data collection unit 101.

[0081] The model learning unit 102 may learn the animation director model from zero by using only training data, or may implement the learning by fine-tuning of a large-scale vision language model such as Vision Language Model (VLM).　The VLM is a model capable of simultaneously understanding and processing visual information (images and video) and text information (language data) (see NPL 1).

[0082] In addition, data in animation production such as key frames and settings is not data that is frequently present on the Internet, and hence the amount of training data in the distributed large-scale vision language model is expected to be small.　Thus, work data in the animation production may be used to additionally prepare a large-scale vision language model dedicated for animation.

[0083] How reliable the corrected key frame and correction comment output from the animation director model are varies depending on input data or output content.　Thus, it is conceivable that how reliable the comment is may be output from the animation director model on the basis of internal representation in the form of numerical values such as reliability level.　In this case, it is conceivable to provide a threshold such that only data with a certain level of reliability or higher is output.

[0084] B-3. Model utilizing unit The model utilizing unit 103 utilizes the animation director model learned by the model learning unit 102 for the drawing work in the actual animation production process.　Fig. 10 illustrates a procedure of processing in which the key animator and the animation director perform work by utilizing the animation director model in the form of a flowchart.

[0085] Creation of key frame: In Step S1001, the key animator performs drawing work as usual, and completes a key frame.　The key animator may use a key frame work UI (see Fig. 11) described later, or may perform drawing as usual.

[0086] Selection of supplemental information: As described above, the key animator uses supplemental information such as plots, screenplay, script, setting, scenario, storyboard, and layout of the animation work as information on context related to the drawing.　In Step S1002, a user (for example, key animator) selects, from the past work database, which of those pieces of data is used as supplemental information when generating a correction comment by using the animation director model.　Alternatively, which of those pieces of data is used as supplemental information may be automatically selected on the basis of related information such as cut information.

[0087] Animation director model: In Step S1003, the key frame created by the key animator is input to the animation director model, thereby obtaining corrections and comments on the key frame as output.　To the animation director model, the key frame created by the key animator in Step S1001 as well as the supplemental information selected in Step S1002 is input.

[0088] Instead of presenting a correction draft obtained from the animation director model to the key animator as it is, only an alert may be issued about a location to be corrected in the key frame, or what kind of correction is desirable may be presented in addition to the alert.

[0089] In this case, instead of presenting all correction drafts output from the animation director model to the key animator at the same time, correction drafts may be presented in the descending order of priority for utilization.　The determination of priority is not particularly limited.　For example, the animation director model may set the priority.　The animation director model may have learn the allocation of priority by the real animation director.

[0090] Furthermore, the reliability of the corrected key frame and the correction comment output from the animation director model varies depending on input data or output content.　Thus, how reliable the comment is output from the animation director model on the basis of internal representation in the form of numerical values such as reliability level.　Then, in Step S1003, among correction drafts output from the animation director model, only a given number of the ones with higher levels of reliability or the ones exceeding a given threshold may be presented for utilization.

[0091] Self-confirmation by key animator: In Step S1004, the key animator confirms the corrections and comments on the key frame output from the animation director model in Step S1003, and determines whether to directly shift to the confirmation by the animation director.

[0092] In this case, when the key animator selects to add corrections on the key frame on the basis of the corrections and comments output from the animation director model (NG in Step S1004), the flow returns to Step S1001, and the key animator corrects the key frame on the basis of the corrections and comments by the animation director model.　For the correction of the key frame, the key frame work UI (see Fig. 11) is used.　Each time the key animator corrects the key frame, a correction layer is added in the key frame work UI (described later).　Note that the key animator does not need to follow all corrections and comments output from the animation director model, and may make corrections only on a part concerned among the corrections and comments.

[0093] On the other hand, when the key animator selects not to correct the key frame (OK in Step S1004), the flow proceeds to subsequent Step S1005.

[0094] Confirmation by animation director: In Step S1005, the animation director receives the key frame that has passed the self-confirmation by the key animator in Step S1004, and the animation director adds corrections and comments on the key frame by himself or herself.　The animation director may use the annotation UI (see Fig. 8) to add corrections and comments on the key frame, or may write corrections and comments in a correction sheet in a conventional manner.

[0095] In this case, when the animation director wants to correct the key frame that has been self-confirmed by the key animator, that is, there is animation director's correction (NG in Step S1005), rework to the key animator occurs.　In other words, the flow returns to Step S1001, where the key frame is returned to the key animator together with corrections and comments, and an instruction to correct the key frame is made to the key animator.

[0096] On the other hand, when the animation director confirms and approves the key frame that has passed the self-confirmation by the key animator (OK in Step S1005), the work of the key frame is finished, and the key frame is transferred to the subsequent step.

[0097] The work data in the model utilizing unit 103 can be substituted by data collection.　In other words, when the animation director uses the annotation UI (described above) to perform animation director's correction, training data consisting of a set of data of [pre-correction key frame, corrected key frame, comment, and supplemental information] for each part in the key frame is concurrently collected, and is saved in the correction database.

[0098] In the flowchart illustrated in Fig. 10, Steps S1001 to S1004 are the work on the key animator side, and only Step S1005 is the work by the animation director.　In the case where the key frame is corrected on the basis of corrections and comments output from the animation director model in Step S1004 and the case where the key frame is corrected on the basis of corrections and comments added by the animation director in Step S1005, the flow similarly returns to Step S1001 and the key animator performs the key frame.　Note that the loop of Steps S1001 to S1004 is the work closed by the key animator, and hence even if the loop is repeated a plurality of times, the work can be performed efficiently in a short period of time.　On the other hand, the rework after the confirmation by the animation director in Step S1005 requires time, and long time is required if the rework is repeated a plurality of times.　Furthermore, the animation director confirms key frames by a plurality of key animators, and hence even if a key frame that has passed self-confirmation by a key animator is shifted to confirmation by the animation director, there is a standby time until the animation director starts the confirmation.

[0099] According to the present disclosure, the key animator uses the animation director model, and hence the step can be shifted to the confirmation by the animation director after a part that is likely to be corrected by the animation director has been corrected.　Thus, according to the present disclosure, although the animation director's correction cannot be completely omitted, the number of reworks can be expected to be reduced by the self-confirmation by the key animator utilizing the animation director model.　According to the present disclosure, the interactions between the animator and the animation director can be made efficient to reduce the work time.

[0100] Key frame work UI: Fig. 11 illustrates a configuration example of a key frame work UI screen used for the key animator to create and correct key frames.　The key frame work UI screen is configured in consideration of the application of the animation director model.　It should be noted that results of animation director's correction by the real animation director can be imported to the key frame work UI screen and used for the correction work of the key frame in response to the animation director's correction.　The key frame work UI screen is assumed to be utilized for the creation and correction of the key frame by the key animator in Step S1001 in the flowchart illustrated in Fig. 10.

[0101] An area indicated by reference numeral 1101 on the left side of the screen is a drawing field where a user (key animator) can perform drawing work.　The drawing field 1101 may have the same functions as those of general drawing software, which is not directly related to the present disclosure, and hence description of the drawing functions is omitted.

[0102] At the lower right of the screen, a "request for confirmation to animation director model" button indicated by reference numeral 1102 is disposed.　When the user pushes the "request for confirmation to animation director model" button 1102 after creating the key frame in the drawing field 1101, the key frame created in the drawing field 1101 is input to the animation director model, and corrections and comments on the key frame from the animation director model can be obtained.

[0103] When the "request for confirmation to animation director model" button 1102 is pushed, in background, the key frame is input to the animation director model together with supplemental information in the drawing field 1101, and corrections and comments on the key frame output from the animation director model can be obtained.　Then, in the drawing field 1101, a "correction layer" for displaying the corrected key frame output from the animation director model is added.　Highlight indicating a correction part from the animation director model is superimposed on the correction layer (not shown).　Furthermore, in an "animation director comment" field indicated by reference numeral 1103 at the upper right of the screen, comments acquired from the animation director model are displayed as text.

[0104] In the "key animator self-confirmation" at Step S1004 in the flowchart illustrated in Fig. 10, when the key animator selects to add correction on the key frame on the basis of the corrections and comments output from the animation director model, the flow returns to Step S1001, and the processing of the correction by the key animator and the acquisition of corrections and comments from the animation director model is repeated.　Each time the user repeats the processing of the correction of the key frame and the acquisition of corrections and comments from the animation director model is repeated, correction layers indicating corrected key frames such as correction layer 1, correction layer 2, … are sequentially added.　Then, at the lower end of the drawing field 1101, layer tabs 1111, 1112, 1113, 1113… for selecting layers are provided.　By selecting the tab of a desired layer, the key animator causes the layer of the key frame to be corrected to be displayed on top of the drawing field 1101, and can make corrections.　Furthermore, a selected particular layer may be displayed in a translucent manner.　On the other hand, in the "animation director comment" field 1103, comments for each of the correction layers such as correction layer 1, correction layer 2, … are displayed in a timeline manner.

[0105] The configurations and functions of the key frame work UI screen have been described in consideration of the application of the animation director model to the creation of the key frame, but it should be understood that even when the animation director model is not applied and correction instructions and comments are mainly received from the actual animation director, the key frame work UI screen illustrated in Fig. 11 can be used.　Furthermore, the key frame work UI screen may be configured such that corrections and comments from the actual animation director are imported.　For example, after a key frame is corrected by using the animation director model and the self-confirmation by the key animator is finished, when the "request for confirmation to animation director model" button 1102 is pushed, the key frame drawn in the drawing field 1101 is transferred to a tablet terminal or a PC of the actual animation director.　Then, the animation director uses the annotation UI to perform animation director's correction (described above), and correction instructions and comments input by using the annotation UI are imported to the key frame work UI screen.　On the key frame work UI screen, a correction layer where the correction location from the animation director is highlighted is added, and comments from the animation director are displayed in the "animation director comment" field 1103.

[0106] In the example illustrated in Fig. 11, it is assumed that the correction layer 1 and the correction layer 2 are corrections by the animation director model and the last correction layer 3 is the result of importing animation director's correction by an animation director (animation director xx) that has passed the self-confirmation by the key animator.　In the "animation director comment" field 1103, in order to discriminate comments generated by the animation director model and comments imported by the animation director's correction, each comment is provided with a creator of the comment such as "(model output)" and "(animation director xx)".

[0107] Note that, as described above, the key animator uses supplemental information such as plots, screenplay, script, setting, scenario, storyboard, and layout of the animation work as information on context related to drawing.　The key frame work UI may further include a UI component (not shown in Fig. 11) for selecting supplemental information used for the drawing.　In this case, the animation director model inputs a key frame created in the drawing field 1101 and supplemental information, and outputs corrections and comments on the key frame.　The feature that the animation director model inputs the key frame and supplemental information and outputs corrections and comments on the key frame is as already described above with reference to Fig. 9.

[0108] C. Modifications In Section C, modifications and applications of the information processing system 100 are summarized.

[0109] (1) In the information processing system 100, training data for an animation director model may be acquired from time-series corrections concluded by a key animator him / herself rather than from corrections and comments communicated between the key animator and the animation director.　In such a case, training data can be utilized from the initial stage of the drawing work, and an animation director model that can be used for assistance for a lower level of beginners can be constructed.　For example, on the key frame work UI screen illustrated in Fig. 11, a correction layer for displaying a corrected key frame concluded by the key animator him / herself may be added in the drawing field 1101, and correction comments by the key animator him / herself may be added in the comment field 1103.　In the "animation director comment" field 1103, the correction comment by the key animator him / herself is provided with indication that the creator of the comment is the key animator him / herself, such as "(key animator yy)".

[0110] (2) In the above, the embodiment of the present disclosure in which the "animation director model"　that intervenes in the work flow of the "creation" of the key frame in animation by the key animator, the "feedback" of corrections and comments by the animation director, and the "correction" of the key frame by the key animator is constructed and operated has been described.　The present disclosure is applicable to various production processes by substituting a model depending on a target flow.　For example, a model in which the present disclosure is applied to 3DCG production can be learned and utilized.

[0111] (3) An animation director in animation production has a role of unifying the drawings of animation.　From this viewpoint, the animation director model constructed in the present disclosure is intended to mainly point out the overall consistency in interactions between the key animator and the animation director.　On the other hand, there are expressions that may not be drawn in animation, and hence the animation director model needs to point out in other viewpoints.　Examples of the application include a case that, when there are marks that may not be drawn in view of compliance, the animation director model receives inputs of such viewpoints, checks the key frame, checks whether there is a mark that may not be drawn by the key animator, and points out the key animator.

[0112] (4) In the above, the embodiment of the present disclosure in which the animation director model intervenes in the presentation of corrections and comments from the animation director to the key animator has been described.　In the subsequent work, feedback obtained by discussing completed video with a plurality of persons (such as rush check) is conceivable.　It is conceivable to provide an animation director model that intervenes such a step to accumulate comments from a plurality of persons and uses the comments for model learning, thereby performing correction feedbacks in various viewpoints.

[0113] (5) In animations, there are a large number of works based on cartoons and novels.　In a case where the animation director model outputs corrections and comments on key frames in such works, the model may be learned by not only referring to original information but also referring to information that occurs during the original production, for example, correction comments between the original author and the editor.

[0114] D. Configuration of information processing device Fig. 12 illustrates a hardware configuration example of an information processing device 2000 capable of operating as the information processing system 100 according to the present disclosure.　The illustrated information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013.　The information processing device 2000 is configured by, for example, a personal computer, but a part of the functions may be configured by an information terminal such as a tablet and a smartphone.

[0115] The CPU 2001 controls the general operation of the information processing device 2000 in accordance with various kinds of programs.　Note that, in a case where processing with high calculation load such as the learning of the animation director model (described above) is implemented on the information processing device 2000, it is desired that the CPU 2001 be a multi-core CPU (for example, Apple M1 Max) and that the information processing device 2000 further include, in addition to the CPU 2001, a multicore processor (for example, NVIDIA R6000) such as GPU (Graphics Processing Unit) and GPGPU (General-purpose computing on graphics processing units).　Note that those are correctly simply referred to as "CPU 2001" in the following for the sake of description.

[0116] The ROM 2002 stores programs (such as basic input / output system) and arithmetic parameters used by the CPU 2001 therein in a non-volatile manner.　The RAM 2003 is used to load a program used in execution of the CPU 2001 or temporarily store parameters such as work data that changes as appropriate during the execution of the program.　Examples of the program to be loaded to the RAM 2003 and executed in the CPU 2001 include various application programs and an operating system (OS).

[0117] The CPU 2001, the ROM 2002, and the RAM 2003 are mutually connected by a host bus 2004 configured by a CPU bus.　Then, with cooperative operations with the ROM 2002 and the RAM 2003, the CPU 2001 can execute various kinds of application programs under execution environments provided by the OS, thereby implementing various functions and service.　In a case where the information processing device 2000 is a personal computer, the OS is, for example, Windows (registered trademark) by Microsoft Corporation, Unix (registered trademark), or its successor OS.　For example, application programs as described below are executed on the information processing device 2000.

[0118] Data collection application: an application program for collecting training data for learning an animation director model, which includes codes for executing processing of converting, in addition to key frames before and after animation director's correction necessary for learning the animation director model, supplemental information from existing data and processing of newly creating training data for creating an animation director model.　Model learning application: Application including a code for executing processing of learning an animation director model on the basis of training data collected by the data collection application. Model utilizing application: Application including a code for executing processing of performing drawing work by using the animation director model learned by the model learning application.

[0119] The host bus 2004 is connected to the expansion bus 2006 through the bridge 2005.　The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or a PCI Express, and the bridge 2005 is based on PCI standards.　Note that the information processing device 2000 does not need to be configured such that circuit components are separated by the host bus 2004, the bridge 2005, and the expansion bus 2006, and almost all the circuit components may be mutually connected by a single bus (not shown).

[0120] The interface unit 2007 connects peripheral devices such as the input unit 2008, the output unit 2009, the storage unit 2010, the drive 2011, and the communication unit 2013 in accordance with standards of the expansion bus 2006.　However, not all of the peripheral devices illustrated in Fig. 12 are essential, and the information processing device 2000 may further include a peripheral device (not shown).　Furthermore, the peripheral devices may be built in the main body of the information processing device 2000, or a part of the peripheral devices may be externally connected to the main body of the information processing device 2000.

[0121] The input unit 2008 is formed by an input control circuit configured to generate an input signal on the basis of input from a user and output the signal to the CPU 2001.　In a case where the information processing device 2000 is a personal computer, the input unit 2008 may include a keyboard, a mouse, and a touch panel, and may include a camera and a microphone.　Furthermore, the output unit 2009 includes, for example, a display device such as a liquid crystal display (LCD) device, an organic EL (Electro-Luminescence) display device, or an LED (Light Emitting Diode) and an acoustic output device such as a speaker.　The input unit 2008 is used to input moving images to be processed, and the output unit 2009 is used to display a GUI screen.

[0122] The storage unit 2010 stores therein files of programs (such as applications and OS) executed by the CPU 2001 and various kinds of data.　For example, the storage unit 2010 is configured by a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may include an external storage device.

[0123] A removable storage medium 2012 is a storage medium configured in the form of a cartridge, such as a micro SD card.　The drive 2011 performs reading and writing operation on the removable storage medium 113 mounted.　The drive 2011 outputs data read from the removable recording medium 2012 to the RAM 2003 or the storage unit 2010, and writes data on the RAM 2003 and the storage unit 2010 into the removable recording medium 2012.

[0124] The communication unit 2013 is a device configured to perform wireless communication, such as Wi-Fi (registered trademark), Bluetooth (registered trademark), and cellular communication networks such as 4G and 5G.　Furthermore, the communication unit 2013 may be provided with terminals for USB (Universal Serial Bus) and HDMI (registered trademark) (High-Definition Multimedia Interface), and may further have a function of performing communication with a USB device such as a scanner and a printer and HDMI (registered trademark) communication with a display.　A program to be executed on the information processing device 2000 is installed, for example, from the outside through the communication unit 2013.

[0125] The present disclosure has been described above in detail with reference to the particular embodiments.　However, the present disclosure is not intended to be interpreted as being limited to the above-mentioned embodiments, and it is obvious that a person skilled in the art could modify and substitute the embodiments within the range not departing from the gist of the present disclosure.　Furthermore, the effects described herein are merely an example, and the effects obtained by the present disclosure are not limited.　Additional effects that are not described in the present description may be obtained.

[0126] For the simplicity of description, the embodiment in which the present disclosure is applied to the animation production process based on a single-layered structure of an animator and an animation director has been described in the present description, but the present disclosure is similarly applicable to an animation production process with a multilayered structure where there are a plurality of animation directors and there is a chief animation director who oversees the animation directors.

[0127] Furthermore, the present disclosure has been described in the present description, mainly focusing on the embodiment in which the animation director model that can be applied to the key frame work in the drawing work in the animation production process is constructed and operated, but the gist of the present disclosure is not limited thereto.　The present disclosure is applicable to the construction and operation of a learning model that is applied to work other than the key frame work in the animation production process for efficient work.　Furthermore, the scope of application of the present disclosure is not limited to animation production, and can be extended to production of various video content such as live-action moving images (such as drama, movie, and theatrical show), and production of content other than video, such as cartoons, novels, and musical compositions.　In addition, the scope of application of the present disclosure is not limited to production of content, and the present disclosure can construct and operate learning models for acquiring, guiding, and instructing engaged persons in various business fields such as education, sports, craftwork, manufacturing, and medical cares.

[0128] In other words, the present disclosure has been described by way of exemplification, and the content described in the present description should not be interpreted in a limited manner.　To determine the gist of the present disclosure, the scope of the claims should be taken into consideration.

[0129] A series of processing described in the present description can be executed by hardware, software, or a combined configuration of hardware and software.　In the case of executing the processing by software, a program having a processing sequence related to the implementation of the present disclosure recorded therein is installed in a memory in a computer embedded in dedicated hardware, and executed.　The program may be installed in a general-purpose computer capable of executing various kinds of processing so that the processing related to the implementation of the present disclosure is executed.

[0130] The program can be stored in advance in a recording medium mounted in the computer such as an HDD, an SSD, and a ROM as a recording medium.　Alternatively, the program can be temporarily or permanently stored in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disc, a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disk, or a USB (Universal Serial Bus) memory.　By using such a removable recording medium, the program related to the implementation of the present disclosure can be provided as what is called package software.

[0131] Furthermore, the program may be transferred from a download site to the computer in a wireless or wired manner through a network such as a WAN (Wide Area Network) represented by cellular network, a LAN (Local Area Network), and the Internet.　The computer can receive the transferred program and install the program in a large-capacity storage device such as an HDD and an SSD in the computer.

[0132] Note that the present disclosure can employ the following configurations.

[0133] (1) An information processing system, comprising: 　　　circuitry configured to 　　　　　　obtain a frame created by an animator and supplemental information associated with an animation work; 　　　　　　input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; 　　　　　　generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and 　　　　　　output the frame, the correction layer, and the corresponding comments in distinct visual layers. 　　　　　　(2) The system of (1), wherein 　　　the circuitry is configured to receive a corrected version of the frame that selectively incorporates the proposed modifications. 　　　(3) The system of any of (1) to (2), wherein 　　　the supplemental information includes at least one of a script, storyboard, layout, scenario, or setting. (4) The system of (3), wherein 　　　the supplemental information is selected based on cut metadata associated with the frame. 　　　(5) The system of any of (1) to (4), wherein 　　　the correction layer comprises a visual representation that highlights regions of the frame associated with the proposed modifications. 　　　(6) The system of (5), wherein 　　　the visual representation is presented with a bounding box around a part identified for correction. 　　　(7) The system of any of (1) to (6), wherein 　　　 the machine-learned model comprises a vision-language model trained using annotated frames and associated comments. 　　　 (8) The system of any of (1) to (8), wherein 　　　the comments are displayed adjacent to the distinct visual layers within a user interface. 　　　(9) The system of any of (1) to (8), wherein 　　　the comments include priority data indicating an order for implementing corrections. (10) The system of any of (1) to (9), wherein 　　　the circuitry is configured to filter the proposed modifications based on a confidence threshold associated with model output. 　　　(11) The system of any of (1) to (10), wherein 　　　the model outputs a reliability score indicating confidence in each proposed comment. 　　　(12) The system of any of (1) to (11), wherein 　　　the distinct visual layers include model-generated layers and animation-director-generated layers. (13) The system of any of (1) to (12), wherein 　　　the model is retrained over time using feedback from human reviewers. (14) The system of any of (1) to (13), wherein 　　the model is trained using corrections associated with one or more animation directors. (15) The system of (14), wherein 　　the model is configured to generate proposed corrections and comments that reflect stylistic preferences or correction trends specific to the one or more animation directors. (16) The system of any of (1) to (15), wherein 　　　the model is trained using historical corrections extracted via optical character recognition from correction sheets. 　　　(17) The system of any of (1) to (16), wherein 　　　the correction layer is generated for each part of the frame. 　　　 (18) The system of any of (1) to (17), wherein 　　　the circuitry is configured to retrieve supplemental information from a past work database. (19) A method of processing animation data, the method comprising: 　　　obtaining a frame created by an animator and supplemental information associated with an animation work; 　　　inputting the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; 　　　generating, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and 　　　outputting the frame, the correction layer, and the corresponding comments in distinct visual layers. 　　　(20) One or more non-transitory computer readable media including instructions, which when executed by circuitry, cause the circuitry to: 　　　obtain a frame created by an animator and supplemental information associated with an animation work; 　　　input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments; 　　　generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and 　　　output the frame, the correction layer, and the corresponding comments in distinct visual layers. (21) An information processing system, including: an acquisition unit configured to acquire a key frame related to animation production and created by a first user; and an output unit configured to input the key frame to a machine learning model in which a relation of a correction matter on the key frame by a second user different from the first user has been learned, and output the correction matter on the key frame. (22) The information processing system according to (21), in which: the acquisition unit is configured to further acquire supplemental information on the key frame; and the output unit is configured to input the supplemental information, together with the key frame, to the machine learning model, and output the correction matter on the key frame. (23) The information processing system according to (22), in which the supplemental information is information including at least one of plot information, playbook information, script information, setting information, scenario information, storyboard information, and layout information on the animation production. (24) The information processing system according to any one of (21) to (23), in which the output unit is configured to output, as the correction matter on the key frame, at least one of a corrected key frame obtained by correcting the key frame and a comment on the correction of the key frame. (25) The information processing system according to any one of (21) to (24), further including: a collection unit configured to collect training data for learning the machine learning model; and a learning unit configured to learn the machine learning model by using the training data collected by the collection unit. (26) The information processing system according to (25), further including a correction matter input unit configured to input the correction matter on the key frame by the second user, in which the collection unit is configured to collect the training data on the basis of information input to the correction matter input unit by the second user. (27) The information processing system according to (26), in which the collection unit is configured to disassemble the key frame into parts, present to the second user a user interface, which is used for the second user to select a part that is to be corrected and write a correction matter on the selected part, and collect, on the basis of the input on the user interface by the second user, training data including a relation of a correction matter on the key frame for each part. (28) The information processing system according to (27), in which the user interface includes: a key frame display unit configured to display a key frame before correction or after correction; a correction unit configured to select a part that is to be corrected in the key frame and write a correction matter on the selected part; and a confirmation unit configured to confirm correction content written by the correction unit. (29) The information processing system according to any one of (26) to (28), in which the collection unit is configured to disassemble the key frame into parts on the basis of pre-correction key frames created by the first user in the past, corrected key frames corrected by the first user on the basis of correction matters on the key frame by the second user, and supplemental information, detect a correction matter on each part by the second user, and collect training data including a relation of a correction matter on the key frame for each part. (30) The information processing system according to any one of (21) to (29), in which the acquisition unit is configured to acquire the key frame through a key frame work unit configured to perform work on the key frame by the first user. (31) The information processing system according to (30), in which the first user performs correction work on the key frame on the basis of a correction matter on the key frame through the key frame work unit. (32) The information processing system according to (31), in which the key frame work unit is configured to present correction matters respectively by the second user and the machine learning model in a layered manner. (33) The information processing system according to (32), wherein the key frame work unit includes a user interface including a key frame display unit configured to display a key frame that is to be corrected for each layer, and a correction matter presentation unit configured to present a correction matter on the key frame for each layer. (34) An information processing method, including: an acquisition step of acquiring a key frame related to animation production and created by a first user; and an output step of inputting the key frame to a machine learning model in which a relation of a correction matter on the key frame by a second user different from the first user, and outputting the correction matter on the key frame. (35) A computer program written in a computer-readable form for causing a computer to function as: an acquisition unit configured to acquire a key frame related to animation production and created by a first user; and an output unit configured to input the key frame to a machine learning model in which a relation of a correction matter on the key frame by a second user different from the first user, and output the correction matter on the key frame.

[0134] 100 Information processing system 101 Data collection unit 102 Model learning unit 103 Model utilizing unit 202 Parts disassembly unit 202 Annotation unit 203 Correction database 301 Input unit 302 Parts disassembly unit 303 Comment detection unit 304 Parts / comment association unit 305 Data storage unit 306 Past work database 901 Image encoder 902 Text encoder 903 Image decoder 904 Text decoder 905 Work database 2000 Information processing device 2001 CPU 2002 ROM 2003 RAM 2004 Host bus 2005 Bridge 2006 Expansion bus 2007 Interface unit 2008 Input unit 2009 Output unit 2010 Storage unit 2011 Drive 2012 Removable recording medium 2013 Communication unit

Claims

An information processing system, comprising:　　　circuitry configured to　　　　　　obtain a frame created by an animator and supplemental information associated with an animation work;　　　　　　input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments;　　　　　　generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and　　　　　　output the frame, the correction layer, and the corresponding comments in distinct visual layers.　　　　　　　　　The system of claim 1, wherein　　　the circuitry is configured to receive a corrected version of the frame that selectively incorporates the proposed modifications.　　　　　　The system of claim 1, wherein　　　the supplemental information includes at least one of a script, storyboard, layout, scenario, or setting.　　　　　　The system of claim 3, wherein　　　the supplemental information is selected based on cut metadata associated with the frame.　　　　　　The system of claim 1, wherein　　　the correction layer comprises a visual representation that highlights regions of the frame associated with the proposed modifications.　　　　　　The system of claim 5, wherein　　　the visual representation is presented with a bounding box around a part identified for correction.　　　　　　 The system of claim 1, wherein　　　 the machine-learned model comprises a vision-language model trained using annotated frames and associated comments.　　　　　　The system of claim 1, wherein　　　the comments are displayed adjacent to the distinct visual layers within a user interface.　　　　　　The system of claim 1, wherein　　　the comments include priority data indicating an order for implementing corrections.　　　The system of claim 1, wherein　　　the circuitry is configured to filter the proposed modifications based on a confidence threshold associated with model output.　　　　　　The system of claim 1, wherein　　　the model outputs a reliability score indicating confidence in each proposed comment.　　　　　　The system of claim 1, wherein　　　the distinct visual layers include model-generated layers and animation-director-generated layers.　　　　　　The system of claim 1, wherein　　　the model is retrained over time using feedback from human reviewers.　　The system of claim 1, wherein　　the model is trained using corrections associated with one or more animation directors.　　The system of claim 14, wherein　　the model is configured to generate proposed corrections and comments that reflect stylistic preferences or correction trends specific to the one or more animation directors.　　　The system of claim 1, wherein　　　the model is trained using historical corrections extracted via optical character recognition from correction sheets.　　　　　　The system of claim 1, wherein　　　the correction layer is generated for each part of the frame.　　　　　　The system of claim 1, wherein　　　the circuitry is configured to retrieve supplemental information from a past work database.　　　　　　A method of processing animation data, the method comprising:　　　obtaining a frame created by an animator and supplemental information associated with an animation work;　　　inputting the frame and the supplemental information into a machine-learned model trained to predict corrections and comments;　　　generating, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and　　　outputting the frame, the correction layer, and the corresponding comments in distinct visual layers.　　　　　　One or more non-transitory computer readable media including instructions, which when executed by circuitry, cause the circuitry to:　　　obtain a frame created by an animator and supplemental information associated with an animation work;　　　input the frame and the supplemental information into a machine-learned model trained to predict corrections and comments;　　　generate, based on output of the model, a correction layer including proposed modifications to the frame and corresponding comments; and　　　output the frame, the correction layer, and the corresponding comments in distinct visual layers.

Images