Video evaluation system, video evaluation method, and program

Abstract

To provide technology that enables more efficient capture of highly effective video footage. [Solution] One aspect of the present invention is a video evaluation system comprising: a simulation device that performs a real-environment simulation with respect to a real environment, which is the actual environment in which a user takes video footage to present to a viewer, and simulates the state of the real environment and the taking of video footage in the real environment; and an evaluation device that evaluates the results of the real-environment simulation from a first evaluation perspective relating to the purpose of presenting the video footage to the viewer. The real-environment simulation reproduces the real environment in a virtual space and outputs video footage taken in the virtual space by a virtual camera as the simulation result. The evaluation device outputs an evaluation result of the video footage for the purpose by inputting the video footage taken in the real-environment simulation into a predetermined trained model.

Description

Technical Field

[0001] The present invention relates to the technology of a moving image evaluation system, a moving image evaluation method, and a program.

Background Art

[0002] Conventionally, for a production such as a moving image, a technique has been proposed in which an effect (presentation effect) that the production gives to a presentation target is estimated using a learned model constructed by machine learning, and the production is evaluated based on the estimation result (see, for example, Patent Document 1 and Non-Patent Document 1). According to such a technique, a user can efficiently produce a production with a high presentation effect.

[0003] On the other hand, conventionally, a technique has been proposed for improving the efficiency of simulating the shooting of a moving image in a real environment (see, for example, Non-Patent Document 2). According to the technique of Non-Patent Document 2, since the user can perform shooting in the real environment after determining a convincing shooting pattern by simulation, a desired moving image can be efficiently created.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] However, conventional technologies sometimes fail to efficiently capture highly effective video images that meet the intended purpose. For example, in the technologies described in Patent Document 1 and Non-Patent Document 1, increasing the number of objects to be evaluated in order to enhance the presentation effect leads to a bottleneck in the shooting process in the actual environment, resulting in a trade-off between quality improvement and efficiency. This remains true even when much of the shooting is replaced with simulation using the technology described in Non-Patent Document 2.

[0007] In view of the above circumstances, the present invention aims to provide a technology that enables the more efficient capture of moving images that are effective in presenting a specific purpose. [Means for solving the problem]

[0008] One aspect of the present invention is a video evaluation system comprising: a simulation device that performs a real-environment simulation with respect to a real environment, which is the actual environment in which a user takes video footage to present to a viewer, and simulates the state of the real environment and the taking of video footage in the real environment; and an evaluation device that evaluates the results of the real-environment simulation from a first evaluation perspective relating to the purpose of presenting the video footage to the viewer, wherein the real-environment simulation reproduces the real environment in a virtual space and outputs video footage taken in the virtual space by a virtual camera as the simulation result, and the evaluation device evaluates the results of the real-environment simulation based on the video footage as the simulation result or the parameters of the real-environment simulation.

[0009] One aspect of the present invention is the above-described video evaluation system, wherein the evaluation device takes video footage captured in the virtual space as input and outputs an evaluation result for the video footage for the aforementioned purpose to a trained model that has been trained to output an evaluation result for the video footage from the first evaluation viewpoint of the video footage.

[0010] One aspect of the present invention is the above-described video evaluation system, wherein the evaluation device takes the parameters to be evaluated as input to a trained model that has been trained to output evaluation results from a first evaluation viewpoint of a video acquired by a real-world simulation using the parameters to be evaluated, thereby outputting evaluation results for the aforementioned purpose of the video acquired by a real-world simulation using the parameters to be evaluated.

[0011] One aspect of the present invention is the above-described motion image evaluation system, further comprising a shooting plan generation device that generates a shooting plan in which information for capturing motion images obtained as a result of the real-environment simulation in the real environment is described in a manner that is understandable to humans, based on the parameters, wherein the evaluation device outputs an evaluation result of the motion images for the purpose by inputting the shooting plan to be evaluated into a trained model that has been trained to output the result of evaluating motion images obtained by a real-environment simulation using the parameters related to the shooting plan from a first evaluation perspective, with the shooting plan to be evaluated as input.

[0012] One aspect of the present invention is the above-described motion image evaluation system, further comprising a scanning device for reading information of the actual environment, wherein the simulation device reproduces the actual environment in a virtual space using the scan results obtained by the scanning device for reading information of the actual environment.

[0013] One aspect of the present invention is the above-described motion image evaluation system, wherein the simulation device reproduces sound, light, or weather in the real environment in the virtual space during the real environment simulation.

[0014] One aspect of the present invention is the above-described motion image evaluation system, further comprising a parameter generation device that generates parameters used by the simulation device to perform the real-world simulation, wherein the parameter generation device has a function to generate a new parameter set by changing part or all of the parameter set, and repeatedly performs the modification of the parameter set by the parameter generation device and the execution of the real-world simulation with the modified parameter set by the simulation device.

[0015] One aspect of the present invention is the above-described motion image evaluation system, wherein the parameter generation device is This method involves inputting a third parameter set, which is the target of estimation, into a trained model that has learned the relationship between a first parameter set of the real-world simulation and the evaluation results of the real-world simulation using a second parameter set, which has some parameter values ​​different from the first parameter set and obtained higher evaluation results than the evaluation results of the first parameter set. This allows for the estimation of a fourth parameter set that can obtain higher evaluation results than the third parameter set.

[0016] One aspect of the present invention is the above-described motion image evaluation system, wherein the parameter generation device takes a motion image as input and estimates parameters for reproducing the motion image in the real-world simulation by inputting the motion image to be reproduced into a trained model that has been trained to take a motion image as input and output parameters for acquiring the motion image as the result of the real-world simulation.

[0017] One aspect of the present invention is the above-described video evaluation system, wherein the user inputs a second evaluation perspective of the real-world simulation in text format to the evaluation device via the user's terminal device, and the evaluation device evaluates the results of the real-world simulation based on the first or second evaluation perspective.

[0018] One aspect of the present invention is a video evaluation method comprising: a first step in which a simulation device performs a real-environment simulation with respect to a real environment, which is the actual environment in which a user takes video footage to present to a viewer, simulating the state of the real environment and the taking of the video footage in the real environment; and a second step in which an evaluation device evaluates the results of the real-environment simulation from a first evaluation viewpoint relating to the purpose of presenting the video footage to the viewer, wherein the real-environment simulation reproduces the real environment in a virtual space and outputs video footage taken in the virtual space by a virtual camera as the simulation result, and the second step evaluates the results of the real-environment simulation based on the video footage as the simulation result or the real-environment simulation parameters.

[0019] One aspect of the present invention is a program for causing one or more processors to perform a real-environment simulation, which simulates the state of a real environment and the shooting of the video in the real environment, with respect to a real environment which is the actual environment in which a user shoots video to present to a viewer, and a second step of evaluating the results of the real-environment simulation from a first evaluation perspective relating to the purpose of presenting the video to the viewer, wherein the real-environment simulation reproduces the real environment in a virtual space and outputs a video as the simulation result, which is shot in the virtual space by a virtual camera, and the second step evaluates the results of the real-environment simulation based on the video as the simulation result or the real-environment simulation parameters. [Effects of the Invention]

[0020] According to the present invention, it becomes possible to more efficiently capture a moving image with a high presentation effect that matches the purpose.

Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing an example of the system configuration of the moving image evaluation system 1A of the first embodiment. [Figure 2] It is a diagram showing an example of the functional configuration of the simulation device 300 of the first embodiment. [Figure 3] It is a diagram showing an example of the functional configuration of the shooting plan generation device 400 of the first embodiment. [Figure 4] It is a diagram showing an example of the functional configuration of the evaluation device 500 of the first embodiment. [Figure 5] It is an image diagram (part 1) showing an outline of the actual environment simulation. [Figure 6] It is an image diagram (part 2) showing an outline of the actual environment simulation. [Figure 7] It is a diagram showing an example of the processing flow of the moving image evaluation system 1A of the first embodiment. [Figure 8] It is a diagram showing an example of the system configuration of the moving image evaluation system 1B of the second embodiment. [Figure 9] It is a diagram showing an example of the system configuration of the moving image evaluation system 1C of the third embodiment. [Figure 10] It is a diagram for explaining a third modification example of the moving image evaluation system 1 of the embodiment.

Modes for Carrying Out the Invention

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0023] <First Embodiment> Figure 1 shows an example of the system configuration of the video evaluation system 1A of the first embodiment. The video evaluation system 1A is a system that simulates the shooting of video in a real environment R, evaluates the simulation results (video acquired in the simulation) and presents them to the user U. The video evaluation system 1A of the embodiment enables the user U to efficiently produce video with a high presentation effect that matches the purpose by applying the shooting conditions for which a highly rated simulation result was obtained to the real environment R. The video may be an image, a video that plays multiple time-series images, or both. For example, the video may be something that the user U presents to viewers for the above purpose. As an example, the video may be, for example, a video that promotes an advertisement, in which case the purpose may be to enhance the effectiveness of the advertisement.

[0024] The video evaluation system 1A includes, for example, a user terminal device 100, a parameter generation device 200, a simulation device 300, a shooting plan generation device 400, and an evaluation device 500. The user terminal device 100, the parameter generation device 200, the simulation device 300, the shooting plan generation device 400, and the evaluation device 500 can communicate with each other via a network NW. The network NW may be a wireless communication network or a wired communication network. The network NW may be configured using, for example, the Internet or a local area network (LAN). The network NW may be configured by combining multiple networks.

[0025] The user terminal device 100 is a terminal device used by user U of the video evaluation system 1A. For example, the user terminal device 100 may be a smartphone, tablet, personal computer, or other terminal device. A user interface operates on the user terminal device 100. User U can communicate with the video evaluation system 1A by operating the user interface. The user interface may be a dedicated application program or a web application provided via a web browser.

[0026] The parameter generation device 200 is a device that generates parameters for the process (hereinafter referred to as "real-world simulation process") in which the simulation device 300, described later, simulates the capture of moving images in the real environment R in a virtual space. The parameter generation device 200 receives parameter setting operations from the user terminal device 100 and supplies the set parameters to the simulation device 300 and the shooting plan generation device 400. The parameters are for reproducing the shooting in the real environment R in a virtual space. More specifically, the parameters include parameters for virtually reproducing the state of the space to be shot in the real environment R (space reproduction parameters) and parameters for shooting the state of the virtually reproduced space (virtual space) with a virtual camera (shooting parameters). The space reproduction parameters include parameters related to the shape and decoration of the virtual space, and parameters related to the placement and movement of objects in the virtual space. The shooting parameters include parameters related to the control of camera work. 3D objects include static objects such as tables, chairs, and desks, as well as dynamic objects that move within a virtual space (such as people walking, animals running, machines in motion, and other moving objects).

[0027] In addition to generating and modifying parameters based on user settings, the parameter generator 200 can also generate a new parameter set by modifying a portion of the parameter set used in a single real-world simulation (hereinafter referred to as the "parameter set"). The parameter generator 200 may randomly select the parameters to be modified, or it may select the parameters to be modified based on the simulation results of the original parameter set. Furthermore, the parameter generator 200 may randomly change the parameter values, or it may determine the modified parameter values ​​based on the simulation results of the original parameter set.

[0028] The simulation device 300 is a device that performs a real-world simulation. More specifically, the simulation device 300 performs a first simulation that reproduces the state of the virtual space in a time series, and a second simulation that captures the state of the virtual space reproduced by the first simulation in a time series. The simulation device 300 supplies the video images captured by the second simulation to the evaluation device 500 as simulation results. The simulation device 300 may also supply the evaluation device 500 with the simulation results including the parameter set used in the real-world simulation.

[0029] The shooting plan generation device 400 is a device that generates information (hereinafter referred to as "shooting plan") that shows a plan for user U to reproduce the simulation results in the real environment R. The shooting plan generation device 400 generates a shooting plan based on parameters supplied from the parameter generation device 200. For example, the shooting plan includes information for reproducing the simulation environment reproduced in the virtual space in the real environment R. For example, if the real environment simulation reproduces acting by a virtual actor (an example of a dynamic object defined by parameters), the shooting plan may include information such as a script or storyboard that instructs the real actor performing in the real environment R on the content of the virtual actor's performance (e.g., position, movement, facial expressions, lines, etc.). The shooting plan generation device 400 supplies the generated shooting plan to the evaluation device 500.

[0030] As described above, the state of the virtual space reproduced in the real-world simulation is based on the parameters generated by the parameter generation device 200; therefore, the shooting plan is essentially synonymous with the parameters of the real-world simulation. However, parameters are data expressed in a manner that the simulation device 300 can interpret, and are generally not expressed in a manner that a person can visually understand. Therefore, the shooting plan generation device 400 generates the shooting plan by converting the content and meaning of the parameters of the real-world simulation into a format that a person can visually understand. The shooting plan only needs to contain the information necessary for user U to perform shooting in the real-world environment R; it does not necessarily need to contain information corresponding to all parameters.

[0031] The evaluation device 500 is a device that evaluates the simulation results from the simulation device 300. More specifically, the evaluation device 500 scores each of the video images provided by the simulation device 300 as a simulation result, determining how well it matches the objective. Based on the scores of multiple video images generated by real-world simulations for the same objective, the evaluation device 500 determines which video images to present to the user U and supplies a shooting plan for those video images to the user terminal device 100.

[0032] User U can create motion images that meet the purpose (high presentation effect) by performing shooting in a real environment using the shooting plan supplied to the user terminal device 100.

[0033] Figure 2 shows an example of the functional configuration of the simulation device 300 of the first embodiment. The simulation device 300 includes, for example, a parameter input unit 310, a storage unit 320, and a control unit 330. The control unit 330 is configured using, for example, a processor such as a CPU (Central Processing Unit) and memory. The control unit 330 functions as a first simulation execution unit 331 and a second simulation execution unit 332 when the processor executes a program. Note that all or part of the functions of the control unit 330 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs: Solid State Drives), and storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0034] The parameter input unit 310 includes a network interface for communicating with other devices via a network NW and accepts parameter inputs from the parameter generation device 200 for performing a real-world simulation. The network interface may be a wireless communication device or a wired communication device. The parameter input unit 310 stores the input parameters in the storage unit 320.

[0035] The storage unit 320 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 320 stores the parameters of the real-world simulation input from the parameter generation device 200 via the parameter input unit 310. The storage unit 320 may be used as an area to store a program for performing the real-world simulation, an area to store temporary data generated during the execution of the real-world simulation, or an area to store the execution results of the real-world simulation.

[0036] The control unit 330 controls the first simulation execution unit 331 and the second simulation execution unit 332 to realize a real-world simulation. The first simulation execution unit 331 executes the first simulation, and the second simulation execution unit 332 executes the second simulation. By running the first and second simulations simultaneously, the control unit 330 generates moving images of the virtual space captured in time series. The control unit 330 supplies the moving images generated by the second simulation execution unit 332 to the evaluation device 500 as the result of the real-world simulation.

[0037] Figure 3 shows an example of the functional configuration of the imaging plan generation device 400 of the first embodiment. The imaging plan generation device 400 includes, for example, a parameter input unit 410, a storage unit 420, and an imaging plan generation unit 430. The imaging plan generation unit 430 is configured using, for example, a processor such as a CPU (Central Processing Unit) and memory. The imaging plan generation unit 430 is realized by the processor executing a program. Note that all or part of the imaging plan generation unit 430 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs: Solid State Drives), and storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0038] The parameter input unit 410 includes a network interface for communicating with other devices via a network NW and accepts parameter input from the parameter generation device 200 for generating an imaging plan. The network interface may be a wireless communication device or a wired communication device. The parameter input unit 410 stores the input parameters in the storage unit 420.

[0039] The storage unit 420 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 420 stores the parameters of the real-world simulation input from the parameter generation device 200 via the parameter input unit 410. The storage unit 420 may be used as an area to store a program for performing the process of generating an imaging plan, as an area to store temporary data generated during the execution of the process of generating an imaging plan, or as an area to store the generated imaging plan.

[0040] The shooting plan generation unit 430 generates a shooting plan based on the parameters of the real-world simulation input from the parameter generation device 200. For example, the shooting plan generation unit 430 may generate a shooting plan by converting the parameters of the virtual space into a predetermined format (such as a script or storyboard) that is understandable to humans. The conversion may be performed using a rule-based conversion model, or using a trained model that has learned the relationship between parameters and format through machine learning. Furthermore, the shooting plan generation unit 430 may be configured to output a shooting plan with improved readability by using a natural language processing model such as an LLM (Large Language Model). The shooting plan generation unit 430 supplies the generated shooting plan to the evaluation device 500. Any machine learning model may be used, such as a neural network, deep learning, or reinforcement learning.

[0041] Figure 4 shows an example of the functional configuration of the evaluation device 500 according to the first embodiment. The evaluation device 500 includes, for example, a simulation result input unit 510, a storage unit 520, and a simulation result evaluation unit 530. The simulation result evaluation unit 530 is configured using, for example, a processor such as a CPU (Central Processing Unit) and memory. The simulation result evaluation unit 530 is realized by the processor executing a program. Note that all or part of the functions of the simulation result evaluation unit 530 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs: Solid State Drives), and storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0042] The simulation result input unit 510 includes a network interface for communicating with other devices via a network NW and receives input of real-world simulation results (video) from the simulation device 300. The network interface may be a wireless communication device or a wired communication device. The simulation result input unit 510 stores the input simulation results in the storage unit 320.

[0043] The storage unit 520 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 520 stores information such as the results of the real-world simulation input from the simulation device 300 via the simulation result input unit 510, and the evaluation model 521 used to evaluate the simulation results. The storage unit 520 may be used as an area to store a program for performing evaluation processing of the simulation results, as an area to store temporary data generated in the evaluation process, or as an area to store the execution results of the real-world simulation.

[0044] The simulation result evaluation unit 530 evaluates the results of the real-world simulation input from the simulation device 300 using the evaluation model 521. The evaluation model 521 is a trained model that has been trained by machine learning to take the simulation result video as input and output an evaluation score for the video against predetermined evaluation criteria. For example, the evaluation model 521 can be constructed by supervised learning using pairs of video data and evaluation scores as training data. The video data used as training data may be those captured by the simulation, or those captured in the real environment based on a shooting plan or simulation results. The evaluation scores used as training data may reflect the advertising effect obtained by actually presenting the video captured in the real environment or simulation to viewers, or they may reflect the results of the advertiser's evaluation of the video. Furthermore, if there are multiple evaluation criteria, for example, the evaluation model 521 may be configured as a set of multiple trained models with different evaluation criteria. Also, for example, in training the evaluation model 521, features extracted from the video data may be used instead of the video data itself. Any machine learning model may be used, such as neural networks, deep learning, or reinforcement learning.

[0045] Furthermore, the evaluation model 521 may be configured as a multimodal LLM capable of processing multiple information sources in combination, such as text data, image data, video data, and audio data. In this case, for example, the evaluation model 521 may be configured to perform processing to recognize the content and features of the video images of the simulation results, and to score the recognition results by evaluating them against evaluation criteria. For example, LENS, IDEFICS, and GPT-4o can be cited as algorithms for the multimodal LLM. With such a multimodal LLM, the simulation result evaluation unit 530 can evaluate the simulation results more flexibly and accurately by utilizing various data related to the real-world simulation in addition to the video images of the simulation results. For example, the simulation result evaluation unit 530 may be configured to evaluate the simulation results using parameters used in the real-world simulation or shooting plans generated based on those parameters.

[0046] The simulation result evaluation unit 530 performs evaluation processing using the evaluation model 521 for each of the simulation results to be evaluated, thereby obtaining an evaluation score for each of the simulation results. The simulation result evaluation unit 530 ranks the simulation results based on the obtained evaluation scores, determines the shooting plan to be presented to the user U based on the ranking results, and supplies the determined shooting plan to the user terminal device 100.

[0047] Figure 5 is an illustrative diagram showing the general outline of the real-world simulation. Figure 5 illustrates how the real-world simulation is performed in two parts simultaneously: a first simulation that reproduces the state of the virtual space VR, and a second simulation that captures the state of the virtual space VR reproduced by the first simulation using virtual cameras CV1 to CV4. The first simulation execution unit 331 generates the virtual space VR by applying the spatial reproduction parameters generated by the parameter generation device 200 to a 3D simulation model. The second simulation execution unit 332 reproduces the camera work when cameras CV1 to CV4 capture the virtual space VR by applying the shooting parameters generated by the parameter generation device 200 to a 3D camera model.

[0048] As shown in Figure 6, the first simulation execution unit 331 may be configured to reproduce not only the subject and its movements in the virtual space VR, but also other audiovisual elements related to the appeal of the image, such as sound and light. For example, sound effects, dialogue, background sounds, and sounds emitted by objects may be reproduced, as may light emitted by objects, changes in light, and effects using light. In addition, environmental elements such as weather may be reproduced by combining sound and visual effects, for example.

[0049] Furthermore, the second simulation execution unit 332 may be configured to record audio reproduced in the virtual space VR in conjunction with imaging of the virtual space VR. In this case, the simulation device 300 can simulate shooting in a manner closer to the actual environment R, and the evaluation device 500 can perform a highly accurate evaluation of the simulation result video at a level closer to the actual environment R.

[0050] In this case, the parameter generation device 200 may also be configured to generate combinations of parameters (parameter sets) that include variations in sound and visual effects. This allows the user U to obtain a shooting plan with a higher presentation effect from among shooting plans that correspond to a wider range of shooting patterns, and enables the capture of moving images that better match the purpose in the actual environment R.

[0051] Figure 7 shows an example of the processing flow of the video evaluation system 1A of the first embodiment. First, user U operates the user terminal device 100 to set parameters related to the real-world simulation (S101). Next, the parameter generation device 200 generates parameters for the real-world simulation based on the settings in S101 (S102). Here, we will describe the case where the parameter generation device 200 generates multiple parameter sets. The parameter generation device 200 selects one of the multiple parameter sets and supplies it to the simulation device 300 and the shooting plan generation device 400 (S103, S104).

[0052] Next, the simulation device 300 executes a real-world simulation using the parameter set input from the parameter generation device 200 in S103 (S105). The simulation device 300 supplies the results of the real-world simulation executed in S105 to the evaluation device 500 (S106). In S106, in addition to a video of the simulation results, the parameter set corresponding to the executed real-world simulation may also be supplied.

[0053] Meanwhile, the shooting plan generation device 400 generates an shooting plan in S103 based on the parameter set input from the parameter generation device 200 (S107). The shooting plan generation device 400 supplies the shooting plan generated in S107 to the evaluation device 500 (S108). Subsequently, the evaluation device 500 performs evaluation processing on the simulation results input from the simulation device 300 in S106 (S109). The evaluation device 500 saves the simulation results in S109 in association with the shooting plan generated in S107.

[0054] Up to this point, the processing corresponds to one parameter set selected and input in S103. The processing group S103 to S109 (dashed line) is executed for each of the multiple parameter sets generated in S102. Once the processing in S103 to S109 has been performed for all of the multiple parameter sets generated in S102, the evaluation device 500 then determines a shooting plan to present to the user U based on the evaluation results of multiple real-world simulations corresponding to the multiple parameter sets (S110). For example, the evaluation device 500 may determine the shooting plan corresponding to the simulation result with the highest evaluation score as the shooting plan to present to the user U, or it may determine the shooting plan corresponding to the simulation result with an evaluation score above a certain threshold as the shooting plan to present to the user U. The evaluation device 500 supplies the shooting plan determined in S110 to the user terminal device 100 (S111). Then, by performing shooting in the real environment using the shooting plan supplied in S111, the user U can efficiently capture moving images that are effective for presentation and meet the purpose.

[0055] With the video evaluation system 1A of the first embodiment configured in this way, shooting patterns applicable to shooting in a real environment can be explored by repeatedly evaluating them through simulation, making it possible to shoot highly effective video images that match the purpose more efficiently. Furthermore, with the video evaluation system 1A of the first embodiment, shooting patterns (parameters) related to simulation results that have obtained high evaluation scores are presented to the user in the form of a shooting plan that is easy for those involved in shooting to understand, so that the user can perform shooting in a real environment more efficiently.

[0056] <Second Embodiment> Figure 8 shows an example of the system configuration of the second embodiment of the video evaluation system 1B. The second embodiment of the video evaluation system 1B differs from the first embodiment of the video evaluation system 1A in that it further includes a scanning device 600. The scanning device 600 is a device that reads (scans) the state of the real environment R and supplies it to the simulation device 300. More specifically, the scanning device 600 captures images of people and objects present in the real environment R and supplies the video images as scan results to the simulation device 300. The scanning device 600 may also record sounds occurring in the real environment R and supply the sound data as scan results to the simulation device 300.

[0057] The simulation device 300 performs a real-world simulation based on the parameters generated by the parameter generation device 200 and the scan results of the real environment R supplied by the scanning device 600. By using the scan results of the real environment R in addition to the parameters of the real-world simulation, the simulation device 300 can reproduce a virtual space that is closer to the real environment R. Furthermore, by reproducing a virtual space that is closer to the real environment R, it is expected that the difference between the video footage captured in the simulation and the video footage captured when that simulation is reproduced in the real environment R will be smaller. As a result, if the video footage captured in the simulation is evaluated as matching the objective, the probability that the video footage captured when that simulation is reproduced in the real environment R will also match the objective increases, thus improving the reliability and practicality of the evaluation device 500. On another level, it is expected that the degree of agreement between the evaluation score for the video footage captured in the simulation and the evaluation score for the video footage captured when that simulation is reproduced in the real environment R will increase, thus also improving the reliability and practicality of the evaluation device 500.

[0058] <Third Embodiment> Figure 9 shows an example of the system configuration of the video evaluation system 1C of the third embodiment. The video evaluation system 1C of the third embodiment differs from the video evaluation system 1A of the first embodiment in that the simulation device 300 does not supply the results (video) of the real-world simulation to the evaluation device 500, but only supplies the parameters of the real-world simulation. Furthermore, the video evaluation system 1C of the third embodiment differs from the video evaluation system 1A of the first embodiment in that the evaluation device 500 does not perform evaluation processing using the results of the real-world simulation as input, but rather performs evaluation processing using the parameters of the real-world simulation as input.

[0059] As described above, the video footage resulting from the real-world simulation is generated by capturing a virtual space reproduced based on the parameters of the real-world simulation. Therefore, it can be assumed that there is a correlation between the content of the video footage resulting from the real-world simulation and the parameters of the real-world simulation. Based on this idea, the video footage evaluation system 1C of the third embodiment estimates the evaluation results of the simulation video footage using the parameters of the real-world simulation.

[0060] For example, by learning training data that combines the evaluation results of the video evaluation system 1A of the first embodiment with corresponding parameters using machine learning, an estimation model can be constructed that takes the parameters of a real-world simulation as input and outputs an estimated value of the evaluation score. Any machine learning model may be used, such as a neural network, deep learning, or reinforcement learning. By storing the estimation model constructed in this way as the evaluation model 521 in the memory unit 520 of the evaluation device 500 in advance, the evaluation device 500 can estimate the evaluation score based on the parameters of the real-world simulation. Furthermore, with this configuration, the time required for the real-world simulation can be reduced, so that a shooting plan for capturing video that matches the purpose can be provided to the user U in a shorter time. In addition, when the evaluation device 500 performs evaluation by inputting video, even elements related to the quality of the simulation, such as the realism of rendering, may affect the evaluation result. In contrast, when the evaluation device 500 performs evaluation by inputting parameters or a shooting plan, the elements related to the quality of the simulation mentioned above can be excluded from the evaluation, so an evaluation that is not affected by elements specific to the simulation can be achieved, and the reliability of the evaluation device 500 can be improved.

[0061] According to at least one embodiment described above, it is possible to more efficiently capture moving images that are suitable for the purpose and have a high presentation effect.

[0062] <Variation> The following describes modifications of the motion image evaluation systems 1A to 1C of the embodiment. In the following, unless otherwise specified, motion image evaluation systems 1A to 1C will be collectively referred to as motion image evaluation system 1.

[0063] (First variation) The parameter generation device 200 may be configured to have a function (hereinafter referred to as the "parameter modification function") that estimates parameters that should be changed in order to improve the evaluation score of the simulation results for the parameter set of the real-world simulation, and modifies the estimated parameters to be changed to generate a new parameter set. For example, the estimation of parameters to be changed can be achieved by a trained model (hereinafter referred to as the "parameter modification estimation model") that has learned the relationship between the evaluation score of the real-world simulation using the parameter set of the real-world simulation (hereinafter referred to as the "first parameter set") and a parameter set with some parameter values ​​different from the first parameter set (hereinafter referred to as the "second parameter set") through machine learning. Here, the second parameter set is a parameter set that obtained a higher evaluation score than the evaluation score of the first parameter set. The parameter modification estimation model is configured to output the difference between the first parameter set and the second parameter set. The parameter modification estimation model may also be configured to output the second parameter set.

[0064] The parameter generation device 200 can obtain a parameter set that is expected to yield a higher evaluation score than the input parameter set by inputting the parameter set to be estimated into the parameter estimation model for the parameter to be changed, as learned in this manner. In this case, the video evaluation system 1 may be configured to repeatedly perform estimation of the parameter to be changed and execution of a real-world simulation with the parameter set in which the parameter to be changed has been modified. This allows the user U to efficiently generate videos with a higher presentation effect.

[0065] (Second variation) The parameter generation device 200 may be configured to have the function of presenting parameters (hereinafter referred to as "reproduction parameters") for reproducing a video image specified by the user U in the virtual space of the real environment R. For example, the estimation of reproduction parameters can be achieved by a trained model (hereinafter referred to as the "reproduction parameter estimation model") that has learned the relationship between the parameters of a real-environment simulation and the results (video image) of the real-environment simulation performed with those parameters using machine learning. Here, the reproduction parameter estimation model is configured to take a video image of the simulation result as input and output a set of parameters that are (presumably) capable of obtaining that video image.

[0066] Generally, when capturing video footage, even if a sample of the desired video is available, the camera work and composition required to capture a video similar to that sample may be unknown, necessitating repeated trial and error in real-world simulations, potentially increasing the cost of capture. In contrast, with the parameter generation device 200 of the second modified example, user U can input a video footage they wish to use as a reference into the parameter generation device 200 to obtain the parameters needed to acquire that video footage through real-world simulations, thereby efficiently generating video footage with a higher presentation effect. Furthermore, when training a reproduction parameter estimation model using the results of real-world simulations, it is desirable that the sample video footage to be captured is taken in a real-world environment R or a virtual space that mimics the real-world environment R.

[0067] (Third variation) Figure 10 illustrates a third modification of the video evaluation system 1 of the embodiment. In the third modification, the user terminal device 100 receives input of evaluation viewpoints for the simulation results and supplies the input evaluation viewpoints to the evaluation device 500 in text format. The user terminal device 100 may receive input of evaluation viewpoints for each individual real-world simulation, may receive input of new evaluation viewpoints to override the default evaluation viewpoints, or may receive input of new evaluation viewpoints to be added to the default evaluation viewpoints. In this case, for example, by configuring the evaluation model 521 using a multimodal LLM, the evaluation device 500 can obtain an evaluation score that takes the evaluation viewpoints into account for the simulation results by inputting the evaluation viewpoint information supplied from the user terminal device 100 into the evaluation model 521.

[0068] According to the third modified version of the video evaluation system 1, user U can perform real-world simulations while flexibly changing evaluation criteria, taking into account past simulation results, etc., thereby enabling more efficient capture of video images that are highly effective and match the objective.

[0069] Furthermore, the video evaluation system 1 of the embodiment may be configured by appropriately combining the first to third embodiments and modifications thereof. For example, it may take one or more of the following as inputs: video images which are the results of a real-world simulation, parameters of the real-world simulation, and a shooting plan based on those parameters, and output the evaluation results of the real-world simulation.

[0070] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0071] This invention is applicable to applications that support the creation of moving images having a predetermined presentation effect. [Explanation of Symbols]

[0072] 1. 1A~1C Video Image Evaluation System 100 User terminal devices 200 Parameter Generator 300 Simulation Devices 310 Parameter Input Section 320 Storage section 330 Control Unit 331 First Simulation Execution Unit 332 Second Simulation Execution Unit 400 Shooting Plan Generator 410 Parameter Input Section 420 Storage section 430 Shooting Plan Generation Department 500 Evaluation device 510 Simulation Result Input Section 520 Storage section 521 Evaluation Models 530 Simulation Result Evaluation Department 600 Scanning Devices

Claims

1. A simulation device that performs a real-world simulation, which simulates the state of the real-world environment and the shooting of the video in the real-world environment, with respect to the actual environment in which the user takes video footage to present to the viewer. An evaluation device for evaluating the results of the real-world simulation from a first evaluation perspective relating to the purpose of presenting the aforementioned moving images to the viewer, Equipped with, The aforementioned real-world simulation reproduces the real-world environment in a virtual space and outputs moving images captured by a virtual camera in the virtual space as the simulation result. The evaluation device evaluates the results of the real-world simulation based on the motion image as a simulation result or the parameters of the real-world simulation. Video evaluation system.

2. The evaluation device takes video footage of the virtual space as input and outputs evaluation results for the video footage for the aforementioned purpose to a trained model that has been trained to output evaluation results for the video footage from a first evaluation viewpoint. The video evaluation system according to claim 1.

3. The evaluation device outputs the evaluation result for the objective of the video image acquired by a real-world simulation using the parameters to be evaluated, by inputting the parameters to be evaluated into a trained model that has been trained to take the parameters as input and output an evaluation result from a first evaluation viewpoint of the video image acquired by a real-world simulation using the parameters to be evaluated. The video evaluation system according to claim 1.

4. The device further comprises a shooting plan generation apparatus that generates a shooting plan in which information for capturing moving images obtained as a result of the real-world simulation in the real-world environment is described in a manner that is understandable to humans, based on the aforementioned parameters. The evaluation device takes the shooting plan as input and outputs the results of evaluating the video image for the purpose of the shooting plan by inputting the shooting plan to a trained model that has been trained to output the results of evaluating the video image acquired by a real-world simulation using the parameters related to the shooting plan from a first evaluation perspective. The video evaluation system according to claim 1.

5. The device further includes a scanning device that reads information from the aforementioned real environment, The simulation device reproduces the real environment in the virtual space using the scan results obtained by the scanning device from reading information about the real environment. The video evaluation system according to claim 1.

6. The simulation device reproduces sound, light, or weather in the real environment in the virtual space during the real environment simulation. The video evaluation system according to claim 1.

7. The simulation device further comprises a parameter generation device that generates parameters used to perform the real-world simulation, The parameter generation device has a function to generate a new parameter set by changing part or all of the parameter set, The process involves repeatedly performing a parameter set modification using the parameter generation device and then executing the real-world simulation using the modified parameter set with the simulation device. The video evaluation system according to claim 1.

8. The parameter generation device is The first parameter set of the aforementioned real-world simulation, The evaluation results of a real-world simulation using a second parameter set, which has some parameter values ​​different from the first parameter set, and which obtained a higher evaluation result than the evaluation result of the first parameter set, By inputting the third parameter set, which is the target of estimation, into a pre-trained model that has learned the relationship between the three parameters, a fourth parameter set is estimated that can obtain a higher evaluation result than the third parameter set. The video evaluation system according to claim 7.

9. The parameter generation device takes a video as input and estimates the parameters for reproducing the video in the real-world simulation by inputting the video to be reproduced into a trained model that has been trained to output parameters for acquiring the video as a result of the real-world simulation. The video evaluation system according to claim 7.

10. The second evaluation perspective of the real-world simulation is input to the evaluation device in text format via the user's terminal device. The evaluation device evaluates the results of the real-world simulation based on the first evaluation viewpoint or the second evaluation viewpoint. The video evaluation system according to claim 1.

11. The simulation device performs a real-environment simulation, which simulates the state of the real environment and the shooting of the video in the real environment, with respect to the actual environment in which the user takes video to present to the viewer. The evaluation device performs a second step of evaluating the results of the real-world simulation from a first evaluation perspective relating to the purpose of presenting the moving image to the viewer, A video evaluation method that includes, The aforementioned real-world simulation reproduces the real-world environment in a virtual space and outputs moving images captured by a virtual camera in the virtual space as the simulation result. The second step involves evaluating the results of the real-world simulation based on the video image as a simulation result or the parameters of the real-world simulation. Methods for evaluating moving images.

12. One or more processors, Regarding the actual environment in which the user takes motion images to present to viewers, the first step is to perform a real-environment simulation that simulates the state of the real environment and the taking of motion images in the real environment. A second step involves evaluating the results of the real-world simulation from a first evaluation perspective relating to the purpose of presenting the aforementioned moving images to the viewers, A program to execute, The aforementioned real-world simulation reproduces the real-world environment in a virtual space and outputs moving images captured by a virtual camera in the virtual space as the simulation result. The second step involves evaluating the results of the real-world simulation based on the video image as a simulation result or the parameters of the real-world simulation. program.

Images