Simulation device, image processing device, computer program, and storage medium

Abstract

To provide a simulation device that can quickly acquire an appropriate photographing condition by using a virtual space.SOLUTION: A simulation device has: information input means that inputs information on an event to be photographed, photography information, and a desired photographic image; virtual space construction means that constructs a virtual space in which the state of the event is reproduced by using a subject model based on the information input by the information input means; simulation image creation means that creates an image obtained when photographing is performed in an event venue by using the virtual space; evaluation means that evaluates the similarity between the simulation image and the desired photographic image; and notification means that notifies a photographing condition according to the similarity evaluated by the evaluation means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a simulation device using a virtual space. [Background technology]

[0002] To efficiently photograph events such as sports days and weddings, it is effective to conduct location hunting (hereinafter referred to as location scouting) in advance, where you visit the venue and consider shooting conditions such as shooting position and angle of view to obtain the desired images. Location scouting is the process of bringing multiple pieces of photography equipment to the event venue in advance and taking repeated test shots to ensure smooth shooting on the day of the event.

[0003] Meanwhile, simulation technology that uses three-dimensional measurement data to recreate spaces such as event venues is becoming more widespread. For example, Patent Document 1 discloses a simulation technology that uses images of the event venue taken from multiple viewpoints to generate image data of the event venue viewed from any position and any direction. Simulation services that allow users to view the event venue from any position and any direction have also emerged. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-113281 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while simulations using the above-mentioned arbitrary viewpoint image synthesis technology allow users to check the state of the venue, it is difficult to consider the position and size of the actual people who will be the subjects when creating a captured image. Furthermore, to find the shooting conditions, such as the shooting position, direction, and angle of view, that will produce the desired image, users must navigate around the simulation screen using input operations such as a mouse or keyboard, which can be very time-consuming. An object of the present invention is to provide a simulation device that can quickly obtain appropriate shooting conditions using a virtual space. [Means for solving the problem]

[0006] In the simulation device, an information input means for inputting information about the event to be photographed and a desired photographed image; The aforementioned Information about the event to be photographed a virtual space construction means for constructing a virtual space that simulates the event using a subject model based on the above; a simulation image generating means for generating, using the virtual space, a plurality of simulation images obtained by photographing the event under different photographing conditions; an evaluation means for sequentially evaluating the similarity between the plurality of simulation images taken under different photographing conditions and the desired photographed image; a notification means for selecting the simulation images whose similarity is equal to or greater than a predetermined value, which are sequentially evaluated by the evaluation means, and notifying the user of the photographing conditions of the selected simulation images; With death, the photographing conditions include a movable speed; the notification means calculates a plurality of movement routes for sequentially photographing under predetermined photographing conditions according to the degree of similarity based on the possible moving speed; A UI is provided for allowing a user to select one of the plurality of travel routes. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a simulation device that can quickly obtain appropriate shooting conditions using a virtual space. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a simulation device according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a venue layout for an athletic meet in the first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of subject position information in a footrace in the first embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example of a method for specifying the direction of a subject in the first embodiment. [Figure 5] FIG. 4 is a diagram illustrating an example of a format of subject position information in the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of photographing position information in the first embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of photographing angle of view information in the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a desired captured image in the first embodiment. [Figure 9] 1 is a flowchart of a virtual space construction process according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of coordinate transformation of a venue layout in the first embodiment. [Figure 11] FIG. 2 is a diagram showing an image of a subject model in the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of model placement in a virtual space in the first embodiment. [Figure 13] 1 is a flowchart of a simulation image generation process in the first embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of camera coordinate system setting in the first embodiment. [Figure 15] 4 is a diagram showing an example of projection of a subject model image onto a projection surface in the first embodiment. FIG. [Figure 16] FIG. 2 is a diagram showing an example of a simulation image generated in the first embodiment. [Figure 17] 10 is a flowchart of an evaluation value calculation process in the first embodiment. [Figure 18] 10 is a flowchart of a similarity calculation process according to the first embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a graph for calculating a similarity regarding the number of subjects in the first embodiment. [Figure 20] FIG. 10 is a diagram showing an example of a graph for calculating a similarity regarding a face position in the first embodiment. [Figure 21] FIG. 10 is a diagram showing an example of a graph for calculating a similarity degree relating to face size in the first embodiment. [Figure 22] FIG. 10 is a diagram illustrating an image of a method for calculating a similarity regarding a face direction in the first embodiment. [Figure 23] FIG. 10 is a diagram illustrating an example of setting a weighting coefficient for each desired captured image in the first embodiment. [Figure 24] FIG. 10 is a block diagram showing an example of the configuration of a simulation device according to second and third embodiments. [Figure 25] FIG. 10 is a diagram showing an image of an area where entry is permitted in a foot race in Examples 2 and 3. [Figure 26] FIG. 10 is an image diagram of position information to be selected sequentially in the second and third embodiments. [Figure 27] 10 is a processing flowchart for calculating a photographing procedure in the second and third embodiments. [Figure 28] 10 is an image diagram of a display on a monitor in Examples 2 and 3. FIG. [Figure 29] FIG. 10 is an image diagram of the parent-teacher seats in Examples 2 and 3. [Figure 30] FIG. 10 is an image diagram of how to obtain an interference evaluation value in Examples 2 and 3. [Figure 31] FIG. 10 is a block diagram illustrating an example of the configuration of a simulation device according to a fourth embodiment. [Figure 32] 13 is a flowchart showing a process for displaying information about the installation position and orientation of a fixed camera in the fourth embodiment. [Figure 33] FIG. 13 is a diagram showing an example of input of event information in the fourth embodiment. [Figure 34] FIG. 13 is a diagram showing an example of input of photographing information in the fourth embodiment. [Figure 35] FIG. 10 is a diagram showing an example of a desired captured image in the fourth embodiment. [Figure 36] FIG. 13 is a conceptual diagram illustrating how photographing information is associated with the similarity of students in the fourth embodiment. [Figure 37] FIG. 10 is a diagram showing a display example in the fourth embodiment. [Figure 38] 13 is a flowchart showing a process for displaying information about the installation position and orientation of a fixed camera in the fifth embodiment. [Figure 39] FIG. 13 is a diagram illustrating an example of selection of the position of a fixed camera in the fifth embodiment. [Figure 40] FIG. 13 is a diagram illustrating an example of selecting the positions and number of fixed cameras in the fifth embodiment. [Figure 41] FIG. 13 is a conceptual diagram of a fixed camera and a subject to be tracked in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. [Example]

[0010] 1 is a block diagram showing an example of the configuration of a simulation device according to embodiment 1. In this embodiment, for the sake of convenience, an elementary school sports day is assumed as the event, but the same applies to other events. In FIG. 1, the simulation device has information input means 100 for inputting information about the event to be photographed, photographing information, desired photographed images, and the like.

[0011] The system also includes virtual space construction means 101 for constructing a virtual space that reproduces (assumes) the event situation using a subject model based on information input by information input means 100. The system also includes simulation image generation means 102 for generating a simulation image using the virtual space from an image obtained by photographing the event venue.

[0012] Furthermore, it has an evaluation means 103 that evaluates the similarity between the simulation image and a desired captured image, and a notification means 104 that collects shooting conditions according to the similarity so that the evaluation result by the evaluation means 103 is high, and notifies the user of the collected shooting conditions. Here, the functions of the virtual space construction means 101, simulation image generation means 102, evaluation means 103, notification means 104, etc. are realized by a computer (not shown) executing a computer program. Note that each function may also be configured by a discrete circuit.

[0013] The information input means 100 is a means for inputting program information (event program information) of the event to be photographed, photography information, desired photographed images, etc. An event refers to an event that brings together a large number of people, such as an athletic meet, athletic competition, race, play, school performance, festival, live music event, or wedding. Event program information includes the layout of the venue where the event is held, the content and order of the event, subject position information indicating movement information of subjects participating in the event, and the like.

[0014] The venue layout is a bird's-eye view of the venue where the event will be held. Fig. 2 is a diagram showing an example of the venue layout for the athletic meet in Example 1. Reference numeral 200 in Fig. 2 denotes the athletic field that will be the event venue, and is represented in a two-dimensional (X, Y) coordinate space with the upper left corner as the origin (0,0). Reference numeral 201 in Fig. 2 denotes the track that will be used as the running path and boundary lines for each event at the athletic meet.

[0015] FIG. 3 is a diagram showing an example of subject position information in a footrace in Example 1. The subject position information is information about at least one of the position and orientation of each subject (competitor or participant) at each time that an event is taking place. An overview of subject position information will be explained using FIG. 3, taking a footrace at an athletic meet as an example. Numerals 301, 302, 303, and 304 in the figure represent four students (subjects) participating in the race, and as times progress from T0, T1, and T2, the figure shows how each student moves from the start of a track drawn in a playground 300 toward the finish line.

[0016] Here, the position of each student is expressed by XY coordinates, similar to the venue layout in Fig. 1. Fig. 4 is a diagram showing an example of a method for specifying the subject orientation in the first embodiment, and the orientation of the students is expressed by the direction of arrows (D0 to D15) that equally divide 360 degrees as shown in Fig. 4. Fig. 5 is a diagram showing an example of the format of subject position information in the first embodiment, and the subject position information input to the information input means 100 is composed of a combination (set) of position and orientation information for multiple subjects at each time, as shown in Fig. 5.

[0017] The subject position information can be preset and saved in advance in the information input means 100 for each event program (for each sport, such as a foot race or tug-of-war, in the case of an athletic meet). This allows the user to select desired subject position information from samples of subject position information preset in advance according to the event program.

[0018] The content and sequence of events will be entered as program information for the event, but we will not discuss the User Interface (hereafter abbreviated as UI) here. You can input pre-quantified data, or you can have the user input information via a Graphical User Interface (hereafter abbreviated as GUI). The photographing information includes at least one of the photographing position, the photographing direction, and the photographing angle of view.

[0019] Fig. 6 is a diagram showing an example of photography position information in Example 1, and the photography positions are information about areas in an event venue where photography equipment such as photographers or drones can enter (enter). In Fig. 6, P0 to P15 indicated by black circles around a playground 600 indicate photography positions. Each photography position is expressed by XY coordinates, similar to the venue layout in Fig. 1.

[0020] The shooting direction is the direction in which the cameraman or the camera equipment such as a drone takes the picture (camera direction), and is expressed by the direction of the arrows (D0 to D15) that equally divide 360 degrees, as in Figure 4.

[0021] 7 is a diagram showing an example of shooting angle of view information in Example 1, where the shooting angle of view is information about the angle of view of the lens attached to the camera that performs shooting. As shown in Fig. 7, the shooting angle of view is set as an angle with respect to the optical axis Z for discrete lens focal lengths ranging from wide angle (F0: 16 mm) to super telephoto (F12: 800 mm) when converted to a sensor size of 35 mm. Furthermore, the shooting information also includes information about the camera used for shooting and the number of shots to be taken.

[0022] FIG. 8 is a diagram showing an example of a desired captured image in the first embodiment. The desired captured image is information including the number of subjects in the image that the user wants to capture, the position of their faces, the size of their faces, and the direction of their faces. There are various possible methods for inputting the desired captured image, but images or illustrations of the same event held in the past are input as samples. In this embodiment, a large number of sample images such as those shown in FIGS. 8(A), (B), and (C) are prepared, and the number of subjects, the position of their faces, the size of their faces, and the direction of their faces are associated with and stored for each sample image.

[0023] When a user selects a desired photographed image from among the sample images, at least one piece of information associated with the selected sample image, such as the number of subjects, face position, face size, and face direction, is set and stored as input information. Multiple desired photographed images can be selected. The virtual space construction means 101 generates a virtual space that reproduces the event situation using a model based on the event information input to the information input means 100. Fig. 9 is a flowchart of the virtual space construction in the first embodiment, which is realized by a computer (not shown) executing a computer program stored in memory. The flow of generating the virtual space will be explained using Fig. 9.

[0024] First, in step S901 of Fig. 9, the virtual space construction means 101 converts the two-dimensional venue layout input to the information input means 100 into a three-dimensional coordinate space (xyz coordinate system) as shown in Fig. 10. Here, Fig. 10 is a diagram showing an example of the coordinate conversion of the venue layout in the first embodiment.

[0025] Next, in step S 902 , the virtual space construction means 101 places the subject model in a three-dimensional coordinate space based on the subject position information input to the information input means 100 . Fig. 11 is a diagram showing an image of a subject model in Example 1, where 1101 is a model of a student who will be the subject of the sports day, and is represented in a three-dimensional coordinate space by, for example, a polyhedron (polygon). 1102 in the figure is a front mark that indicates the direction of the student's face. Multiple subject models in Fig. 11 are arranged in a three-dimensional coordinate space based on subject position information.

[0026] Fig. 12 is a diagram showing an example of model placement in a virtual space in Example 1. Fig. 12 shows the virtual space near the start line at time T0. The positions and orientations of subject models 1201, 1202, 1203, and 1204 change over time based on subject position information. Furthermore, the virtual space construction means 101 takes a picture of the current scenery and recognizes the spatial position and shape of the taken image.

[0027] The simulation image generating means 102 generates a simulation image using the virtual space from an image that would be obtained if photographed at the event venue. Fig. 13 is a flowchart of the simulation image generation in the first embodiment, which is realized by a computer (not shown) executing a computer program stored in memory. The flow of the simulation image generation will be described using the flowchart in Fig. 13. In step S1301, a position for taking a photograph is determined within the three-dimensional coordinate space in which the subject model is placed. The photographing position is input to information input means 100 and selected from photographing positions P0 to P15 in FIG.

[0028] Next, in step S1302, the orientation of the camera that will take pictures in the three-dimensional coordinate space is set. The camera orientation is input to information input means 100 and selected from among the image-taking orientations D0 to D15 in FIG. Next, in step S1303, the angle of view of the camera that will take a photograph in the three-dimensional coordinate space is set. The angle of view is input to information input means 100 and selected from the photographing angles of view F0 to F12 in FIG.

[0029] Next, in step S1304, a camera coordinate system (XYZ coordinate system) is set with the shooting position (xc, yc, zc) set in step S1301 as the origin and the shooting direction set in step S1302 as the Z axis. Then, the coordinate position of the subject model in the xyz coordinate system is converted into the camera coordinate system (XYZ coordinate system) as shown in FIG. FIG. 14 is a diagram showing an example of camera coordinate system setting in the first embodiment, in which the coordinate position of the front mark of the subject model 1401 placed in a three-dimensional coordinate space is (xo, yo, zo) in the xyz coordinate system, and the shooting position is (xc, yc, zc) in the xyz coordinate system.

[0030] Here, the coordinate position (Xo, Yo, Zo) of the front mark of the subject model in the camera coordinate system (XYZ coordinate system) is given by the following formulas 1 to 3. Xo=d×sinθ (Formula 1) Yo = zo (Equation 2) Zo = d × cosθ (Equation 3)

[0031] Here, d is the distance between the camera position (xc, yc, zc) and the front mark (xo, yo, zo), and θ is the angle between the shooting direction and the line connecting the camera position (xc, yc, zc) and the front mark (xo, yo, zo). Next, in step S1305, a simulation image is generated by projecting an image of the subject onto a projection plane set in the camera coordinate system (XYZ coordinate system).

[0032] Fig. 15 is a diagram showing an example of projection of a subject model image onto a projection plane in Example 1. As shown in Fig. 15, the projection plane is set at Z=1 in the camera coordinate system. The imaging position in the camera coordinate system, that is, the intersection (uo, vo) of the projection plane and the straight line connecting the origin (0,0,0) and the front mark (Xo,Yo,Zo) is the imaging position of the front mark, and is calculated using the following equations 4 and 5. uo=Xo / Zo (Equation 4) vo=Yo / Zo (Equation 5) Similarly, all areas of the subject model placed in the three-dimensional virtual space are projected onto the projection surface using the above formula.

[0033] However, if the angle θ between the shooting direction (Z axis) and the line connecting the shooting position and the subject position is larger than the shooting angle of view set in step S1303, the space is outside the shooting area and is therefore not subject to projection. By carrying out the above steps S1301 to S1305, a simulation image such as that shown in FIG. 16 can be generated.

[0034] Figure 16 shows an example of a simulation image generated in Example 1, and Figure 16(A) is a simulation image obtained when shooting is performed at time T0 from shooting position P13 in the direction of shooting direction D4 with a shooting angle of view F0. FIG. 16B is a simulation image obtained when an image is captured at time T2 from a capturing position P10 in a capturing direction D1 with an angle of view F6.

[0035] Furthermore, the simulation image generation means 102 may create a predetermined pose based on the content of the event, based on a polygonal image of a model of a person to be the subject. The orientation of the pose may then be changed to match the orientation of the spatial position and shape recognized from the current scenery, and the resulting simulation image may be superimposed on the current scenery image to create a simulation image. This allows for the generation of a more realistic simulation image. The method of posing a 3D model is similar to that used in 3D games and is already widely known, so a detailed description will be omitted here.

[0036] In addition, the process of recognizing spatial positions and shapes on landscape images and superimposing models is widely known and is called AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), etc., so a detailed explanation will be omitted here. The evaluation means 103 sequentially evaluates simulation images obtained when the virtual space is photographed under various conditions, based on the event program information, photography information, and information on desired photographed images input to the information input means 100. Fig. 17 is a flowchart of the calculation of the evaluation value in the first embodiment, and the flow for sequentially evaluating the photography will be described with reference to Fig. 17. Steps S1701 and S1702 are a loop for setting various shooting conditions.

[0037] In this example, the imaging conditions are changed in order and in a brute force manner, but the method for changing the imaging conditions is not limited to this. It is also possible to narrow down and change only the necessary conditions efficiently using some algorithm. In Figure 17, the shooting conditions that can be changed are the elapsed time of the event, the camera position, the camera direction, and the focal length of the camera lens, with time indicated by T, position by P, direction by D, and focal length by F, respectively.

[0038] Step S1701 is a loop for comprehensively changing and evaluating T (time) in the range of T0 to TN. Step S1702 is a loop for comprehensively changing and evaluating P (position) in the range of P0 to PN. Step S1703 is a loop for comprehensively sequentially changing and evaluating D (direction) in the range of D0 to DN. Step S1704 is a loop for comprehensively changing and evaluating F (focal length) in the range of F0 to FN, where N is an integer.

[0039] In step S1705, the virtual space construction means 101 constructs a virtual space under conditions based on the loop variable T. Step S1706 is a process for generating a simulation image when photographing is performed under photographing conditions based on the loop variables P, D, and F.

[0040] In step S1707, the evaluation means 103 obtains shooting information (number of subjects, face positions, face sizes, and face orientations) from the simulation image generated in step S1706 by image recognition, and then calculates an evaluation value based on the similarity to the closest desired captured image among the multiple desired captured images selected. The similarity is a value obtained by multiplying the item-specific similarity obtained for each item of the photographing information in the range of 0% (no similarity) to 100% (match) by the number of all items. Details will be described later.

[0041] Steps S1708 and S1709 are processes in which the notification means 104 notifies the user of the simulation image and the shooting conditions based on the evaluation result calculated in step S1707. Details will be described later. Steps S1710 to S1713 represent a determination as to whether or not to continue the loop processing regarding time T, position P, direction D, and focal length F, and the processing from steps S1705 to S1709 is repeated until all conditions are met.

[0042] Next, FIG. 18 is a flowchart of similarity calculation in the first embodiment, and the evaluation of similarity by the evaluation means 103 in step S1707 will be described in detail with reference to FIG. As shown in the flowchart of FIG. 18, the evaluation of similarity is performed in two steps: step S1801, where similarity is calculated for each item; and step S1802, where weighted addition (weighted addition) of similarity for each item is performed.

[0043] First, the calculation of similarity for each item in step S1801 will be described. In step S1801, similarity is calculated for each item, namely, the number of subjects in the simulation image, face position, face size, and face direction. That is, the similarity for the number of subjects (Mnum), face position (Mpos), face size (Msize), and face direction (Mdir) are calculated.

[0044] The similarity (Mnum) of the number of subjects is defined as the absolute value of mn, where n is the number of subjects in a photo taken under the shooting conditions (T, P, D, F) of interest, and m is the number of subjects associated with the desired photographed image. Similarity is calculated from this difference in the number of people. 19 is a diagram showing an example of a graph for calculating similarity regarding the number of people of subjects in Example 1, with the horizontal axis representing the difference in the number of people and the vertical axis representing similarity. Note that in FIG. 19, the larger the difference in the number of people, the smaller the similarity linearly becomes, but this is just an example and a curve may also be used.

[0045] The similarity of the face position (Mpos) is calculated by finding the Euclidean distance between the center position (Xc, Yc) of the subject's face in the photograph taken under the target photographing conditions (T, P, D, F) and the center position (Xs, Ys) of the face associated with the desired photographed image. The similarity is calculated from this Euclidean distance.

[0046] Fig. 20 is a diagram showing an example of a graph for calculating similarity regarding face positions in Example 1, with the horizontal axis representing Euclidean distance and the vertical axis representing similarity. The curve shown in the graph in Fig. 20 is an example, and a different shape may be used. The shooting conditions of interest include multiple faces, and the desired captured image also includes multiple faces, but the combination that results in the smallest Euclidean distance is paired and used as the similarity. The similarity of face size (Msize) is calculated using the face size indicated by the radius from the center of the face. The similarity is calculated using the combination of pairs obtained when calculating the similarity of face position.

[0047] If the face size of the subject in photography based on the photography conditions (T, P, D, F) of interest is Rc and the face size associated with the desired photographed image is Rs, the absolute value of Rs-Rc is defined as the face size difference Rd. Fig. 21 is a diagram showing an example of a graph for calculating similarity regarding face size in Example 1, with the face size difference Rd on the horizontal axis and the similarity on the vertical axis. The curve shown in the graph of FIG. 21 is also an example, and other shapes may be used.

[0048] The similarity of face direction (Mdir) evaluates the similarity between the face direction of the subject in the photograph taken under the photographing conditions (T, P, D, F) of interest and the face direction associated with the desired photographed image, each as a directional vector in three-dimensional space. FIG. 22 is a diagram illustrating an image of a method for calculating a similarity regarding a face direction in the first embodiment; An XY plane 900 including the center coordinates of the face of interest is extracted, a vector indicating the orientation of the face is projected onto the plane, and the direction in which the projected vector faces is determined.

[0049] The orientation of the subject's face during photography under the current photography conditions is superimposed on the orientation of the face associated with the desired photographed image, and the smaller of the angles formed by these two vectors is determined. The absolute value of the angle difference is between 0° and 360°, but if it exceeds 180°, the smaller angle is determined by subtracting the angle from 360°.

[0050] The smaller angle value of 0° to 180° corresponds to 100% to 0%, thereby determining the similarity on the XY plane 900. The similarity is similarly determined for the XZ plane 901, and the value obtained by multiplying the similarity on the XY plane 900 by the similarity on the XZ plane 901 is determined as the similarity between the orientations of the two faces.

[0051] Next, the weighted addition of similarity for each item in step S1802 will be described. The similarity (Mtotal) of the entire image between the desired image information and the simulation image is obtained by weighting and adding the item-specific similarities (Mnum, Mpos, Mdir, Msize) calculated in S1801 using weighting coefficients for each item-specific similarity. If the weighting factor for the number of faces is Wnum, the weighting factor for the position of the face is Wpos, the weighting factor for the direction of the face is Wdir, and the weighting factor for the size of the face is Wsize, Mtotal can be expressed by, for example, Equation 6. Mtotal=Wnum×Mnum+Wpos×Mpos+Wsize×Msize+Wdir×Mdir (Formula 6)

[0052] Here, the values of Wnum, Wpos, Wsize, and Wdir are set to different values depending on the selected desired image information. FIG. 23 is a diagram showing an example of setting a weighting factor for each desired captured image in the first embodiment. Figure 23(A) is an example of a desired photographic image, which gives an overall impression of the image. In this case, by increasing the weighting of the number and size of faces, it is possible to extract an image that is closer to the desired photographic image, which includes multiple subjects (students) and the background.

[0053] 23(B) is an example of a desired photographic image, which is a photographic image of a scene where a specific event occurred. In the case of such an image, by setting all weighting factors equally, the individual similarities can be evaluated in a balanced manner, and an image that is closest to the desired photographic image can be extracted. Figure 23(C) is an example of a desired photographic image, which is a close-up of the subject's facial expression. In the case of such an image, by setting a large weighting factor for the size and orientation of the face, it is possible to extract a simulation image in which the subject's facial expression can be more clearly confirmed. When performing the weighting calculation, the similarity is calculated for at least two of the number of subjects, face position, face size, and face direction, and the evaluation value is calculated using a weighting coefficient set according to the desired captured image.

[0054] Next, the details of the processing of the notification means 104 will be explained. The notification means 104 compares the similarity (Mtotal) calculated by the evaluation means 103 with a preset threshold value Th. Only when Mtotal is greater than Th, i.e., when the similarity is relatively high, does it notify the user of the specified shooting conditions. That is, it notifies the user of the simulation image and the shooting position (P), direction (D), angle of view (F), and program time (T) used when generating the simulation image. Notification methods include output to a display device (not shown), printout using a printing device, and data transmission to external media or a device connected to a network.

[0055] As described above, according to the first embodiment of the present invention, it is possible to automatically extract the photographing conditions for obtaining a desired photographic image using a simulation, and therefore it is possible to extract the photographing conditions for obtaining a desired photographic image more accurately without having to bring a large amount of photographing equipment to the site in advance. Furthermore, by evaluating the similarity of the number of subjects, the position and size of the face, and the orientation using different weights depending on the desired captured image, it becomes possible to extract the shooting conditions that will more effectively obtain an image that is close to the desired captured image. [Example]

[0056] Next, a second embodiment of the present invention will be described. In the second embodiment, in addition to the photographing position and photographing conditions for obtaining a desired photographed image, a moving route for sequentially photographing under the photographing conditions is calculated and notified to the user. Fig. 24 is a block diagram showing an example of the configuration of a simulation device in Examples 2 and 3. 2400 to 2403 and 2405 in Fig. 24 have substantially the same configuration as 100 to 103 and 104 in Fig. 1, and the basic configuration is almost the same as that in Example 1, except that a shooting procedure calculation means 2404 is added. There are also differences in the input to the information input means 2400 and the processing contents of the evaluation means 2403.

[0057] The following description will focus on the differences from the first embodiment. First, as event information input to the information input means 2400, information including areas that can be entered (stood) for photography, areas that cannot be entered (restricted), and the positions and sizes of surrounding obstacles is added. The areas that can be entered for photography, areas that cannot be entered, and the positions and sizes of surrounding obstacles will be input based on information provided by the event organizer or the like.

[0058] FIG. 25 is a diagram showing an image of an area that can be entered to take photos during a footrace in the second and third embodiments. There is a race track 2500 drawn on the grounds of an elementary school that will be the event venue, child seats 2501 arranged along the perimeter, children waiting their turn to run 2502, and parent seats 2503 for parents who have come to watch the children. In addition, there are areas (objects) 2504 that cannot be entered, such as tents and school buildings, and the area that can be entered, which does not overlap with these and takes into account the area used for the race, is input using an input means such as a GUI.

[0059] Next, a description will be given of the evaluation means 2403. The processing flow of the evaluation means 2403 is approximately the same as the flow shown in the flowchart of FIG. Furthermore, the photographing position in step S1702 is evaluated in more detail based on information such as the area that may be entered for photographing, which information is additionally input by the information input means 2400.

[0060] For example, the event venue is divided into meshes at regular intervals, and areas at the intersections of the meshes where it is permitted to enter for photography are selected in order. Fig. 26 is an image diagram of the position information to be sequentially selected in Examples 2 and 3. The intersection of the meshes existing inside the intrusion-permitted area 2505 is set as position information 2600. As shown in Fig. 26, there are multiple pieces of this position information 2600. Evaluation of photography is performed sequentially for all of this position information 2600.

[0061] The photographing procedure calculation means 2404 is a means for determining candidates for travel routes based on the evaluation values determined for each photographing condition (T, P, D, F). FIG. 27 is a flowchart showing the process of calculating the photographing procedure in the second and third embodiments. In the loop of step S2700 in FIG. 27, the evaluation values of the shooting conditions (T, P, D, F) are scanned in order from time T0 to TN. At T0, a predetermined number of shooting conditions (T0, P, D, F) are selected in order of highest evaluation value in step S2701. For times other than T0, the search is narrowed down to only positions P that are within the time difference between the time of interest and the time immediately before, from the position P selected at the previous time. Then, the shooting conditions (T, P, D, F) are scanned, and a predetermined number of shooting conditions (T0, P, D, F) are selected in order of highest evaluation value in step S2701.

[0062] In step S2703, the shooting conditions (T0, P, D, F) with the highest evaluation values obtained in step S2701 are linked in chronological order to determine the shooting order. Since a predetermined number of conditions are selected per time, the same evaluation value rankings are linked together to calculate a predetermined number of movement routes. 24 displays the determined multiple travel routes superimposed on a captured background image (live view image) on a monitor (display device) of a camera, head-mounted display, PC, smartphone, tablet, etc. Then, the user can select one of the multiple travel routes.

[0063] FIG. 28 is an image diagram of the display on the monitor in the second and third embodiments. The display contents are composed of a subject model 2800, a movement route display 2801, a photographing background 2802, a selected movement route display 2803 which displays the number of the selected movement route, a photographing point 2804, a photographing direction 2805, and a monitor screen 2806. This is the minimum display, and other displays may also be included. There are two types of display content: one for location scouting and one for actual shooting. Each is explained below.

[0064] When scouting locations before an event, the current scenery is photographed, for example, with a camera mounted on a smartphone, and displayed in real time as a photographed background 2802 (live view image) on the monitor screen 2806. A subject model 2800 in a virtual space, a movement route display 2801, a selected movement route display 2803, a photographing point 2804, and a photographing direction 205 are displayed on the monitor so as to be superimposed on the photographed background 2802. The subject model 2800 is a reproduced image of a 3D model reproduced by the virtual space construction means 2401, and is generated using computer graphics (hereinafter referred to as CG) etc. The movement route display 2801 is one of the predetermined number of calculated movement routes superimposed and displayed so that its position matches the background imaged on the shooting background 2802.

[0065] To realize such a display, known technologies such as AR, VR, and MR are employed. The selected travel route display 2803 displays an icon indicating which route the travel route display 2801 corresponds to among the predetermined number of calculated travel routes. Another feature is that it has a UI that allows the user to select one of the multiple travel routes.

[0066] The photographing information may include the possible moving speed, the notification means may calculate a plurality of moving routes based on the possible moving speed, and the user may select one of the plurality of moving routes. The photographing point 2804 is displayed superimposed on the position determined to have a high evaluation value. The photographing direction 2805 is displayed in association with the photographing point 2804, and indicates which way the person needs to face when photographing at the photographing point 2804.

[0067] On the other hand, in the display during actual photography, the current scenery is photographed, for example, by a camera mounted on a smartphone, and displayed in real time on the monitor screen 2806 as a photography background 2802. A travel route display 2801, a selected travel route display 2803, a photography point 2804, and a photography direction 2805 are displayed so as to be superimposed on the photography background 2802.

[0068] The travel route display 2801 is one of the predetermined number of calculated travel routes superimposed so that its position matches the background imaged on the photographed background 2802. This travel route display 2801 highlights only the time near the position corresponding to the current time, and travel routes for other times are dimmed or hidden. The selected travel route display 2803 displays an icon indicating which route the travel route display 2801 corresponds to among the predetermined number of calculated travel routes.

[0069] The photographing point 2804 is displayed superimposed on the position determined to have a high evaluation value. The photographing direction 2805 is displayed in association with the photographing point 2804, and indicates which way the person needs to face when photographing at the photographing point 2804. In this way, when scouting locations, the 3D model can be used to check whether the calculated travel route is suitable for the type of filming, and during actual filming, the travel route confirmed during location scouting can be viewed as needed while filming.

[0070] This reduces the burden of the travel route, out of the burden of the photography and travel route that the photographer bears when taking a photograph, and allows the photographer to concentrate more on taking photographs. The cameras used for actual shooting may be not only general still cameras or video cameras, but also remote cameras such as drones. [Example]

[0071] Next, a third embodiment of the present invention will be described. In Example 3, in addition to the travel route calculated in Example 2, an interference evaluation value for event viewers is also calculated, and a travel route that will result in a roughly uniform interference evaluation value for a large number of event viewers (i.e., a photography route that does not interfere with event viewing) is notified. The basic configuration of this embodiment is the same as that of the second embodiment, so the following description will focus on the differences.

[0072] The information input means 2400 inputs location information of event viewers in addition to the input data described in the first and second embodiments. Event viewers are participants who come to watch an event, such as parents in the case of an elementary school athletic meet, and spectators in the case of an event such as a live music concert. Location information of these event viewers is also input. For convenience, this embodiment assumes an elementary school athletic meet, and the event viewers are parents.

[0073] The evaluation means 2403 obtains an evaluation value of interference with the event viewers in addition to the evaluations described in the first and second embodiments. FIG. 29 is an image diagram of the parent seats in the second and third embodiments. For example, as shown in Figure 29, the event venue is divided into meshes at regular intervals, and the intersections of the meshes that are inside the parent-teacher seats are selected in order.

[0074] Furthermore, the resolution of the mesh does not have to be the same as that of the mesh used in the second embodiment, but in this embodiment, a mesh with the same resolution is used. FIG. 30 is an image diagram showing how to calculate the obstruction evaluation value in the second and third embodiments, and an example of parent location information 3001 will be described with reference to FIG. 30(A).

[0075] An obstruction evaluation value is calculated for one piece of position information 3000 selected from the intersections of the meshes that are selected sequentially. It should be noted that each parent's location information 3001 has an associated count value, and the initial value is set to 1. Next, a method for determining the interference evaluation value will be described with reference to FIG.

[0076] First, one of the subject model positions 3004 is connected by a straight line to the selected position information 3000, which is the shooting position. A straight line 3010 is extended from the end of the straight line on the side of the selected position information 3000, and the tentative count value for within range 3011 is set to the associated count value +1, which is set to the range centered at subject model position 3004 and expanded by a predetermined angle θ. On the other hand, for outside range 3012, which is outside the expanded range by θ, the associated count value itself is set to the tentative count value.

[0077] The total value A of the count value + 1 linked to all parent location information 3001 is calculated, and the total value B of the temporary count value of all parent location information 3001 is calculated, and (AB) / A is treated as the similarity. If there are multiple subject models assumed to be the subject, this value is calculated for all of those subject models, and the value obtained by multiplying all of the ratios is used as the interference evaluation value, which is treated in the same way as the similarity.

[0078] Furthermore, the overall similarity is calculated using the method described in the first embodiment, and the possible count value is elevated to the original count value based on the selected position information 3000 that was actually adopted when determining the travel route.

[0079] By processing this in chronological order, the count values are determined sequentially. This is a mechanism for minimizing the possibility of complaints in cases where a parent may complain if they linger too long in front of a specific parent on the day of the sports day. By treating the interference evaluation value in the same way as similarity and taking it into account based on the method of this embodiment, the problem of unintentionally preventing specific parents from continuing to watch the children compete can be alleviated. The notification means may calculate an optimal travel route based on the obstruction evaluation value. [Example]

[0080] Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, the optimal arrangement of multiple fixed cameras is calculated by simulation. FIG. 31 is a block diagram showing an example of the configuration of a simulation device according to the fourth embodiment. 31 have substantially the same configuration as 100 to 103 and 104 in FIG. 1, with the addition of an evaluation result storage means 3104 and a camera installation information selection means 3106.

[0081] The differences from the first embodiment will be explained. First, the evaluation means 3103 in FIG. 31 compares the photographed image input by the information input means 3100 with the simulation image generated by the simulation image generation means 3102, and calculates the similarity for each subject model. The evaluation result storage means 3104 is composed of a memory or the like, and stores the similarity for each subject model output by the evaluation means 3103, the score obtained by adding up the similarities, and the shooting information input to the information input means 3100 as the evaluation result.

[0082] The camera installation information selection means 3106 selects the photographic information based on the photographic information input to the information input means 3100 and the evaluation results stored in association with each other in the evaluation result storage means 3104 . The notification means 3105 is composed of a display device such as a liquid crystal panel, and displays the photographic information selected by the camera installation information selection means 3106 as the installation information of the fixed camera.

[0083] Next, the process of displaying information about the installation position and orientation of the fixed camera in this embodiment will be described with reference to Figs. 32 to 37, taking as an example the process when four students A to D hold a dance at an athletic meet. FIG. 32 shows a flowchart of the process for displaying information about the installation position and orientation of the fixed camera in this embodiment.

[0084] First, in step S3201 of FIG. 32, the area of the playground as seen from above, the dance performances of students A to D, and the direction of their faces are input as event information to the information input means 3100, and the process proceeds to step S3202. Figure 33 shows an example of event information input in Example 4. Using Figure 33, we will specifically explain the area of the playground as seen from above, the positions of students A to D during their dance performances, and the direction of their faces, which are input as event information in step S3201.

[0085] 3301 indicates the area of the playground, and is entered as two-dimensional coordinates such as G(Xg, Yg) with the upper left corner as the origin when viewed from above. 3311 to 3314 indicate the positions and facial directions of students A to D, respectively. 3300 shows a table in which the positions and facial directions of students A to D are entered. MApos(Xn,Yn) indicates the position of the center of student A's face and is entered in two-dimensional coordinates. MBpos(Xn,Yn) indicates the position of the center of student B's face and is entered in two-dimensional coordinates. MCpos(Xn,Yn) indicates the position of the center of student C's face and is entered in two-dimensional coordinates. MDpos(Xn,Yn) indicates the position of the center of student D's face and is entered in two-dimensional coordinates. n is an integer.

[0086] Enter the angle of the face direction, with north being 0 degrees, east being 90 degrees, south being 180 degrees, and west being 270 degrees. MAang(Θn) is the angle that indicates the face direction of student A. MBang(Θn) is the angle that indicates the face direction of student B. MCang(Θn) is the angle that indicates the face direction of student C. MDang(Θn) is the angle that indicates the face direction of student D.

[0087] Figure 33(A) shows the positions and orientations of students A to D at the start of the dance at time T0. Figure 33(B) shows the positions and orientations of students A to D during the dance performance at time T1 (after time T1 has elapsed). Figure 33(C) shows the positions and orientations of students A to D at the end of the dance at time T2. In this way, event information is set by inputting the positions and orientations of students A to D at each time.

[0088] In this embodiment, the area of the playground and the positions of the students are set using two-dimensional coordinates as viewed from above, but they may also be set using three-dimensional coordinates. Also, in this embodiment, the orientation of the students is set using a single-axis angle as viewed from above, but they may also be set using three-axis angles. Also, in this embodiment, three states, T0 to T2, are input, but by increasing the number of states, it is possible to set more detailed event information that reflects changes over time.

[0089] Furthermore, as event information, a priority of 0% (no priority) to 100% (highest priority) may be input as main subject information for each student. By adding a priority, it becomes possible to perform a weighted evaluation by multiplying the priority by the similarity for each of students A to D calculated by evaluation means 3103, which will be described later. Then, the notification means can notify information regarding the position and orientation of the camera based on the priority. In step S3202, the photographing position, photographing direction, and photographing angle of the fixed camera are input as photographing information to the information input means 3100, and the process proceeds to step S3203.

[0090] Next, Fig. 34 is a diagram showing an example of inputting shooting information in the fourth embodiment, and a method for inputting the shooting position, shooting direction, and shooting angle of view of a fixed camera will be specifically described using Fig. 34. Fig. 34(A) shows the shooting position, and the positions of the fixed camera that can be set within the playground area 3401 are input as shooting positions P0 (Xp0, Yp0) to P9 (Xp9, Yp9) in two-dimensional coordinates. Fig. 34(B) shows the shooting direction, and the shooting orientations D0 to D15 of the fixed camera are input. Fig. 34(C) shows the shooting angle of view, and F, which indicates the shooting angle of the fixed camera, is input.

[0091] The number of fixed cameras and lens information (for example, focal length) may be added as shooting information. By adding the number of fixed cameras and lens information, the shooting information selected by the camera installation information selection means 3106 (described later) can be set based on the number and lens information, making it possible to display based on the number of fixed cameras that can be prepared and the lens information. That is, the notification means can notify information about the positions of the cameras based on the similarity and the number of cameras or lens information.

[0092] 35 is a diagram showing an example of a desired photographed image in Example 4, and in step S3203, the image in Figure 35, for example, is input as the desired photographed image by the information input means 3100, and the process proceeds to step S3204. Note that in this example, one type of image is input as the desired photographed image, but multiple images may also be input. In step S3204, the virtual space constructing means 3101 constructs a virtual space that reproduces the dance situation based on the event information input in step S3201, and the process proceeds to step S3205.

[0093] In step S3205, a simulation image is generated based on the desired photographed image input in step S3202 and the virtual space constructed in step S3204, and the process proceeds to step S3206. In step S3206, the evaluation means 3103 calculates the similarity for each of the students A to D based on the desired photographed image input in step S3203 and the simulation image generated in step S3205, and the process proceeds to step S3207.

[0094] In step S3207, the similarity for each of students A to D calculated in step S3206 and the shooting information of the fixed camera used to generate the simulation image in step S3205 are stored as associated evaluation results in the evaluation result storage means 3104, and the process proceeds to step S3208.

[0095] FIG. 36 is a conceptual diagram of associating photographic information with the similarity of students in the fourth embodiment, showing the concept of associating and storing photographic information from a fixed camera with the similarity of each of students A to D. The shooting information (shooting position, shooting direction, shooting angle, time) of the fixed camera used to generate the simulation image is set as the parameters of the captured image. The evaluation result is stored in association with the similarity to the simulation image generated using the parameters of the captured image described above, and a score that is the sum of the similarities of each student.

[0096] In step S3208, the input information input to the information input means 3100 is compared with the evaluation results stored in the evaluation result storage means 3104 to determine whether the evaluation is complete. If it is determined that the evaluation is complete, proceed to step S3209; otherwise, set the input information as unevaluated and return to step S3205.

[0097] In step S3209, the camera installation information selection means 3106 sorts the evaluation results stored in the evaluation result storage means 3104, for example, in descending order of score, and displays the position, direction, and number of fixed cameras on the display device, thereby terminating the process. Here, Fig. 37 shows a display example in Example 4. Fig. 37(A) shows a display example in which the positions of four fixed cameras are superimposed on the screen so as to maximize the similarity between students A to D.

[0098] 37(B) shows a display example in which the position of one fixed camera is superimposed on the screen so that the score becomes the maximum value when the number of fixed cameras is set to 1 in advance in step S3202. Note that a specific method for realizing the display example is to use a known method such as AR, VR, or MR. Also, the display example in this embodiment is just one example, and it is sufficient if the user is notified of the camera installation information, and an icon display or the like may also be used.

[0099] As described above, the image processing device according to this embodiment can determine the number, positions, and orientations of fixed cameras in a shorter time than conventional methods, making it possible to efficiently perform location scouting for event photography. The camera used for actual shooting may not only be a fixed camera, but also a general still camera, video camera, or remote camera such as a drone. [Example]

[0100] Next, a fifth embodiment of the present invention will be described. In Example 5, when a fixed camera with a subject tracking function is installed, setting information is provided for determining the installation position and orientation of at least one camera so that the model to be photographed is included in the desired captured image.

[0101] The process of displaying information about the installation position and orientation of the fixed camera in this embodiment will be described using Figs. 38 to 41, taking as an example the process when four students A to D hold a dance at a sports day. FIG. 38 shows a flowchart of the process for displaying information about the installation position and orientation of the fixed camera in this embodiment. Steps S3801 to S3808 in FIG. 38 are the same as steps S3201 to S3208 in FIG. 32 of the fourth embodiment, so only the different parts from S3809 onwards will be explained.

[0102] In step S3809, the camera installation information selection means 106 sorts the evaluation results stored in the evaluation result storage means 3104, for example, in descending order of score, and selects the position, direction, and number of fixed cameras, and the process proceeds to step S3810. Here, Fig. 39 is a diagram showing an example of selection of the position of a fixed camera in the fifth embodiment, and shows an example of selection by the camera installation information selection means 106. Note that Fig. 39 shows an example of selection when one fixed camera is placed at position (P5) so as to maximize the similarity between students A to D, and shows an example of display similar to Fig. 37(B).

[0103] In step S3810, multiple simulation images are generated under the conditions that the desired captured image input in step S3802, the virtual space constructed in step S3804, the positions and number of fixed cameras selected in step S3809 are fixed, and the shooting directions D0 to D15 of each camera are variable, and then the process proceeds to step S3811. In step S3811, the evaluation means 103 calculates the similarity for each of the students A to D for each condition based on the simulation image generated in step S3810, and the process proceeds to step S3812.

[0104] In step S3812, the similarity for each of students A to D calculated in step S3811 and the shooting information of the fixed camera used to generate the simulation image are stored as associated evaluation results in the evaluation result storage means 3104, and the process proceeds to step S3813. In step S3813, the similarity calculated from the evaluation results stored in the evaluation result storage means 3104 is evaluated, and information for selecting students A to D to follow for each camera in chronological order is displayed on the display device, and the process ends.

[0105] Next, a specific example of the process in step S3813 will be shown with reference to FIGS. Figure 40 shows an example of selecting the position and number of fixed cameras in Example 5, and shows the student with the highest hourly score when one fixed camera is placed at P5 to maximize the similarity between students A to D. Figure 40(A) shows that the highest score for a camera placed at position P5 at time T0 is student D1014. Figure 40(B) shows that the highest score for a camera placed at position P5 at time T1 is student D1014.

[0106] 40(C) shows that the maximum score of the camera placed at position P5 at time T2 is student C 1013. Therefore, the main subject tracked by the camera placed at position P5 is student D 1014 at times T0 and T1 for each time series, and student C 1013 is selected at T2.

[0107] 41 is a conceptual diagram of a fixed camera and a subject to be tracked in Example 5, and shows an example of a display displayed on a display device. In this example, a table is displayed that associates a camera installed at position P5 with the time and information on the main subject to be tracked, so that the selected main subject is switched halfway between times T1 and T2. Note that specific display employs known methods such as AR, VR, MR, etc. Also, the display example in this embodiment is just an example, and it is sufficient to notify the user of camera installation information and main subject information to be tracked, and icon display may also be used.

[0108] In this embodiment, the images from one camera are sorted so that they have the greatest similarity to the desired captured image. However, even if four cameras are installed, the images can be sorted in the same way so that the similarity to the desired captured image is maximized. Furthermore, when multiple cameras are installed and two or more cameras select the same main subject at the same time, the main subject selection of the camera with the highest similarity may be made valid, and the other cameras may be sorted so that the similarity of another main subject becomes the highest.

[0109] As described above, the image processing device according to this embodiment makes it possible to efficiently perform location scouting for event photography by determining the number, positions, and orientations of fixed cameras in a shorter time than conventional methods. The camera used for actual shooting may not only be a fixed camera, but also a general still camera, video camera, or remote camera such as a drone.

[0110] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. A computer program that realizes all or part of the control in this embodiment and the functions of the above-described embodiment may be supplied to the simulation device via a network or various storage media. The computer (or CPU, MPU, etc.) in the simulation device may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0111] 100 Information input means 101 Virtual Space Construction Methods 102 Simulation image generation means 103 Evaluation Instruments 104 Means of Notification

Claims

1. an information input means for inputting information about the event to be photographed and a desired photographed image; a virtual space construction means for constructing a virtual space that simulates the event using a subject model based on information about the event to be photographed; a simulation image generating means for generating, using the virtual space, a plurality of simulation images obtained by photographing the event under different photographing conditions; an evaluation means for sequentially evaluating the similarity between the plurality of simulation images taken under different photographing conditions and the desired photographed image; a notification means for selecting the simulation images whose similarity is equal to or greater than a predetermined value, which are sequentially evaluated by the evaluation means, and notifying the user of the photographing conditions of the selected simulation images; and the photographing conditions include a movable speed; the notification means calculates a plurality of movement routes for sequentially photographing under predetermined photographing conditions according to the degree of similarity based on the possible moving speed; A simulation device comprising a UI for a user to select one of the plurality of travel routes.

2. An information input means for inputting information about an event to be photographed and a desired photographed image; a virtual space construction means for constructing a virtual space that simulates the event using a subject model based on information about the event to be photographed; a simulation image generating means for generating, using the virtual space, a plurality of simulation images obtained by photographing the event under different photographing conditions; an evaluation means for sequentially evaluating the similarity between the plurality of simulation images taken under different photographing conditions and the desired photographed image; a notification means for selecting the simulation images whose similarity is equal to or greater than a predetermined value, which are sequentially evaluated by the evaluation means, and notifying the user of the photographing conditions of the selected simulation images; and The event information includes location information of event attendees; the evaluation means includes an evaluation value of disruption to the event viewers, The simulation device is characterized in that the notification means calculates a movement route for sequentially taking photographs under predetermined photographing conditions according to the similarity based on the obstruction evaluation value.

3. 3. The simulation device according to claim 1, wherein the event information includes at least one of program information including the content of the event and the order of the event, and information about an accessible area.

4. 4. The simulation device according to claim 1, wherein the photographing conditions include at least one of a photographing position, a photographing direction, and a photographing angle of view.

5. 4. The simulation device according to claim 3, wherein the program information includes subject position information relating to at least one of the layout of a venue where the event is held, the position of a subject at each time, and the orientation of the subject.

6. 6. The simulation device according to claim 5, wherein the subject position information can be selected from preset samples.

7. 7. The simulation device according to claim 1, wherein the desired photographed image is associated with at least one of the number of subjects, face positions, face sizes, and face orientations.

8. 8. The simulation device according to claim 1, wherein the desired photographed image can be selected from preset samples.

9. The simulation device according to any one of claims 1 to 8, characterized in that the evaluation means calculates similarities between the simulation image and the desired captured image for at least two of the number of subjects, face position, face size, and face orientation, and calculates an evaluation value using a weighting coefficient set according to the desired captured image.

10. The simulation device according to any one of claims 1 to 9, wherein the notification means notifies the user of a predetermined simulation image and predetermined shooting conditions according to the similarity by displaying, printing, or distributing the image.

11. 11. The simulation device according to claim 1, wherein the notification means displays the travel route by superimposing it on a live view image.

12. The simulation device according to any one of claims 1 to 10, characterized in that the notification means displays the movement route and a model of the subject in the virtual space on a monitor before the event, superimposed on a live view image.

13. The simulation device according to any one of claims 1 to 12, characterized in that the notification means notifies information regarding a position and orientation of a camera so that a predetermined subject model is included in the desired captured image based on the similarity.

14. the information input means inputs main subject information including a priority of the main subject; 14. The simulation device according to claim 1, wherein the notification means notifies information relating to the position and orientation of a camera based on the priority.

15. 15. The simulation device according to claim 13, wherein the notification means notifies information about the positions of the cameras based on the similarity and the number of cameras or lens information.

16. A computer program for controlling each means of the simulation device according to any one of claims 1 to 15 by a computer.

17. A computer-readable storage medium storing the computer program according to claim 16.

Images