CG video superimposing device

Abstract

A CG video superimposing device according to one embodiment of the present invention comprises a live video acquisition unit, a CG video generation unit, a luminance difference calculation unit, a video superimposition unit, and a video presentation unit. The live video acquisition unit acquires a live video. The CG video generation unit generates a plurality of CG videos using different rendering conditions. The luminance difference calculation unit calculates a difference from the live video for each of the CG videos. The video superimposition unit generates a superimposed video in which a CG video having the smallest difference from the live video among the CG videos is superimposed on the live video. The video presentation unit presents the superimposed video.

Description

CG image overlay device

[0001] The present invention relates to a CG image superimposing device that superimposes a CG image on a live-action image.

[0002] One method for virtually experiencing the field of view when operating a mobile machine such as a vehicle involves synthesizing CG (Computer Graphics) images with live-action footage that has actually been captured. For example, Non-Patent Document 1 describes a method in which camera parameters such as the camera position are identified based on an A4 sheet of paper captured in the live-action footage, a CG image is generated based on the identified camera parameters, and the generated CG image is superimposed on the live-action footage. However, with this method, only the camera parameters can be reflected from the live-action footage into the CG image. In other words, information about the live-action footage other than the camera parameters is not reflected in the CG image.

[0003] Furthermore, Non-Patent Document 2 describes a method for recognizing roads shown in live-action footage and comparing road geometry information with intersection and road width information stored in storage. With this method, the only information that can be reflected from the live-action footage into the CG image is the position of the arrow indicating the direction of travel. In other words, information from the live-action footage other than the position of the arrow indicating the direction of travel is not reflected in the CG image.

[0004] Yuji Nakazawa and three others, "A System for Combining Photographed Images and Computer Graphics Images Based on Image Feature Points," Journal of the Institute of Image Information and Television Engineers, Vol. 51, No. 7, pp. 1086-1095, 1997, Internet: <https: / / www.jstage.jst.go.jp / article / itej1997 / 51 / 7 / 51_7_1086 / _article / -char / ja / > Hiroaki Sawano and one other, "Real-time Image Processing Using an In-Vehicle Camera and Its Application to Car Navigation Systems Using an AR Technology-Based Display Method," Journal of the Society of Arts and Sciences, Vol. 5, No. 2, pp. 57-68, Internet: <https: / / www.jstage.jst.go.jp / article / artsci / 5 / 2 / 5_2_57 / _pdf / -char / ja>

[0005] When practicing the operation of a vehicle, using an actual vehicle can sometimes prevent sufficient practice opportunities due to factors such as the environment or the absence of an instructor. Therefore, using a simulator is effective. For example, windsurfing involves gliding at high speeds of around 60 km / h several kilometers from the coast, making it difficult for an instructor to be nearby while operating a real vehicle. Furthermore, practice is not possible in weak wind environments. These problems can be solved by using a simulator.

[0006] When operating a vehicle, it is important to appropriately control speed. For this reason, for example, automobile driving simulators are equipped with a speedometer, allowing the driver to control speed using the accelerator and brake while checking the speedometer. On the other hand, in vehicles such as windsurfing, where it is difficult to install a speedometer, simulator occupants use wave crests displayed on a 360-degree video using a virtual reality head-mounted display (VRHMD) as a clue to determine speed. However, there are cases where a sufficient number of wave crests are not displayed in the live video, or where the visibility of the wave crests is insufficient due to low resolution of the VRHMD.

[0007] The present invention has been made in light of the above-mentioned circumstances, and its object is to present a superimposed image in which an appropriate CG image is superimposed on a real-life image.

[0008] In order to achieve the above object, one aspect of a CG image superimposition device according to the present invention comprises a live-action image acquisition unit, a CG image generation unit, a brightness difference calculation unit, an image superimposition unit, and an image presentation unit. The live-action image acquisition unit acquires live-action images. The CG image generation unit generates multiple CG images using different rendering conditions. The brightness difference calculation unit calculates the difference in brightness value between each CG image and the live-action image. The image superimposition unit generates a superimposed image by superimposing the CG image with the smallest difference from the live-action image on the live-action image. The image presentation unit presents the superimposed image.

[0009] According to one aspect of the present invention, for example, a CG image simulating an object for judging the moving speed of a vehicle during training is superimposed on a peripheral image and presented to the trainee. Therefore, even if an object used for speed judgment is difficult to see or missing in the acquired peripheral image, an image in which an object simulating the object used for speed judgment is superimposed on the peripheral image can be presented to the trainee. The trainee can judge the speed based on the movement of the object displayed together with the peripheral image.

[0010] That is, according to one aspect of the present invention, it is possible to present a superimposed image in which an appropriate CG image is superimposed on a real-life image.

[0011] FIG. 1 is a diagram illustrating an example of the configuration of a system including a CG image superimposing device according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a CG image superimposing device and a simulator according to an embodiment. FIG. 3 is a diagram illustrating an example of how live-action video is captured using an imaging device according to an embodiment. FIG. 4 is a flowchart illustrating an example of a processing procedure for a simulation video presentation process performed by the CG image superimposing device according to an embodiment. FIG. 5 is a diagram illustrating an example of live-action video according to an embodiment. FIG. 6 is a diagram illustrating an example of a live-action video of a frame different from that shown in FIG. 5. FIG. 7 is a diagram illustrating an example of a mask image used in a mask process according to an embodiment. FIG. 8 is a diagram illustrating the difference in luminance values ​​between a CG image and a live-action video for each frame according to an embodiment. FIG. 9 is a diagram illustrating the difference in luminance values ​​between a CG image and a live-action video for each frame different from that shown in FIG. 8. FIG. 10 is a diagram illustrating the difference in luminance values ​​between a CG image and a live-action video for each frame different from that shown in FIGS. 8 and 9. FIG. 11 is a diagram illustrating the difference in luminance values ​​between a CG image and a live-action video for each frame different from that shown in FIGS. 8 to 10. FIG. 12 is a diagram illustrating an example of a superimposed video according to an embodiment.

[0012] A CG image superimposition device according to an embodiment of the present disclosure will be described in detail below with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0013] (Configuration Example) Fig. 1 is a diagram showing an example of the configuration of a windsurfing simulation system including a CG image superimposition device according to one embodiment. The windsurfing simulation system is a system for training windsurfers in maneuvering using a simulator that mimics the shape, movement, and field of view of a windsurfing machine. A windsurfing machine is an example of a mobile machine. The configuration according to this embodiment may also be applied to mobile machines other than windsurfing machines.

[0014] The windsurfing simulation system includes a simulator SS, a display device DS, a steering simulation image generation device CS, a position measurement device PS, and an imaging device CM. The steering simulation image generation device CS is connected to the simulator SS, the display device DS, the position measurement device PS, and the imaging device CM via wired or wireless connections. The steering simulation image generation device CS generates simulation images used for windsurfing steering training using data received from the position measurement device PS and the imaging device CM. The simulation images are images in which a CG image (described below) simulating wave crests is superimposed on a real-life image (described below) received from the imaging device CM. The simulation images are an example of superimposed images, and the steering simulation image generation device CS is an example of a CG image superimposition device. The simulator SS and the display device DS are used when conducting windsurfing steering training using the simulation images generated by the steering simulation image generation device CS.

[0015] 2 is a diagram showing an example of the configuration of the simulator SS. The simulator SS includes a surfing section 100 on which a user undergoing piloting training boards, and a drive section 200 that supports the surfing section 100. The surfing section 100 includes a board 101 provided with a platform on which the user stands, a mast 102 erected on the board 101, and a sail 103 attached to the mast 102.

[0016] The driving unit 200 includes an actuator 201 for reproducing the four-axial movements of pitch, roll, yaw, and heave that occur in an actual windsurfing machine. The driving unit 200 swings the board 101 and mast 102 of the surfing club 100 in the directions of the four axes by operating the actuator 201 in response to a drive control signal output from a simulator control device (not shown).

[0017] The drive unit 200 is not limited to the four axes mentioned above, but may be capable of supporting five or more axes, including the inclination angle of the mast section relative to the board, or may be capable of supporting only two axes, pitch and yaw, or only three axes, pitch, roll, and yaw.

[0018] The display device DS is, for example, a virtual reality head mounted display (VRHMD) and is worn on the user's head. The display device DS displays the simulation video output from the operation simulation video generation device CS. The display device DS may also include a speaker that outputs audio data.

[0019] The maneuvering simulation video generation device CS generates simulation videos used for windsurfing maneuver training using information received from the imaging device CM and the position measurement device PS. Fig. 3 shows how a real windsurfing device 300 is used to capture surrounding video used in the simulation video. The real windsurfing device 300 includes a board 301 provided with a platform on which a user stands, a mast 302 erected on the board 301, and a sail 303 attached to the mast 302. When generating the actual video, an instructor wearing the imaging device CM steers the real windsurfing device 300 and glides across the water.

[0020] The position measurement device PS is attached to the actual windsurfing machine 300. The position measurement device PS is attached to, for example, the mast 302. When the actual windsurfing machine is being used to glide on the water, the position measurement device PS continuously measures position information indicating latitude and longitude using a GPS (Global Positioning System) sensor, and outputs a collection of the measured latitude and longitude data as movement trajectory data to the maneuvering simulation image generation device CS.

[0021] The imaging device CM is attached to the head of a user riding the windsurfing machine 300. While the user is gliding on the water using the windsurfing machine 300, the imaging device CM captures surrounding images using, for example, an omnidirectional (360-degree) camera attached to the operator's head, and outputs the captured surrounding image data to the maneuvering simulation image generation device CS. Hereinafter, the surrounding image data will be referred to as "actual image." Note that the imaging device CM is not limited to capturing omnidirectional images, and may also capture panoramic images at least within the range of the operator's field of view.

[0022] The maneuvering simulation image generating device CS generates a simulation image using the movement trajectory data received from the position measurement device PS and the actual image acquired from the imaging device CM. The maneuvering simulation image generating device CS is realized as an information processing terminal such as a personal computer.

[0023] The movement trajectory data can be used to calculate the movement speed of the actual windsurfing equipment. If there is no need to calculate the movement speed, the position measurement device PS does not need to be used.

[0024] Next, the configuration of a maneuvering simulation image generation device CS, which is a CG image superimposition device according to this embodiment, will be described. The maneuvering simulation image generation device CS includes a control unit 1, a program storage unit 2, a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4. The control unit 1 includes a hardware processor such as a central processing unit (CPU). The control unit 1 is connected to each of the program storage unit 2, the data storage unit 3, and the input / output I / F unit 4 via buses.

[0025] The simulator SS and the display device DS are connected to the input / output I / F unit 4 via signal cables such as USB (Universal Serial Bus) cables. The input / output I / F unit 4 transmits and receives control signals and simulation images between the simulator SS and the display device DS.

[0026] The above-mentioned position measurement device PS and imaging device CM can be connected to the input / output I / F unit 4. The input / output I / F unit 4 receives the above-mentioned movement trajectory data from the position measurement device PS. The input / output I / F unit 4 also receives the above-mentioned actual video from the imaging device CM.

[0027] Note that a wireless interface that adopts a low-power wireless data transmission standard such as Bluetooth (registered trademark) or Wi-Fi (registered trademark), or a wireless interface of a public mobile radio communication network, may be used as the input / output I / F unit 4. Use of a wireless interface can eliminate the need for signal cables connecting the simulator SS, display device DS, position measurement device PS, and imaging device CM, thereby increasing the degree of freedom for the user.

[0028] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various control processes according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0029] The data storage unit 3 is, for example, a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from at any time with a volatile memory such as a RAM (Random Access Memory). The data storage unit 3 stores the above-mentioned movement trajectory data, real-life video, simulation video, etc. The data storage unit 3 also stores information used when generating the simulation video and information temporarily generated when generating the simulation video.

[0030] The control unit 1 includes, as processing functions, a live-action image acquisition unit 11, a CG image generation unit 12, a brightness difference calculation unit 13, an image superimposition unit 14, and an image output unit 15. Each of the above processing units 11 to 15 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the above processing units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0031] The live-action video acquisition unit 11 receives live-action video captured by the imaging device CM via the input / output I / F unit 4 and stores the acquired live-action video in the data storage unit 3. The live-action video is, for example, video data including multiple 360-degree images. The number of frames of the live-action video is, for example, 200 frames.

[0032] The CG image generation unit 12 generates CG images and stores the generated CG images in the data storage unit 3. Each CG image is video data including multiple CG images. Each CG image has the same number of frames as the live-action image. The CG image has the same shape and the same number of pixels as the live-action image. Each CG image includes an object (hereinafter referred to as a wave crest object) that simulates the crest of a wave. The wave crest object is an example of an object that simulates a feature in the live-action image that is difficult to see or a feature missing from the live-action image. Each wave crest object is generated (rendered) to have a shape and color similar to the shape of an actual wave crest. Furthermore, the wave crest object is positioned in the CG image within a range corresponding to the sea surface in the live-action image. Furthermore, the wave crest object is generated to have high visibility when superimposed on the sea surface in the live-action image. For example, the wave crest object has a certain size or more and is formed in a color that is easily distinguishable from the ocean surface in the live-action image. More specifically, each CG image is generated using a three-dimensional structure including a combination of polygons (polygons) each made up of triangles, each of whose three vertices has three-dimensional coordinates (x, y, z), and rectangular objects. The wave crest object is represented as a white rectangle on a plane at the height of the ocean surface (e.g., y = 0). The size of the wave crest object in the direction of the windsurfing (e.g., z-axis) is set to, for example, half the wavelength (e.g., 28 m). The distance between adjacent wave crest objects is set to, for example, half the wavelength (e.g., 28 m). The size of the wave crest object in a direction perpendicular to the direction of the windsurfing and the height direction (e.g., x-axis) is set to, for example, a size that ensures sufficient visibility (e.g., 100 m). The CG image is generated by calculating the position of the apex of the wave crest object as seen from the viewpoint ((x, y, z) = (0, 1.7, 0)) of a person at a height of 1.7 m using perspective transformation.

[0033] It is also preferable that a sufficient number of wave crest objects are included in the entire CG image so that the occupant of the simulator SS can easily judge the speed. It is also preferable that the moving speed of the wave crest objects and the distance between the wave crest objects (hereinafter also referred to as the wavelength of the wave crest objects) in the entire CG image are set to values ​​close to the moving speed of the actual wave crests and the distance between the actual wave crests so that an unnatural feeling does not appear when superimposed on the actual image.

[0034] The CG image generation unit 12 also generates a plurality of CG images using different rendering conditions. The CG image generation unit 12 generates a plurality of CG images by changing one or more parameters related to the rendering of the wave crest object. Examples of the parameters that can be changed include the spacing between the wave crest objects and the movement speed of the wave crest objects. When changing the parameters, it is preferable to change the parameters within the range that the actual wave crest can take.

[0035] The brightness difference calculation unit 13 calculates a total difference value between each of the multiple CG images and the live-action image, and stores the calculated total difference value in the data storage unit 3, linking it to each CG image. The total difference value is a feature that indicates the degree of difference between the CG image and the live-action image. The total difference value is a value obtained by adding up the absolute values ​​of the differences in brightness between the CG image and the live-action image, calculated for each frame. The difference in brightness between the CG image and the live-action image is calculated using pixel values ​​at the same position in the same frame. It is preferable to calculate the total difference value after performing grayscale processing or masking processing on both the live-action image and the CG image.

[0036] The image superimposing unit 14 generates a simulation image by superimposing the object image on the real-life image, and stores the generated simulation image in the data storage unit 3. At this time, the image superimposing unit 14 selects the CG image with the smallest total difference value from the multiple CG images stored in the data storage unit 3, and superimposes the selected CG image on the real-life image. In this way, a simulation image is generated in which the CG image with the smallest total difference value from the real-life image is superimposed on the real-life image.

[0037] The video output unit 15 reads out a simulation video from the data storage unit 3 in response to an operation signal received from the simulator SS, and outputs the read out simulation video to the display device DS via the input / output I / F unit 4. The video output unit 15 is an example of a video presentation unit.

[0038] (Operation Example) Next, an operation example of the piloting simulation image generation device CS configured as described above will be described. FIG. 4 is a flowchart showing an example of the procedure for simulation image presentation processing by the piloting simulation image generation device CS. The simulation image presentation processing is a series of processes from generating a simulation image based on live-action image captured using a real machine to transmitting the generated simulation image to the display device DS worn by the pilot of the simulator SS. Note that the processing procedure for the reliability determination processing described below is merely an example, and each process can be modified as appropriate as possible. Furthermore, steps in the processing procedure described below can be omitted, replaced, or added as appropriate depending on the embodiment.

[0039] First, the process of generating a simulation video will be described.

[0040] When generating the simulation video, first, while the instructor is gliding on the water using a real windsurfing machine, an imaging device CM attached to the instructor's head captures an omnidirectional video of the surroundings as a real video, and the imaging device CM transmits the captured real video to the operation simulation video generation device CS.

[0041] The live-action video acquisition unit 11 acquires live-action video from the imaging device CM via the input / output I / F unit 4. Then, the brightness difference calculation unit 13 converts the acquired live-action video into grayscale (step S101). FIGS. 5 and 6 each show an example of live-action video J. FIGS. 5 and 6 show live-action images of different frames. FIGS. 5 and 6 are 360-degree images. As shown in FIGS. 5 and 6, the live-action video J displays a windsurfing machine 401, the sea surface 402, and the sky 403. The live-action video J also includes frames in which wave crests 404 are displayed, as in FIG. 5, and frames in which wave crests 404 are not displayed, as in FIG. 6.

[0042] Here, the moving speed of the wave crest 404 is sufficiently small compared to the moving speed of the windsurfing, so the wave crest 404 can be considered to be stationary relative to the windsurfing. Generally, there are no other stationary objects on the sea surface, so the operator of the simulator SS checks the moving status of the wave crest 404 to confirm the current moving speed of the windsurfing being operated. However, in the actual image J, there are cases where the wave crest is not shown, as shown in Figure 6, or where the wave crest 404 is not large enough, as shown in Figure 5, making it difficult to visually recognize the wave crest 404.

[0043] Next, the brightness difference calculation unit 13 performs masking on the grayscaled live-action video J to generate a live-action mask video (step S102). At this time, the live-action video acquisition unit 11 performs masking so that the brightness value of an area in the live-action video J where a windsurfing board or sail may be present (hereinafter referred to as a first area) is 0, and the brightness value of an area in the live-action video J where the sky, sea, and wave crests are present (hereinafter referred to as a second area) is 1. FIG. 7 is a diagram showing an example of a mask image M used for masking. The mask image M is a two-dimensional image having a pixel count corresponding to each frame image of the live-action video J. In FIG. 7, the first area 501 is shown in black, and the second area 502 is shown in white. The ranges of the first area 501 and the second area 502 may be input by a user via the input / output I / F unit 4, or may be automatically set by image analysis of the live-action video J. The live-action mask image generated by the masking process has a brightness value of 0 in areas of live-action image J where a windsurfing board or sail may be present, while maintaining the brightness values ​​of areas where the sky, sea, and wave crests are present, resulting in an image that shows only the sky, sea, and wave crests in grayscale.

[0044] Next, the CG image generation unit 12 generates a plurality of CG image generation conditions (step S103). In this case, the CG image generation unit 12 generates a plurality of rendering conditions in which one or more parameters related to the rendering of the wave crest object differ. Table 1 below shows an example of the generated rendering conditions. Table 1 shows an example of generating four rendering conditions C1 to C4 in which two parameters, the wavelength of the wave crest object and the moving speed of the wave crest object, are each changed in two stages. Each of the conditions C1 to C4 uses a numerical value close to that of an actual wave crest.

[0045]

[0046] The wavelength [L] of a wave crest object is the distance between multiple wave crest objects. The wavelength [L] is calculated, for example, using the following formula (1). In formula (1), "T" is the period [s], "g" is the gravitational acceleration [m / s^2], and "h" is the water depth [m]. In shallow waters where windsurfing competitions are held, the water depth is known to be approximately 5 [m] and the wave period T is 8 to 14 [seconds]. Therefore, using formula (1), the wavelength L of an actual wave crest is estimated to be 56 to 98 [m]. For this reason, in conditions C1 to C4, the wavelength [L] of the wave crest object is changed in two stages within the range of 56 to 98 [m].

[0047]

[0048] The actual moving speed [V] of the wave crest can be estimated using the moving speed of the windsurfing and the moving speed of the actual wave crest itself. Because the moving speed of the wave crest itself is sufficiently slower than the moving speed of the windsurfing, the moving speed [V] of the actual wave crest can be considered to be equal to the moving speed of the windsurfing. Since the moving speed of a windsurfing is known to be 50 to 60 km / h, it is advisable to set the moving speed [V] of the wave crest object in the video to be 50 to 60 km / h. For this reason, in conditions C1 to C4, the moving speed [V] of the wave crest object is changed in two stages within the range of 50 to 60 m.

[0049] Next, the CG image generation unit 12 generates a CG image based on the set rendering conditions (step S104). At this time, the CG image generation unit 12 generates the CG image using one of conditions C1 to C4. Here, we will explain the case where CG image I1 is generated using condition C1.

[0050] Next, the brightness difference calculation unit 13 converts the CG image I1 into a grayscale image (step S105), in a manner similar to the grayscale conversion performed on the real-life image J in step S101.

[0051] Next, the brightness difference calculation unit 13 performs masking on the grayscaled CG image I1 (step S106). At this time, the CG image generation unit 12 uses the mask image used for the live-action image J in step S102 to perform masking so that the brightness value of the area corresponding to the first area of ​​the live-action image J becomes 0 and the brightness value of the area corresponding to the second area of ​​the live-action image J becomes 1. After the masking process, the CG image I1 has a brightness value of 0 in the part corresponding to the first area of ​​the live-action image J and maintains the brightness value of the part corresponding to the second area of ​​the live-action image J, resulting in an image that includes only grayscale wave crest objects that are placed in the areas where the sky, sea, and wave crests are present.

[0052] Next, the brightness difference calculation unit 13 calculates a total difference value d1 for the masked CG image I1 relative to the masked live-action image J (step S107). The total difference value d1 is the sum of the absolute values ​​of the differences in brightness between the CG image and the live-action image at the same coordinates in the same frame, for all coordinates, and then the sum for all frames.

[0053] The control unit 1 repeatedly executes the processes of steps S103 to S107 until the total difference values ​​(d1 to d4) for all conditions (C1 to C4) are calculated (step S108-No). As a result, four CG images I1 to I4 are generated, each differing in at least one of the wavelength and the moving speed of the wave crest object, and grayscale image processing and masking processing are executed on each of the four CG images I1 to I4. Then, the total difference values ​​d1 to d4 from the live-action image J are calculated for each of the CG images I1 to I4.

[0054] 8 to 11 are diagrams showing the difference in luminance values ​​between each of CG images I1 to I4 and live-action image J for each frame. FIG. 8 shows the absolute value of the difference in luminance values ​​between CG image I1 and live-action image J, FIG. 9 shows the absolute value of the difference in luminance values ​​between CG image I2 and live-action image J, FIG. 10 shows the absolute value of the difference in luminance values ​​between CG image I3 and live-action image J, and FIG. 11 shows the absolute value of the difference in luminance values ​​between CG image I4 and live-action image J. The horizontal axis in FIGS. 8 to 11 indicates the frame number. The vertical axis in FIGS. 8 to 11 indicates the sum of the absolute values ​​of the difference in luminance values ​​at the same coordinates between the CG image and the live-action image in each frame, for all coordinates.

[0055] The following equation (2) is used to calculate the total difference value d ｎ The process of calculating the total difference value d ｎ "n" in the above is an identification number for the CG images I1 to I4, and in this case is any one of the numbers 1 to 4.

[0056]

[0057] In equation (2), "m" represents the number of frames in the live-action image J and the CG image, "x" represents the x-coordinate in the live-action image J and the CG image, and "x" represents the y-coordinate in the live-action image J and the CG image. Furthermore, J(m, x, y) in equation (2) represents the luminance value of the coordinates (x, y) of the mth frame of the live-action image in the grayscaled live-action image J, and I(n, m, x, y) represents the luminance value of the coordinates (x, y) of the mth frame of the CG image in the grayscaled CG image Cn. Furthermore, M(x, y) in equation (2) represents the luminance value of the coordinates (x, y) in the mask image of FIG. 7.

[0058] The total difference value d shown in equation (2) ｎ is calculated by calculating the absolute value of the difference between the brightness value of the real image and the brightness value of the CG image at the same coordinates in the same frame, summing the calculated absolute values ​​of the differences for all coordinates, and then summing the sum for all frames. ｎ is a value in which the influence of the area where windsurfing is present is eliminated by masking, and only the area where the sea, sky, and wave crests are present is affected. In other words, the total difference values ​​d1 to d4 are values ​​that indicate the difference in brightness between the wave crest objects contained in each CG image I1 to I4 and the areas in the live-action image J where the sea surface and wave crests are present. Therefore, the greater the difference between the actual wave crests contained in the live-action image and the wave crest objects contained in the CG image, the larger the total difference value. In other words, the closer the CG image is generated under rendering conditions that are closer to those of actual wave crests, the smaller the total difference value will be.

[0059] Table 2 below shows the total difference values ​​d1 to d4 calculated by the processing in steps S101 to S108.

[0060]

[0061] Once the total difference values ​​d1 to d4 of the CG images I1 to I4 have been calculated for all conditions C1 to C4 (step S108—Yes), the image superimposition unit 14 generates a simulation image by superimposing one of the CG images on the live-action image J (step S109). At this time, the image superimposition unit 14 generates a simulation image by superimposing CG image I2, which has the smallest total difference value among the four CG images I1 to I4 generated under different conditions, on the live-action image J. FIG. 12 is a diagram showing an example of the generated simulation image S. As shown in FIG. 12, in the generated simulation image S, a wave crest object 605 is superimposed on the sea surface 402 in the live-action image J. The simulation image generated by superimposing CG image I2 on the live-action image J has a smaller difference in overall brightness between the live-action image J and the CG image including the wave crest object than a simulation image generated by superimposing any one of the other CG images I1, I3, and I4 on the live-action image J. Therefore, the simulation video S is an image in which a wave crest object that resembles an actual wave crest is superimposed, compared to a simulation video generated by superimposing any one of the other CG videos I1, I3, and I4 on the live-action video J. The video superimposing unit 14 stores the generated simulation video in the data storage unit 3.

[0062] When training using the simulator SS is started, the video output unit 15 transmits the simulation video to the display device DS via the input / output I / F unit 4, and causes the display device DS to display the simulation video (step S110). The displayed simulation video is used for training using the simulator SS.

[0063] (Actions and Effects) The maneuvering simulation image generation device CS of this embodiment acquires live-action images, generates multiple CG images using different rendering conditions, calculates the difference between each of the CG images and the live-action image, and generates a simulation image by superimposing the CG image with the smallest total absolute value (total difference value) on the live-action image, and can present the generated simulation image. The maneuvering simulation image generation device CS is an example of a CG image superimposition device. As the difference from the live-action image, for example, the total value (total difference value) of the absolute values ​​of the differences in luminance values ​​between each CG image and the live-action image can be used. Note that the difference from the live-action image may also be the sum of squares of the differences in luminance values ​​between the CG image and the live-action image (residual sum of squares), or the like.

[0064] The live-action video is, for example, video captured by an instructor wearing an imaging device CM while piloting the windsurfing machine 300 and gliding across the water, capturing the instructor's 360-degree field of view. The windsurfing machine 300 is an example of a real mobile machine. The CG video is, for example, video including a wave crest object that simulates a wave crest. The wave crest object is an example of an object that simulates a feature in the live-action video that is difficult to see or a feature that is missing from the live-action video. The simulation video is, for example, a superimposed video in which one of the CG videos including a wave crest object is superimposed on the live-action video.

[0065] With the above configuration, the maneuvering simulation image generating device CS according to this embodiment can generate a simulation image in which wave crest objects simulating wave crests are superimposed on live-action image data, and display the generated simulation image on a display device DS, such as a VRHMD, worn on the user's head. By using CG images, a sufficient number of highly visible wave crest objects can be generated. Therefore, even if a sufficient number of wave crests are not included in the live-action image data or if the resolution of the VRHMD is low and the visibility of the wave crests in the live-action image is insufficient, an image in which a sufficient number of highly visible wave crest objects are superimposed on the live-action image data can be displayed on the display device DS. In other words, a simulation image that appropriately reflects features that are difficult to see or missing from the live-action image data can be generated and displayed on the display device DS. This allows a user of the simulator SS wearing the display device DS to easily determine the moving speed by visually observing the movement of the wave crest objects.

[0066] Furthermore, in this embodiment, a simulation image is generated using the CG image with the smallest total difference value from among multiple CG images generated under different rendering conditions. The total difference value becomes smaller as the wave crest object included in the CG image is closer to the wave crest in the live-action image. Therefore, it is possible to select a CG image that includes a wave crest object that is closer to the actual wave crest in the live-action image, thereby reducing the sense of discomfort felt by the user when the wave crest object is superimposed on the live-action image.

[0067] Furthermore, in this embodiment, a masking process is performed on both the live-action video and the CG video, which sets the brightness value of the area (first area) in which the actual windsurfing equipment is displayed in the live-action video to zero, and then the total difference value is calculated. By performing the masking process on the live-action video, the influence of the actual windsurfing equipment in the live-action video can be ignored, and the difference between the wave crest object in the CG video and the wave crest in the live-action video can be accurately calculated. For example, if the masking process is not performed on the live-action video, in a scene in which the windsurfing sail moves backward in the live-action video, a CG video having a wave crest object moving at a speed close to that of the sail may be selected. According to this embodiment, it is possible to select a CG video including a wave crest object that is close to the actual wave crest in the live-action video without being affected by the sail's moving speed. Furthermore, by performing a similar masking process on the CG video, it is possible to prevent a wave crest object generated in front of the windsurfing sail or board from being displayed in the superimposed video.

[0068] The above configuration may also be applied to mobile machines other than windsurfing. For example, by superimposing a CG image including an object for determining the moving speed of the flying drone on a live image taken by a flying drone for photography, the moving speed of the flying drone can be easily determined.

[0069] The technical idea of ​​the present application can be executed by a computer based on a program that causes the computer to execute instructions shown in the processing procedures shown in the above-described embodiments and modifications. The technical idea of ​​the present application can also be realized as a program that can be provided over a network. The technical idea of ​​the present application can also be realized as a recording medium on which the program is recorded.

[0070] This invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0071] DESCRIPTION OF SYMBOLS 1...Control unit 2...Program memory unit 3...Data memory unit 4...Input / output I / F unit 11...Actual image acquisition unit 12...CG image generation unit 13...Brightness difference calculation unit 14...Image superposition unit 15...Image output unit 100...Surfing unit 300...Actual aircraft 101, 301...Board 102, 302...Mast 103, 303...Sail 200...Drive unit 201...Actuator J...Actual image 401...Actual aircraft 402...Sea surface 403...Sky 404...Wave crest M...Mask image 501...First area 502...Second area S...Simulation image 605...Wave crest object

Claims

1. A CG image superimposition device comprising: a live-action image acquisition unit that acquires live-action image; a CG image generation unit that generates a plurality of CG images using different rendering conditions; a brightness difference calculation unit that calculates the difference between each of the CG images and the live-action image; an image superimposition unit that generates a superimposed image by superimposing the CG image that has the smallest difference from the live-action image on the live-action image; and an image presentation unit that presents the superimposed image.

2. The CG image superimposition device according to claim 1, wherein the CG image includes an object that simulates an object that is difficult to see or is missing in the live-action image.

3. A CG image superimposition device as described in claim 2, wherein the live-action image is an image capturing the field of view of an operator when operating an actual mobile machine, the CG image is displayed on the live-action image and includes an object simulating an object used to determine the speed of the mobile machine, and the superimposed image is used for training in operating the mobile machine.

4. The CG image superimposition device described in claim 3, wherein the brightness difference calculation unit performs mask processing on each of the live-action image and the CG image, and calculates, for each of the CG images on which the mask processing has been performed, the absolute value of the difference in brightness value between the live-action image on which the mask processing has been performed and the CG image on which the mask processing has been performed, the mask processing on the live-action image is a process of setting to 0 the brightness value of an area in the live-action image in which the moving machine is shown, and the mask processing on the CG image is a process of setting to 0 the brightness value of a position in the CG image corresponding to the area.

Images