Mobile machine simulation device

Abstract

One embodiment of a mobile machine simulation device according to the present invention simulates a mobile machine operated by an operator. The mobile machine simulation device comprises a first display control unit, an extraction unit, a calculation unit, a highlighting unit, and a second display control unit. During steering training by a simulator of the mobile machine, the first display control unit causes a display unit to display video of a range including the viewing angle of the operator when the mobile machine is steered. The extraction unit extracts a feature point from the video. The calculation unit calculates the speed of the feature point. When the speed of the feature point substantially matches the speed of the mobile machine, the highlighting unit highlights a vicinity region including the feature point in the video. When the vicinity region is highlighted, the second display control unit causes a highlighted video in which the vicinity region is highlighted to be displayed instead of the video by the display unit.

Description

Mobile Machine Simulator

[0001] One aspect of the present invention relates to a mobile machine simulator.

[0002] Mobile machine simulators are known that can train users to operate mobile machines by simulating a mobile machine such as a human-operated vehicle or windsurfing. For safety reasons, it is important for these types of mobile machine simulators to display an image that allows users to perceive the speed of the mobile machine. Regarding this, for example, three display methods (i) to (iii) are known.

[0003] (i) The first method is to display an image including a speedometer in addition to the forward view seen from the driver's seat of a vehicle in a driving simulator (see, for example, Non-Patent Document 1).

[0004] (ii) The second method is to place four LED arrays radially around the periphery of a screen presented in the central visual field, and stimulate the peripheral visual field, which is superior in motion perception, with the movement of light, thereby making the subject experience the speed of the vehicle. Note that the increase or decrease in perceived speed corresponds to the increase or decrease in the speed of the light (see, for example, Non-Patent Document 2).

[0005] (iii) The third method is to display an image of a virtual guided vehicle with computer graphics (CG) superimposed on it in the field of view of a user wearing an HMD (head mounted display) in order to move the user's vehicle on an appropriate course and at an appropriate speed (see, for example, Non-Patent Document 3).

[0006] Keiichi Ogawa and three others, "Application of a Simple Driving Simulator to the Analysis of Driving Behavior During Traffic Light Changes," Civil Engineering Planning Research Papers, Vol. 26, No. 5, pp. 865-871, 2009, Internet <URL: https: / / www.jstage.jst.go.jp / article / journalip1984 / 26 / 0 / 26_865 / _pdf / -char / ja> Keisuke Nakajima and one other, "A System for Enhancing the Sensation of Speed ​​in Sports Videos Based on Dynamic Perceptual Characteristics of Peripheral Vision," Information Processing Society of Japan Research Report, Vol. 2013-HCI-152, No. 8, 2013, Internet: <https: / / ipsj.ixsq.nii.ac.jp / ej / ?action=repository_uri&item_id=90536&file_id=1&file_no=1> Ryota Kimura and two others, “Augmented Reality-Based Steering Assistance System for Welfare Vehicles,” Japan Society of Mechanical Engineers, No. 17-2 Proceedings of the 2017 JSME Conference on Robotics and Mechatronics, 2A1-J08, 2017, Internet: <https: / / www.jstage.jst.go.jp / article / jsmermd / 2017 / 0 / 2017_2A1-J08 / _pdf / -char / ja>

[0007] However, the above three display methods (i) to (iii) each have the following disadvantages: For example, display method (i) has the disadvantage that the user's line of sight is directed toward the speedometer, making it difficult to see the forward field of view.

[0008] Furthermore, display method (ii) has the disadvantage that it is difficult to sense the speed when operating a real vehicle without an LED array.Similarly, display method (iii) has the disadvantage that it is difficult to perceive the speed when operating a real vehicle without a virtual guided vehicle.

[0009] The present invention has been made in consideration of the above circumstances, and provides a technique that can display an image that allows a user to perceive the speed of a mobile machine without relying on the display of a speedometer.

[0010] In order to solve the above-mentioned problems, one aspect of a mobile machine simulator according to the present invention simulates a mobile machine operated by a driver. The mobile machine simulator includes a first display control unit, an extraction unit, a calculation unit, an emphasis unit, and a second display control unit. The first display control unit causes a display unit to display an image captured from a range including the driver's field of view when operating the mobile machine during operation training using the mobile machine simulator. The extraction unit extracts feature points from the image. The calculation unit calculates the speed of the feature points. When the speed of the feature points approximately matches the speed of the mobile machine, the emphasis unit emphasizes a nearby region including the feature points in the image. When the nearby region is emphasized, the second display control unit causes the display unit to display an emphasized image in which the nearby region is emphasized, instead of the image.

[0011] According to one aspect of the present invention, feature points are extracted from an image of a mobile machine being operated, a nearby area including feature points whose speed approximately matches the speed of the mobile machine is emphasized, and an image in which the nearby area is emphasized is displayed. This makes it possible to notify the speed of the mobile machine from the movement of the nearby area that approximately matches the speed of the mobile machine. Therefore, it is possible to display an image that allows the user to perceive the speed of the mobile machine without relying on the speedometer display.

[0012] That is, according to one aspect of the present invention, it is possible to display an image that allows the user to perceive the speed of the mobile machine without relying on the display of a speedometer.

[0013] FIG. 1 is a schematic diagram showing an example of the configuration of a simulation system including a mobile machine simulator according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a simulation system including a mobile machine simulator according to the first embodiment. FIG. 3 is a schematic diagram for explaining the position measurement device and the imaging device shown in FIG. 2. FIG. 4 is a block diagram showing an example of the configuration of the control unit shown in FIG. 2. FIG. 5 is a flowchart for explaining an example of operation of the first embodiment. FIG. 6 is a schematic diagram showing an example of a pre-processing image in the first embodiment. FIG. 7 is a schematic diagram showing an image to be processed among the pre-processing images in FIG. 6. FIG. 8 is a schematic diagram showing an image not to be processed among the pre-processing images in FIG. 6. FIG. 9 is a schematic diagram showing the result of feature point detection for the image in FIG. 7. FIG. 10 is a schematic diagram showing the result of optical flow detection for the image in FIG. 9. FIG. 11 is a schematic diagram showing the sea surface from the side in an example of operation of the first embodiment. FIG. 12 is a schematic diagram showing the sea surface from above in an example of operation of the first embodiment. FIG. 13 is a schematic diagram showing a mask image generated from the image in FIG. 10. Fig. 14 is a schematic diagram showing the results of Gaussian blur processing on the mask image of Fig. 13. Fig. 15 is a schematic diagram showing a contrast-enhanced image of the image of Fig. 7. Fig. 16 is a schematic diagram showing a composite image obtained by mask-combining the contrast-enhanced image of Fig. 15 with the image of Fig. 7. Fig. 17 is a schematic diagram showing a simulation image obtained by combining the composite image of Fig. 16 with the image of Fig. 8. Fig. 18 is a schematic diagram showing another example of a pre-processed image in the first embodiment. Fig. 19 is a flowchart for explaining an example of operation of the second embodiment. Fig. 20 is a graph for explaining an example of operation of the second embodiment. Fig. 21 is a graph showing the classification results for the graph of Fig. 20.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] 1 is a schematic diagram showing an example of the configuration of a simulation system equipped with a mobile machine simulator according to a first embodiment of the present invention. This simulation system is configured to reproduce the movement of windsurfing, and includes a simulator SS, a display device DS, and a mobile machine simulator CS.

[0016] The simulator SS includes a surfing section 100 on which a user undergoing piloting training boards, and a driving section 200 that supports the surfing section 100. The surfing section 100 includes a board 101 provided with a platform on which the user stands, a mast 102 erected on the board 101, and a sail 103 that is supported by the mast 102 and catches the wind. The user boards the board 101 and grasps the boom to conduct piloting training, such as operating the mast 102 and how to put their weight on it.

[0017] The driving unit 200 includes an actuator 201 for reproducing the four-axial movements of pitch, roll, yaw, and heave that occur in an actual windsurfing machine. The driving unit 200 swings the board 101 and mast 102 of the surfing club 100 in the directions of the four axes by operating the actuator 201 in response to a drive control signal output from a simulator control device (not shown).

[0018] The drive unit 200 is not limited to the four axes mentioned above, but may be capable of supporting five or more axes, including the inclination angle of the mast 102 relative to the board 101, or may be capable of supporting only two axes, pitch and yaw, or only three axes, pitch, roll, and yaw.

[0019] The display device DS is, for example, a VRHMD (Virtual Reality Head Mounted Display) and is used by being worn on the user's head. However, the display device DS is not limited to this, and may be a device that is not worn on the user's head, such as a projector or a liquid crystal display. The display device DS displays a simulation image output from the mobile machine simulator CS. The display device DS may also be equipped with a speaker that outputs audio data. The display device DS is an example of a display unit.

[0020] The mobile machine simulator CS is a device that simulates windsurfing as a mobile machine operated by a pilot. Note that the mobile machine is not limited to a windsurfing machine, and any machine that moves in response to the pilot's operation, such as a vehicle, can be used. Furthermore, the mobile machine simulator CS is, for example, composed of a personal computer, as shown in FIG. 2. Furthermore, the mobile machine simulator CS may be composed of a server computer installed on the Web or the cloud.

[0021] The mobile machine simulator CS includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and an input / output I / F unit 4 are connected to the control unit 1 via a bus 5. "I / F" is an abbreviation for "interface."

[0022] The simulator SS and the display device DS are connected to the input / output I / F unit 4 via signal cables such as USB (Universal Serial Bus) cables, etc. The input / output I / F unit 4 transmits and receives control signals and simulation video data between the simulator SS and the display device DS.

[0023] The input / output I / F unit 4 can also be connected to a position measurement device PS and an imaging device CM. As shown in FIG. 3 , the position measurement device PS is installed, for example, at the top of a mast 302 attached to a board 301 of a windsurfing machine 300. The mast 302 supports a sail 303 that catches the wind. The direction and speed of the machine 300 are controlled by a pilot on the board 301, who grips the boom to tilt the mast 302 or change the center of gravity on the board 301. The position measurement device PS measures the latitude and longitude representing the position of the machine 300 being operated by the pilot, for example, using a GPS (Global Positioning System) sensor, and outputs a collection of the measured latitude and longitude data as movement trajectory data to the mobile machine simulator CS. The position measurement device PS can be installed not only at the top of the mast 302 but also at any location where radio waves can be received.

[0024] The imaging device CM captures surrounding images using, for example, an omnidirectional (360-degree) camera attached to the head of the operator while the actual windsurfing machine 300 is gliding on the water, and outputs the captured surrounding image data to the mobile machine simulator CS. Note that the imaging device CM is not limited to capturing omnidirectional images, and may capture panoramic images at least within the range of the operator's field of view angle.

[0025] Note that a wireless interface adopting a low-power wireless data transmission standard such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) may be used as the input / output I / F unit 4, or a wireless interface compatible with a public mobile wireless communication network standard such as 4G or 5G may be used. The use of a wireless interface can eliminate the need for signal cables connecting the simulator SS, display device DS, position measurement device PS, and imaging device CM, thereby increasing the degree of freedom for the user.

[0026] The program storage unit 2 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written and read as needed with a nonvolatile memory such as a read-only memory (ROM) as a storage medium. It stores middleware such as an operating system (OS), as well as application programs necessary for executing various control operations according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The program may be installed in advance on the computer from a network or a non-transitory computer-readable storage medium, or may be pre-recorded on the computer. In either case, the program is executed by the processor to cause the computer to function as the mobile machine simulator CS.

[0027] The data storage unit 3 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium.

[0028] 4, the control unit 1 includes an acquisition unit 11, an extraction unit 12, a calculation unit 13, an emphasis unit 14, and a display control unit 15. The control unit 1 is not limited to the processes of the above units 11 to 15, but can also execute any process in response to a user operation.

[0029] Each of the above units 11 to 15 is realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the above units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0030] The acquisition unit 11 receives the movement trajectory data measured by the position measurement device PS via the input / output I / F unit 4 when the actual windsurfing machine 300 operated by the operator is gliding on the water, and stores the received movement trajectory data in the data storage unit 3.

[0031] Similarly, the acquisition unit 11 receives, via the input / output I / F unit 4, video of the surrounding scenery captured by the imaging device CM while the actual machine 300 is gliding on the water, and stores the received video in the data storage unit 3. The video stored in the data storage unit 3 is an example of video captured over a range including the operator's field of view when operating the mobile machine. This video is displayed on the display device DS by the display control unit 15.

[0032] The extraction unit 12 extracts feature points from the displayed image. For example, the image may include wave crests on the water surface over which the mobile machine has moved. In this case, the extraction unit 12 may detect corners scattered on the wave crests from the image including the wave crests and extract the detected corners as feature points.

[0033] The calculation unit 13 calculates the velocity of the extracted feature point. The velocity of the feature point is, for example, the amount of movement of the feature point for each frame of the video.

[0034] When the speed of a feature point substantially matches the speed of the mobile machine, the highlighting unit 14 highlights a nearby area in the video that includes the feature point. For example, the highlighting unit 14 may determine that the speed of the feature point substantially matches the speed of the mobile machine when the amount of movement of the feature point for each frame and the distance traveled by the mobile machine for each frame are within a predetermined range of the distance.

[0035] The display control unit 15 causes the display device DS to display an image captured within a range including the field of view of the operator when operating the mobile machine during operation training using the mobile machine simulator SS. When the nearby area is highlighted, the display control unit 15 causes the display device DS to display an highlighted image in which the nearby area is highlighted, instead of the image. The display control unit 15 is an example of a first display control unit and a second display control unit.

[0036] Next, an example of the operation of the mobile machine simulator configured as above will be described with reference to the flowchart of Fig. 5 and the schematic diagrams of Fig. 6 to Fig. 18. The following description will be given taking the case of training windsurfing operation as an example.

[0037] First, when recording data from the position measurement device PS and the imaging device CM, an instructor, acting as the operator, boards the actual windsurfing machine 300 and actually drives it on the sea. At this time, the control unit 1 of the mobile machine simulator CS receives the movement trajectory data measured by the position measurement device PS via the input / output I / F unit 4 and stores the movement trajectory data in the data storage unit 3. Similarly, the control unit 1 receives an image captured by the imaging device CM via the input / output I / F unit 4 and stores the image in the data storage unit 3.

[0038] On the other hand, when training in operation using the mobile machine simulator CS, a user wearing the display device DS boards the surfing section 100 of the simulator SS and trains in operating the mast 102 and how to put their weight on it while watching video taken on the sea. The following operation example is for processing video taken when the actual machine 300 is traveling on the sea for training in the surfing section 100. The video is processed by the control section 1 of the mobile machine simulator CS in the following order:

[0039] First, the control unit 1 reads from the data storage unit 3 an image of a range including the instructor's field of view when operating the actual mobile machine 300, and displays the image on the display device DS.

[0040] (Step ST1) The control unit 1 extracts stationary objects from the displayed image by masking. Here, the objects included in the image can be classified into the following four types (a) to (d).

[0041] (a) The first type is an object that is stationary in world coordinates and located near the imaging device CM. It is also an object that can be considered to be stationary at a relatively slow speed compared to the speed of windsurfing, such as a wave crest on the ocean surface. When viewed from the imaging device CM, the object in (a) appears to be moving backward at the speed of windsurfing.

[0042] (b) The second type is an object that is stationary in world coordinates and located far from the imaging device CM, such as land or a building seen from offshore. The object in (b) appears to be almost stationary when viewed from the imaging device CM.

[0043] (c) The third type is an object that moves together with the imaging device CM, such as an instructor on board a windsurfing boat or a part of the actual machine 300. The object (c) appears to be almost stationary when viewed from the imaging device CM.

[0044] (d) The fourth type is an object that moves independently of the windsurfing being carried, such as another windsurfing object or an airplane.

[0045] As shown in Fig. 6, in order to extract the object (a) from the pre-processing image I of each frame of video, the control unit 1 extracts an area where the object (a) may exist by performing mask processing using a mask image. The control unit 1 also generates an image I1 to be processed by filling in the surrounding area other than the object in the pre-processing image I with black, as shown in Fig. 7. The control unit 1 also uses an inverted mask image to extract the surrounding area from the pre-processing image I, and then fills in the area other than the surrounding area with black, as shown in Fig. 8, to generate an image I2 not to be processed.

[0046] The mask image is created in advance based on positions where the above (b) objects (such as the pilot or the actual aircraft 300) are not reflected, and positions where the above (c) and (d) objects (such as land, buildings, and the sky) are not reflected.

[0047] Note that step ST1 is not essential. Step ST1 is a process for accurately performing step ST4 (described later) for calculating the optical flow and step ST7 (determining whether the optical flow substantially matches the speed of the actual device 300), and may be omitted.

[0048] (Step ST2) The control unit 1 extracts feature points from a processing target image I1 that indicates a part of the n-th frame of the pre-processing image I among the images captured when the actual mobile machine 300 is operated. For example, the control unit 1 converts the processing target image I1 into a grayscale image, and detects corners from the grayscale image using a Harris corner detector to obtain the feature points.

[0049] Here, the luminance of the image I1 to be processed is I(x, y), and the value obtained by differentiating the luminance in the x-axis direction is I ｘ (x, y), differentiated in the y-axis direction is I ｙ Assuming (x, y), the energy function M is expressed by the following equation (1).

[0050]

[0051] The score R indicating whether a pixel in the image I1 to be processed is a corner is given by the following equation (2).

[0052]

[0053] Here, det(M) and trace(M) are as shown in the following equations (3) and (4).

[0054]

[0055] In addition, in formula (2), k is a coefficient determined empirically, and k=0.04 to 0.06.

[0056] The control unit 1 calculates a score R for each pixel of the image I1 to be processed, and extracts points where R>threshold as feature points. As a result, a plurality of feature points fp are extracted from the image I1 to be processed, as shown in Fig. 9, for example. In Fig. 9, corners scattered on the wave crests are extracted as feature points from the image I1 that includes the wave crests of the water surface over which the mobile machine has moved.

[0057] (Step ST3) Similarly, the control unit 1 extracts feature points fp from the image I1 to be processed, which represents a part of the pre-processing image I of the (n+1)th frame.

[0058] (Step ST4) The control unit 1 calculates the optical flow between the nth and n+1th frame images I1. That is, the control unit 1 calculates the correspondence between the feature points fp of the nth frame image I1 and the feature points fp of the n+1th frame image I1. For example, the control unit 1 calculates the optical flow using the Lucas-Kanade method.

[0059] The brightness of the image I1 to be processed is I, and the brightness obtained by differentiating it in the x-axis direction is I ｘ , the luminance differentiated in the y-axis direction is I ｙ Then, the following equation (5) holds true.

[0060]

[0061] where Δx is the amount of movement of the feature point in the x direction, Δy is the amount of movement in the y direction, and Δt is the number of frames between images, which is 1 since they are adjacent frames.

[0062] Next, the feature point and its eight neighbors p １ , p ２ , ..., p ９ Focusing on this, it is expressed as the following equations (6) to (8).

[0063]

[0064] As a result, the amount of movement of each feature point from the processing target image I1 can be calculated as an optical flow op, as shown in Fig. 10. In Fig. 10, the circled end of each optical flow op is the end point, and the opposite end is the start point.

[0065] (Step ST5) The control unit 1 creates a mask image Mk that is initialized to black in its entirety. That is, the control unit 1 generates a mask image Mk in which all pixels are black in order to later combine images.

[0066] (Step ST6) The control unit 1 converts each optical flow op into world coordinates. For example, the coordinates of a feature point in the nth frame are (x1, y1), and the coordinates of a feature point in the n+1th frame are (x1 + Δx, y1 + Δy) = (x2, y2). The movement amount of the feature point between the frames (x2 - x1, y2 - y1) is the optical flow op on the image plane. Therefore, the control unit 1 converts each optical flow op on the image plane into the world coordinate system as shown below.

[0067] First, it is assumed that the height of the image capturing device CM at the time of capturing an image is known, and that the object (a) extracted in step ST1, which is a clue for perceiving the speed, is located above the sea surface.

[0068] Conversion to the world coordinate system will be explained below with reference to Figures 11 and 12. Specifically, an example will be described in which each image constituting the captured video has a resolution of 1920 x 1080 pixels and is an equirectangular projection in which the X axis corresponds to a pan angle of -180 degrees to 180 degrees and the Y axis corresponds to a tilt angle of -90 degrees to 90 degrees.

[0069] Since the center of the image is (x, y) = (960, 540), the point (x, y) on the image can be converted into a pan angle θ = (x - 960) ÷ 1920 × 360 and a tilt angle φ = (y - 540) ÷ 1080 × 180. This allows, for example, the coordinates (x, y) of the feature point fp on the image to be converted into the tilt angle φ.

[0070] The height Hc of the imaging device CM is also calculated. For example, if the imaging device CM is attached to the head of a windsurfing pilot, the height Hc is set to a fixed value such as 2 m based on the pilot's height. If the imaging device CM is installed on a flying object such as a drone, the altitude Hc of the flying object is calculated using a position measurement device PS such as a GPS.

[0071] Furthermore, the height Hg of the surface on which the feature point fp exists is calculated. For example, in the case of windsurfing, the feature point fp is assumed to exist on the sea surface, and Hg = 0 is set. In the case of a moving object traveling on the ground, the latitude and longitude are calculated using a position measurement device PS such as a GPS, and the height Hg of the ground is obtained using altitude information from a map. Note that either the height Hc or Hg may be calculated first.

[0072] Next, the relative height H = Hc - Hg is calculated from the difference between the height Hc of the image capture device CM and the height Hg of the plane on which the feature point fp is located. In the example of Fig. 11, the relative height H is 2 m. The horizontal distance L between the image capture device CM and the feature point fp is calculated as shown in equation (9) using the relative height H and the tilt angle φ.

[0073]

[0074] As shown in FIG. 12, the horizontal X distance x ｗ , horizontal Y distance y ｗ are calculated as shown in equations (10) and (11), respectively.

[0075]

[0076] The coordinates (x1, y1) of the feature point fp in the nth frame are converted into world coordinates and expressed as (x1 ｗ , y1 ｗ ), the coordinates (x2, y2) of the feature point fp in the n+1th frame are converted into world coordinates and expressed as (x2 ｗ , y2 ｗ ), the movement of the feature point fp between frames is (x2 ｗ -x1 ｗ , y2 ｗ -y1 ｗ ) In this way, the optical flow op is transformed into world coordinates.

[0077] (Step ST7) The control unit 1 determines whether the converted optical flow substantially matches the speed of the mobile machine.

[0078] For example, the speed of the actual mobile machine 300 is determined from the position measurement device PS, an airspeed indicator, etc. The determined speed of the actual machine 300 is then divided by the frame rate at which the video is captured to determine the distance traveled by the actual machine 300 per frame. For example, if the speed of the actual machine 300 is 40 km / h and the frame rate of the video is 30 fps, the distance d traveled by the actual machine 300 between frames is calculated as follows:

[0079]

[0080] The control unit 1 determines that an optical flow op in which the movement amount of the feature point fp is close to the distance d, for example, an optical flow op in the range of ±20%, substantially matches the speed of the actual mobile machine 300.

[0081]

[0082] The optical flow op that satisfies this determination formula is used in the process of step ST8.

[0083] (Step ST8) The control unit 1 changes the vicinity of the optical flow end point of the mask image Mk to white.

[0084] For example, a mask image Mk is initialized entirely to black, and the vicinity of the end point (x2, y2) of the optical flow op, which approximately matches the speed of the actual device 300, is colored white. For example, an area within a radius of 30 pixels centered on the end point (x2, y2) is filled with white. Gaussian blurring may also be applied to smooth the whitened boundary. FIG. 13 shows an example of a mask image M1 in which the vicinity of the end point of the optical flow is filled with white. FIG. 14 shows an example of a mask image M2 in which a Gaussian blurring with a diameter of 101 pixels is further applied to the mask image M1.

[0085] (Step ST9) The control unit 1 determines whether all optical flows op have been processed. If the result of this determination is no, the control unit 1 returns to step ST7 and repeats steps ST7 to ST9 for unprocessed optical flows. On the other hand, if the result of the determination in step ST9 is that all optical flows op have been processed, the control unit 1 proceeds to step ST10.

[0086] (Step ST10) The control unit 1 enhances the contrast of the image of the (n+1)th frame.

[0087] The image is represented by RGB brightness values ​​(I ｒ , I ｇ , I ｂ) matrix. The control unit 1 creates a contrast-enhanced image by converting the brightness value of the image of the (n+1)th frame so that the contrast is enhanced and the image stands out. For example, as shown in equation (12), the brightness value I ｒ I' ｒ Convert to.

[0088]

[0089] However, the luminance value I'r after conversion must be within the range of 0 to 255.

[0090] Furthermore, the green luminance value I ｇ , blue luminance value I ｂ is also converted into a contrast-enhanced image using the same equations (13) and (14).

[0091]

[0092] As a result, the unprocessed image I1 shown in FIG. 7 is converted into a contrast-enhanced image I' as shown in FIG.

[0093] (Step ST11) The control unit 1 combines the original image before processing and the contrast-enhanced image using a mask.

[0094] For example, the control unit 1 combines the unprocessed image I1 of the (n+1)th frame shown in FIG. 7 with the contrast-enhanced image I' shown in FIG. 15 using the mask image M1 shown in FIG. 13 as shown in equation (15). This results in a combined image I" as shown in FIG. 16. Note that when the mask image M2 is used, the combined image I" can be obtained using equation (16).

[0095]

[0096] The synthetic image I″ is an image in which the contrast near the end point of the optical flow op, which approximately coincides with the speed of the actual device 300, is enhanced.

[0097] The control unit 1 combines this composite image I'' with the image I2 that is not the processing target, to generate an enhanced image I3 of the (n+1)th frame, as shown in FIG.

[0098] Similarly, the control unit 1 repeatedly executes steps ST1 to ST11 while updating the frame number, and causes the display device DS to display the enhanced image I3 of each frame generated from the unprocessed image I of each frame as an enhanced video.

[0099] For example, suppose that an actual image of the surroundings captured while windsurfing is actually being performed is read from the data storage unit 3 and displayed on the display device DS, as shown in Fig. 6. In this case, when the speed corresponding to the operation of the simulator SS is approximately the same as the speed of the actual aircraft 300, the control unit 1 emphasizes the contrast of the wave crests moving backward at a speed corresponding to the operation. As a result, an emphasized image, for example, as shown in Fig. 16, is generated, and this emphasized image is displayed on the display device DS in place of the actual image.

[0100] Therefore, by viewing the highlighted image displayed on the display device DS, the user can have a simulated experience of windsurfing, and can also get a sense of speed from the backward movement of the highlighted wave crests.

[0101] That is, the display device DS displays the surrounding image flowing backward as the windsurfing progresses, along with the wave crests flowing backward at the same speed, with enhanced contrast, so that the user can clearly perceive the traveling speed from the surrounding image flowing backward as well as the wave crests flowing backward in the same manner.

[0102] As described above, in the first embodiment, the display control unit 15 causes the display device DS to display an image captured over a range including the field of view of the operator when operating a mobile machine during operation training using the mobile machine simulator SS. The extraction unit 12 extracts feature points from the displayed image. The calculation unit 13 calculates the speed of the extracted feature points. When the speed of the feature points approximately matches the speed of the mobile machine, the highlighting unit 14 highlights a nearby area in the image that includes the feature points. When the nearby area is highlighted, the display control unit 15 causes the display device DS to display an highlighted image in which the nearby area is highlighted, instead of the image.

[0103] This allows a user undergoing pilot training to be informed of the speed of the mobile machine from the movement of the nearby area that roughly matches the speed of the mobile machine, and therefore allows an image that allows the user to perceive the speed of the mobile machine without relying on the speedometer display to be displayed.

[0104] Additionally, the mobile machine simulator estimates the speed of objects captured in the video from the speed of feature points in the video and highlights objects that are deemed to be stationary. This allows the user to train in reading the speed of the mobile machine from its relative speed to stationary objects without taking their eyes off the road. Furthermore, since the objects displayed during piloting training using the simulator SS are also visible when piloting the actual machine 300 without using the simulator SS, the speed reading skills acquired with the simulator SS can be effectively utilized when piloting the actual machine 300.

[0105] Furthermore, according to the first embodiment, the velocity of the feature point may be the amount of movement of the feature point for each frame of the video. Based on the amount of movement and the distance traveled by the mobile machine for each frame, the highlighting unit 14 may determine that the two approximate matches are true if the amount of movement is within a predetermined range of the distance. In this case, in addition to the effects described above, by estimating the velocity of the feature point extracted from the video as the amount of movement of the feature point for each frame and using the distance traveled for each frame as the velocity of the mobile machine, the velocity of the feature point and the velocity of the mobile machine can be easily compared.

[0106] According to the first embodiment, the video may include wave crests on the water surface over which the mobile machine is moving. The extraction unit 12 may also detect corners scattered on the wave crests from the video including the wave crests and extract the detected corners as feature points. In this case, in addition to the effects described above, it is possible to display video that particularly allows the user to perceive the speed of the mobile machine moving over the water surface.

[0107] Additionally, there are not enough wave crests at sea to provide clues for determining the speed. In contrast, according to the first embodiment, a configuration for highlighting wave crests that move backward at a speed that approximately matches the speed of the mobile machine makes it possible to compensate for situations where there are only a few wave crests at sea. Furthermore, by generating an enhanced image in which the wave crests are highlighted from an unprocessed image in which the wave crests are not within the field of view and it is difficult to determine the speed, the speed of the mobile machine can be perceived.

[0108] Second Embodiment The second embodiment is a modification of step ST7 of the first embodiment, and uses clustering when determining whether the converted optical flow substantially matches the speed of the actual device 300.

[0109] Accordingly, it is assumed that the extraction unit 12 in the control unit 1 extracts a plurality of feature points fp from the video. Furthermore, the speed of the feature points fp calculated by the calculation unit 13 is the amount of movement of each of the plurality of feature points fp for each frame of the video.

[0110] The highlighting unit 14 classifies each of the movement amounts into two clusters, finds the center of gravity of each of the two clusters, and determines that the movement amounts classified into the cluster having the center of gravity farthest from the origin are approximately the same.

[0111] As described above, each of these units 12 to 14 is realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Similarly, some or all of the above units 12 to 14 may be realized using hardware such as an LSI or an ASIC.

[0112] Next, an example of the operation of the mobile machine simulator configured as above will be described with reference to the flowchart of FIG. 19 and the graphs of FIGS. 20 and 21.

[0113] Now, it is assumed that steps ST1 to ST6 have been executed in the same manner as described above.

[0114] (Step ST7A) After step ST6, step ST7A is executed, which includes steps ST7A-1 to ST7A-4.

[0115] The control unit 1 classifies each optical flow op into two clusters (step ST7A-1). For example, as shown in FIG. 20, the control unit 1 plots the optical flow (Δx, Δy) in the world coordinate system calculated in step ST6 on a graph. Note that (Δx, Δy)=(x2 ｗ -x1 ｗ , y2 ｗ -y1 ｗ ) In FIG. 20, the vertical axis represents Δy [m], and the horizontal axis represents Δx [m]. Thereafter, the control unit 1 classifies the movement amount of each of the multiple feature points fp for each frame of the video into one of two clusters c1 and c2, as shown in FIG. 21. FIG. 21 shows the classification results using the k-means method with the number of clusters set to 2, with cluster c1 represented by a black circle and cluster c2 represented by a black triangle.

[0116] After step ST7A-1, the control unit 1 calculates the center of gravity of the optical flow op for each of clusters c1 and c2 (step ST7A-2). In this example, the center of gravity of the optical flow included in cluster c1 is obtained as (x, y) = (0.002650579, -0.020086477). The center of gravity of the optical flow included in cluster c2 is obtained as (x, y) = (0.088532707, -0.387144047).

[0117] After step ST7A-2, the control unit 1 calculates the distance from the center of gravity of the optical flow to the origin (step ST7A-3). In this example, the distance from the center of gravity of the optical flow included in cluster c1 to the origin is 0.020260605. A short distance from the origin to the center of gravity means that the optical flow is small, and is therefore estimated to represent a feature point of an object that moves together with the image capture device CM. On the other hand, the distance from the center of gravity of the optical flow included in cluster c2 to the origin is 0.397137952. A long distance from the origin to the center of gravity means that the optical flow is large, and is therefore estimated to represent a feature point of an object that is stationary in world coordinates and appears to recede as the image capture device CM moves forward.

[0118] After step ST7A-3, the control unit 1 selects the cluster with the longer distance to the origin from the two clusters c1 and c2 (step ST7A-4). In this example, the control unit 1 selects the optical flow included in cluster c2, which has the longer distance from the origin to the center of gravity, as the optical flow that substantially matches the speed of the actual device 300. This completes step 7A, which includes steps ST7A-1 to ST7A-4.

[0119] (Step ST8) After step ST7A, the control unit 1 executes the above-mentioned step ST8 for the optical flows included in the selected cluster. That is, the control unit 1 makes the vicinity of the end points of the optical flows included in the selected cluster white in the mask image Mk.

[0120] (Step ST9) The control unit 1 determines whether all optical flows op included in the selected cluster have been processed. If the result of this determination is no, the control unit 1 returns to step ST8 and executes step ST8 for unprocessed optical flows. On the other hand, if the result of the determination in step ST9 is that all optical flows op have been processed, the control unit 1 proceeds to step ST10.

[0121] (Step ST10 and after) As described above, the control unit 1 then executes the processes from step ST10 onwards.

[0122] As described above, according to the second embodiment, the extraction unit 12 extracts multiple feature points fp from a video. The velocity of the feature points fp is the amount of movement of each of the multiple feature points fp for each frame of the video. The enhancement unit 14 classifies each of the amounts of movement into two clusters, calculates the center of gravity of each of the two clusters, and determines that the amounts of movement of the cluster with the center of gravity farthest from the origin substantially coincide with each other. Therefore, even with a configuration in which the amount of movement of the feature points is classified into two clusters, the same effect as in the first embodiment can be achieved.

[0123] Other Embodiments The functional configuration, processing procedures, and processing contents of the mobile machine simulator CS can be modified in various ways without departing from the spirit and scope of the present invention. For example, the image captured by the imaging device CM is not limited to images captured by a 360-degree camera, and any image within a range that includes the operator's field of view can be used.

[0124] For example, in the above embodiment, step ST1 is performed to extract an area from the pre-processed image I where the object (a) may exist, but as mentioned above, step ST1 may be omitted.

[0125] For example, a mobile machine operated by a driver is not limited to a windsurfing machine that moves on the surface of the sea without a human-powered or mechanical power source, but may also be a ship that moves on the surface of the sea with a human-powered or mechanical power source. Furthermore, the mobile machine is not limited to the surface of the sea, and any machine that moves on any water surface, such as a lake, pond, or river, can be used. Similarly, any mobile machine that can be operated by a driver, regardless of whether it has a power source, such as a vehicle or sled that moves on land or an airplane that moves in the air, can be used.

[0126] For example, the corner detection method is not limited to the Harris corner detector, and any other method can be used as appropriate.Similarly, the feature point extraction method is not limited to the corner detection method, and any other method can be used depending on the feature points contained in the captured live video.

[0127] Furthermore, for example, the optical flow detection method is not limited to the Lucas-Kanade method, and various other methods can be used.

[0128] Furthermore, for example, the clustering method is not limited to the k-means method, and any method can be used. Furthermore, the number of clusters is not limited to two, and clustering can be performed with three or more clusters.

[0129] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0130] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0131] SS...Simulator 100...Surfing part 101...Board 102...Mast 103...Sail 200...Drive part 201...Actuator 300...Actual machine CS...Mobile machine simulator DS...Display device PS...Position measurement device CM...Image capture device 1...Control part 2...Program memory part 3...Data memory part 4...Input / output I / F part 5...Bus 11...Acquisition part 12...Extraction part 13...Calculation part 14...Emphasis part 15...Display control part fp...Feature point op...Optical flow I...Unprocessed image I'...Contrast-enhanced image I"...Synthesized image c1, c2...Cluster

Claims

1. A mobile machine simulator that simulates a mobile machine operated by a driver, comprising: a first display control unit that causes a display unit to display an image captured within a range including the field of view of the driver when operating the mobile machine during operation training using a simulator of the mobile machine; an extraction unit that extracts feature points from the image; a calculation unit that calculates the speed of the feature points; an emphasis unit that, when the speed of the feature points approximately matches the speed of the mobile machine, emphasizes a nearby area in the image that includes the feature points; and a second display control unit that, when the nearby area is emphasized, causes the display unit to display an emphasized image in which the nearby area is emphasized, instead of the image.

2. The mobile machine simulation device of claim 1, wherein the velocity of the feature point is the amount of movement of the feature point for each frame of the video, and the highlighting unit determines that the approximate match occurs if the amount of movement is within a predetermined range of the distance based on the amount of movement and the distance the mobile machine moves for each frame.

3. The mobile machine simulator of claim 1, wherein the extraction unit extracts a plurality of feature points from the video, the velocity of the feature points being the amount of movement of each of the plurality of feature points for each frame of the video, and the emphasis unit classifies the amount of movement into two clusters, determines the center of gravity of each of the two clusters, and determines that the amount of movement classified into the cluster having the center of gravity farthest from the origin is approximately the same.

4. A mobile machine simulation device as described in any one of claims 1 to 3, wherein the image includes wave crests on the water surface over which the mobile machine has moved, and the extraction unit detects corners scattered on the wave crests from the image including the wave crests and extracts the detected corners as the feature points.

Images