Video processing device, video processing method, and video processing program

Abstract

A user-worn video processing device according to one embodiment of the present invention comprises: an image acquisition unit that acquires a captured image captured by a camera, the captured image containing a moving body and an environment; a moving body detection unit that detects the moving body from the captured image; a line-of-sight estimation unit that estimates the line of sight of a user on the basis of the captured image; a line-of-sight movement detection unit that acquires sensor data from a sensor provided in the video processing device and detects a movement of the line of sight of the user with respect to the estimated line of sight as a start point on the basis of the sensor data; a determination unit that determines, on the basis of the detected movement of the line of sight of the user, whether the moving body is a moving body that the user is riding; a drawing unit that computes the appearance of an AR content on the basis of the estimated movement of the line of sight; and an output control unit that causes the computed AR content to be displayed.

Description

[Technical Field]

[0001] The present invention relates to a video processing device, a video processing method, and a video processing program. [Background technology]

[0002] Users of an Augmented Reality (AR) system can view the real world through a mobile terminal or AR device. In this case, content such as navigation information or 3D data (hereinafter referred to as AR content) is presented as additional information to the real world. In other words, users of the AR system can see the AR content superimposed on the real world, and can use the information in this content. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Explains the mechanisms and principles of AR (Augmented Reality) technology in an easy-to-understand way, and is available online.<URL:https: / / tech-camp.in / note / technology / 18791 / #i-3> [Non-patent document 2] Object Detection, Internet<URL:https: / / ja.wikipedia.org / wiki / %E7%89%A9%E4%BD%93%E6%A4%9C%E5%87%BA> [Non-patent document 3] Badminton Competition x Ultra-Realistic Communication Technology Kirari!, NTT Technical Journal vol.33 No.10 pp.24-29, 2021 [Non-patent document 4] Vehicle speed detection from a single motion blurred image, Internet<URL:https: / / dl.acm.org / doi / abs / 10.1016 / j.imavis.2007.04.004> Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when a user using an AR system is traveling on a moving vehicle, the image displayed by the AR device is a mixture of the camera image showing the environment (the scenery in front of the bicycle) and the moving vehicle (part of the bicycle's body).

[0005] In a scene where a vehicle similar to the one the user is riding in is traveling side by side, the AR device recognizes not only the vehicle the user is riding in but also the vehicle the user is not riding in. As a result, the AR device displays the AR content in the correct position, but also in a location corresponding to the vehicle the user is not riding in, which is a problem.

[0006] This invention was made with the above-mentioned circumstances in mind, and its purpose is to provide a technology that, when a mobile body that the user is not riding in is recognized in addition to a mobile body that the user is riding in, can prevent AR content from being displayed in locations corresponding to the mobile body that the user is not riding in. [Means for solving the problem]

[0007] In order to solve the above problem, one aspect of the present invention is an image processing device worn by a user, comprising: an image acquisition unit that acquires a captured image including a moving object and an environment captured by a camera; a moving object detection unit that detects the moving object from the captured image; a gaze estimation unit that estimates the user's gaze based on the captured image; a gaze movement detection unit that acquires sensor data from a sensor provided in the image processing device and detects movement of the user's gaze starting from the estimated gaze based on the sensor data; a determination unit that determines whether the moving object is a moving object ridden by the user based on the detected movement of the user's gaze; a rendering unit that calculates how AR content will appear based on the estimated movement of the gaze; and an output control unit that controls the calculated AR content to be displayed. [Effects of the Invention]

[0008] According to one aspect of the present invention, even if a moving object with a similar appearance is captured in the captured image, the AR content can be displayed only in the location corresponding to the moving object the user is riding on, thereby making it possible to accurately present the AR content to the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of a video processing device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of the video processing device according to the embodiment in relation to the hardware configuration shown in FIG. [Figure 3] FIG. 3 is a flowchart showing an example of an operation performed by the video processing device to display AR content only at the correct position in a captured image. [Figure 4] FIG. 4 is a diagram showing an example of a captured image. [Figure 5] FIG. 5 is a diagram showing an example in which a moving object is detected in a captured image. [Figure 6]FIG. 6 is a diagram showing an example of the "feature point space" and the "AR content space" stored in the space storage unit. [Figure 7] FIG. 7 is a diagram showing an example of the "range of the line of sight origin with the moving body on which the user is riding as the target" and the "range of the line of sight direction with the moving body of the user as the target." [Figure 8] FIG. 8 is a diagram showing an example of how the calculated AR content appears. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Hereinafter, elements that are identical or similar to elements already described will be designated by the same or similar reference numerals, and duplicate descriptions will generally be omitted. For example, when there are multiple identical or similar elements, a common reference numeral may be used to describe each element without distinguishing between them, or a subnumber may be used in addition to the common reference numeral to describe each element with distinction between them.

[0011] [Embodiment] (composition) FIG. 1 is a block diagram showing an example of a hardware configuration of a video processing device 1 according to an embodiment. The image processing device 1 is a computer that analyzes input data and generates and outputs output data. The image processing device 1 may be, for example, an AR device including AR glasses, smart glasses, or other wearable devices. In other words, the image processing device 1 may be a device worn by a user.

[0012] As shown in Fig. 1, the video processing device 1 includes a control unit 10, a program storage unit 20, a data storage unit 30, a communication interface 40, and an input / output interface 50. The control unit 10, the program storage unit 20, the data storage unit 30, the communication interface 40, and the input / output interface 50 are communicatively connected to one another via a bus. Furthermore, the communication interface 40 may be communicatively connected to an external device via a network. Furthermore, the input / output interface 50 is communicatively connected to an input device 2, an output device 3, a camera 4, and an inertial sensor 5.

[0013] The control unit 10 controls the video processing device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 10 may be an integrated circuit capable of executing various programs.

[0014] The program storage unit 20 may use, as a storage medium, a combination of nonvolatile memory that can be written to and read from at any time, such as an EPROM (Erasable Programmable Read Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and a nonvolatile memory such as a ROM (Read Only Memory). The program storage unit 20 stores programs necessary for executing various processes. That is, the control unit 10 can realize various controls and operations by reading and executing the programs stored in the program storage unit 20.

[0015] The data storage unit 30 is a storage that uses a combination of nonvolatile memory such as a HDD or memory card, which can be written to and read from at any time, and volatile memory such as RAM (Random Access Memory), as a storage medium. The data storage unit 30 is used to store data acquired and generated in the process of the control unit 10 executing programs and performing various processes.

[0016] The communication interface 40 includes one or more wired or wireless communication modules. For example, the communication interface 40 includes a communication module for wired or wireless connection to an external device via a network. The communication interface 40 may also include a wireless communication module for wireless connection to an external device such as a Wi-Fi access point or base station. Furthermore, the communication interface 40 may also include a wireless communication module for wireless connection to an external device using short-range wireless technology. In other words, the communication interface 40 may be any general communication interface as long as it can communicate with an external device under the control of the control unit 10 and send and receive various information including past performance data.

[0017] The input / output interface 50 is connected to the input device 2, the output device 3, the camera 4, the inertial sensor 5, etc. The input / output interface 50 is an interface that enables transmission and reception of information between the input device 2, the output device 3, and the multiple cameras 4 and the inertial sensors 5. The input / output interface 50 may be integrated with the communication interface 40. For example, the video processing device 1 and at least one of the input device 2, the output device 3, the camera 4, and the inertial sensor 5 may be wirelessly connected using short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology.

[0018] The input device 2 may include, for example, a keyboard, a pointing device, or the like for the user to input various information including past performance data to the video processing device 1. The input device 2 may also include a reader for reading data to be stored in the program storage unit 20 or the data storage unit 30 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0019] The output device 3 includes a display that displays images captured by the camera 4 and AR content. The output device 3 may be integrated with the image processing device 1. For example, if the image processing device 1 is AR glasses or smart glasses, the output device 3 is part of the glasses.

[0020] The camera 4 is capable of capturing images of the environment, such as scenery, and may be a general camera 4 that can be attached to the video processing device 1. Here, the environment refers to a scenery that is generally captured. The camera 4 may be integrated with the video processing device 1. The camera 4 may output the captured image to the control unit 10 of the video processing device 1 via the input / output interface 50.

[0021] The inertial sensor 5 includes, for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc. For example, when the video processing device 1 is an AR device, the inertial sensor 5 senses the moving speed and head movement of a user wearing the AR device, and outputs sensor data corresponding to the sensing to the control unit 10.

[0022] FIG. 2 is a block diagram showing the software configuration of the video processing device 1 according to the embodiment in relation to the hardware configuration shown in FIG. The control unit 10 includes an image acquisition unit 101, a moving object detection unit 102, a feature point extraction unit 103, a gaze estimation unit 104, a gaze movement estimation unit 105, a sensor data control unit 106, an object determination unit 107, an AR content drawing unit 108, and an output control unit 109.

[0023] The image acquisition unit 101 acquires the captured image captured by the camera 4. The image acquisition unit 101 may store the captured image in the image storage unit 301.

[0024] The moving object detection unit 102 detects a moving object from a captured image. The moving object detection unit 102 detects a moving object MB captured in the captured image. The moving object may be any object, such as a motorized bicycle, an electric bicycle, a motorcycle, or a vehicle. Furthermore, the moving object may include only the moving object, or may also include a part of the body of a user riding the moving object, such as an arm. Furthermore, a detection method may use a general technique. Furthermore, the moving object detection unit 102 may delete from the captured image any part of the environment that is other than the detected moving object.

[0025] The feature point extraction unit 103 extracts feature points from a moving object in a captured image. For example, the feature point extraction unit 103 may extract, as feature points, points that are located near a feature point space stored in a space storage unit 302 (described later).

[0026] The gaze estimation unit 104 estimates the gaze position by matching the feature points extracted by the feature point extraction unit 103 with the feature point space stored in the space storage unit 302. The method for estimating the gaze position will be described in detail later.

[0027] The gaze movement estimation unit 105 estimates gaze movement. The gaze movement estimation unit 105 estimates gaze movement that follows the movement of the user's head by moving the three-dimensional movement measured by the sensor data control unit 106 (described later) to a starting point of the gaze position received from the gaze estimation unit 104. That is, the gaze movement estimation unit 105 estimates the movement of the user's gaze from the gaze position estimated by the gaze estimation unit 104 based on the sensor data.

[0028] The sensor data control unit 106 acquires sensor data from the inertial sensor 5. Then, the sensor data control unit 106 measures the user's head movement, body movement, etc. from the acquired sensor data. For example, the sensor data control unit 106 measures the user's three-dimensional movement (e.g., the user's head movement) based on the sensor data.

[0029] The object determination unit 107 determines whether the object is a moving body in which the user is riding. For example, the object determination unit 107 determines whether the moving body is a moving body in which the user is riding, based on the detected movement of the user's line of sight. Specifically, the object determination unit 107 determines that the moving body is a moving body in which the user is riding, if the origin of the line of sight of the detected user's line of sight is within a predetermined range, and the direction of the detected user's line of sight is within a range in which the moving body in which the user is riding is visible from the predetermined range. The predetermined range may be a cubic range that is higher than the sitting position of the moving body by the average sitting height of a human and takes into account the riding posture of the user.

[0030] The AR content rendering unit 108 calculates how the AR content will appear. The AR content rendering unit 108 sets the estimated gaze movement included in the gaze movement information received from the gaze estimation unit 104, i.e., the gaze of the destination, in the AR content space corresponding to the feature point space stored in the space storage unit 302, and calculates how the AR content will appear in the set space.

[0031] The output control unit 109 outputs the AR content information. The output control unit 109 controls the output device 3 to render the AR content. For example, the output control unit 109 controls the output device 3 to display the adjusted AR content on AR glasses or the like.

[0032] The data storage unit 30 includes an image storage unit 301 and a spatial storage unit 302 .

[0033] The image storage unit 301 may store the captured image acquired by the image acquisition unit 101. Here, the captured image stored in the image storage unit 301 may include information about the longitude and latitude in the real world where the captured image was captured, which information was acquired by the video processing device 1. Furthermore, the image storage unit 301 may automatically delete the captured image after a predetermined time has elapsed.

[0034] The space storage unit 302 stores a feature point space and an AR content space corresponding to the feature point space. The feature point space may be located at a predetermined position in the captured image, and may be set at each part of a moving object, for example. Then, an AR content space corresponding to the feature point space may be set.

[0035] (operation) First, a method for displaying AR content on a video processing device 1 (a mobile terminal or an AR device) used by a user in a typical AR system will be described.

[0036] The video processing device 1 extracts feature points from the captured image captured by the camera 4. Furthermore, the video processing device 1 estimates the user's line of sight (position and direction) by comparing data (hereinafter referred to as a feature point space) in which feature points are also extracted from previously captured images (for example, captured images from the previous frame or frames several frames earlier) with the extracted feature points. Here, the feature point space is a space constructed for a "surrounding space" set based on a predetermined usage scene. Therefore, the captured image is also a capture of the surrounding space. Therefore, it is assumed that the positional relationship between both the feature point space and the captured image is identified based on the "surrounding space."

[0037] The video processing device 1 tracks the movement of the user's head based on the inertial data received from the inertial sensor 5, and follows the above-mentioned user's line of sight in real time.

[0038] Furthermore, the video processing device 1 positions the line of sight of the user, which is being tracked in real time, in the AR content space created in advance, and calculates how the AR content will appear from there.

[0039] Then, the video processing device 1 causes the output device 3 to render the calculated appearance.

[0040] In this way, the AR system generates AR content and displays the AR content on the smartphone or AR glasses that are the image processing device 1. However, as described above, with this method, if multiple moving objects appear in the image captured by the camera 4, the image processing device 1 not only displays the AR content in the correct position, but may also display the AR content in a location corresponding to a moving object that the user is not riding on.

[0041] Therefore, the following describes the operation of the video processing device 1 for displaying AR content at the correct position even in a captured image that includes multiple moving objects.

[0042] FIG. 3 is a flowchart showing an example of the operation of the video processing device 1 to display AR content only at the correct position in the captured image. The control unit 10 of the video processing device 1 reads out and executes the program stored in the program storage unit 20, thereby realizing the operation of this flowchart.

[0043] This operation flow is started, for example, when the user inputs an instruction to display AR content, or when a predetermined condition is met and the control unit 10 outputs an instruction to display AR content. Alternatively, this operation flow may be started when the video processing device 1 is started and the camera 4 acquires a captured image. In this operation, the moving object is assumed to be a bicycle.

[0044] In step ST101, the image acquisition unit 101 acquires a captured image captured by the camera 4. The image acquisition unit 101 may store the captured image in the image storage unit 301. The captured image includes an environment and a moving object. Here, the environment may be a general landscape, as described above. Therefore, the environment refers to the part excluding the moving object.

[0045] FIG. 4 is a diagram showing an example of a captured image. In the example of Fig. 4, the image is captured while the user is riding a bicycle, which is a moving object. Therefore, the captured image includes the moving object and the environment. Here, the camera 4 is a camera 4 provided in the AR glasses, which are the image processing device 1, and the captured image is captured by this camera 4.

[0046] In addition, in the example of FIG. 4, for simplicity, the user's arms and the handlebars and wheels of the bicycle are shown as shaded areas.

[0047] In step ST102, the moving object detection unit 102 detects a moving object MB from the captured image. Here, the moving object detection unit 102 may detect the moving object MB by performing object detection using a general method. For example, the object detection method may be the object detection method disclosed in Non-Patent Document 2. Therefore, a detailed description of the object detection method will be omitted here. Furthermore, the moving object detection unit 102 leaves only those parts that have not been detected as moving object MBs and deletes the other parts. In other words, the moving object detection unit 102 deletes the environmental parts in the captured image.

[0048] FIG. 5 is a diagram showing an example when a moving object MB is detected in a photographed image. In the example of Figure 5, the bicycle and the user are detected as a moving object MB. That is, the part detected as a moving object MB includes the handlebars and wheels of the bicycle as well as the user's arms. Then, as shown in Figure 5, the environmental part is deleted.

[0049] In step ST103, the gaze estimation unit 104 estimates the gaze of the user. First, it is assumed that a "feature point space" in which each part of a bicycle, which is a moving object MB, is set as a feature point is stored in the space storage unit 302. It is also assumed that an "AR content space" in which a speedometer is placed as AR content at the center of the handlebar in the feature point space is stored in the space storage unit 302.

[0050] FIG. 6 is a diagram showing an example of the “feature point space” and the “AR content space” stored in the space storage unit 302. As shown in FIG.

[0051] In the example of Fig. 6, the "feature point space" is indicated by the reference symbol CP, and the AR content is indicated by the reference symbol ARC. Fig. 6 is just one example, and the space storage unit 302 may of course store multiple such diagrams.

[0052] The feature point extraction unit 103 extracts feature points from the captured image from which the environment portion has been deleted in step ST102. That is, the feature point extraction unit 103 extracts feature points within the moving object MB. Here, the feature point extraction method may be a general method. The feature point extraction unit 104 surveys the entire captured image and extracts specific features such as the boundary (edge) of an object or the corner (corner) of an object. Here, the object may be the boundary of an object, etc. For example, in the example of FIG. 6, it is possible to extract the boundary of the moving object MB as a feature point.

[0053] Then, the gaze estimation unit 104 estimates the position of the gaze by matching the feature points extracted by the feature point extraction unit 103 with the feature point space stored in the space storage unit 302. For example, the gaze estimation unit 104 may estimate the gaze using a vision-based AR technology or the like. The vision-based AR technology may be a general technology such as PTAM, SmartAR, or Microsoft Hololens, which is a markerless AR technology. Therefore, a detailed description of the AR technology will be omitted here. The gaze estimation unit 104 outputs the estimated gaze position to the gaze movement estimation unit 105.

[0054] In step ST104, the gaze movement estimation unit 105 estimates gaze movement. First, the sensor data control unit 106 acquires sensor data from the inertial sensor 5. Then, the sensor data control unit 106 measures the user's head movement, body movement, etc. from the acquired sensor data. Specifically, for example, the inertial sensor 5 is an inertial measurement unit (IMU), and the sensor data control unit 106 acquires sensor data such as acceleration, angular velocity, and geomagnetism from the inertial sensor 5. Then, the sensor data control unit 106 may measure the user's three-dimensional movement (e.g., the user's head movement) based on this data. Then, the sensor data control unit 106 outputs the measurement result to the gaze movement estimation unit 105.

[0055] The gaze movement estimation unit 105 estimates gaze movement that follows the movement of the user's head by moving the three-dimensional movement measured by the sensor data control unit 106 to a starting point of the gaze position received from the gaze estimation unit 104. That is, the gaze movement estimation unit 105 estimates the movement of the user's gaze from a starting point of the gaze position estimated by the gaze estimation unit 104, based on the sensor data. Then, the gaze movement estimation unit 105 outputs gaze movement information including the estimated gaze movement to the object determination unit 107.

[0056] In step ST105, the object determination unit 107 determines whether the object is a moving object MB on which the user is riding. For example, the object determination unit 107 determines whether the moving object MB is a moving object on which the user is riding, based on the detected movement of the user's line of sight. Specifically, the process is as follows.

[0057] First, the object determination unit 107 defines the range in which the head of the user riding on the moving body MB is located as the "range of the origin of the line of sight of the moving body MB that the user is riding as the object." Specifically, the "range of the origin of the line of sight of the moving body MB that the user is riding as the object" is defined as the range of a cube that is higher than the seating position of the moving body MB (for example, the saddle of a bicycle) by the average sitting height of a typical person and that takes into account the posture of the user while riding.

[0058] Furthermore, the range of the line of sight from which the user can see the moving body MB on which the user is riding is defined as the "range of the line of sight direction of the user with the moving body MB as the target." Specifically, the "range of the line of sight direction of the user with the moving body MB as the target" is defined by a cone of the range in which the moving body MB can be seen from the user's head.

[0059] The object determination unit 107 determines whether the starting point of the gaze movement estimated by the gaze movement estimation unit 105 in step ST105 (i.e., the starting point of the gaze after the movement) is within a predetermined range, i.e., the "range of the starting point of the gaze with the moving body the user is riding as the object," and whether the direction of the gaze (i.e., the direction of the gaze after the gaze movement) is within the "range of the direction of the gaze with the user's moving body as the object."

[0060] If it is determined that the object is included, it is determined that the object is a moving object MB on which the user is riding. In this case, the object determination unit 107 outputs the line of sight movement information to the AR content rendering unit 108. Then, the processing proceeds to step ST106. On the other hand, if it is determined that the object is not included, it is determined that the object is a moving object MB on which the user is not riding. Therefore, the processing ends. In other words, the AR content is not displayed at a location corresponding to the moving object MB on which it is determined that the user is not riding.

[0061] FIG. 7 is a diagram showing an example of the "range of the line of sight origin with the moving body MB on which the user is riding as the target" and the "range of the line of sight direction with the moving body MB of the user as the target."

[0062] In Figure 7, the "range of the origin of the line of sight with the moving body MB on which the user is riding as the target" is represented by the reference symbol Ori, and the "range of the direction of the user's line of sight with the moving body MB as the target" is represented by the reference symbol Di.

[0063] Furthermore, the line of sight in (a) of Figure 7 shows an example in which the origin of the line of sight is included in the "range of origin of the line of sight with the moving body MB on which the user is riding as the target" and the direction of the line of sight is included in the "range of direction of the line of sight with the moving body MB of the user as the target."

[0064] On the other hand, the line of sight in (b1) of Figure 7 shows an example in which the origin of the line of sight is included in the "range of origin of the line of sight with the moving body MB on which the user is riding as the target," but the direction of the line of sight is not included in the "range of direction of the line of sight with the moving body MB on which the user is riding as the target." Furthermore, the line of sight in (b2) of Figure 7 shows an example in which the origin of the line of sight is not included in the "range of origin of the line of sight with the moving body MB on which the user is riding as the target," but the direction of the line of sight is included in the "range of direction of the line of sight with the moving body MB on which the user is riding as the target."

[0065] The line of sight in (a) of FIG. 7 is determined to be the line of sight of the user, whereas (b1) and (b2) are determined not to be the line of sight of the user.

[0066] In step ST106, the AR content rendering unit 108 calculates how the AR content will appear. The AR content rendering unit 108 sets the estimated gaze movement included in the gaze movement information received from the gaze estimation unit 104, i.e., the gaze of the destination, in the AR content space corresponding to the feature point space stored in the space storage unit 302, and calculates how the AR content will appear in the set space.

[0067] FIG. 8 is a diagram showing an example of how the calculated AR content appears. As shown in FIG. 8, the AR content rendering unit 108 adjusts the appearance of the AR content based on the movement information of the line of sight movement, and outputs AR content information for rendering the adjusted AR content to the output control unit 109.

[0068] In step ST107, the output control unit 109 outputs the AR content information. The output control unit 109 controls the output device 3 to render the AR content. For example, the output control unit 109 controls the adjusted AR content to be displayed on AR glasses or the like.

[0069] (Effects of the embodiment) According to the embodiment, the video processing device 1 can display AR content only in a location corresponding to the moving body MB on which the user is riding, even if a moving body MB with a similar appearance is captured in the captured image, thereby enabling the video processing device 1 to accurately present the AR content to the user.

[0070] [Other embodiments] In the above embodiment, an example has been described in which a captured image taken by the camera 4 included in the video processing device 1 is used, but the captured image is not limited to being taken by the camera 4 included in the video processing device 1. For example, it may be an independent camera 4 connected to the video processing device 1. However, it is assumed that the camera 4 is installed in a location where it can capture an image that allows the user's line of sight to be estimated (for example, above the user's head, etc.).

[0071] The techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be distributed by transmitting the program via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that implements this device loads the program stored on the storage medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by controlling the operation of the software means. The term "storage medium" as used herein is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0072] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in combination as appropriate as possible, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements. [Explanation of symbols]

[0073] 1...Video processing device 2...Input device 3...Output device 4. Camera 5...Inertial sensor 10...Control unit 101...Image acquisition unit 102...Moving object detection unit 103...Feature point extraction unit 104... Gaze estimation section 105... Gaze movement estimation unit 106...Sensor data control unit 107...Object determination unit 108…AR content drawing section 109...Output control section 20...Program memory section 30...Data storage unit 301...Image storage unit 302…Spatial storage unit 40...Communication interface 50...Input / output interface MB: Mobile

Claims

1. A video processing device worn by a user, an image acquisition unit that acquires a captured image including a moving object and an environment captured by a camera; a moving object detection unit that detects the moving object from the captured image; a gaze estimation unit that estimates a gaze of the user based on the captured image; a gaze movement detection unit that acquires sensor data from a sensor included in the video processing device and detects a movement of the user's gaze from the estimated gaze point based on the sensor data; a determination unit that determines whether the moving body is a moving body in which the user is riding, based on the detected movement of the user's line of sight; a rendering unit that, when it is determined that the moving body is a moving body in which the user is riding, calculates how the AR content appears based on the estimated movement of the line of sight; an output control unit that controls the rendering of the calculated appearance of the AR content; A video processing device comprising:

2. The image processing device according to claim 1 , wherein the moving object detection unit deletes a portion of the environment excluding the detected moving object from the captured image.

3. The video processing device of claim 1, wherein the determination unit determines that the moving body is the moving body the user is riding in if the origin of the gaze of the detected user to which the gaze is moving is within a predetermined range and the direction of the detected user's gaze is within a range in which the moving body the user is riding in is visible from the predetermined range.

4. The image processing device according to claim 3 , wherein the predetermined range is a cubic range that is higher than the seating position of the moving body by an average sitting height of a human and that takes into consideration the riding posture of the user.

5. a feature point extraction unit that extracts feature points within the moving object; a storage unit that stores the feature point space; The image processing device according to claim 1 , further comprising: a gaze estimation unit that estimates a gaze based on the extracted feature points and the feature point space.

6. the storage unit further stores an AR content space corresponding to the feature point space; The image processing device according to claim 5 , wherein the rendering unit sets a destination of the line of sight in the AR content space, and calculates how the AR content will appear in the set space.

7. A video processing method executed by a processor of a video processing device worn by a user, comprising: Acquiring a captured image including a moving object and an environment captured by a camera; Detecting the moving object from the captured image; estimating a line of sight of the user based on the captured image; acquiring sensor data from a sensor included in the video processing device; Detecting a movement of the user's gaze starting from the estimated gaze point based on the sensor data; determining whether the moving object is a moving object in which the user is riding, based on the detected movement of the user's line of sight; When it is determined that the moving body is a moving body in which the user is riding, calculating how the AR content appears based on the estimated movement of the line of sight; Controlling the rendering of the calculated appearance of the AR content; A video processing method comprising:

8. 1. A video processing program comprising instructions to be executed by a processor of a video processing device worn by a user, the instructions comprising: Acquiring a captured image including a moving object and an environment captured by a camera; Detecting the moving object from the captured image; estimating a line of sight of the user based on the captured image; acquiring sensor data from a sensor included in the video processing device; Detecting a movement of the user's gaze starting from the estimated gaze point based on the sensor data; determining whether the moving object is a moving object in which the user is riding, based on the detected movement of the user's line of sight; When it is determined that the moving body is a moving body in which the user is riding, calculating how the AR content appears based on the estimated movement of the line of sight; Controlling the rendering of the calculated appearance of the AR content; A video processing program comprising:

Images