Information processing device and information processing method

Abstract

To keep accurately tracking a position and a posture of an object using a captured image.SOLUTION: An image generation device tracks a state of a head-mounted display by Visual SLAM, on the basis of a captured image of surrounding space of a user wearing the head-mounted display. When a play area is set (a), a key frame used for collation with a current frame is determined (S50), and a portion thereof is classified as undiscardable (S52). When an application is executed (b), a stored key frame data is read and used for tracking (S60). Key frames are added along with a movement of the user and data of discardable key frames are discarded (S64).SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method for identifying the state of an object based on a captured image.

Background Art

[0002] Image display systems that allow a user wearing a head-mounted display to view a target space from a free viewpoint have become widespread. For example, there are known electronic contents that realize virtual reality (VR: Virtual Reality) by using a virtual three-dimensional space as a display target and displaying an image corresponding to the user's line-of-sight direction on the head-mounted display. In addition, a walk-through system has been developed in which a user wearing a head-mounted display can physically move to virtually walk around in the space displayed as an image.

[0003] Techniques for tracking the position and orientation of an object that allows free movement and performing information processing corresponding thereto are required not only in head-mounted displays but also in various fields such as autonomous mobile robots and automobiles (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For users wearing head-mounted displays who cannot see their surroundings, it is desirable to accurately determine the positional relationship between the user and surrounding objects and to be able to provide appropriate warnings and movement restrictions in order to enjoy content without the risk of collision with surrounding objects. Such risk avoidance and guidance in the correct direction of movement are necessary in various fields, as mentioned above. In the case of head-mounted displays, it is also necessary to accurately and continuously grasp the position and posture of the user's head in order to enhance immersion and realism in the displayed world, especially in VR technology that changes the field of view according to the user's movements.

[0006] This invention has been made in view of these problems, and its purpose is to provide a technology that can accurately track the position and orientation of an object using captured images. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the present invention relates to an information processing device. This information processing device is characterized by comprising: a state information acquisition unit that extracts feature points from the latest frame of a moving image being captured, and acquires state information of the position and orientation of a device equipped with a camera that is capturing a moving image based on the relationship between the corresponding feature points in a past frame to be compared and the points on the subject that they represent; a registration information generation unit that designates a frame that satisfies predetermined conditions from the latest frame as a key frame to be used as a reference for past frames, classifies and registers it into one of a plurality of groups for which different discard rules are set, and discards one of the registered key frame data according to the discard rule; and a registration information storage unit that stores the key frame data together with classification information.

[0008] Another aspect of the present invention relates to an information processing method. This information processing method is characterized by including the steps of: extracting feature points from the latest frame of a moving image being captured, and obtaining position and orientation information of a device equipped with a camera capturing a moving image based on the relationship between the corresponding feature points in a past frame to be compared and the points on the subject they represent; designating frames that satisfy predetermined conditions from the latest frames as key frames to be used as a reference for past frames, classifying and registering them into one of a plurality of groups for which different discard rules are set; discarding any of the registered key frame data according to the discard rules; and storing the key frame data together with the classification information in a storage device.

[0009] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0010] According to the present invention, the position and orientation of an object can be accurately tracked using captured images. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the appearance of the head-mounted display in this embodiment. [Figure 2] This figure shows an example of the configuration of the image display system in this embodiment. [Figure 3] This figure illustrates an example of the image world that the image generation device displays on the head-mounted display in this embodiment. [Figure 4] This is a diagram illustrating the principles of Visual SLAM. [Figure 5] This figure shows the internal circuit configuration of the image generation device in this embodiment. [Figure 6]This figure shows the internal circuit configuration of the head-mounted display in this embodiment. [Figure 7] This block diagram shows the functional blocks of the image generation device in this embodiment. [Figure 8] This figure shows an example of a frame image captured by a stereo camera in this embodiment, and keyframe data obtained therefrom. [Figure 9] This figure illustrates the data structure of the registration information stored in the registration information storage unit in this embodiment. [Figure 10] This figure shows an example of spatial division for evaluating the spatial coverage of keyframes in this embodiment. [Figure 11] This flowchart shows the processing procedure by which the image generation device sets the play area in this embodiment. [Figure 12] This is an overview view showing an example of the state change of the head-mounted display during the processing period from S14 to S18 in Figure 11. [Figure 13] This flowchart shows the processing procedure by which the image generation device executes the application in this embodiment. [Figure 14] This diagram illustrates the progression of keyframes in this embodiment. [Modes for carrying out the invention]

[0012] This embodiment relates to a technique for determining the position and orientation of a mobile device equipped with a camera by analyzing images captured by the camera. To that extent, the type of mobile device is not particularly limited, but a head-mounted display will be described as a typical example. In the following description, at least one of the position and orientation of a head-mounted display may be collectively referred to as the "state" of the head-mounted display.

[0013] FIG. 1 shows an external appearance example of a head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that wraps around the head when worn by the user to achieve fixation of the device.

[0014] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and a display panel is provided inside so as to face the eyes when worn. In this example, the display panel of the head-mounted display 100 is assumed to be non-transmissive. That is, the head-mounted display 100 is a light non-transmissive type head-mounted display.

[0015] Inside the housing 108, an eyepiece lens may be provided that is located between the display panel and the user's eyes when the head-mounted display 100 is worn and expands the user's viewing angle. The head-mounted display 100 may further include speakers or earphones at positions corresponding to the user's ears when worn. Also, the head-mounted display 100 incorporates a motion sensor and detects the translational and rotational movements of the head of the user wearing the head-mounted display 100, and thus the position and posture at each moment.

[0016] In addition, the head-mounted display 100 includes a stereo camera 110 on the front surface of the housing 108. The stereo camera 110 captures a moving image of the surrounding real space in a field of view corresponding to the user's line of sight. If the captured image is immediately displayed, so-called video see-through can be realized, where the state of the real space in the direction the user is facing can be seen as it is. Furthermore, if a virtual object is drawn on the image of the real object shown in the captured image, augmented reality (AR) can be realized.

[0017] Figure 2 shows an example configuration of the image display system according to this embodiment. The image display system 10 comprises a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 via wireless communication. The image generation device 200 may also be connected to a server via a network. In that case, the server may provide the image generation device 200 with data for online applications such as games that multiple users can participate in via the network.

[0018] The image generation device 200 is an information processing device that, based on the position and orientation of the head-mounted display 100, identifies the position and direction of the user's viewpoint and gaze, generates a display image to match the user's field of view, and outputs it to the head-mounted display 100. For example, the image generation device 200 may generate a display image of the virtual world, which is the setting of an electronic game, while the game is progressing, or it may display moving images for viewing or information provision, regardless of whether they are virtual or real worlds. Furthermore, by displaying a panoramic image with a wide field of view centered on the user's viewpoint on the head-mounted display 100, the user can be given a deep sense of immersion in the displayed world. The image generation device 200 may be a home game console or a PC.

[0019] The controller 140 is held in the user's hand and is a controller (e.g., a game controller) that receives user input for controlling image generation in the image generation device 200 and image display in the head-mounted display 100. The controller 140 is connected to the image generation device 200 by wireless communication. As a variation, either the head-mounted display 100 or the controller 140, or both, may be connected to the image generation device 200 by wired communication via a signal cable or the like.

[0020] Figure 3 illustrates an example of an image world displayed on a head-mounted display 100 by an image generation device 200. In this example, a state is created in which user 12 is in a virtual room. As shown in the figure, objects such as walls, floors, windows, tables, and objects on the tables are placed in the world coordinate system that defines the virtual space. The image generation device 200 defines a view screen 14 in the world coordinate system according to the position and direction of the user 12's viewpoint, and renders the display image by representing the images of the objects on it.

[0021] The image generation device 200 acquires the state of the head-mounted display 100 at a predetermined rate and changes the position and orientation of the view screen 14 accordingly. This allows the image to be displayed on the head-mounted display 100 within the field of view corresponding to the user's gaze. Furthermore, the image generation device 200 can generate stereo images with parallax and display these stereo images in the left and right areas of the display panel of the head-mounted display 100, allowing the user 12 to experience the virtual space in 3D. This allows the user 12 to experience virtual reality as if they were actually in a room within the displayed world.

[0022] In this embodiment, the image generation device 200 sequentially acquires the state of the head-mounted display 100, and consequently the position and posture of the user's head wearing it, based on images captured by at least the stereo camera 110. The acquired information can be used to configure the view screen 14, as well as to configure the play area, which defines the range of the real world in which the user can move. The play area is the range of the real world in which the user, who is viewing the virtual world through the head-mounted display 100, can move around, and is, for example, a range in which safe movement is guaranteed without colliding with surrounding objects.

[0023] Visual SLAM (Simultaneous Localization and Mapping) is a known technique for simultaneously estimating the self-position of a mobile object equipped with a camera and creating an environmental map using captured images. Figure 4 is a diagram illustrating the principle of Visual SLAM. Camera 22 is mounted on the mobile object and captures video of the real space 26 within its field of view while changing its position and orientation. Here, suppose feature points 28a and 28b representing points 24 on the same subject are extracted from frame 20a captured at a certain time and frame 20b captured at time Δt.

[0024] The shift in the positional coordinates of the corresponding feature points 28a and 28b (hereinafter sometimes referred to as "corresponding points") in each frame plane depends on the change in the position and orientation of the camera 22 at time Δt. Specifically, if R and T are matrices representing the changes due to the rotational and translational motions of the camera 22, respectively, and P1 and P2 are the three-dimensional vectors from the camera 22 to point 24 at each time step, then the following relationship holds. P1 = R·P2 + T

[0025] By utilizing this relationship and extracting multiple corresponding points in two frames that are sequentially preceding or succeeding in time, and solving a system of equations, the changes in the position and orientation of camera 22 during that time can be identified. Furthermore, by minimizing the error in the derivation result through recursive calculation, the 3D information of the subject surface in real space 26, such as point 24, can be accurately constructed. If camera 22 is a stereo camera 110, the 3D position coordinates of point 24, etc., can be determined independently at each time point, making calculations such as extracting corresponding points easier.

[0026] However, even if camera 22 is a monocular camera, the Visual SLAM algorithm is well-established. Therefore, for the purpose of tracking the state of the head-mounted display 100, the camera equipped on the head-mounted display 100 is not limited to a stereo camera 110. Furthermore, numerous Visual SLAM algorithms have been proposed, and any of them can be adopted. In any case, according to the principle illustrated, the state change of camera 22 from the previous time can be derived at the same rate as the frame rate of the video, but this information is relative.

[0027] Therefore, by repeatedly performing the same process and adding the derived state changes to the state obtained at the previous time, errors accumulate, and it is conceivable that tracking accuracy will deteriorate over time, or eventually tracking will fail. To address this, feature points from past frames, whose state has already been identified, are saved in association with the state of the camera 22. These feature points are then used to compare with the feature points of the current frame (latest frame), thereby eliminating the accumulation of errors at a predetermined timing. In other words, by using past state information as a reference and regularly setting a timing to acquire the state of the camera 22 as a change from that point, errors that occurred in between are canceled out.

[0028] This helps to suppress the increase in errors even if tracking is prolonged. The frame used as a reference for state changes in this way is called a "key frame". When the state of camera 22 is obtained using a key frame as a reference, the frame at that time (the current frame) itself can also function as a key frame. In order to obtain the required number of correspondence points with the key frame by comparing it with the current frame, it is desirable to secure a key frame obtained in a state as close as possible to the state of camera 22 that is capturing the current frame.

[0029] In other words, in order to accurately track the state of camera 22, the wider the range of movement allowed for the mobile body equipped with camera 22, the more keyframes need to be saved. On the other hand, if keyframes are saved frequently in accordance with the movement of the mobile body during operation, the memory will eventually be depleted. For this reason, it is conceivable to discard data starting with the oldest keyframes, but in this case, an imbalance may occur in the state of camera 22 for which keyframes have been saved. For example, if the position and orientation of camera 22 do not change significantly for a long period of time, only keyframes representing that position and orientation will remain.

[0030] If camera 22 suddenly changes direction or moves in a direction different from its previous tendency, there may not be enough keyframes with sufficient corresponding points, potentially leading to a matching failure. As an initial process, it is conceivable to acquire a variety of keyframes in advance to completely cover the range of movement, and then track the state of camera 22 without changing the keyframes in subsequent processing. However, especially in the case of the head-mounted display 100, the user needs to move until keyframes that satisfy the conditions are obtained, which is a considerable amount of work.

[0031] In this embodiment, the basic principle is to acquire keyframes along with state information when setting up the play area or when running the application. The acquired keyframes are then classified into multiple groups with different discard rules. In the simplest form, they are divided into keyframes that cannot be discarded and keyframes that can be discarded. For example, as the former, one keyframe is selected for each of the multiple states that cover the directions that camera 22 can face.

[0032] This allows us to select a keyframe that closely matches the current state, regardless of how the orientation of camera 22 changes. As long as we secure at least a minimum number of such keyframes, we can accurately acquire the state of camera 22 even if the rest are biased towards keyframes obtained in a state close to the current state. In other words, such keyframes can be discarded in chronological order. The classification once determined can be fixed or updated as needed. Specific examples will be given later.

[0033] Figure 5 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0034] The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and also receives data input from the controller 140. The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory.

[0035] The CPU 222 controls the entire image generation device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., VR game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing.

[0036] Figure 6 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 120, main memory 122, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, and a stereo camera 110.

[0037] The CPU 120 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the image generation device 200 to the display unit 124 and the audio output unit 126. The main memory 122 stores the programs and data necessary for processing by the CPU 120.

[0038] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, and which expand the user's field of view.

[0039] The audio output unit 126 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, allowing the user to hear audio. The communication unit 132 is an interface for sending and receiving data with the image generation device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®. The motion sensor 134 includes a gyro sensor and an accelerometer, and acquires the angular velocity and acceleration of the head-mounted display 100.

[0040] As shown in Figure 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints, with a field of view corresponding to the user's viewpoint. The frames of the moving images captured by the stereo camera 110 show objects that are in the direction of the user's line of sight (typically in front of the user). The values ​​measured by the motion sensor 134 and the data of the images captured by the stereo camera 110 are transmitted to the image generation device 200 via the communication unit 132 as needed.

[0041] Figure 7 is a block diagram showing the functional blocks of the image generation device. As described above, the image generation device 200 performs general information processing such as managing the progress of electronic games and communicating with a server. Figure 7 mainly shows the function of acquiring state information of the head-mounted display 100 and the functions realized using that state information. At least some of the functions of the image generation device 200 shown in Figure 7 may be implemented on a server connected to the image generation device 200 via a network. In addition, the function of acquiring state information of the head-mounted display 100 from captured images may be implemented separately as an information processing device, or it may be implemented on the head-mounted display 100 itself.

[0042] Furthermore, the multiple functional blocks shown in Figure 7 can be implemented in hardware using the configuration shown in Figure 5, such as the CPU 222, GPU 224, main memory 226, and storage unit 234, and in software using a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these.

[0043] The image generation device 200 comprises a data processing unit 250 and a data storage unit 252. The data processing unit 250 performs various data processing operations. The data processing unit 250 transmits and receives data to and from the head-mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in Figure 5. The data storage unit 252 is implemented by the storage unit 234 shown in Figure 5 and stores data that is referenced or updated by the data processing unit 250. In other words, the data storage unit 252 functions as a non-volatile memory device.

[0044] The data storage unit 252 includes an App storage unit 254, a play area storage unit 256, and a registration information storage unit 258. The App storage unit 254 stores data such as programs and object models necessary for running applications that involve image display, such as VR games. The play area storage unit 256 stores data related to the play area. The data related to the play area includes data indicating the positions of the point cloud that constitutes the boundary of the play area (for example, the coordinate values ​​of each point in the world coordinate system).

[0045] The registration information storage unit 258 stores registration data for acquiring the position and orientation of the head-mounted display 100, and by extension, the head of the user wearing the head-mounted display 100. Specifically, the registration information storage unit 258 stores the keyframe data mentioned above in association with data of an environmental map (hereinafter referred to as the "map") that represents the structure of the object surface in three-dimensional real space.

[0046] Map data, for example, is information about the three-dimensional position coordinates of a group of points on the surface of objects that make up the room in which the user plays the VR game. Each point is associated with a feature point extracted from a keyframe. The keyframe data is associated with the state of the stereo camera 110 when it was acquired.

[0047] The data processing unit 250 comprises a system unit 260, an application execution unit 290, and a display control unit 292. The functions of these multiple functional blocks may be implemented in a computer program. The CPU 222 and GPU 224 of the image generation device 200 may perform the functions of the multiple functional blocks by reading the computer program from the storage unit 234 or recording medium into the main memory 226 and executing it.

[0048] The system unit 260 performs system processing related to the head-mounted display 100. The system unit 260 provides common services to multiple applications for the head-mounted display 100 (e.g., VR games). The system unit 260 includes an image acquisition unit 262, an image analysis unit 272, and a play area control unit 264.

[0049] The image acquisition unit 262 sequentially acquires frame data of images captured by the stereo camera 110, which is transmitted from the head-mounted display 100. The frequency at which the image acquisition unit 262 acquires data may be the same as the frame rate of the stereo camera 110, or it may be lower.

[0050] The image analysis unit 272 sequentially acquires state information of the head-mounted display 100 using the Visual SLAM method described above. The state information of the head-mounted display 100 is used to set and reset the play area prior to the execution of applications such as VR games. The state information is also used to set the view screen during application execution and to warn users when they approach the play area boundary. Therefore, the image analysis unit 272 appropriately supplies the acquired state information to the play area control unit 264, the App execution unit 290, and the display control unit 292 according to the situation at that time.

[0051] In detail, the image analysis unit 272 comprises a registration information generation unit 274 and a state information acquisition unit 276. The registration information generation unit 274 generates registration information consisting of keyframe data and map data based on the frame data of the captured image acquired by the captured image acquisition unit 262. Finally, the registration information generation unit 274 stores the generated registration information in the registration information storage unit 258 so that it can be read out at any time later.

[0052] The state information acquisition unit 276 acquires information on the state of the head-mounted display 100, i.e., its position and orientation, at each time point based on the data of each frame of the captured image acquired by the captured image acquisition unit 262 and the registered information. The registered information may be information being generated by the registered information generation unit 274, or information read from the registered information storage unit 258 by the registered information generation unit 274. The state information acquisition unit 276 may also integrate the measured values ​​from the motion sensor 134 built into the head-mounted display 100 to obtain the state information.

[0053] The registration information generation unit 274 may extract frames that meet predetermined criteria from among the frames used by the state information acquisition unit 276 to acquire the state of the head-mounted display 100, and save them as new keyframes. In this case, the registration information generation unit 274 classifies the keyframes into one of several groups for which different discard rules are set, based on predetermined criteria.

[0054] Furthermore, the registration information generation unit 274 discards saved keyframe data selected according to the discard rules set for each group, as needed. For example, when the data size of keyframes reaches its limit and new keyframes cannot be saved, the registration information generation unit 274 discards keyframes in order from the first registered keyframe in a group where a FIFO (First-In First-Out) discard rule is set.

[0055] The registration information generation unit 274 may also, once a predetermined number of keyframes belonging to a group that cannot be discarded have been saved, determine the keyframes that constitute that group and classify all subsequent keyframes into a group that can be discarded. In any case, by creating differences in the rules for discarding keyframes, the registration information storage unit 258 is controlled to ensure that the keyframes necessary for obtaining status information are secured while preventing insufficient capacity.

[0056] The play area control unit 264 sets a real-world area where the user can move safely as the play area, and during the application execution phase, it issues warnings as appropriate when the user approaches the boundary of that area. At each of these stages, the play area control unit 264 sequentially acquires state information of the head-mounted display 100 from the image analysis unit 272 and processes it based on that information.

[0057] For example, when setting up a play area, the play area control unit 264 instructs the user wearing the head-mounted display 100 to look around. As a result, real objects such as furniture, walls, and floors around the user are captured by the stereo camera 110. The image analysis unit 272 sequentially acquires frames of the captured images and generates registration information based on them. The play area control unit 264 refers to the map data in the registration information and automatically determines the area of ​​the floor surface that will not collide with furniture, walls, etc., as the play area.

[0058] The play area control unit 264 may accept user editing operations for the play area by displaying an image representing the boundaries of the play area, once determined, on the head-mounted display 100. In this case, the play area control unit 264 acquires the content of the user operation via the controller 140 and changes the shape of the play area accordingly. The play area control unit 264 finally stores the data related to the determined play area in the play area storage unit 256.

[0059] Furthermore, the registration information generation unit 274 of the image analysis unit 272 may designate keyframes saved during the play area setup, or keyframes selected from said keyframes according to predetermined criteria, as keyframes that cannot be discarded. As mentioned above, during the play area setup, the camera is intentionally designed to capture the area around the user, making it easier to obtain keyframes that are not biased towards any particular state.

[0060] The App execution unit 290 reads and executes application data, such as a VR game selected by the user, from the App storage unit 254. During this process, the App execution unit 290 sequentially acquires state information of the head-mounted display 100 from the image analysis unit 272, sets the view screen at the corresponding position and orientation, and renders the display image. This allows the world of the displayed object to be represented with a field of view that corresponds to the user's head movements.

[0061] The display control unit 292 sequentially transmits data from various images generated by the App execution unit 290, such as VR images and AR image frames, to the head-mounted display 100. The display control unit 292 also transmits to the head-mounted display 100, as needed, images instructing the user to look around when setting the play area, images showing the appearance of the tentatively selected play area and accepting edits, and images warning of approaching the boundary of the play area.

[0062] Figure 8 shows an example of a frame image captured by the stereo camera 110 and the keyframe data obtained therefrom. In reality, a pair of images captured by the stereo camera 110 is obtained for each frame, but only one of them is schematically shown in the figure. The state information acquisition unit 276 extracts multiple feature points contained in the frame image 40 using known methods such as corner detection and methods based on brightness gradients. As described above, by comparing these with the frame image from the previous time and obtaining corresponding points, the state information of the stereo camera 110 at the time the frame image 40 was captured is identified.

[0063] On the other hand, the registration information generation unit 274 selects a keyframe if it extracts a predetermined number of feature points, for example, 24 or more, from a single frame image 40. This is because frames with many feature points make it easier to detect corresponding points by comparing them with other frames. However, the selection criteria for keyframes are not limited to the number of feature points; they may also include frame images where the state of the stereo camera 110 at the time of capture differs by a threshold or more, or a combination of multiple criteria may be used.

[0064] The registration information generation unit 274 generates keyframe 42 data, which consists of the position coordinates of a feature point (e.g., feature point 44) extracted from the selected frame image 40 on the image plane, and an image of a predetermined area surrounding it. The registration information generation unit 274 further associates this with the state information of the stereo camera 110 at the time of capture, and uses this as the final keyframe data. However, this does not mean that the components of the keyframe data are limited to these.

[0065] Figure 9 illustrates the data structure of the registration information stored in the registration information storage unit 258. The registration information has a structure in which multiple keyframe data 72a, 72b, 72c, 72d, ... are associated with one map data 70. In the figure, the map data 70 is denoted as "MAP1", and the keyframe data 72a, 72b, 72c, 72d, ... are denoted as "KF1", "KF2", "KF3", "KF4", ..., each having a structure as illustrated below.

[0066] Map data 70 consists of a "Map ID," which is unique identification information assigned to the map, and a list of "Point Information" that makes up the map. Point Information is data that associates a "Point ID," which is unique identification information assigned to each point, with its position coordinates in three-dimensional space (indicated as "3D Position Coordinates" in the diagram).

[0067] Furthermore, each of the keyframe data 72a, 72b, 72c, 72d, ... consists of a "keyframe ID," which is unique identification information assigned to the keyframe; "camera state information," which is the position and orientation of the stereo camera 110 when it was acquired; its "acquisition time"; and a list of "feature point information" that constitutes the keyframe. Here, the camera state information may be represented as a relative state obtained by Visual SLAM, for example, with the initial state as the reference, or it may be an absolute value obtained by integrating the measurement value from the motion sensor 134 built into the head-mounted display 100 with the results from Visual SLAM.

[0068] The feature point information corresponds to the data shown in keyframe 42 of Figure 8, and is data that associates the position coordinates of the extracted feature points in the image plane (labeled "2D position coordinates" in the figure) with the surrounding image of a predetermined size. Each feature point is further associated with a "point ID," which is the identification information of the point it represents in three-dimensional space.

[0069] The data structure of the registration information is not limited to that shown in the diagram, and may include supplementary information as appropriate to efficiently extract corresponding points when comparing with the current frame or to improve the accuracy of the map. The registration information storage unit 258 may also store registration information for multiple map data 70 in a similar structure. For example, if the user has multiple locations to play the game, the play area control unit 264 sets up a play area for each location, and the registration information generation unit 274 acquires and stores the registration information for each location individually.

[0070] In this case, location identification information is also associated with the registration information. The image analysis unit 272 acquires location information entered by the user at the start of gameplay, reads the corresponding registration information, and uses it to acquire state information of the head-mounted display 100. The registration information generation unit 274 may also generate registration information for each state when the furniture arrangement is changed even in the same location, or when the lighting environment differs depending on the time of day.

[0071] In any case, the registration information generation unit 274 stores keyframe data 72a, 72b, 72c, 72d, etc., classified according to predetermined criteria, such as "Group A," "Group B," etc., for each piece of registration information. The registration information generation unit 274 maintains in its internal memory the keyframe discard rules set for each group. For example, there may be two groups, one of which cannot be discarded and the other which is discarded using a FIFO (First-In, First-Out) method.

[0072] In this case, the registration information generation unit 274 first classifies all keyframes obtained in each of the multiple states that cover the directions the user may face into the former category, and thereafter classifies keyframes obtained in overlapping states into the latter category. Alternatively, the "quality" of the keyframes may be evaluated and divided into multiple stages to form groups. For example, the "quality" may be divided into three stages, and keyframes belonging to the lowest quality group may be discarded as a priority. The highest quality group may not be discarded, and the intermediate quality group may be discarded when certain conditions are met.

[0073] The "quality" of newly obtained keyframes may be compared with that of existing keyframes, and the group to which existing keyframes belong may be changed as appropriate. Here, high-quality keyframes are those captured with a difference of a certain value or more from others, keyframes with a large number of feature points, keyframes with a recent acquisition time, keyframes that have been used for matching many times, etc. Any of these may be used as a classification criterion, or two or more of them may be combined as the basis for classification.

[0074] For example, keyframes may be scored from these perspectives and classified according to the range of their total scores. This generally results in the discarding of keyframe data in order of lowest quality. If the number of times a keyframe has been used for matching is used as a classification metric, the registration information generation unit 274 further records the number of matching counts in the keyframe data structure shown in the diagram. The number of groups to classify keyframes, the classification criteria, and the discard rules applied to the groups can be varied and may be optimized according to the characteristics of the application and the scale of the system.

[0075] Figure 10 shows an example of spatial division for evaluating the spatial coverage of keyframes. Here, spatial coverage of keyframes refers to the extent to which the distribution of the state of the head-mounted display 100 at the time each keyframe was captured extends to all possible states of the head-mounted display 100.

[0076] The diagram shows three examples of division patterns, all of which are examples where the user's position in a reference state, such as when setting up the play area, is the center, and the area is equally divided in the yaw direction with gravity as the axis. Pattern 50 consists of division areas made up of four sections with a central angle of 90 degrees. Pattern 52 consists of division areas that are shifted in phase by 45 degrees relative to Pattern 50, based on the direction the user is facing. Pattern 54 consists of division areas made up of sixteen sections with a central angle of 22.5 degrees.

[0077] The registration information generation unit 274 uses at least one of the division areas of patterns 50, 52, and 54 as bins and evaluates the range covered by frames that satisfy the keyframe conditions. Specifically, the registration information generation unit 274 determines that a division area is covered by a keyframe by identifying which division area the direction of the stereo camera 110 was facing when the keyframe was captured. It then counts the total number of division areas covered by the keyframe.

[0078] For example, if all three division patterns shown in the diagram are used, a threshold of "10" is set for the total number of division areas covered by keyframes. If only pattern 54 is used, for example, a threshold of "8" is set. These values ​​are defined as values ​​that can be cleared if the user wearing the head-mounted display 100 looks around 180 degrees, but cannot be cleared if the user does not look around.

[0079] However, the division patterns and thresholds used can be set in various ways depending on the range of directions the user may face when the application is executed. For example, the central angle does not have to be divided equally. When the number of division areas covered by keyframes reaches the threshold, the registration information generation unit 274 determines that the keyframes have covered the space. The registration information generation unit 274 then classifies, for example, the keyframe that first covered a division area into a group that cannot be discarded. At the same time, the play area control unit 264 may make the final decision on the shape of the play area using the map that has been constructed at that point.

[0080] By selecting keyframes that cannot be discarded through this evaluation, the state of keyframes is distributed as much as possible, allowing a wide range to be covered with a small number of frames. The registration information generation unit 274 then classifies the obtained keyframes into groups that are permitted to be discarded. Alternatively, if the quality of newly obtained keyframes is higher for the same classification area, it replaces existing keyframes that cannot be discarded.

[0081] The division pattern is not limited to yaw division; it can also be divided in the pitch direction, which determines the elevation angle, or in the translational direction, such as front, back, left, or right of the user, or a combination of these to divide in multiple directions. However, evaluating comprehensiveness using yaw division, which provides the most variations of feature point information, makes it easier to avoid situations where there are no frames that can be matched to the current frame, even if the number of keyframes that cannot be discarded is small.

[0082] Next, the operation of the image display system with the above configuration will be explained. Figure 11 is a flowchart showing the processing procedure for setting the play area in the image generation device 200. The user can select to initialize or reset the play area in the system settings menu of the head-mounted display 100. If initial initialization or resetting of the play area is selected, the image acquisition unit 262 of the image generation device 200 establishes communication with the head-mounted display 100 and starts acquiring data of frames captured by the stereo camera 110 (S10).

[0083] Next, the play area control unit 264, via the display control unit 292, displays a message on the head-mounted display 100 prompting the user to change the state of the stereo camera 110 (S12). In practice, the play area control unit 264 only needs to display an instruction image on the head-mounted display 100 that instructs the user to look around while wearing the head-mounted display 100.

[0084] The image analysis unit 272 then performs Visual SLAM using the frame images captured in this manner to generate registration information of the user's location (S14). Specifically, the image analysis unit 272 obtains the correspondence of feature points extracted from each frame image and identifies the three-dimensional position coordinates of the points represented by those feature points. Using this position coordinate information, it also obtains the state changes of the stereo camera 110 and reprojects the points accordingly to correct any errors in the position coordinates. In these processes, the image analysis unit 272 selects keyframes from the frame images based on predetermined conditions.

[0085] In parallel with this, the play area control unit 264 constructs a play area in the space surrounding the user (S16). For example, the play area control unit 264 estimates the three-dimensional shape of the user's room using map data included in the registration information. In this case, the play area control unit 264 may detect a plane (typically the floor) perpendicular to the direction of gravity indicated by the motion sensor 134 based on the estimated three-dimensional shape of the room, and detect the result of combining multiple detected planes of the same height as the play area.

[0086] The processes in S14 and S16 are repeated until the image analysis unit 272 determines that it has obtained enough keyframes to cover the surrounding space, based on an evaluation of the keyframe coverage rate for the spatial divisions as illustrated in Figure 10 (N in S18, S14, S16). This ensures that the boundaries of the play area are defined without any gaps, and that sufficient map data and keyframe data are prepared for the play area.

[0087] If the keyframes determine that the surrounding space has been covered (Y in S18), the play area control unit 264 stores the play area data, including the coordinate values ​​of the point cloud that constitute the boundary of the play area, in the play area storage unit 256 (S20). Alternatively, as described above, the play area control unit 264 may give the user the opportunity to edit the play area and then store the edited data in the play area storage unit 256.

[0088] The image analysis unit 272 then associates the map data shown in Figure 9 with multiple keyframe data and stores it in the registration information storage unit 258 (S22). At this time, the image analysis unit 272 classifies the keyframes that first filled each sectioned area shown in Figure 10 into a group that cannot be discarded, and the other keyframes into a group that can be discarded, and stores them. However, as mentioned above, there are various possible modes of classification.

[0089] Figure 12 is an overview view showing an example of the state change of the head-mounted display 100 during the processing period S14 to S18 in Figure 11. Here, the position and direction of the head-mounted display 100 are represented by an isosceles triangle (for example, isosceles triangle 32). The base of the isosceles triangle corresponds to the imaging plane of the stereo camera 110. That is, the line of sight of the head-mounted display 100, and by extension the user, is in the direction of arrow 34, for example.

[0090] Following the instructions displayed in S12, the user wears the head-mounted display 100 on their head and moves around the room 30 while looking around. As a result, the state of the head-mounted display changes as shown in the figure. At this time, the play area control unit 264 may control the head-mounted display 100 to display the moving image in the direction of the user's gaze that the stereo camera 110 is capturing. The user can move safely by showing the user the real space in the direction they are facing.

[0091] The image analysis unit 272 of the image generation device 200 generates registration information based on the frames of the video. The play area control unit 264 uses this information to set the play area 36. The image analysis unit 272 may also have the head-mounted display 100 transmit measurement values ​​from the motion sensor 134, such as angular velocity and acceleration, along with the video frame data. By using this, the image analysis unit 272 can more accurately identify the state information of the stereo camera 110 at each time point.

[0092] Figure 13 is a flowchart showing the processing procedure by which the image generation device 200 executes an application. This flowchart is initiated, for example, when a user starts an application on the image generation device 200. As a prerequisite, it is assumed that at least the play area setup process shown in Figure 11 has been performed. First, the play area control unit 264 reads the play area data stored in the play area storage unit 256, for example, data indicating the shape and size of the play area (S32).

[0093] The image analysis unit 272 also reads the registration information stored in the registration information storage unit 258 (S34). The registration information includes keyframe data that has been acquired through the setting of the play area or the execution of the application that was performed earlier.

[0094] Next, the App execution unit 290 reads the application program data from the App storage unit 254 and starts processing (S36). For example, the App execution unit 290 starts a VR game that progresses according to the user's movements and user operations via the controller 140. Accordingly, the image analysis unit 272 performs Visual SLAM using frames of images captured by the stereo camera 110 that are sequentially transmitted from the head-mounted display 100, and acquires state information of the head-mounted display 100 (S38).

[0095] In this process, the image analysis unit 272 uses the keyframe read in S34 as a reference for matching the current frame at predetermined intervals, such as a set period. This allows for accurate acquisition of the state of the head-mounted display 100 even in the initial stages when sufficient frame data has not yet been obtained. In particular, by using the keyframe acquired when setting the play area as a reference, the user's position relative to the play area can be acquired with high accuracy.

[0096] Furthermore, when a newly acquired frame meets the requirements for a keyframe, the image analysis unit 272 generates keyframe data for that frame and stores it in its internal memory or elsewhere. At this time, if the total size of all keyframe data exceeds the capacity of the corresponding storage area of ​​the registration information storage unit 258, or if any other condition for data discarding is met, the image analysis unit 272 selects keyframes from the group permitted for discarding and discards them.

[0097] For example, the image analysis unit 272 determines which keyframes to discard from the group based on predetermined criteria, such as keyframes with older acquisition times, keyframes that have been used as a matching target infrequently, and keyframes obtained in a state far removed from the current state of the stereo camera 110. The image analysis unit 272 also determines the classification of new keyframes based on predetermined criteria. The registration of new keyframes may change the group to which existing keyframes belong.

[0098] In addition, in the processing of S38, the image analysis unit 272 may, as described above, integrate the measured values ​​from the motion sensor 134 built into the head-mounted display 100 with the results of Visual SLAM to derive the state of the head-mounted display 100. The state information of the head-mounted display 100 is supplied sequentially to the App execution unit 290. Based on the state information of the head-mounted display 100, the App execution unit 290 generates frame data of the displayed image in the field of view corresponding to the user's viewpoint and gaze, and transmits it sequentially to the head-mounted display 100 via the display control unit 292 (S40).

[0099] The head-mounted display 100 displays the transmitted frame data sequentially. The play area control unit 264 of the image generation device 200 executes a predetermined warning process for the user when the user's position in the real world meets the conditions for issuing a warning, for example, when the distance from the head-mounted display 100 worn by the user to the boundary of the play area falls below a predetermined threshold (e.g., 30 centimeters) (Y in S42) (S44).

[0100] For example, the play area control unit 264 supplies the display control unit 292 with an image representing the boundaries of the play area as a three-dimensional object such as a fence. The display control unit 292 may superimpose the image representing the boundaries of the play area onto the application image generated by the App execution unit 290 and display it on the head-mounted display 100 as a display image. The play area control unit 264 may also display a video see-through image on the head-mounted display 100 via the display control unit 292 when the user's position in the real world approaches the boundaries of the play area or crosses the boundaries of the play area.

[0101] If the user's position in the real world does not meet the conditions for issuing a warning (N in S42), the process in S44 is skipped. The processes in S38 to S44 are repeated until the predetermined termination conditions are met, such as when the user stops the execution of the application (N in S46). When the predetermined termination conditions are met (Y in S46), the image analysis unit 272 reflects the newly obtained keyframe data from S38 and the resulting updates to the entire registration information into the registration information storage unit 258 (S48), and terminates all processing.

[0102] Figure 14 illustrates the progression of keyframes in this embodiment. (a) shows keyframes acquired and used when the play area is set, and (b) shows keyframes acquired and used when the application is executed, each represented by an isosceles triangle that represents the state of the head-mounted display 100 at the time of acquisition. Similar to Figure 12, the base of the isosceles triangle corresponds to the imaging plane of the stereo camera 110. The illustrated example also assumes that keyframes are classified into two categories: "non-discardable" and "discardable," with the former represented by a black isosceles triangle (e.g., isosceles triangle 80) and the latter by a white isosceles triangle (e.g., isosceles triangle 82).

[0103] The transition in (a) is realized when the play area is not set or when it becomes necessary to set it up again, along with setting the play area. In the initial stage S50, the user looks around the surrounding space as indicated by the arrow, as instructed by the play area control unit 264. As a result, the stereo camera 110 of the head-mounted display 100 captures moving images while moving the gaze in an approximately radial pattern. The image analysis unit 272 generates registration information using Visual SLAM with the frames of these moving images.

[0104] In this process, the registration information generation unit 274 sequentially determines the frames that satisfy the requirements from among the frames from which feature points have been extracted as keyframes. In the diagram, stage S50 is the initial state, and all keyframes are set to be discardable, but in the subsequent stage S52, the keyframes are classified. For example, the registration information generation unit 274 selects one keyframe at a time from those acquired in the state corresponding to each yaw direction division area shown in Figure 10, classifies them into a group that cannot be discarded, and places the rest into a group that can be discarded.

[0105] Once the play area is set up, the registration information generation unit 274 stores all keyframe data, regardless of classification, in the registration information storage unit 258, associating it with the map data along with the classification information. In practice, for example, data for about 100 keyframes in total is stored, and about half of them are classified as non-discardable. However, this is not intended to limit the embodiments.

[0106] The transition in (b) is realized in conjunction with the application processing. In the initial stage, S60, the registration information generation unit 274 reads the registration information stored in the registration information storage unit 258. This results in a state where keyframes covering all directions are obtained, as enclosed by the dashed line 84, without the user having to look around. Furthermore, when the application is started, new captured images are transmitted from the head-mounted display 100, and keyframes are added as appropriate.

[0107] The state information acquisition unit 276 acquires the current state information of the head-mounted display 100 by matching it with a keyframe obtained in a state close to that of the newly transmitted frame (for example, frame 86). Even in the initial stages when the number of newly transmitted frames is small, the availability of a variety of keyframes allows for the acquisition of state information efficiently and with stable accuracy, regardless of the state of the head-mounted display 100.

[0108] The registration information generation unit 274 basically inherits the classification of keyframes made previously, such as when setting the play area, and determines the discard rules. The registration information generation unit 274 also classifies new keyframes acquired up to that point in S62, when the initial state of the head-mounted display 100 is acquired. At this time, the registration information generation unit 274 may increase the number of keyframes that cannot be discarded. The registration information generation unit 274 may also change the group configuration, such as making keyframes that were originally classified as non-discardable into a group that can be discarded.

[0109] As the application, such as a game, progresses and the user's position and posture change significantly, the number of keyframes in a state close to that change increases, as shown in S64. When the number of keyframes reaches the upper limit set considering the capacity of the registration information storage unit 258, the registration information generation unit 274 sequentially discards keyframes that have been deemed discardable up to that point. This allows the registration information storage unit 258 to prioritize storing keyframes that are likely to be matched, even if its storage capacity is limited.

[0110] Furthermore, by retaining keyframes selected according to predetermined criteria as non-discardable, even if the user's movement patterns suddenly change, highly accurate state tracking can be maintained without requiring time-consuming or failing matching. Once application processing is complete, the registration information generation unit 274 stores all keyframe data, regardless of classification, in the registration information storage unit 258, associating it with the map data along with the classification information. As a result, the same transition as in (b) is repeated in the next application processing.

[0111] According to the embodiment described above, in a technology that acquires the position and orientation of a moving object and an environmental map from frames of captured images using Visual SLAM, errors in state information are suppressed by selecting a reference keyframe and using it for comparison with the current frame. Here, keyframes are classified into multiple groups with different discard rules according to predetermined criteria. This makes it possible to control whether or not to discard keyframes, the time until discarding, and the priority of saving based on the quality and characteristics of the keyframes and their compatibility with the actual state of the moving object.

[0112] As a result, even with limited memory capacity, highly necessary keyframes can be saved in a readily accessible state, allowing for accurate acquisition of state information from the early stages when shooting time is limited. When applied to head-mounted displays that display images of a virtual world in response to user movements, such as VR games, keyframes are acquired when setting the play area. This allows for the constant presence of keyframes that cover the entire space, tailored to each user's environment, enabling highly accurate acquisition of state information, and consequently, the generation of display images and warnings of dangers associated with movement.

[0113] While tracking the state, new keyframe data is generated and registered, and existing keyframes are discarded as appropriate according to rules, thus maintaining an optimized keyframe structure at all times. Therefore, there is no need to prepare a large number of keyframes in advance to anticipate various situations, which reduces the effort required by the user during initial setup and also reduces the amount of memory needed.

[0114] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]

[0115] 10 Image display system, 100 Head-mounted display, 200 Image generation device, 222 CPU, 226 Main memory, 258 Registered information storage unit, 264 Play area control unit, 272 Image analysis unit, 274 Registered information generation unit, 276 Status information acquisition unit, 292 Display control unit.

Claims

1. A state information acquisition unit that extracts feature points from the latest frame of a video image showing the user's surroundings that is being captured by a camera equipped in a head mounted display, and acquires state information about the position and attitude of the head mounted display based on the relationship between the corresponding feature points in a past frame to be compared and the points on a subject that they represent; a registration information generating unit that defines a frame that satisfies a predetermined condition among the latest frames as a key frame to be used as a reference for the past frames, classifies the frame into one of a plurality of groups having different discard rules, registers the group, and discards any of the registered key frame data in accordance with the discard rule; a registration information storage unit that stores the key frame data together with the classification information; a play area control unit that sets a play area in which a user can move based on a three-dimensional environmental map that is constructed together with the state information and is made up of a set of the points; Equipped with The information processing device is characterized in that the registration information generation unit selects the key frames to be classified into a group that cannot be discarded from the frames of the video captured when the play area is set.

2. The information processing device according to claim 1, characterized in that the registration information generation unit selects the key frames to be classified into a group that cannot be discarded based on the distribution of the state of the head mounted display when each of the key frames was captured.

3. The information processing device according to claim 2, characterized in that the registration information generation unit classifies one of the key frames taken in each direction of a partitioned area formed by dividing space in a yaw direction centered on the head-mounted display and with the direction of gravity as its axis, into the group that cannot be discarded.

4. The information processing device according to any one of claims 1 to 3, characterized in that the registration information generation unit classifies the key frames based on at least one of the number of extracted feature points, the acquisition time, the number of times used for matching, and a difference from others in the state of the device when the image was captured.

5. The information processing device according to any one of claims 1 to 4, characterized in that when a new key frame causes the number of registered key frames to exceed the upper limit, the registration information generation unit discards the key frame that was registered first among the registered key frames belonging to one of the groups.

6. An application execution unit that generates a display image in a field of view corresponding to a line of sight of a user based on state information of the head mounted display, 6. The information processing device according to claim 1, wherein the state information acquisition unit acquires state information of the head mounted display using the key frame data stored in the registration information storage unit before the application execution unit starts processing.

7. A step of extracting feature points from the latest frame of a video image showing the space around the user being captured by a camera equipped in a head mounted display, and acquiring state information of the position and attitude of the head mounted display based on the relationship between the corresponding feature points in a past frame to be compared and the points on the subject that they represent; determining a frame that satisfies a predetermined condition among the latest frames as a key frame to be used as a reference for the past frames, and classifying and registering the frame in one of a plurality of groups having different discard rules; discarding any of the registered key frame data in accordance with the discarding rule; storing said key frame data together with said classification information in a storage device; A step of setting a play area in which a user can move based on a three-dimensional environmental map composed of the set of points constructed together with the state information; Including, The information processing method is characterized in that the registration step selects the key frames to be classified into a group that cannot be discarded from the frames of the video captured when the play area is set.

8. A function of extracting feature points from the latest frame of a video image showing the space around the user, which is being captured by a camera equipped in the head mounted display, and acquiring status information of the position and attitude of the head mounted display based on the relationship between the corresponding feature points in a past frame to be compared and the points on the subject that they represent; a function of defining a frame that satisfies a predetermined condition among the latest frames as a key frame to be used as a reference for the past frames, and classifying and registering the frame into one of a plurality of groups having different discard rules; A function of discarding any of the registered key frame data in accordance with the discarding rule; storing the key frame data together with the classification information in a storage device; A function of setting a play area in which a user can move based on a three-dimensional environmental map composed of a set of the points constructed together with the state information; This is realized on a computer. A computer program characterized in that the registration function selects the key frames to be classified into a group that cannot be discarded from the frames of the video captured when the play area is set.

Images