Information processing device, control method, and program

Abstract

An information processing device 1X mainly has a feature point information acquisition means 42X, a field estimation information acquisition means 53Xa, and a label modification means 53Xb. The feature point information acquisition means 42X acquires feature point information that is determined on the basis of an image including at least a portion of a target field, and that indicates candidate positions and candidate labels for a plurality of feature points of the field. The field estimation information acquisition means 53Xa acquires field estimation information indicating results of estimating the positions and labels of the plurality of feature points in a first coordinate system, which is the coordinate system used as a reference by the display device having the camera that captures the image. The label modification means 53Xb modifies the candidate labels on the basis of the feature point information and the field estimation information.

Description

[Technical field]

[0001] The present invention relates to the technical field of an information processing device, a control method, and a storage medium that perform processing related to spatial perception in augmented reality (AR). [Background technology]

[0002] In a device that provides augmented reality, there is a technology that determines the display position of an image (so-called AR image) that is displayed superimposed on a scene viewed by a user based on an image captured by a camera. For example, Patent Document 1 describes a technology that estimates the positions of each feature point on a field and calibrates AR based on the estimation results in order to realize AR in watching sports. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2021 / 033314 Summary of the Invention [Problem to be solved by the invention]

[0004] When processing such as calibration is performed by extracting feature points of a target field as in Patent Document 1, the accuracy of the extraction of the feature points affects the accuracy of subsequent processing such as calibration.

[0005] In view of the above-mentioned problems, one of the main objects of the present disclosure is to provide an information processing device, a control method, and a storage medium that are capable of suitably correcting feature point information to be used. [Means for solving the problem]

[0006] One aspect of the information processing device is Has multiple structural features determined based on an image including at least a portion of the field; For each classification indicating which of the plurality of feature points the feature point belongs to,Candidate positions of the feature points; and This is a label that represents the classification. Candidate labels Group of A feature point information acquisition means for acquiring feature point information indicating a field estimation information acquisition means for acquiring field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera that captures the image; a label correction means for correcting the candidate labels based on the feature point information and the field estimation information; The information processing device has the following features.

[0007] Another aspect of the information processing device is Has multiple structural features determined based on an image including at least a portion of the field; For each classification indicating which of the plurality of feature points the feature point belongs to, Candidate positions of the feature points; and This is a label that represents the classification. Candidate labels Group of A feature point information acquisition means for acquiring feature point information indicating a field estimation information generating means for generating field estimation information indicating estimation results of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera that captures the image, based on at least some of the sets of the candidate positions and the candidate labels indicated by the feature point information, information on positional relationships of the plurality of feature points, and anisotropy information on anisotropy of displacement of the plurality of feature points over time; The information processing device has the following features.

[0008] One aspect of the control method includes: The computer Has multiple structural features determined based on an image including at least a portion of the field; For each classification indicating which of the plurality of feature points the feature point belongs to, Candidate positions of the feature points; and This is a label that represents the classification. Candidate labels Group of Feature point information indicating obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; revising the candidate labels based on the feature point information and the field estimation information; It is a control method.

[0009] One aspect of the program is Has multiple structural features determined based on an image including at least a portion of the field; For each classification indicating which of the plurality of feature points the feature point belongs to, Candidate positions of the feature points; and This is a label that represents the classification. Candidate labels Group of Feature point information indicating obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; The program causes a computer to execute a process of correcting the candidate label based on the feature point information and the field estimation information. Effect of the Invention

[0010] As an example of an effect of the present disclosure, it becomes possible to suitably correct feature point information to be used. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a display device according to a first embodiment. [Diagram 2] 1 shows an example of a data structure of structure data. [Diagram 3] 4 is a block diagram showing a functional configuration of a control unit. FIG. [Figure 4] (A) A first label definition example of a structural feature point when the target field is a tennis court, (B) A second label definition example of a structural feature point when the target field is a tennis court. [Diagram 5] 1A is a first example of a label definition for a structural feature point when the target field is a pool, and FIG. 1B is a second example of a label definition for a structural feature point when the target field is a pool. [Figure 6]4 is an example of a functional block of a field estimation unit. [Figure 7] 4 is an example of a functional block of an erroneous label detection unit. [Figure 8] This shows the positional relationship between the device coordinate system and the field coordinate system after adjustment so that the height axes of the device coordinate system and the field coordinate system are parallel. [Figure 9] 1 is a diagram showing the shooting position of a camera, a captured image, and a target field in a device coordinate system and a field coordinate system. [Figure 10] FIG. 13 is a diagram showing the results of clustering the six generated candidate fields in the device coordinate system. [Figure 11] 1 is a diagram clearly showing a float that is designated as a structural feature among ropes attached to a pool. [Figure 12] 13 is a diagram showing an estimated position of a target field in a device coordinate system obtained by an optimization process based on feature point candidate positions excluding erroneous feature point candidate positions. FIG. [Figure 13] 13 is a diagram showing an isotropic optimization solution and an anisotropic optimization solution of a target field on the xz plane of a device coordinate system. FIG. [Figure 14] 1A is a diagram showing feature point candidate positions together with candidate label numbers on a field estimation plane based on first field estimation information when the target field is a tennis court, and (B) is a diagram showing sets of feature point candidate positions and candidate label numbers after correcting the candidate labels on a field estimation plane based on the first field estimation information. [Figure 15] 10 is an example of a flowchart illustrating an overview of a virtual object display process executed by a control unit in the first embodiment. [Figure 16] 10 is a flowchart illustrating an example of a detailed procedure of a calibration process. [Figure 17] 13 is an example of a flowchart illustrating details of a feature point error detection process executed in step S22. [Figure 18]13 is an example of a flowchart illustrating details of a first field estimation process. [Figure 19] 13 is an example of a flowchart illustrating details of a label correction process. [Figure 20] 13 shows the configuration of a display system in a second embodiment. [Figure 21] FIG. 11 is a block diagram of a server device according to a second embodiment. [Figure 22] 13 is an example of a flowchart illustrating a processing procedure executed by a control unit of the server device in the second embodiment. [Figure 23] 13 shows a schematic configuration of an information processing device according to a third embodiment. [Figure 24] 13 is an example of a flowchart according to the third embodiment. [Diagram 25] 13 shows a schematic configuration of an information processing device according to a fourth embodiment. [Figure 26] 13 is an example of a flowchart according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of an information processing device, a control method, and a storage medium will be described with reference to the drawings.

[0013] First Embodiment (1) Outline configuration FIG. 1 is a schematic diagram of a display device 1 according to a first embodiment. The display device 1 is a device that can be worn by a user, and is, for example, a see-through type configured as glasses, and is configured to be wearable on the user's head. The display device 1 realizes augmented reality (AR) by displaying visual information superimposed on a real landscape when watching a sports game or a play (including a concert). The visual information is a virtual object represented in two or three dimensions, and will hereinafter be referred to as a "virtual object." The display device 1 may display a virtual object to only one eye of the user, or may display a virtual object to both eyes.

[0014] In this embodiment, it is assumed that there is a place or structure (hereinafter also referred to as a "target field") where sports, plays, etc. are performed, and the display device 1 superimposes and displays a virtual object, which is additional information that assists a user when watching a sports game or a play, on or around the target field. The target field corresponds to, for example, a field that is a target for watching a sports game (e.g., a tennis court, a swimming pool, a stadium, etc.), or a field that is a target for watching a play (e.g., a theater, a concert hall, a multipurpose hall, various stages, etc.). As will be described later, the target field has a plurality of structural (i.e., characteristic in shape) feature points (also referred to as "structural feature points"). The target field functions as a reference object for calibrating the display device 1.

[0015] Virtual objects include, for example, in the case of tennis, a score board displayed above the tennis court, in the case of competitive swimming, world record lines superimposed in real time on the pool during a swimming competition, and virtual performers superimposed on the stage during a theater performance.

[0016] The display device 1 includes a light source unit 10, an optical element 11, a communication unit 12, an input unit 13, a storage unit 14, a camera 15, a position and orientation detection sensor 16, and a control unit 17.

[0017] The light source unit 10 has a light source such as a laser light source or an LCD (Liquid Crystal Display) light source, and emits light based on a drive signal supplied from the control unit 17. The optical element 11 has a predetermined transmittance, transmits at least a part of external light to make it incident on the user's eyeball, and reflects at least a part of the light from the light source unit 10 toward the user's eyeball. As a result, a virtual image corresponding to a virtual object formed by the display device 1 is superimposed on the scenery and is visually recognized by the user. The optical element 11 may be a half mirror whose transmittance and reflectance are approximately equal, or may be a mirror whose transmittance and reflectance are not equal (so-called beam splitter).

[0018] The communication unit 12 transmits and receives data to and from external devices under the control of the control unit 17. For example, when a user uses the display device 1 to watch a sporting event or a play, the communication unit 12 receives information about virtual objects to be displayed by the display device 1 from a server device managed by a promoter under the control of the control unit 17.

[0019] The input unit 13 generates an input signal based on a user's operation and transmits it to the control unit 17. The input unit 13 is, for example, a button, a cross key, a voice input device, or the like that allows the user to give instructions to the display device 1.

[0020] The camera 15 generates an image of an area in front of the display device 1 under the control of the control unit 17, and supplies the generated image (also called a “captured image Im”) to the control unit 17.

[0021] The position and orientation detection sensor 16 is a sensor (sensor group) that detects the position and orientation (direction) of the display device 1, and includes, for example, a positioning sensor such as a Global Positioning Satellite (GPS) receiver, and an orientation detection sensor that detects a change in the relative orientation of the display device 1, such as a gyro sensor, an acceleration sensor, and an Inertial Measurement Unit (IMU). The position and orientation detection sensor 16 supplies the generated detection signal related to the position and orientation of the display device 1 to the control unit 17. As will be described later, the control unit 17 detects the amount of change in the position and orientation of the display device 1 from the start-up or the like based on the detection signal supplied from the position and orientation detection sensor 16. Note that, instead of detecting the position of the display device 1 from the positioning sensor, the control unit 17 may specify the position of the display device 1 based on, for example, a signal received from a beacon terminal or a wireless LAN device installed in the venue. In another example, the control unit 17 may specify the position of the display device 1 based on a known position estimation technique using an AR marker. In these cases, the position and orientation detection sensor 16 may not include a positioning sensor.

[0022] The control unit 17 includes a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a volatile memory that functions as a working memory for the processor, and performs overall control of the display device 1.

[0023] For example, the control unit 17 performs a calibration process for associating the real world space with the space recognized by the display device 1 based on the structural feature points of the target field recognized from the captured image Im at the timing of displaying the virtual object. In this calibration process, the control unit 17 generates coordinate conversion information for converting a coordinate system of a three-dimensional space based on the display device 1 (also called a "device coordinate system") into a coordinate system of a three-dimensional space based on the target field (also called a "field coordinate system"). The device coordinate system is an example of a "first coordinate system", and the field coordinate system is an example of a "second coordinate system". The calibration process will be described in detail later. Then, the control unit 17 generates a drive signal for driving the light source unit 10 based on the above-mentioned coordinate conversion information, and supplies the drive signal to the light source unit 10 to cause the optical element 11 to emit light for displaying a virtual object on the light source unit 10 (also called a "display light"). As a result, the control unit 17 allows the user to visually recognize the virtual object.

[0024] The storage unit 14 is a non-volatile memory that stores various information required for the control unit 17 to control the display device 1. The storage unit 14 may include a removable storage medium such as a flash memory. The storage unit 14 also stores programs executed by the control unit 17.

[0025] The storage unit 14 also includes a sensor data storage unit 20, a parameter storage unit 21, and a structure data storage unit 22.

[0026] The sensor data storage unit 20 stores the captured image Im generated by the camera 15 in association with the amount of change in the position and orientation of the display device 1 from the time when the device coordinate system was set (for example, when the display device 1 was started) when the captured image Im was generated. In this case, for example, the control unit 17 constantly calculates the amount of change in the current position and orientation based on the position and orientation when the device coordinate system was set, based on the detection signal of the position and orientation detection sensor 16. Then, when storing the captured image Im generated by the camera 15 in the sensor data storage unit 20, the control unit 17 stores the amount of change in position and orientation Ap calculated when the captured image Im was generated in association with the captured image Im in the sensor data storage unit 20. For example, the control unit 17 stores a combination of the captured image Im and the amount of change in position and orientation Ap for the latest predetermined time or number of times in the sensor data storage unit 20. The information stored in the sensor data storage unit 20 is used in the calibration process.

[0027] The parameter storage unit 21 stores parameters of an inference device (also called a "feature extractor") used when extracting position information of structural feature points of a target field and classification information of the structural feature points from a captured image Im in a calibration process. The above-mentioned feature extractor is a learning model that is trained to output position information of structural feature points in an image for each classification (i.e., label) of a structural feature point to be extracted when a captured image Im is input. The above-mentioned position information may be map information on an image indicating the reliability of each coordinate value of a structural feature point, or may be coordinate values ​​indicating the position of a structural feature point in an image in pixel or subpixel units. When the above-mentioned feature extractor outputs map information for each classification of a structural feature point, for example, the top m coordinate values ​​(m is an integer of 1 or more) whose reliability exceeds a certain threshold for each classification of a structural feature point are adopted as the positions of the structural feature points. The integer m is "1" in the examples of Figs. 4(A) and 5(A) described later, and "2" in the examples of Figs. 4(B) and 5(B) described later. The learning model used for training the feature extractor may be a learning model based on a neural network, or may be another type of learning model such as a support vector machine, or may be a combination of these. For example, when the above-mentioned learning model is a neural network such as a convolutional neural network, the parameter storage unit 21 stores various parameters such as the layer structure, the neuron structure of each layer, the number of filters and the filter size in each layer, and the weight of each element of each filter.

[0028] Furthermore, the parameter storage unit 21 stores parameters related to the camera 15 necessary for displaying a virtual object, such as the focal length, internal parameters, principal point, and size information of the captured image Im of the camera 15.

[0029] The structure data storage unit 22 stores structure data that is data related to the structure of a target field. Fig. 2 shows an example of the data structure of the structure data. The structure data has size information, registered feature point information, and anisotropy information.

[0030] The size information is information about the size of the target field. The registered feature point information is information about the structural feature points of the target field, and includes individual information for each structural feature point measured in advance. Here, since the target field has N structural feature points (N is an integer equal to or greater than 2), the registered feature point information includes information (first registered feature point information to Nth registered feature point information) corresponding to each of these N structural feature points. The registered feature point information includes at least a label indicating the classification of the target structural feature point, and registered position information indicating the position of the target structural feature point in the field coordinate system. The registered position information is coordinate information expressed in the field coordinate system, and is set, for example, so that the position of any one of the structural feature points is the origin. The registered feature point information is used in the calibration process. Note that the length between any two structural feature points can be calculated based on the coordinate values ​​indicated by the registered position information.

[0031] Anisotropy information is information relating to the anisotropy of displacement of a structural feature point of a target field over time. For example, when a structural feature point of a target field displaces over time, anisotropy information is information indicating a direction in which it is easy to displace and / or a direction in which it is difficult to displace (and the degree of displacement). These directions may be represented by vectors in the field coordinate system, for example, or may be specified based on the positional relationship between structural feature points, such as "the direction from structural feature point A to structural feature point B."

[0032] In addition, the structural data may include information specifying a structural feature point that serves as the origin in the field coordinate system, and information specifying each of the directions of the three axes of the field coordinate system.

[0033] The configuration of the display device 1 shown in FIG. 1 is an example, and various modifications may be made to this configuration. For example, the display device 1 may further include a speaker that outputs sound based on the control of the control unit 17. The display device 1 may also include a gaze detection camera for determining whether or not a virtual object is displayed and for changing the display position of the virtual object according to the position of the user's gaze. In yet another example, the storage unit 14 may not include the sensor data storage unit 20. In this case, the control unit 17 performs a calibration process using the captured image Im instantly acquired from the camera 15 and the position and orientation change amount Ap calculated based on the detection signal of the position and orientation detection sensor 16.

[0034] In yet another example, the display device 1 may not need to detect the position of the display device 1 using the position and orientation detection sensor 16 or the like. In general, the user rarely moves while watching a sporting event, a play, or the like, and a change in the position of the display device 1 has a smaller effect on the display of a virtual object than a change in the orientation of the display device 1. In consideration of the above, the position and orientation detection sensor 16 may be configured with a sensor that detects the orientation of the display device 1, and the control unit 17 may calculate only the amount of change in the orientation of the display device 1 from the time the device coordinate system was set as the position and orientation change amount Ap.

[0035] (2) Functional Blocks Fig. 3 is a block diagram showing the functional configuration of the control unit 17. As shown in Fig. 3, the control unit 17 functionally has a virtual object acquisition unit 41, a feature extraction unit 42, a coordinate conversion information generation unit 43, a reflection unit 44, and a light source control unit 45. Note that in Fig. 3, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to that shown in Fig. 3. The same applies to other functional block diagrams described later.

[0036] The virtual object acquisition unit 41 acquires information (also called "designated display information Id") that designates a virtual object to be superimposed on a landscape and its display position. This virtual object may be information for drawing a two-dimensional object (two-dimensional drawing information) or information for drawing a three-dimensional object (three-dimensional drawing information). For example, when a server device managed by an exhibitor and the display device 1 are capable of communicating with each other, the virtual object acquisition unit 41 acquires, as the designated display information Id, distribution information that is distributed by push type or pull type at a predetermined timing from the server device. In this case, the designated display information Id includes information that designates a display position (for example, information indicating coordinate values ​​in a field coordinate system) in addition to the virtual object. In another example, information indicating a combination of a virtual object, a display position, and a display condition thereof may be stored in advance in the storage unit 14. In this case, when the virtual object acquisition unit 41 determines that the stored display condition is satisfied, the virtual object acquisition unit 41 acquires, as the designated display information Id, a combination of a virtual object and a display position corresponding to the satisfied display condition.

[0037] The feature extraction unit 42 generates feature point information "IF" based on the captured image Im acquired from the sensor data storage unit 20. In this case, the feature extraction unit 42 inputs the captured image Im (for example, the latest captured image Im) to a feature extractor configured based on parameters extracted from the parameter storage unit 21, and generates feature point information IF from information output by the feature extractor. For example, in this case, the feature extractor outputs the position (for example, coordinate value) of the structural feature point in the input captured image Im for each label indicating the classification of the structural feature point, and the feature extraction unit 42 generates feature point information IF indicating a plurality of combinations in which the positions of the structural feature points in the above-mentioned captured image Im are candidate positions of the structural feature points (also referred to as "feature point candidate positions") and the labels corresponding to the positions are candidate labels (also referred to as "candidate labels"). Note that, when the feature extractor outputs normalized values ​​so as not to depend on the image size as the coordinate values ​​of the structural feature points, the feature extraction unit 42 calculates the feature point candidate positions by multiplying the coordinate values ​​by the image size of the captured image Im.

[0038] The coordinate conversion information generating unit 43 generates coordinate conversion information "Ic" between the device coordinate system and the field coordinate system based on the structure data extracted from the structure data storage unit 22, the feature point information IF, the position and orientation change amount Ap at the time of generating the captured image Im from which the feature extraction was performed, and the parameters of the camera 15. The coordinate conversion information Ic is, for example, a combination of a rotation matrix and a translation vector that are generally used to perform coordinate conversion between three-dimensional spaces. Note that the coordinate conversion information Ic is not limited to information used when converting the field coordinate system to the device coordinate system, and may be information used when converting the device coordinate system to the field coordinate system. Here, the rotation matrix and the translation vector for converting from the field coordinate system to the device coordinate system can be converted into a rotation matrix (the inverse matrix of the above-mentioned rotation matrix) and a translation vector (the above-mentioned translation vector with the sign inverted) for converting from the device coordinate system to the field coordinate system.

[0039] The coordinate conversion information generating unit 43 also has a field estimation unit 46. The field estimation unit 46 estimates the position of the target field in the device coordinate system. The estimation method by the field estimation unit 46 will be described later. The coordinate conversion information generating unit 43 generates coordinate conversion information Ic based on the position of the target field in the device coordinate system estimated by the field estimation unit 46 and the position of the target field in the field coordinate system indicated by the structure data storage unit 22.

[0040] The reflecting unit 44 reflects the coordinate conversion information Ic supplied from the coordinate conversion information generating unit 43 on the designated display information Id supplied from the virtual object acquiring unit 41, thereby generating a display signal “Sd” indicating a virtual object to be projected onto the optical element 11. In this case, the reflecting unit 44 matches the device coordinate system with the field coordinate system using the coordinate conversion information Ic, and then generates a display signal Sd based on the designated display information Id. The light source control unit 45 generates a drive signal that indicates the drive timing and light amount for driving the light source of the light source unit 10 (for example, each light source corresponding to RGB) based on the display signal Sd supplied from the reflecting unit 44, and supplies the generated drive signal to the light source unit 10.

[0041] Note that the explanation of each process (i.e., the process of the reflection unit 44 and the light source control unit 45) after the calibration is completed (i.e., after the coordinate conversion information Ic is calculated) is just an example, and a virtual object may be displayed by superimposing the virtual object at a desired landscape position using any method applied in existing AR products, etc. For example, examples of documents disclosing such technology include JP2015-116336A and JP2016-525741A. As shown in these documents, the display device 1 detects the user's line of sight, and performs control so that the virtual object is appropriately viewed.

[0042] Each of the components of the virtual object acquisition unit 41, the feature extraction unit 42, the coordinate conversion information generation unit 43, the reflection unit 44, and the light source control unit 45 described in FIG. 3 can be realized, for example, by the control unit 17 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. Note that at least a part of each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of each of these components may be realized by using an integrated circuit programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Also, at least a part of each of the components may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each of the components may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0043] (3) Example of label definition for structural feature points Fig. 4(A) is a first label definition example of a structural feature point when the target field is a tennis court, and Fig. 4(B) is a second label definition example of a structural feature point when the target field is a tennis court. In Fig. 4(A) and Fig. 4(B), circles and corresponding label numbers are clearly indicated at the positions of the structural feature points.

[0044] In the first label definition example shown in Fig. 4(A), serial numbers from 0 to 13 are assigned as labels to each of the 14 structural feature points, starting from the corner. On the other hand, in the second label definition example shown in Fig. 4(B), the same label is assigned to symmetrical structural feature points. As a result, each label from 0 to 5 is assigned to two structural feature points that are in symmetrical positions.

[0045] Fig. 5(A) is a first label definition example of a structural feature point when the target field is a swimming pool, and Fig. 5(B) is a second label definition example of a structural feature point when the target field is a swimming pool. In Fig. 5(A) and Fig. 5(B), circles and corresponding label numbers are clearly indicated at the positions of the structural feature points. When the target field is a swimming pool, floats of a specified color attached at specified distance intervals to the rope that divides the course are selected as the structural feature points.

[0046] In the first label definition example shown in Fig. 5(A), serial numbers from 0 to 24 are assigned as labels to the 25 structural feature points, starting from the corners. On the other hand, in the second label definition example shown in Fig. 5(B), the same label is assigned to symmetrical structural feature points. As a result, labels from 0 to 11 are assigned to two structural feature points that are in symmetrical positions.

[0047] (4) Field Estimation Processing (4-1) overview Fig. 6 is a functional block diagram of the field estimation unit 46 showing an overview of the processing of the field estimation unit 46. As shown in Fig. 6, the field estimation unit 46 functionally has a feature point error detection unit 51, a first field estimation unit 52, a label correction unit 53, and a second field estimation unit 54. Although not shown in Fig. 6, each component of the field estimation unit 46 performs each process by referring to the position and orientation change amount Ap stored in the sensor data storage unit 20 and the camera parameters (including internal parameters and the size of the captured image Im) stored in the parameter storage unit 21.

[0048] The feature point error detection unit 51 detects feature point candidate positions (also called "error feature point candidate positions") representing the extraction results of erroneous structural feature points, based on the feature point information IF supplied from the feature extraction unit 42 and the structure data stored in the structure data storage unit 22. In this case, the feature point error detection unit 51 may perform processing to make the vertical axes of the field coordinate system and the device coordinate system parallel, based on the output signal of the acceleration sensor 160 included in the position and orientation detection sensor 16. Then, the feature point error detection unit 51 supplies the detection results of the error feature point candidate positions to the first field estimation unit 52 and the label correction unit 53.

[0049] The first field estimation unit 52 estimates the position of the target field in the device coordinate system based on the feature point information IF, the detection result of the error feature point candidate position by the feature point error detection unit 51, and the structure data stored in the structure data storage unit 22. Specifically, as described below, when the structural feature point of the target field is displaced anisotropically (biasedly in a certain direction), the first field estimation unit 52 determines the estimated position of the target field in the device coordinate system by performing optimization taking into account the anisotropic displacement. In this case, the first field estimation unit 52 may perform processing to make the vertical axis of the field coordinate system and the device coordinate system parallel to each other based on the output signal of the acceleration sensor 160.

[0050] The estimated position of the target field in the device coordinate system calculated by the first field estimation unit 52 is a provisional estimated position for use in processing by the label correction unit 53, and the final estimated position of the target field is determined by the second field estimation unit 54. The first field estimation unit 52 supplies information indicating the estimated position of the target field (also referred to as "first field estimation information") to the label correction unit 53.

[0051] The label correction unit 53 corrects the candidate labels of the feature point information IF based on the first field estimation information supplied from the first field estimation unit 52, the feature point information IF, and the structure data stored in the structure data storage unit 22. In this case, the label correction unit 53 performs a process of changing the candidate labels of error feature point candidate positions (also called "label error feature point candidate positions") that indicate the positions of correct structural feature points by changing the candidate labels among the error feature point candidate positions. Then, the label correction unit 53 supplies the feature point information IF with the candidate labels of the label error feature point candidate positions changed to the second field estimation unit 54.

[0052] The second field estimation unit 54 estimates the position of the target field based on the feature point information IF reflecting the correction by the label correction unit 53 and the structure data stored in the structure data storage unit 22. Here, the estimated position of the target field by the second field estimation unit 54 reflects the processing results of the feature point error detection unit 51, the first field estimation unit 52, and the label correction unit 53, and becomes the estimated position of the target field finally output by the field estimation unit 46. Hereinafter, the information indicating the estimated position of the target field by the second field estimation unit 54 is also called "second field estimation information". Then, based on the second field estimation information output by the second field estimation unit 54, coordinate conversion information Ic is generated.

[0053] (4-2) Minutiae Error Detection 7 is an example of a functional block of the feature point error detection unit 51. Functionally, the feature point error detection unit 51 has an axis adjustment unit 510, a candidate field generation unit 511, an estimated field determination unit 512, and an erroneous label detection unit 513.

[0054] The axis adjustment unit 510 recognizes the vertical direction in the device coordinate system based on the output signal of the acceleration sensor 160. Then, based on the recognized vertical direction, the axis adjustment unit 510 adjusts the height direction axes of the device coordinate system and the field coordinate system so that they are parallel to each other.

[0055] The candidate field generating unit 511 executes a process of generating candidates (also called "candidate fields") representing the target field in the device coordinate system multiple times, using two of the feature point candidate positions indicated by the feature point information IF. In this case, the candidate field generating unit 511 may generate n candidate fields from n sets (pairs) of feature point candidate positions randomly selected n times (n is an integer equal to or greater than 3), or may generate candidate fields from all sets of feature point candidate positions. Note that the candidate field generating unit 511 identifies, for example, the estimated positions of structural feature points in the device coordinate system for each label and the estimated surface of the target field in the device coordinate system as the candidate fields.

[0056] Here, a supplementary explanation will be given on the case where a candidate field is generated from a set of all possible feature point positions. In the case of a label definition in which the label name and the position of the structural feature point are uniquely determined (i.e., the case of the first label definition example shown in FIG. 4(A) and FIG. 5(A)), if the number of feature point candidate positions is N, the candidate field generation unit 511 generates N C 2 From a set of N C 2 On the other hand, in the case of a label definition in which the label name and the position of the structural feature point are not uniquely determined, multiple field estimations are required from one pair. For example, in the case of the label definition examples shown in FIG. 4(B) and FIG. 5(B) (i.e., label definitions in which point-symmetric labels are made the same), the candidate field generation unit 511 generates a total of N C 2 ×2 candidate fields are created. In general, in a label definition example where "M" (M is an integer of 3 or more) possible feature point positions are possible for one label, the minimum estimated number of candidate fields when N structural feature points are given will differ depending on the symmetry of the label definition, etc. Specifically, if processing efficiency is not taken into consideration, the minimum estimated number of candidate fields will be: N C 2 The result is ×M×M.

[0057] Furthermore, in the case of the label definition examples shown in FIG. 4B and FIG. 5B (that is, the label definition in which the point-symmetric labels are the same and M=2), as described above,M C 2 Here is a supplementary explanation of the generation of 2 x 2 candidate fields. For example, if the labels 0,1 in Figure 4(B) are considered as a pair (hereinafter written as (0,1)), when converted to the first label definition example in Figure 4(A), the following 2 x 2 = 4 pairs exist. (0,1), (0,12), (13,1), (13,12)

[0058] On the other hand, in the second label definition example, since the labels are point-symmetric, the field estimation results (candidate fields) when (0,1) is converted to (0,1) or (13,12) are essentially the same, with a 180° rotation relationship. The same is true when (0,1) is converted to (0,12) or (13,1). Taking the above into consideration, in the second label definition example, which results in point-symmetric labels, it is necessary and sufficient to estimate two candidate fields converted by fixing one of the labels before conversion (for example, when 0 is fixed, there are two possible cases: (0,1) and (0,12)).

[0059] The estimation field determination unit 512 determines a target field (also called an "estimated field") in the device coordinate system to be finally output as an estimation result from a plurality of candidate fields. In this case, the estimation field determination unit 512 performs clustering of the candidate fields, and determines an estimated field from the candidate fields that belong to the main cluster to which the largest number of candidate fields belong. In other words, the estimation field determination unit 512 determines the estimated positions of the structural feature points of each label of the target field in the device coordinate system (and the estimated surface of the target field).

[0060] The error label detection unit 513 determines the error feature point candidate positions based on the estimated field determined by the estimated field determination unit 512. In this case, for example, the error label detection unit 513 recognizes the points obtained by projecting each feature point candidate position onto the estimated field as each feature point candidate position in the device coordinate system based on the shooting position in the device coordinate system specified based on the position and orientation change amount Ap and the parameters (including internal parameters) of the camera 15. Then, the error label detection unit 513 compares the positions of the structural feature points in the estimated field with the feature point candidate positions in the device coordinate system for each label, and determines the feature point candidate positions that are a predetermined distance or more as the error feature point candidate positions. The above-mentioned predetermined distance is, for example, a preset compatibility value stored in the storage unit 14 or the like. Furthermore, the field estimation unit 46 ends the estimation when the number of structural feature points that do not correspond to the error feature point candidate positions becomes equal to or less than a predetermined threshold value (for example, 5), and determines that the estimation of the target field has failed. In this case, the field estimation unit 46 executes a field estimation process based on the captured image Im that is newly acquired.

[0061] Next, the processing of the axis adjustment unit 510 will be specifically described with reference to FIG.

[0062] Figure 8 shows the positional relationship between the device coordinate system and the field coordinate system after adjustment so that the height axes of the device coordinate system and the field coordinate system are parallel. Here, the target field is a tennis court, and the field coordinate system has axes in the short and long directions of the tennis court and in the perpendicular directions to these. Hereinafter, the three axes of the device coordinate system are referred to as the x-axis, y-axis, and z-axis, and the three axes of the field coordinate system are referred to as the x^-axis, y^-axis, and z^-axis.

[0063] In this case, the axis adjustment unit 510 moves at least one of the device coordinate system or the field coordinate system so that the y axis of the device coordinate system and the y axis of the field coordinate system are parallel to each other based on the output signal of the acceleration sensor 160. For example, if the y^ axis of the field coordinate system is already parallel to the vertical direction, the axis adjustment unit 510 rotates the device coordinate system so that the y axis of the device coordinate system is parallel to the vertical direction. In another example, if the y^ axis of the field coordinate system is not parallel to the vertical direction, the axis adjustment unit 510 rotates both the device coordinate system and the field coordinate system by an appropriate angle so that the y axis of the device coordinate system and the y^ axis of the field coordinate system are parallel to the vertical direction.

[0064] It should be noted that the axis adjustment unit 510 is not an essential component of the feature point error detection unit 51. For example, instead of the feature point error detection unit 51 having the axis adjustment unit 510, the candidate field generation unit 511 may generate a candidate field from three (or four or more) feature point candidate positions.

[0065] Next, the process of generating candidate fields by the candidate field generating section 511 will be specifically described with reference to FIG.

[0066] 9 is a diagram showing the shooting position of the camera 15, the captured image Im, and the target field in the device coordinate system and the field coordinate system. Here, the y axis of the device coordinate system and the y^ axis of the field coordinate system are adjusted to be parallel, and these axes are shifted by an offset "Y". In this case, the candidate field generating unit 511 can determine the candidate field by calculating the offset Y and the angle between the x axis and the x^ axis (or the angle between the z axis and the z^ axis) from any two points of the feature point candidate positions.

[0067] Here, a supplementary explanation will be given on the process of determining a candidate field from two arbitrary feature point candidate positions.

[0068] FIG. 9 shows an overview of a process for generating a candidate field from two feature point candidate positions of label numbers 6 and 7 based on the first label definition example shown in FIG. 4(A). The distance between the two feature point candidate positions of label numbers 6 and 7 (see arrow 90) can be specified by referring to the structure data. Then, the candidate field generating unit 511 determines a candidate field based on the shooting position (including the shooting direction) of the camera 15 in the device coordinate system, the parameters of the camera 15, the feature point candidate positions in the captured image Im of label numbers 6 and 7, and the distance indicated by the arrow 90. The shooting position (including the shooting direction) of the camera 15 in the device coordinate system is specified based on the position and orientation change amount Ap.

[0069] Furthermore, the candidate field generating unit 511 needs to execute the process of determining candidate fields from sets of feature point candidate positions multiple times, but these processes can be calculated in parallel using matrix processing. Therefore, the candidate field generating unit 511 can calculate candidate fields for each of the multiple sets of generated feature point candidate positions by parallel processing.

[0070] Next, a supplementary explanation will be given of the clustering process performed by the estimation field determination unit 512.

[0071] The estimated field determination unit 512 performs clustering of the candidate fields by using any clustering method. Such clustering methods include, but are not limited to, the single link method, the complete link method, the group average method, the Ward method, the centroid method, the weighted method, and the median method. In this case, the estimated field determination unit 512 performs the above-mentioned clustering based on, for example, the four-dimensional coordinate value (vector) of the three-dimensional center of gravity coordinate in the device coordinate system of each candidate field and the angle representing the orientation on the xz plane (the orientation in the long direction or the short direction in the case of a tennis court). As the above-mentioned "three-dimensional center of gravity coordinate in the device coordinate system of the candidate field", for example, the center of gravity coordinate of the estimated position in the device coordinate system of the structural feature point of all the labels constituting the candidate field is calculated. Note that, when the axis adjustment based on the vertical direction is not performed, the above-mentioned four-dimensional coordinate value becomes the six-dimensional coordinate value of the three-dimensional center of gravity coordinate in the device coordinate system of each candidate field and the Euler angles (yaw, pitch, roll).

[0072] 10 is a diagram showing the result of clustering the six generated candidate fields "C1" to "C6" in the device coordinate system. In this case, the estimated field determination unit 512 classifies the candidate fields into a first cluster CL1, a second cluster CL2, and a third cluster CL3 by using an arbitrary clustering method. Here, the first cluster CL1 is the main cluster, and the four candidate fields C1, C3, C4, and C6, which are the most numerous, belong to it. Meanwhile, the second cluster CL2 is made up of the candidate field C2, and the third cluster CL3 is made up of the candidate field C5.

[0073] In this case, the estimated field determination unit 512 determines that the candidate field belonging to the first cluster CL1, which is the main cluster, is the correct estimated result of the target field, and determines the estimated field based on the candidate fields C1, C3, C4, and C6. For example, the estimated field determination unit 512 calculates the center of gravity (average vector) of the above-mentioned four-dimensional coordinate values ​​of each of the candidate fields belonging to the main cluster, and selects the candidate field corresponding to the four-dimensional coordinate value closest to the center of gravity as the estimated field. In another example, the estimated field determination unit 512 may determine the estimated field by integrating the candidate fields belonging to the main cluster based on an analysis process such as the least squares method using the average (including a weighted average according to the distance from the above-mentioned center of gravity) of the candidate fields belonging to the main cluster or other statistical processing or the positional relationship of the structural feature points specified from the structural data (i.e., the model of the target field). In this case, the estimated field determination unit 512 may determine the estimated field by, for example, setting a model or constraint conditions that represent the positional relationship (distance) between each structural feature point of the target field indicated by the structure data, and performing optimization (including analysis by the least squares method, etc.) to minimize the sum of errors between the structural feature points of the estimated field and the point group of feature point candidate positions of the candidate field belonging to the main cluster for each label. In yet another example, the estimated field determination unit 512 may determine, as the estimated field, a candidate field randomly selected from the candidate fields belonging to the main cluster. In this way, the estimated field determination unit 512 determines the estimated field based on the integration or selection of the candidate fields belonging to the main cluster.

[0074] Next, a supplementary explanation will be given of the technical effects based on the processing of the minutiae error detection unit 51.

[0075] As a comparative example, consider a case where plausible structural feature points and labels are estimated based on RANSAC, instead of the processes related to the generation of candidate fields by the candidate field generation unit 511 and the determination of an estimated field by the estimated field determination unit 512. In this comparative example, a suitable set (pair) of feature point candidate positions that enables field estimation is randomly selected from all structural feature points, and a set of structural feature point positions and labels that are likely to be true values ​​is identified. In this comparative example, a target field is estimated by optimization such that as many feature point candidate positions as possible match the estimated positions of structural feature points. In this case, there is a problem that the amount of calculation per trial is large, and the processing time can be long depending on the required accuracy.

[0076] Taking the above into consideration, the feature point error detection unit 51 according to this embodiment calculates candidate fields based on a set of multiple feature point candidate positions, and then determines an estimated field by clustering. This enables the feature point error detection unit 51 to process candidate fields in parallel, and can shorten the processing time compared to the comparative example using RANSAC. Furthermore, the feature point error detection unit 51 can obtain an accurate estimation result of the target field by excluding erroneously extracted feature point candidate positions.

[0077] (4-3) First Field Estimation Section Next, the processing of the first field estimation unit 52 will be described. When the structural feature points of the target field are displaced anisotropically (biased in a certain direction), the first field estimation unit 52 performs optimization taking into consideration the anisotropic displacement to determine the estimated position of the target field. Here, the target field in which the structural feature points are displaced anisotropically (biased in a certain direction) is, for example, the (sports) swimming pool shown in Fig. 5(A) and Fig. 5(B). A target field having such properties can be regarded as a quasi-rigid body having properties similar to those of a rigid body.

[0078] FIG. 11 is a diagram clearly showing floats that are designated as structural feature points among the ropes attached to the pool. Here, the floats that are structural feature points are hatched. As shown in FIG. 11, the floats attached to the ropes are less likely to move in the direction in which the ropes are lined up (i.e., the short side direction of the pool) and are more likely to move in the direction in which the ropes extend (i.e., the long side direction of the pool). In this way, when the pool is the target field, the displacement of the structural feature points over time is not isotropic. Taking the above into consideration, the first field estimation unit 52 performs optimization that takes into account the anisotropic displacement in estimating the target field in which the structural feature points displace anisotropically over time.

[0079] FIG. 12 is a diagram showing, in the device coordinate system, estimated positions of the target field obtained by optimization processing based on feature point candidate positions excluding the error feature point candidate positions. Here, the target field has, as structural feature points, the vertices of each lattice (total of 12 vertices) when the target field is divided into a lattice. Then, the first field estimation unit 52 estimates the target field based on nine feature point candidate positions "P2" to "P7", "P10" to "P12" excluding the three error feature point candidate positions. In this case, the first field estimation unit 52 specifies the feature point candidate positions P2 to P7, P10 to P12 (dashed lines shown in FIG. 12) projected onto the device coordinate system based on the shooting position (including the shooting direction) in the device coordinate system specified based on the position and orientation change amount Ap and the parameters (including the internal parameters) of the camera 15. Then, the candidate field generation unit 51 refers to the structural data to identify the relative positional relationship (model of the target field) that indicates the distance between the structural feature points of the target field, and determines the feature point candidate positions P2 to P7, P10 to P12 and the entire estimated field in the device coordinate system so that the positional relationship between the identified structural feature points is maintained.

[0080] In this way, the first field estimation unit 52 estimates the target field in the device coordinate system based on at least some of the pairs of feature point candidate positions and candidate labels indicated by the feature point information IF and the structure data, on the premise that there are no errors in the candidate labels of the feature point candidate positions used and that distortion caused by the internal parameters of the camera 15, etc. has been adjusted.

[0081] Next, the optimization process based on the anisotropy information contained in the structure data will be specifically described.

[0082] 13 is a diagram showing the isotropic optimization solution and the anisotropic optimization solution of the target field on the xz plane of the device coordinate system. Here, the isotropic optimization solution is shown by a dashed line, and the anisotropic optimization solution is shown by a solid line. Note that here, the first field estimation unit 52 is assumed to have already adjusted the y-axis of the device coordinate system to be parallel to the vertical direction (i.e., the normal direction of the target field) based on the same process as the axis adjustment unit 510 (i.e., based on the output signal of the acceleration sensor 160).

[0083] Here, the first field estimation unit 52 refers to the anisotropy information, and when it is determined that the structural feature points of the target field do not displace anisotropically, performs isotropic optimization. In this case, the first field estimation unit 52 calculates the distance (i.e., error) between the estimated position of the structural feature point of the target field and the feature point candidate position in the device coordinate system for each label, and calculates the above-mentioned solution of the estimated position that minimizes the evaluation function by using the sum of the distances for each label. Note that in this case, the first field estimation unit 52 calculates the above-mentioned solution under the constraint that the relative positional relationship of each structural feature point based on the structural data stored in the structural data storage unit 22 is satisfied. Note that, for example, the above-mentioned calculation may be performed by the least squares method by regarding the above-mentioned constraint condition as a model of the target field.

[0084] On the other hand, when the first field estimation unit 52 refers to the anisotropic information and determines that the structural feature point of the target field is displaced anisotropically, it performs anisotropic optimization. In this case, the first field estimation unit 52 first calculates the distances (i.e., errors) in the x-axis direction and the z-axis direction between the estimated position of the structural feature point of the target field to be estimated and the feature point candidate position in the device coordinate system for each label, and calculates the sum of the distances in the x-axis direction and the sum of the distances in the z-axis direction. Then, the first field estimation unit 52 sets the sum of the calculated sum of the distances in the x-axis direction and the sum of the distances in the z-axis direction multiplied by different weighting coefficients (the coefficient corresponding to the x-axis is "ax" and the coefficient corresponding to the y-axis is "ay") as the evaluation function. These coefficients may be included in the anisotropic information, for example, and the weighting coefficient in the direction with the larger displacement and the weighting coefficient in the direction with the smaller displacement may be set to fixed values ​​regardless of the target field. The first field estimation unit 52 then calculates the solution of the estimated position that minimizes the evaluation function, using the relative positional relationship of each structural feature point based on the structural data as a constraint condition. Note that the calculation may be performed by using the weighted least squares method, for example, by regarding the constraint condition as a model of the target field.

[0085] The above-mentioned optimization method is merely an example, and isotropic or anisotropic optimization may be performed for each coordinate axis based on an arbitrary evaluation function.

[0086] 13, the first field estimation unit 52 refers to the anisotropy information of the structure data and recognizes that the structural feature points are likely to move in the x-axis direction (i.e., the short-side direction of the target field) and are unlikely to move in the z-axis direction (i.e., the long-side direction of the target field). Therefore, in anisotropic optimization, the first field estimation unit 52 sets the weighting coefficient ax corresponding to the error in the x-axis higher than the weighting coefficient ay corresponding to the error in the y-axis, and determines the estimated position of the target field so as to emphasize alignment in the x-axis direction. As a result, the estimated position of the target field corresponding to the solution of the anisotropic optimization (see solid line) has less deviation in the x-axis direction compared to the estimated position of the target field corresponding to the solution of the isotropic optimization (see dashed line).

[0087] In this way, the first field estimation unit 52 can accurately estimate the target field by taking into consideration the ease of displacement for each direction of the structural feature points of the target field. Note that, when the first field estimation unit 52 refers to the anisotropy information of the structure data and determines that no displacement occurs in the structural feature points of the target field or that the displacement is isotropic, it estimates the target field by performing the above-mentioned isotropic optimization.

[0088] Then, the first field estimation unit 52 supplies the estimation result of the target field as first field estimation information to the label correction unit 53. Here, the first field estimation information is information indicating, for example, a set of estimated positions in the device coordinate system of structural feature points of the target field and labels, and an estimated surface of the target field in the device coordinate system specified from the estimated positions of these structural feature points (also referred to as a "field estimation surface").

[0089] Note that the process of adjusting the y-axis of the device coordinate system so that it is parallel to the vertical direction (i.e., the normal direction of the target field) is not essential, and the first field estimation unit 52 may estimate the target field by the above-mentioned optimization without performing this process. In this case, the first field estimation unit 52 performs the above-mentioned optimization based on the distance in three-dimensional space between the estimated position of the structural feature point of the target field and the feature point candidate position in the device coordinate system.

[0090] (4-4) Label correction section The label correction unit 53 corrects the candidate labels of the feature point information IF based on the first field estimation information and the feature point information IF supplied from the first field estimation unit 52. In the following description, it is assumed that distortion caused by the internal parameters of the camera 15 and the like has already been adjusted.

[0091] FIG. 14(A) is a diagram showing candidate feature point positions, together with candidate label numbers, on a field estimation surface based on the first field estimation information when the target field is a tennis court. Here, it is assumed that each candidate feature point position has been converted to a position in the device coordinate system based on the shooting position and parameters of the camera 15 by applying the processing of the first field estimation unit 52 described with reference to FIG. 12 and the like. Note that in FIG. 14(A), as an example, a label based on the first label definition example of the tennis court shown in FIG. 4(A) is assigned to each structural feature point. In FIG. 14(A), candidate error feature point positions are indicated by dashed circles, and other candidate feature point positions are indicated by solid circles. Here, the candidate error feature point positions are candidate feature point positions corresponding to candidate labels 7 and 8.

[0092] 14B is a diagram showing a set of feature point candidate positions and candidate label numbers after the first field estimation unit 52 corrects the candidate labels on a field estimation plane based on the first field estimation information. In this case, the first field estimation unit 52 compares a set of structural feature points and labels specified based on the first field estimation information with a set of feature point candidate positions and candidate labels indicated by the feature point information IF. In this case, the first field estimation unit 52 calculates, for each feature point candidate position indicated by the feature point information IF, a projection position on the field estimation plane indicated by the first field estimation information (for example, feature point candidate positions P2 to P7, P10 to P12 in FIG. 12) as each feature point candidate position in the device coordinate system. In this case, the first field estimation unit 52 calculates each feature point candidate position in the device coordinate system taking into account distortion caused by internal parameters of the camera 15 and the like.

[0093] Then, for labels 0 to 6 and 9 to 13, the first field estimation unit 52 determines that these feature point candidate positions are consistent with the first field estimation information (i.e., match the structural feature point positions indicated by the first field estimation information) because the distance between the structural feature point positions based on the first field estimation information and the feature point candidate positions in the device coordinate system corresponding to the same labels is within a predetermined threshold. The above-mentioned predetermined distance is, for example, a distance at which the positions can be regarded as being the same position, and the matching value is stored in advance in the storage unit 14 or the like.

[0094] On the other hand, the first field estimation unit 52 determines that the distance between the positions of the structural feature points based on the first field estimation information and the feature point candidate positions in the device coordinate system corresponding to the same label is longer than a predetermined threshold for labels 7 and 8. Therefore, in this case, the first field estimation unit 52 determines that these feature point candidate positions are not consistent with the first field estimation information.

[0095] The first field estimation section 52 may determine whether or not each feature point candidate position is consistent with the first field estimation information based on the detection result of the error feature point candidate position by the feature point error detection section 51. In this case, the first field estimation section 52 determines that the error feature point candidate position detected by the feature point error detection section 51 does not match the first field estimation information, and determines that the feature point candidate positions other than the error feature point candidate position detected by the feature point error detection section 51 are consistent with the first field estimation information.

[0096] Next, the first field estimation unit 52 further determines whether or not there is a feature point candidate position that is consistent with the first field estimation information by correcting the candidate label among the feature point candidate positions that are inconsistent with the first field estimation information (here, the feature point candidate positions with candidate labels 7 and 8).Then, since the distance between the feature point candidate position with candidate label 7 in the device coordinate system and the position of an arbitrary structural feature point based on the first field estimation information is longer than the above-mentioned predetermined threshold, the first field estimation unit 52 determines that the error in the feature point candidate position with candidate label 7 cannot be corrected by correcting the candidate label.

[0097] On the other hand, since the distance between the feature point candidate position in the device coordinate system of candidate label 8 and the position of the structural feature point of label 7 based on the first field estimation information is less than the above-mentioned predetermined threshold, the first field estimation unit 52 determines that the error in the feature point candidate position of candidate label 8 can be corrected by correcting the candidate label to 7 (i.e., it is consistent with the first field estimation information). Therefore, in this case, the first field estimation unit 52 determines that the feature point candidate position of candidate label 8 is a label error feature point candidate position, and changes the candidate label of the feature point candidate position of candidate label 8 to 7, as shown in Fig. 14(B). This allows the first field estimation unit 52 to appropriately correct the error in the feature point information IF.

[0098] In this way, the first field estimation unit 52 corrects the candidate labels of label error feature point candidate positions where the error can be corrected simply by correcting the candidate labels, among the feature point candidate positions that do not match the first field estimation information. This allows the first field estimation unit 52 to appropriately correct the feature point candidate positions.

[0099] Thereafter, the label correction unit 53 generates second field estimation information by performing the same processing as the first field estimation unit 52, for example, based on the feature point information IF corrected by the first field estimation unit 52. Here, when generating the second field estimation information, the label correction unit 53 estimates the target field based on the feature point information IF in which the errors in the candidate labels have been appropriately corrected, so that it is possible to suitably generate second field estimation information with higher accuracy than the first field estimation information generated by the first field estimation unit 52.

[0100] Furthermore, the field estimation unit 46 may determine whether the field estimation has been successful or unsuccessful based on the amount of error and / or the number of structural feature points used after correction by the label correction unit 53. For example, the field estimation unit 46 stops the field estimation and determines that the field estimation has failed when any of the following (Case A), (Case B), or (Case C) applies.

[0101] (Case A) When the total number “n” of structural feature points used in field estimation by the second field estimation unit 54, including the structural feature points corrected by the label correction unit 53, is equal to or less than a predetermined threshold value, (Case B) When the ratio (n / Np) of the total number of used points n to the number of feature points “Np” obtained by the feature extraction unit 42 (i.e., the number of feature points indicated by the feature point information IF) is equal to or less than a predetermined threshold value, (Case C) When the label of the corrected structural feature and the labels of other structural feature points used for field estimation are collinear in the structural data (e.g., labels 10, 11, 12, and 13 in Figure 4(A)). In other words, when field estimation is performed using only structural feature points that are structurally arranged to be collinear.

[0102] Then, when the field estimation unit 46 corresponds to any one of (Case A), (Case B), and (Case C) and stops the field estimation, it executes a field estimation process based on the newly acquired captured image Im. Note that by stopping the field estimation in the case of (Case C), it is possible to prevent the execution of a field estimation with low accuracy, since the accuracy is significantly lower than in other cases when performing field estimation using only structural feature points that are structurally on the same straight line. Also, the field estimation unit 46 may determine whether or not the estimation process by the second field transition unit 54 corresponds to (Case A) or (Case B), and determine whether or not the executed field estimation has failed. That is, in this case, similar to the distance comparison with the position of the structural feature point by the label correction unit 53, the field estimation unit 46 may obtain the consistency between the position of the structural feature point obtained by the feature extraction unit 42 and the position of the structural feature point after the field estimation, and determine whether or not the consistent structural feature point corresponds to (Case A) or (Case B).

[0103] Here, a supplementary explanation will be given on the effect based on the processing of the first field estimation unit 52. Generally, when structural feature points of a tennis court or the like are similar to each other, a feature extraction result may be generated in which only the candidate label has an error. In such a case, the display device 1 can appropriately correct the feature extraction result, and can perform a calibration process or the like based on the corrected feature extraction result with high accuracy.

[0104] (5) Processing flow FIG. 15 is an example of a flowchart showing an outline of a process related to a display process of a virtual object executed by the control unit 17 in the first embodiment.

[0105] First, the control unit 17 detects the start-up of the display device 1 (step S11). In this case, the control unit 17 sets a device coordinate system based on the attitude and position of the display device 1 at the time of start-up of the display device 1 (step S12). After that, the control unit 17 acquires a captured image Im generated by the camera 15, and acquires a position and orientation change amount Ap based on a detection signal output by the position and orientation detection sensor 16 (step S13). The control unit 17 stores the combination of the captured image Im and the position and orientation change amount Ap acquired in step S13 in the sensor data storage unit 20.

[0106] Then, the control unit 17 determines whether or not there is a request to display a virtual object (step S14). For example, when the virtual object acquisition unit 41 receives distribution information instructing the display of a virtual object from a server device (not shown) managed by the promoter, the virtual object acquisition unit 41 determines that there is a request to display a virtual object. If there is no request to display a virtual object (step S14; No), the control unit 17 continues to acquire the captured image Im and the position and orientation change amount Ap in step S13.

[0107] On the other hand, if there is a request to display a virtual object (step S14; Yes), the control unit 17 executes a calibration process (step S15). The details of the procedure of this calibration process will be described later with reference to FIG.

[0108] Next, the reflection unit 44 of the control unit 17 generates a display signal Sd for displaying the virtual object designated in the display request and the virtual object corresponding to the display position based on the coordinate conversion information Ic obtained by the calibration process in step S15 (step S16). Note that in this case, as in various conventional AR display products, the control unit 17 actually recognizes the space viewed by the user in the device coordinate system by taking into account the user's line of sight and the position and attitude change amount Ap, and generates the display signal Sd so that the virtual object is displayed at the designated position in the space. Then, the light source control unit 45 of the control unit 17 performs emission control of the light source unit 10 based on the display signal Sd (step S17).

[0109] The processing procedure of the flowchart shown in FIG. 15 is an example, and various modifications can be made to this processing procedure.

[0110] For example, the control unit 17 executes the calibration process of step S15 every time a virtual object display request is received, but this is not limited to the above. Alternatively, the control unit 17 may execute the calibration process only when a predetermined time or more has elapsed since the previous calibration process. In this way, the control unit 17 needs to execute the calibration process at least once after the display device 1 is started up.

[0111] Furthermore, the control unit 17 determines the device coordinate system based on the position and orientation of the display device 1 at the time of starting up the display device 1, but this is not limited to the above. Instead of this, for example, the control unit 17 may determine the device coordinate system based on the position and orientation of the display device 1 at the time of the first display request after starting up the display device 1 (i.e., when the calibration process is executed for the first time). In another example, the control unit 17 may reset the device coordinate system based on the position and orientation of the display device 1 at the time of the display request (i.e., when the calibration process is executed) every time a display request is issued. In this case, it is not necessary to use the position and orientation change amount Ap in the process of generating the coordinate conversion information Ic, which will be described later.

[0112] FIG. 16 is an example of a flowchart showing a detailed procedure of the calibration process in step S15 of FIG.

[0113] First, the feature extraction unit 42 of the control unit 17 generates feature point information IF indicating a set of feature point candidate positions and candidate labels corresponding to each structural feature point of the target field, based on the captured image Im acquired from the sensor data storage unit 20 etc. (Step S21). In this case, the feature extraction unit 42 configures a feature extractor based on parameters acquired from the parameter storage unit 21, and inputs the captured image Im to the feature extractor. Then, the feature extraction unit 42 generates feature point information IF based on information output by the feature extractor.

[0114] Next, the field estimation unit 46 of the coordinate conversion information generation unit 43 executes a feature point error detection process (step S22). Details of the feature point error detection process will be described with reference to FIG.

[0115] Furthermore, the field estimation unit 46 executes a first field estimation process (step S23). The first field estimation process will be described in detail with reference to FIG.

[0116] Then, the field estimation unit 46 executes a label correction process (step S24). Details of the label correction process will be described with reference to Fig. 19. Then, the field estimation unit 46 executes a second field estimation process (step S25). As a result, the field estimation unit 46 generates second field estimation information representing a result of estimating the target field in the device coordinate system with high accuracy based on the error-corrected feature point information IF.

[0117] Then, the coordinate conversion information generation unit 43 generates coordinate conversion information Ic for conversion from the device coordinate system to the field coordinate system based on the second field estimation information generated by the field estimation unit 46 (step S26). In this case, for each label of the structural feature point, the coordinate conversion information generation unit 43 matches the detected position in the device coordinate system of each structural feature point indicated by the second field estimation information acquired in step S25 with the position of the structural feature point in the field coordinate system indicated by the registered position information included in the structural data, and calculates coordinate conversion information Ic such that the matched positions for each label match (i.e., the position error for each label is minimized).

[0118] In this way, the display device 1 compares information extracted from the captured image Im of only structural feature points that have been registered in advance (i.e., labels have already been registered) in the calibration process with information on structural feature points registered in the structure data. This significantly reduces the amount of calculation required for the comparison process to calculate the coordinate transformation information Ic, and makes it possible to calculate robust coordinate transformation information Ic that is not affected by the extraction of noise contained in the captured image Im (i.e., feature points other than the target field). Furthermore, in this embodiment, the above-mentioned comparison is performed based on estimated information of structural feature points in which errors in the feature extraction process have been appropriately corrected, making it possible to calculate accurate coordinate transformation information Ic.

[0119] FIG. 17 is an example of a flowchart showing details of the feature point error detection process executed in step S22.

[0120] First, the feature point error detection unit 51 identifies the vertical direction based on the output signal of the acceleration sensor 160, etc., and adjusts the device coordinate system, etc. based on the identified vertical direction so that one axis (y axis and y^ axis) of the device coordinate system and the field coordinate system are parallel (step S31).

[0121] Next, the feature point error detection unit 51 executes a process of generating a candidate field in the device coordinate system based on the structure data and the selected two feature point candidate positions multiple times (step S32). This generates candidate fields the same number of times as the above-mentioned process has been executed. Note that the feature point error detection unit 51 may execute the process of generating candidate fields in parallel.

[0122] Next, the feature point error detection unit 51 performs clustering on the multiple candidate fields generated in step S32 (step S33). In this case, the feature point error detection unit 51 generates one or more clusters for the multiple candidate fields by an arbitrary clustering method.

[0123] The feature point error detection unit 51 then determines an estimated field based on the candidate field of the main cluster to which the largest number of candidate fields belong (step S34).The feature point error detection unit 51 then projects each feature point candidate position onto the estimated field, and detects error feature point candidate positions by comparing the projected feature point candidate positions with the structural feature points of the estimated field for each label (step S35).

[0124] FIG. 18 is an example of a flowchart showing details of the first field estimation process executed in step S23.

[0125] First, the first field estimation unit 52 specifies the vertical direction based on the output signal of the acceleration sensor 160, etc., and adjusts the device coordinate system, etc., based on the specified vertical direction so that one axis (y axis and y^ axis) of the device coordinate system and the field coordinate system are parallel (step S41). Note that, if one axis of the device coordinate system and the field coordinate system is already parallel based on the processing of step S31, the first field estimation unit 52 does not need to execute the processing of step S41.

[0126] Next, the first field estimation unit 52 acquires anisotropy information included in the structure data stored in the structure data storage unit 22 (step S42). Then, the first field estimation unit 52 performs optimization for estimating the target field based on the anisotropy information and a plurality of feature point candidate positions other than the error feature point candidate positions (step S43). In this case, when the first field estimation unit 52 determines that the structural feature points of the target field are displaced anisotropically based on the anisotropy information, it performs anisotropic optimization (i.e., weights differ depending on the direction). On the other hand, when the first field estimation unit 52 determines that the structural feature points of the target field are displaced isotropically based on the anisotropy information, it performs isotropic optimization (i.e., weights do not change depending on the direction). Then, the first field estimation unit 52 generates first field estimation information representing the estimation result of the target field obtained by the above optimization (step S44).

[0127] FIG. 19 is an example of a flowchart showing details of the label correction process executed in step S24.

[0128] First, the label correction unit 53 specifies a feature point candidate position in the device coordinate system based on the first field estimation information generated by the first field estimation unit 52 in step S44 (step S51). Then, the label correction unit 53 determines whether or not there is a feature point candidate position that is inconsistent with the first field estimation information (step S52). In this case, for example, if there is an erroneous feature point candidate position detected in step S35, the label correction unit 53 determines that there is a feature point candidate position that is inconsistent with the first field estimation information. In another example, the label correction unit 53 compares the position and label of each structural feature point specified from the first field estimation information with the feature point candidate position and label in the device coordinate system specified in step S51 to determine whether or not there is an erroneous feature point candidate position. Then, if there is no feature point candidate position that is inconsistent with the first field estimation information (step S52; No), the label correction unit 53 ends the processing of the flowchart since there is no need to correct the feature point information IF.

[0129] On the other hand, when there is a feature point candidate position that is not consistent with the first field estimation information (step S52; Yes), the label correction unit 53 judges whether or not the feature point candidate position is consistent with the first field estimation information by changing the corresponding candidate label (i.e., whether or not a label error feature point candidate position is present) (step S53). Then, when the label correction unit 53 judges that the feature point candidate position is consistent with the first field estimation information by changing the corresponding candidate label (step S53; Yes), it corrects the candidate label of the feature point candidate position (step S54). After that, the label correction unit 53 generates feature point information IF that reflects the correction of the candidate label in step S54. On the other hand, when it is judged that the target feature point candidate position is not consistent with the first field estimation information even by changing the corresponding candidate label (i.e., there is no label error feature point candidate position) (step S53; No), the label correction unit 53 ends the processing of the flowchart. In this case, label correction unit 53 assumes that the feature point candidate position detected in step S52 has been erroneously estimated, and performs processing such as deleting the feature point candidate position from feature point information IF or attaching a flag indicating that it is an error feature point candidate position.

[0130] <Second embodiment> Fig. 20 shows the configuration of a display system according to the second embodiment. As shown in Fig. 20, the display system according to the second embodiment includes a display device 1A and a server device 2. The second embodiment differs from the first embodiment in that the server device 2 executes a calibration process and the like instead of the display device 1A. Hereinafter, the same components as those in the first embodiment are appropriately designated by the same reference numerals, and the description thereof will be omitted.

[0131] The display device 1A transmits an upload signal "S1" to the server device 2, which is information necessary for the server device 2 to perform a calibration process or the like. In this case, the upload signal S1 includes, for example, a captured image Im generated by the camera 15 and a position and orientation change amount Ap detected based on the output of the position and orientation detection sensor 16. Then, when the display device 1A receives a distribution signal "S2" transmitted from the server device 2, the display device 1A displays a virtual object by controlling the emission of light from the light source unit 10 based on the distribution signal S2. For example, the distribution signal S2 includes information equivalent to the display signal Sd of the first embodiment, and after receiving the distribution signal S2, the display device 1A causes the light source unit 10 to emit light for displaying a virtual object by performing a process similar to that of the light source control unit 45 of the first embodiment.

[0132] Server device 2 is, for example, a server device managed by an exhibitor, and generates distribution signal S2 and distributes distribution signal S2 to display device 1A based on upload signal S1 received from display device 1A. Fig. 21 is a block diagram of server device 2. Server device 2 has input unit 26, control unit 27, communication unit 28, and storage unit 29.

[0133] The storage unit 29 is a non-volatile memory that stores various information required by the control unit 27 to control the server device 2. The storage unit 29 stores a program executed by the control unit 27. The storage unit 29 has a sensor data storage unit 20, a parameter storage unit 21, and a structure data storage unit 22. The sensor data storage unit 20 stores the captured image Im and the position and orientation change amount Ap included in the upload signal S1 under the control of the control unit 27. The storage unit 29 may be an external storage device such as a hard disk connected to or built into the server device 2, or may be a storage medium such as a flash memory. The storage unit 29 may also be a server device that performs data communication with the server device 2 (i.e., a device that stores information that can be referenced from other devices). In this case, the storage unit 29 may be composed of multiple server devices, and the sensor data storage unit 20, the parameter storage unit 21, and the structure data storage unit 22 may be stored in a distributed manner.

[0134] The control unit 27 has a processor such as a CPU or a GPU, a volatile memory functioning as a working memory, and the like, and performs overall control of the server device 2. The control unit 27 generates information on a virtual object to be displayed as a virtual object and a display position (i.e., information corresponding to the designated display information Id in the first embodiment) based on a user input to the input unit 26, etc. Furthermore, the control unit 27 executes a calibration process by referring to the sensor data storage unit 20, the parameter storage unit 21, and the structure data storage unit 22, and generates a distribution signal S2. In this way, the control unit 27 includes functions corresponding to the virtual object acquisition unit 41, the feature extraction unit 42, the coordinate conversion information generation unit 43, and the reflection unit 44 shown in FIG. 3.

[0135] FIG. 22 is an example of a flowchart showing a process procedure executed by the control unit 27 of the server device 2 in the second embodiment.

[0136] First, the control unit 27 receives an upload signal S1 including a captured image Im and a position and orientation change amount Ap from the display device 1A via the communication unit 28 (step S61). In this case, the control unit 27 updates the data to be stored in the sensor data storage unit 20 based on the upload signal S1. Then, the control unit 27 determines whether it is time to display a virtual object (step S62). If it is not time to display the virtual object (step S62; No), the control unit 27 continues to receive the upload signal S1 from the display device 1A in step S61.

[0137] On the other hand, if it is the display timing (step S62; Yes), the control unit 27 executes a calibration process based on the latest upload signal S1 received in step S61, etc. In this case, the control unit 27 executes the flowchart shown in FIG. 16. Then, the control unit 27 generates a distribution signal S2 for the display device 1A to display the virtual object based on the coordinate conversion information Ic obtained in the calibration process (step S64). Then, the control unit 27 transmits the generated distribution signal S2 to the display device 1A via the communication unit 28 (step S65). After that, the display device 1A that has received the distribution signal S2 controls the light source unit 10 based on the distribution signal S2 to display the virtual object.

[0138] As described above, according to the second embodiment as well, the display system can accurately calculate the coordinate conversion information Ic required for displaying a virtual object by the display device 1A, and can allow the user to visually recognize the virtual object in a suitable manner.

[0139] In the second embodiment, the calibration process and the like may be performed by the display device 1A instead of by the server device 2. In this case, the display device 1A executes the process of the flowchart shown in Fig. 16 by appropriately receiving information necessary for the calibration process from the server device 2. Even in this mode, the display system can allow the user of the display device 1A to visually recognize the virtual object in a suitable manner.

[0140] <Third embodiment> Fig. 23 shows a schematic configuration of an information processing device 1X in the third embodiment. As shown in Fig. 23, the information processing device 1X mainly includes a feature point information acquiring means 42X, a field estimation information acquiring means 53Xa, and a label correcting means 53Xb. The information processing device 1X is realized by, for example, the display device 1 or the control unit 17 of the display device 1 in the first embodiment, or the control unit 27 of the server device 2 in the second embodiment. Note that the information processing device 1X may be composed of multiple devices.

[0141] The feature point information acquisition means 42X acquires feature point information indicating candidate positions and candidate labels of a plurality of feature points of a field, which is determined based on an image including at least a part of the target field. The feature point information acquisition means 42X may receive feature point information generated by a processing block (including a device other than the information processing device 1X) other than the feature point information acquisition means 42X, or the feature point information acquisition means 42X may generate the feature point information. In the latter case, the feature point information acquisition means 42X may be, for example, the feature extraction unit 42 in the first or second embodiment.

[0142] The field estimation information acquisition means 53Xa acquires field estimation information indicating the estimation results of the positions and labels of a plurality of feature points in a first coordinate system, which is a coordinate system based on a display device having a camera that captures an image. The field estimation information can be, for example, the "first field estimation information" in the first or second embodiment.

[0143] The label correcting means 53Xb corrects the candidate labels based on the feature point information and the field estimation information. The field estimation information acquiring means 53Xa and the label correcting means 53Xb can be, for example, the label correcting unit 53 in the first or second embodiment.

[0144] 24 is an example of a flowchart in the third embodiment. The feature point information acquisition means 42X acquires feature point information indicating candidate positions and candidate labels of multiple feature points of a field, determined based on an image including at least a part of the target field (step S71). The field estimation information acquisition means 53Xa acquires field estimation information indicating the estimation results of the positions and labels of multiple feature points in a first coordinate system, which is a coordinate system based on a display device having a camera that captures an image (step S72). The label correction means 53Xb corrects the candidate labels based on the feature point information and the field estimation information (step S73).

[0145] According to the third embodiment, the information processing device 1X can appropriately correct feature point information, and can improve the accuracy of various processes that use the feature point information.

[0146] <Fourth embodiment> Fig. 25 shows a schematic configuration of an information processing device 1Y in the fourth embodiment. As shown in Fig. 25, the information processing device 1Y mainly includes a feature point information acquiring means 42Y, a candidate field generating means 511Y, and an estimated field determining means 512Y. The information processing device 1Y is realized by, for example, the display device 1 or the control unit 17 of the display device 1 in the first embodiment, or the control unit 27 of the server device 2 in the second embodiment. Note that the information processing device 1Y may be composed of multiple devices.

[0147] The feature point information acquisition means 42Y acquires feature point information indicating a set of candidate positions and candidate labels of feature points of a field, determined based on an image including at least a part of the target field. The feature point information acquisition means 42Y may receive feature point information generated by a processing block other than the feature point information acquisition means 42Y (including a device other than the information processing device 1Y), or the feature point information acquisition means 42Y may generate the feature point information. In the latter case, the feature point information acquisition means 42Y may be, for example, the feature extraction unit 42 in the first or second embodiment.

[0148] The candidate field generating means 511Y generates a candidate field representing a field candidate in a first coordinate system, which is a coordinate system based on a display device having a camera that captures an image, based on at least two pairs and structure data related to the structure of the field. The candidate field generating means 511Y can be, for example, the candidate field generating unit 511 in the first or second embodiment.

[0149] The estimated field determination means 512Y determines an estimated field representing an estimated field in the first coordinate system based on the generated candidate fields. The estimated field determination means 512Y can be, for example, the estimated field determination unit 512 in the first or second embodiment.

[0150] 26 is an example of a flowchart in the fourth embodiment. The feature point information acquisition means 42Y acquires feature point information indicating a set of candidate positions and candidate labels of feature points of a field, determined based on an image including at least a part of the field (step S81). The candidate field generation means 511Y generates a candidate field representing a candidate of the field in a first coordinate system, which is a coordinate system based on a display device having a camera that captures an image, based on at least two sets and structure data related to the structure of the field (step S82). The estimated field determination means 512Y determines an estimated field representing a field estimated in the first coordinate system based on the generated candidate fields (step S83).

[0151] According to the fourth embodiment, the information processing device 1Y can estimate the field in the first coordinate system with high accuracy even if an error exists in the pair of the candidate position and the candidate label indicated by the feature point information.

[0152] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0153] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0154] [Appendix 1] a feature point information acquiring means for acquiring feature point information indicating candidate positions and candidate labels of a plurality of feature points of a target field, the candidate positions and candidate labels being determined based on an image including at least a portion of the target field; a field estimation information acquisition means for acquiring field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera that captures the image; a label correction means for correcting the candidate labels based on the feature point information and the field estimation information; An information processing device having the above configuration. [Appendix 2] The information processing device described in Appendix 1, wherein the label correction means corrects the candidate label based on a set of a projected position and the candidate label obtained by projecting the candidate position onto the first coordinate system based on the shooting position and parameters of the camera, and the field estimation information. [Appendix 3] The information processing device according to claim 2, wherein, when there is a pair of the projection position and the candidate label that is inconsistent with the field estimation information and that becomes consistent with the field estimation information by changing the candidate label, the label correction means corrects the candidate label of the pair. [Appendix 4] The information processing device according to any one of appendices 1 to 3, wherein the field estimation information acquisition means generates the field estimation information based on at least a portion of the candidate positions and candidate labels indicated by the feature point information and structural data relating to the structure of the field. [Appendix 5] The information processing device described in Appendix 4, wherein the field estimation information acquisition means generates the field estimation information based on the positional relationship of the multiple feature points specified by the structure data and a set of at least a portion of the candidate positions and the candidate labels. [Appendix 6] the structural data includes anisotropy information regarding anisotropy of displacement of the plurality of feature points over time; The information processing device according to claim 5, wherein the field estimation information acquisition means generates the field estimation information based on the positional relationship, the anisotropy information, and the at least some of the sets of the candidate positions and the candidate labels. [Appendix 7] The information processing device described in Appendix 6, wherein when calculating estimation results of the positions of the multiple feature points that satisfy the positional relationship and minimize the error from projected positions of at least some of the candidate positions projected onto the first coordinate system based on the camera's shooting position and parameters, the field estimation information acquisition means determines a weight for each direction related to the error based on the anisotropic information. [Appendix 8] The information processing device according to any one of appendices 4 to 7, wherein the field estimation information acquisition means makes one axis of a second coordinate system, which is a coordinate system adopted in the structure data, parallel to the first coordinate system based on an output signal of a sensor provided on the display device. [Appendix 9] The information processing device according to any one of appendices 1 to 8, further comprising a coordinate conversion information generation means for generating coordinate conversion information relating to coordinate conversion between the first coordinate system and a second coordinate system, which is a coordinate system adopted in the structure data, based on the feature point information reflecting the correction by the label correction means and structure data relating to the structure of the field. [Appendix 10] The information processing device is the display device that displays a virtual object superimposed on a landscape, a light source unit that emits display light for displaying the virtual object; an optical element that reflects at least a part of the display light to allow a viewer to visually recognize the virtual object superimposed on a landscape; The information processing device according to any one of claims 1 to 9, further comprising: [Appendix 11] a field estimation means for estimating the field in the first coordinate system based on the feature point information and the candidate label corrected by the label correction means; a determination means for determining whether the estimation of the field by the field estimation means has failed or been successful; 11. The information processing device according to any one of claims 1 to 10, further comprising: [Appendix 12] The determination means is the total number or arrangement of the feature points used in estimating the field by the field estimation means; the number of the feature points indicated by the feature point information; The number of feature points whose candidate positions do not match the field estimation information The information processing device according to claim 11, wherein the failure or success is determined based on at least one of the above. [Appendix 13] a feature point information acquiring means for acquiring feature point information indicating candidate positions and candidate labels of a plurality of feature points of a target field, the candidate positions and candidate labels being determined based on an image including at least a portion of the target field; a field estimation information generating means for generating field estimation information indicating estimation results of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera that captures the image, based on at least some of the sets of the candidate positions and the candidate labels indicated by the feature point information, information on positional relationships of the plurality of feature points, and anisotropy information on anisotropy of displacement of the plurality of feature points over time; An information processing device having the above configuration. [Appendix 14] The computer obtaining feature point information indicative of candidate locations and candidate labels for a plurality of feature points of a field of interest determined based on an image including at least a portion of the field; obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; revising the candidate labels based on the feature point information and the field estimation information; Control methods. [Appendix 15] obtaining feature point information indicative of candidate locations and candidate labels for a plurality of feature points of a field of interest determined based on an image including at least a portion of the field; obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; A storage medium that stores a program that causes a computer to execute a process of correcting the candidate label based on the feature point information and the field estimation information. [Appendix 16] The computer obtaining feature point information indicative of candidate locations and candidate labels for a plurality of feature points of a field of interest determined based on an image including at least a portion of the field; generating field estimation information indicating estimation results of positions and labels of the plurality of feature points in a first coordinate system, which is a coordinate system based on a display device having a camera that captures the image, based on at least a part of the sets of the candidate positions and the candidate labels indicated by the feature point information, information on the positional relationships of the plurality of feature points, and anisotropy information on anisotropy of displacement of the plurality of feature points; Control methods. [Appendix 17] obtaining feature point information indicative of candidate locations and candidate labels for a plurality of feature points of a field of interest determined based on an image including at least a portion of the field; A storage medium that stores a program for causing a computer to execute a process of generating field estimation information that indicates estimation results of positions and labels of the multiple feature points in a first coordinate system, which is a coordinate system based on a display device having a camera that captures the image, based on at least some of the sets of candidate positions and candidate labels indicated by the feature point information, information on the positional relationships of the multiple feature points, and anisotropy information on the anisotropy of the displacement of the multiple feature points.

[0155] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0156] 1, 1A display device 1X, 1Y Information processing equipment 2. Server equipment 10 Light source unit 11 Optical elements 12 Communications Department 13 Input section 14 Storage section 15 Camera 16 Position and orientation detection sensor 20 Sensor data storage unit 21 Parameter storage section 22 Structure data storage unit

Claims

1. A feature point information acquiring means for acquiring feature point information indicating a pair of candidate positions of a feature point and candidate labels which are labels representing a classification of a plurality of structural feature points, the candidate positions being determined based on an image including at least a portion of a field having the plurality of structural feature points; and a field estimation information acquisition means for acquiring field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera for capturing the image; a label correction means for correcting the candidate labels based on the feature point information and the field estimation information; An information processing device having the above configuration.

2. 2 . The information processing device according to claim 1 , wherein the label correction means corrects the candidate label based on a pair of a projected position and the candidate label obtained by projecting the candidate position onto the first coordinate system based on the shooting position and parameters of the camera, and the field estimation information.

3. The information processing device according to claim 2 , wherein, when there is a pair of the projection position and the candidate label that is inconsistent with the field estimation information and that becomes consistent with the field estimation information by changing the candidate label, the label correction means corrects the candidate label of the pair.

4. The information processing device according to any one of claims 1 to 3, wherein the field estimation information acquisition means generates the field estimation information based on at least a portion of the candidate positions and candidate labels indicated by the feature point information and structural data relating to the structure of the field.

5. The information processing apparatus according to claim 4 , wherein the field estimation information acquisition means generates the field estimation information based on a positional relationship of the plurality of feature points specified by the structure data and a set of at least a portion of the candidate positions and the candidate labels.

6. the structural data includes anisotropy information regarding anisotropy of displacement of the plurality of feature points over time; The information processing apparatus according to claim 5 , wherein the field estimation information acquisition means generates the field estimation information based on the positional relationship, the anisotropy information, and the set of at least a portion of the candidate positions and the candidate labels.

7. The information processing device of claim 6, wherein when calculating an estimation result of the positions of the plurality of feature points that satisfies the positional relationship and minimizes an error between at least some of the candidate positions and a projected position projected onto the first coordinate system based on the camera's shooting position and parameters, the field estimation information acquisition means determines a weight for each direction related to the error based on the anisotropic information.

8. A feature point information acquisition means for acquiring feature point information indicating a pair of candidate positions of a feature point and candidate labels which are labels representing a classification of a plurality of structural feature points, the candidate positions being determined based on an image including at least a portion of a field having the plurality of structural feature points; and a field estimation information generating means for generating field estimation information indicating estimation results of positions and labels of the plurality of feature points in a first coordinate system which is a coordinate system based on a display device having a camera that captures the image, based on at least some of the sets of the candidate positions and the candidate labels indicated by the feature point information, information relating to a positional relationship of the plurality of feature points, and anisotropy information relating to anisotropy of displacement of the plurality of feature points over time; An information processing device having the above configuration.

9. The computer acquiring feature point information indicating a pair of a candidate position of the feature point and a candidate label which is a label representing a classification of the feature point, the candidate position being determined based on an image including at least a portion of a field having a plurality of structural feature points; obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; revising the candidate labels based on the feature point information and the field estimation information; Control methods.

10. Obtaining feature point information indicating a pair of candidate positions of the feature points and candidate labels representing the classification of the feature points, the candidate positions being determined based on an image including at least a portion of a field having a plurality of structural feature points; obtaining field estimation information indicating an estimation result of positions and labels of the plurality of feature points in a first coordinate system, the first coordinate system being a coordinate system based on a display device having a camera that captures the image; A program that causes a computer to execute a process of correcting the candidate label based on the feature point information and the field estimation information.

Images