Information processing device, control method, and program

Abstract

A structure data acquisition means 46X acquires: first structure data concerning a first feature point group that is defined with respect to feature points to be extracted from a field to be targeted; and second structure data concerning a second feature point group other than the first feature point group. A feature point information acquisition means 42X acquires feature point information that is determined on the basis of an image including at least a part of the field and that indicates sets of candidate labels indicating correspondences between positions on the image and the feature points of the first feature point group. A label correction means 53X specifies, on the basis of the second structure data, sets of candidate labels and positions on the image corresponding to the feature points of the second feature point group, and generates correction feature point information obtained by making correction from the specified candidate labels to labels indicating the feature points of the second feature point group. A field inference means 54X infers, on the basis of the first structure data, the second structure data, and the correction feature point information, a field in a coordinate system in which a display device having a camera for photographing the image serves as a reference.

Description

[Technical field]

[0001] The present invention relates to the technical field of an information processing device, a control method, and a storage medium that perform processing related to spatial perception in augmented reality (AR). [Background technology]

[0002] In a device that provides augmented reality, there is a technology that determines the display position of an image (so-called AR image) that is displayed superimposed on a scene viewed by a user based on an image captured by a camera. For example, Patent Document 1 describes a technology that estimates the positions of each feature point on a field and calibrates AR based on the estimation results in order to realize AR in watching sports. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2021 / 033314 Summary of the Invention [Problem to be solved by the invention]

[0004] When processing such as calibration is performed by extracting feature points of a target field as in Patent Document 1, the accuracy of the feature point extraction affects the accuracy of subsequent processing such as calibration. In addition, there are cases where the specifications and features of the target field in learning feature point extraction and the target field in AR display do not completely match, and in such cases, the accuracy of the feature point extraction may decrease.

[0005] In consideration of the above-mentioned problems, one of the main objectives of the present disclosure is to provide an information processing device, a control method, and a storage medium that are capable of suitably estimating a field based on feature point information obtained by feature point extraction. [Means for solving the problem]

[0006] One aspect of the information processing device is a structure data acquisition means for acquiring first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; a feature point information acquiring means for acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; a label correction means for specifying a set of a position on the image corresponding to a feature point of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the specified candidate label to a label indicating the feature point of the second feature point group; a field estimation means for estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; The information processing device has the following features.

[0007] One aspect of the control method includes: The computer The computer obtaining first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; It is a control method.

[0008] program One aspect of the present invention is obtaining first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; and causing a computer to execute a process of estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information. program It is. Effect of the Invention

[0009] As an example of one advantage of the present disclosure, it becomes possible to accurately estimate the field based on the corrected feature point information. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic configuration diagram of a display device according to a first embodiment. [Diagram 2] 4 is a block diagram showing a functional configuration of a control unit. FIG. [Diagram 3] 1A shows an example of a first label definition for a first structural feature point group of a training field that is a tennis court, and 1B shows an example of a label definition for a first structural feature point group and a second structural feature point group of a display target field that is a tennis court. [Figure 4] 1A shows an example of a first label definition for a first structural feature point group of a learning field that is a pool. 1B shows an example of a first label definition for a first structural feature point group and a second structural feature point group of a display target field that is a pool. [Diagram 5] (A) shows an example of a second label definition for a first structural feature point group and a second structural feature point group of a display target field that is a tennis court, (B) shows an example of a second label definition for a first structural feature point group and a second structural feature point group of a display target field that is a swimming pool. [Figure 6] 4 is an example of a functional block of a field estimation unit. [Figure 7] 4 is an example of a functional block of a feature point error detection unit. [Figure 8] This shows the positional relationship between the device coordinate system and the field coordinate system after adjustment so that the height axes of the device coordinate system and the field coordinate system are parallel. [Figure 9] 1 is a diagram showing the shooting position of a camera, a captured image, and a display target field in a device coordinate system and a field coordinate system. [Figure 10] FIG. 13 is a diagram showing the results of clustering the six generated candidate fields in the device coordinate system. [Figure 11] 13 is a diagram showing an estimated position of a field to be displayed in a device coordinate system, which is obtained by an optimization process based on feature extraction positions excluding erroneous feature extraction positions. FIG. [Figure 12] 1A is a diagram showing, on a field estimation plane, feature extraction positions together with candidate label numbers when the field to be displayed is a tennis court, and FIG. 1B is a diagram showing, on a field estimation plane, feature extraction positions and candidate label pairs after the candidate labels are corrected by a first correction unit. [Figure 13] 1A is a diagram showing, on a field estimation plane, feature extraction positions together with candidate label numbers when the field to be displayed is a tennis court, and FIG. 1B is a diagram showing, on a field estimation plane, sets of feature extraction positions and candidate labels after the second correction unit corrects the candidate labels. [Figure 14]FIG. 13 is a diagram showing the correspondence between structural feature points and labels when a pool with a long float in the center is used as a learning field and a display target field. [Figure 15] 10 is an example of a flowchart illustrating an overview of a virtual object display process executed by a control unit in the first embodiment. [Figure 16] 10 is a flowchart illustrating an example of a detailed procedure of a calibration process. [Figure 17] 13 is an example of a flowchart illustrating details of a feature point error detection process executed in step S22. [Figure 18] 13 is an example of a flowchart illustrating details of a first label correction process. [Figure 19] 13 is an example of a flowchart illustrating details of a second label correction process. [Figure 20] 13 shows the configuration of a display system in a second embodiment. [Figure 21] FIG. 11 is a block diagram of a server device according to a second embodiment. [Figure 22] 13 is an example of a flowchart illustrating a processing procedure executed by a control unit of the server device in the second embodiment. [Diagram 23] 13 shows a schematic configuration of an information processing device according to a third embodiment. [Figure 24] 13 is an example of a flowchart according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of an information processing device, a control method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) Outline configuration FIG. 1 is a schematic diagram of a display device 1 according to a first embodiment. The display device 1 is a device that can be worn by a user, and is, for example, a see-through type configured as glasses, and is configured to be wearable on the user's head. The display device 1 realizes augmented reality (AR) by displaying visual information superimposed on a real landscape when watching a sports game or a play (including a concert). The visual information is a virtual object represented in two or three dimensions, and will hereinafter be referred to as a "virtual object." The display device 1 may display a virtual object to only one eye of the user, or may display a virtual object to both eyes.

[0013] In this embodiment, it is assumed that there is a place or structure (hereinafter also referred to as a "field") where sports, plays, etc. are held, and the display device 1 superimposes virtual objects, which serve as additional information to assist the user when watching a sport or a play, on the field or its periphery. Virtual objects include, for example, a score board displayed above a tennis court in the case of tennis, a world record line superimposed in real time on the pool during a swimming race in the case of a swimming competition, and virtual performers superimposed on the stage in a play or the like.

[0014] The field has a plurality of structural (i.e., shape-specific) feature points (also called "structural feature points"). Examples of the field include fields that are of interest in watching sports (e.g., tennis courts, competitive swimming pools, stadiums, etc.) and fields that are of interest in watching plays (e.g., theaters, concert halls, multipurpose halls, various stages, etc.). The field functions as a reference object in the calibration of the display device 1.

[0015] In addition, in this embodiment, even when the specifications of a field displayed on an image used in the learning stage of the feature extraction process for extracting structural feature points on an image differ partially from those of a field displayed on an image captured by the display device 1 in the display stage by the display device 1, an accurate calibration process is performed. Hereinafter, a field displayed on an image used in the learning stage of the feature extraction process is called a "learning field", and a field displayed on an image captured by the display device 1 in the calibration process is called a "display target field". As described later, the display target field is a field used for the same purpose as the learning field, includes at least structural feature points possessed by the learning field, and optionally includes structural feature points not present in the learning field. For example, the learning field is a field corresponding to one type of sport, and the display target field is a field corresponding to multiple types of sports. Other examples will be described later with reference to FIG. 4(A) and FIG. 4(B). The learning field and the display target field may be fields of the same specifications. In this case, for example, feature points that were not labeled in the learning stage of the feature extraction process are labeled in the calibration process. This example will be described later with reference to Figures 3(A), 3(B) and 14. Note that two fields having the same specifications means that the two fields have the same number of corresponding structural feature points.

[0016] Hereinafter, a set of structural feature points of a field labeled in the learning stage of the feature extraction process will be referred to as the "first structural feature point group." Additionally, a set of structural feature points of a field that was not subject to labeling in the learning stage but will be subject to labeling in the calibration process will be referred to as the "second structural feature point group." Additionally, hereinafter, the first structural feature point group will be referred to as having "N1" structural feature points (N1 is an integer equal to or greater than 2) and the second structural feature point group will be referred to as having "N2" structural feature points (N2 is an integer equal to or greater than 1).

[0017] The display device 1 includes a light source unit 10, an optical element 11, a communication unit 12, an input unit 13, a storage unit 14, a camera 15, a position and orientation detection sensor 16, and a control unit 17.

[0018] The light source unit 10 has a light source such as a laser light source or an LCD (Liquid Crystal Display) light source, and emits light based on a drive signal supplied from the control unit 17. The optical element 11 has a predetermined transmittance, transmits at least a part of external light to make it incident on the user's eyeball, and reflects at least a part of the light from the light source unit 10 toward the user's eyeball. As a result, a virtual image corresponding to a virtual object formed by the display device 1 is superimposed on the scenery and is visually recognized by the user. The optical element 11 may be a half mirror whose transmittance and reflectance are approximately equal, or may be a mirror whose transmittance and reflectance are not equal (so-called beam splitter).

[0019] The communication unit 12 transmits and receives data to and from external devices under the control of the control unit 17. For example, when a user uses the display device 1 to watch a sporting event or a play, the communication unit 12 receives information about virtual objects to be displayed by the display device 1 from a server device managed by a promoter under the control of the control unit 17.

[0020] The input unit 13 generates an input signal based on a user's operation and transmits it to the control unit 17. The input unit 13 is, for example, a button, a cross key, a voice input device, or the like that allows the user to give instructions to the display device 1.

[0021] The camera 15 generates an image of an area in front of the display device 1 under the control of the control unit 17, and supplies the generated image (also called a “captured image Im”) to the control unit 17.

[0022] The position and orientation detection sensor 16 is a sensor (sensor group) that detects the position and orientation (direction) of the display device 1, and includes, for example, a positioning sensor such as a Global Positioning Satellite (GPS) receiver, and an orientation detection sensor that detects a change in the relative orientation of the display device 1, such as a gyro sensor, an acceleration sensor, and an Inertial Measurement Unit (IMU). The position and orientation detection sensor 16 supplies the generated detection signal related to the position and orientation of the display device 1 to the control unit 17. As will be described later, the control unit 17 detects the amount of change in the position and orientation of the display device 1 from the start-up or the like based on the detection signal supplied from the position and orientation detection sensor 16. Note that, instead of detecting the position of the display device 1 from the positioning sensor, the control unit 17 may specify the position of the display device 1 based on, for example, a signal received from a beacon terminal or a wireless LAN device installed in the venue. In another example, the control unit 17 may specify the position of the display device 1 based on a known position estimation technique using an AR marker. In these cases, the position and orientation detection sensor 16 may not include a positioning sensor.

[0023] The control unit 17 includes a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a volatile memory that functions as a working memory for the processor, and performs overall control of the display device 1.

[0024] For example, the control unit 17 performs a calibration process for associating the real world space with the space recognized by the display device 1 based on the structural feature points of the display target field recognized from the captured image Im at the timing of displaying the virtual object. In this calibration process, the control unit 17 generates coordinate conversion information for converting a coordinate system of a three-dimensional space based on the display device 1 (also called a "device coordinate system") into a coordinate system of a three-dimensional space based on the display target field (also called a "field coordinate system"). The device coordinate system is an example of a "first coordinate system", and the field coordinate system is an example of a "second coordinate system". The calibration process will be described in detail later. Then, the control unit 17 generates a drive signal for driving the light source unit 10 based on the above-mentioned coordinate conversion information, and supplies the drive signal to the light source unit 10 to cause the optical element 11 to emit light for displaying a virtual object on the light source unit 10 (also called a "display light"). As a result, the control unit 17 allows the user to visually recognize the virtual object.

[0025] The storage unit 14 is a non-volatile memory that stores various information required for the control unit 17 to control the display device 1. The storage unit 14 may include a removable storage medium such as a flash memory. The storage unit 14 also stores programs executed by the control unit 17.

[0026] The storage unit 14 further includes a sensor data storage unit 20, a parameter storage unit 21, a first structure data storage unit 22, and a second structure data storage unit .

[0027] The sensor data storage unit 20 stores the captured image Im generated by the camera 15 in association with the amount of change in the position and orientation of the display device 1 from the time when the device coordinate system was set (for example, when the display device 1 was started) when the captured image Im was generated. In this case, for example, the control unit 17 constantly calculates the amount of change in the current position and orientation based on the position and orientation when the device coordinate system was set, based on the detection signal of the position and orientation detection sensor 16. Then, when storing the captured image Im generated by the camera 15 in the sensor data storage unit 20, the control unit 17 stores the amount of change in position and orientation Ap calculated when the captured image Im was generated in association with the captured image Im in the sensor data storage unit 20. For example, the control unit 17 stores a combination of the captured image Im and the amount of change in position and orientation Ap for the latest predetermined time or number of times in the sensor data storage unit 20. The information stored in the sensor data storage unit 20 is used in the calibration process.

[0028] The parameter storage unit 21 stores parameters of an inference device (also referred to as a "feature extractor") used when extracting position information of structural feature points of a field to be displayed and classification information of the structural feature points from a captured image Im in a calibration process. The feature extractor described above is a learning model that is trained to output position information of structural feature points in an image for each classification (i.e., label) of a structural feature point to be extracted when an image is input. The position information described above may be map information on an image indicating the reliability of each coordinate value of a structural feature point, or may be coordinate values ​​indicating the position of a structural feature point in an image in pixel or sub-pixel units. When the feature extractor described above outputs map information for each classification of structural feature points, for example, the top m (m is an integer equal to or greater than 1) coordinate values ​​whose reliability exceeds a certain threshold for each classification of structural feature points are adopted as the positions of the structural feature points. The integer m is "1" in a label definition example in which a unique label is assigned to each structural feature point (see Figs. 3(A), 3(B), 4(A), and 4(B) described later), and is "2" in a label definition example in which the same label is assigned to symmetrical structural feature points (see Figs. 5(A) and 5(B) described later). The learning model used for training the feature extractor may be a learning model based on a neural network, or may be another type of learning model such as a support vector machine, or may be a combination of these. For example, when the above-mentioned learning model is a neural network such as a convolutional neural network, the parameter storage unit 21 stores various parameters such as the layer structure, the neuron structure of each layer, the number of filters and the filter size in each layer, and the weight of each element of each filter. Also, as described above, an image representing a learning field is used as an image for training the feature extractor.

[0029] Furthermore, the parameter storage unit 21 stores parameters related to the camera 15 necessary for displaying a virtual object, such as the focal length, internal parameters, principal point, and size information of the captured image Im of the camera 15.

[0030] The first structure data storage unit 22 stores the first structure data. The first structure data includes information related to the structures of the N1 structural feature points belonging to the first structural feature point group. Specifically, the first structure data includes at least a label indicating the classification (class) of each structural feature point belonging to the first structural feature point group, and registered position information indicating the position in the field coordinate system of each structural feature point belonging to the first structural feature point group. The registered position information is coordinate information represented by the field coordinate system, and is set, for example, so that the position of any one of the structural feature points is the origin.

[0031] The second structure data storage unit 23 stores the second structure data. The second structure data includes information related to the structures of N2 structural feature points belonging to the second structural feature point group. Specifically, the second structure data includes at least a label indicating the classification of each structural feature point belonging to the second structural feature point group, and registered position information indicating the position in the field coordinate system of each structural feature point belonging to the second structural feature point group. Note that the length between any two structural feature points in the first structural feature point group and the second structural feature point group can be calculated based on the coordinate values ​​indicated by the registered position information. The first structure data and the second structure data are used in the calibration process.

[0032] The storage unit 14 may store various information in addition to the above-mentioned information. For example, the storage unit 14 may include information on the size of the field to be displayed, information specifying a structural feature point that is the origin in the field coordinate system, and information specifying each direction of the three axes of the field coordinate system. The storage unit 14 may also store anisotropy information on the anisotropy of the displacement of the structural feature point of the field over time. The anisotropy information is, for example, information indicating a direction in which the structural feature point of the field is likely to be displaced or / and a direction in which the structural feature point of the field is unlikely to be displaced (and the degree of the displacement) when the structural feature point of the field is displaced over time. These directions may be represented by vectors in the field coordinate system, for example, or may be specified based on the positional relationship between the structural feature points, such as "the direction from structural feature point A to structural feature point B."

[0033] The configuration of the display device 1 shown in FIG. 1 is an example, and various modifications may be made to this configuration. For example, the display device 1 may further include a speaker that outputs sound based on the control of the control unit 17. The display device 1 may also include a gaze detection camera for determining whether or not a virtual object is displayed and for changing the display position of the virtual object according to the position of the user's gaze. In yet another example, the storage unit 14 may not include the sensor data storage unit 20. In this case, the control unit 17 performs a calibration process using the captured image Im instantly acquired from the camera 15 and the position and orientation change amount Ap calculated based on the detection signal of the position and orientation detection sensor 16.

[0034] In yet another example, the display device 1 may not need to detect the position of the display device 1 using the position and orientation detection sensor 16 or the like. In general, the user rarely moves while watching a sporting event, a play, or the like, and a change in the position of the display device 1 has a smaller effect on the display of a virtual object than a change in the orientation of the display device 1. In consideration of the above, the position and orientation detection sensor 16 may be configured with a sensor that detects the orientation of the display device 1, and the control unit 17 may calculate only the amount of change in the orientation of the display device 1 from the time the device coordinate system was set as the position and orientation change amount Ap.

[0035] (2) Functional Blocks Fig. 2 is a block diagram showing the functional configuration of the control unit 17. As shown in Fig. 2, the control unit 17 functionally includes a virtual object acquisition unit 41, a feature extraction unit 42, a coordinate conversion information generation unit 43, a reflection unit 44, and a light source control unit 45. Note that in Fig. 2, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to that shown in Fig. 2. The same applies to other functional block diagrams described later.

[0036] The virtual object acquisition unit 41 acquires information (also called "designated display information Id") that designates a virtual object to be superimposed on a landscape and its display position. This virtual object may be information for drawing a two-dimensional object (two-dimensional drawing information) or information for drawing a three-dimensional object (three-dimensional drawing information). For example, when a server device managed by an exhibitor and the display device 1 are capable of communicating with each other, the virtual object acquisition unit 41 acquires, as the designated display information Id, distribution information that is distributed by push type or pull type at a predetermined timing from the server device. In this case, the designated display information Id includes information that designates a display position (for example, information indicating coordinate values ​​in a field coordinate system) in addition to the virtual object. In another example, information indicating a combination of a virtual object, a display position, and a display condition thereof may be stored in advance in the storage unit 14. In this case, when the virtual object acquisition unit 41 determines that the stored display condition is satisfied, the virtual object acquisition unit 41 acquires, as the designated display information Id, a combination of a virtual object and a display position corresponding to the satisfied display condition.

[0037] The feature extraction unit 42 generates feature point information "IF" based on the captured image Im acquired from the sensor data storage unit 20. In this case, the feature extraction unit 42 inputs the captured image Im (e.g., the latest captured image Im) to a feature extractor configured based on parameters extracted from the parameter storage unit 21, and generates feature point information IF from information output by the feature extractor. For example, in this case, the feature extractor outputs the position (e.g., coordinate value) of the structural feature point on the input captured image Im for each label indicating the classification of the structural feature point, and the feature extraction unit 42 generates feature point information IF indicating a combination of the position on the captured image Im output by the feature extractor (also called "feature extraction position") and a candidate label corresponding to the position (also called "candidate label"). Note that, when the feature extractor outputs a normalized value that does not depend on the image size as the coordinate value of the structural feature point, the feature extraction unit 42 calculates the feature extraction position by multiplying the coordinate value by the image size of the captured image Im.

[0038] The coordinate conversion information generating unit 43 generates coordinate conversion information "Ic" between the device coordinate system and the field coordinate system based on the first structure data and the second structure data, the feature point information IF, the position and orientation change amount Ap at the time of generating the captured image Im from which feature extraction has been performed, and the parameters of the camera 15. The coordinate conversion information Ic is, for example, a combination of a rotation matrix and a translation vector that are generally used to perform coordinate conversion between three-dimensional spaces. Note that the coordinate conversion information Ic is not limited to information used when converting the field coordinate system to the device coordinate system, and may be information used when converting the device coordinate system to the field coordinate system. Here, the rotation matrix and the translation vector for converting from the field coordinate system to the device coordinate system can be converted into a rotation matrix (the inverse matrix of the above-mentioned rotation matrix) and a translation vector (the above-mentioned translation vector with the sign inverted) for converting from the device coordinate system to the field coordinate system.

[0039] The coordinate conversion information generating unit 43 also has a field estimation unit 46. The field estimation unit 46 estimates the position of the field to be displayed in the device coordinate system. The estimation method by the field estimation unit 46 will be described later. The coordinate conversion information generating unit 43 generates coordinate conversion information Ic based on the position of the field to be displayed in the device coordinate system estimated by the field estimation unit 46 and the position of the field to be displayed in the field coordinate system based on the first structure data and the second structure data.

[0040] The reflecting unit 44 reflects the coordinate conversion information Ic supplied from the coordinate conversion information generating unit 43 on the designated display information Id supplied from the virtual object acquiring unit 41, thereby generating a display signal “Sd” indicating a virtual object to be projected onto the optical element 11. In this case, the reflecting unit 44 matches the device coordinate system with the field coordinate system using the coordinate conversion information Ic, and then generates a display signal Sd based on the designated display information Id. The light source control unit 45 generates a drive signal that indicates the drive timing and light amount for driving the light source of the light source unit 10 (for example, each light source corresponding to RGB) based on the display signal Sd supplied from the reflecting unit 44, and supplies the generated drive signal to the light source unit 10.

[0041] Note that the explanation of each process (i.e., the process of the reflection unit 44 and the light source control unit 45) after the calibration is completed (i.e., after the coordinate conversion information Ic is calculated) is just an example, and a virtual object may be displayed by superimposing the virtual object at a desired landscape position using any method applied in existing AR products, etc. For example, examples of documents disclosing such technology include JP2015-116336A and JP2016-525741A. As shown in these documents, the display device 1 detects the user's line of sight, and performs control so that the virtual object is appropriately viewed.

[0042] Each of the components of the virtual object acquisition unit 41, the feature extraction unit 42, the coordinate conversion information generation unit 43, the reflection unit 44, and the light source control unit 45 described in FIG. 2 can be realized, for example, by the control unit 17 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. Note that at least a part of each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of each of these components may be realized by using an integrated circuit programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Also, at least a part of each of the components may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each of the components may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0043] (3) Example of label definition for structural feature points Fig. 3(A) shows an example of label definition of a first structural feature point group of a training field, which is a tennis court, and Fig. 3(B) shows an example of label definition of a first structural feature point group and a second structural feature point group of a display target field, which is a tennis court. In Fig. 3(A) and Fig. 3(B), circles and corresponding label numbers are clearly indicated at the positions of the structural feature points. In this example, the training field and the display target field have the same structural feature points, but the feature extractor has been trained to extract only a portion of these structural feature points (here, only the structural feature points of the singles court).

[0044] As shown in Fig. 3(A), the first structural feature point group that the feature extractor has learned to extract is related to the singles court and there are 10 of them, and these structural feature points are labeled with consecutive numbers from "0" to "9" starting from the corners. On the other hand, as shown in Fig. 3(B), the second structural feature point group includes four structural feature points (here, the four corners of the doubles court) that were not included in the first structural feature point group, and these structural feature points are labeled with numbers from "o0" to "o3" starting from the corners.

[0045] FIG. 4(A) shows an example of label definition of the first structural feature point group of the learning field which is a swimming pool, and FIG. 4(B) shows an example of label definition of the first structural feature point group and the second structural feature point group of the display target field which is a swimming pool. In FIG. 4(A) and FIG. 4(B), circles and corresponding label numbers are clearly indicated at the positions of the structural feature points. When the field is a swimming pool, the centers of floats of a predetermined color that are provided at a predetermined distance interval on the rope that divides the course are selected as the structural feature points. In this example, the learning field is an 8-lane pool, whereas the display target field is a 10-lane pool. Therefore, the display target field has more floats and more structural feature points because it has more lanes than the learning field. There are 25 structural feature points in the first structural feature point group, and these structural feature points are labeled with serial numbers from "0" to "24". On the other hand, the second structural feature point group includes six structural feature points corresponding to the floats of the lanes at both ends that were not included in the first structural feature point group, and these structural feature points are labeled with numbers from "o0" to "o5."

[0046] Here, the effect of defining the structural feature points of the second structural feature point group will be described.

[0047] Each structural feature point of the second structural feature point group is generally arranged in the vicinity of the first structural feature point group, and may be erroneously extracted by the feature extractor as a structural feature point of the first structural feature point group due to the similarity of the structure between the learning field and the display target field, etc. In this case, the structural feature points of the first structural feature point group that should be extracted by the feature extractor are not detected, and the number of structural feature points used to estimate the position of the display target field in the device coordinate system is reduced. As a result, the estimation accuracy decreases and the estimation time increases. This tendency is noticeable when setting an upper limit (m) on the number of estimated structural feature points, and there is a tendency for structural feature points of the first structural feature point group that exist around the erroneously extracted position to go undetected. On the other hand, retraining the feature extractor for all display target fields requires a large data cost, and it is not realistic to deal with various specifications in the learning stage that requires a huge amount of data.

[0048] Taking the above into consideration, in this embodiment, the display device 1 corrects the candidate labels output by the feature extractor by using the second structure data related to other structural feature points of the field to be displayed that are not defined as the first structural feature point group. This enables the display device 1 to accurately estimate the position of the field to be displayed in the device coordinate system by using the structural feature points of the second structural feature point group that have been erroneously extracted by the feature extractor as structural feature points of the first structural feature point group.

[0049] In the examples of Figures 3(A), 3(B), 4(A), and 4(B), label definitions are used that uniquely determine the label name and the position of the structural feature point, but examples of label definitions are not limited to this. For example, label definitions may be used that assign the same label to symmetrical structural feature points. Hereinafter, a label definition that uniquely determines the label name and the position of a structural feature point is referred to as a "first label definition," and a label definition that assigns the same label to symmetrical structural feature points is referred to as a "second label definition."

[0050] Fig. 5(A) is a diagram showing the correspondence between structural feature points and labels when labels determined by the second label definition are assigned to each structural feature point of the field to be displayed shown in Fig. 3(B). In this example, the same label is assigned to symmetrical structural feature points, and as a result, the labels 0 to 4 and o0 to o1 are assigned to two structural feature points at symmetrical positions. Fig. 5(B) is a diagram showing the correspondence between structural feature points and labels when labels determined by the second label definition are assigned to each structural feature point of the field to be displayed shown in Fig. 4(B). In this example, the same label is assigned to symmetrical structural feature points, and as a result, the labels 0 to 12 and o0 to o2 are assigned to two structural feature points at symmetrical positions.

[0051] (4) Field Estimation Processing (4-1) overview Fig. 6 is a functional block diagram of the field estimation unit 46 showing an overview of the processing of the field estimation unit 46. As shown in Fig. 6, the field estimation unit 46 functionally has a feature point error detection unit 51, a first field estimation unit 52, a label correction unit 53, and a second field estimation unit 54. Although not shown in Fig. 6, each component of the field estimation unit 46 performs each process by referring to the position and orientation change amount Ap stored in the sensor data storage unit 20 and the camera parameters (including internal parameters and the size of the captured image Im) stored in the parameter storage unit 21.

[0052] The feature point error detection unit 51 detects a feature extraction position (also referred to as an "error feature extraction position") that indicates an erroneous extraction result of a structural feature point (more specifically, a structural feature point of the first structural feature point group) based on the feature point information IF supplied from the feature extraction unit 42 and the first structure data stored in the first structure data storage unit 22. In this case, the feature point error detection unit 51 may perform a process of making the vertical axis of the field coordinate system and the device coordinate system parallel to each other based on the output signal of the acceleration sensor 160 included in the position and orientation detection sensor 16. Then, the feature point error detection unit 51 supplies the detection result of the erroneous feature extraction position to the first field estimation unit 52 and the label correction unit 53.

[0053] The first field estimation unit 52 estimates the position of the field to be displayed in the device coordinate system, based on the feature point information IF, the detection result of the error feature extraction position by the feature point error detection unit 51, and the first structure data stored in the first structure data storage unit 22. In this case, the first field estimation unit 52 may perform a process of making the vertical axes of the field coordinate system and the device coordinate system parallel to each other, based on the output signal of the acceleration sensor 160.

[0054] The estimated position of the field to be displayed in the device coordinate system calculated by the first field estimation unit 52 is a provisional estimated position for use in processing by the label correction unit 53, and the final estimated position of the field to be displayed is determined by the second field estimation unit 54. In other words, the first field estimation unit 52 generates information indicating a first estimation result of the field to be displayed in the device coordinate system (also referred to as "first field estimation information"), and the second field estimation unit 54 generates information indicating a second estimation result of the field to be displayed in the device coordinate system (also referred to as "second field estimation information"). The first field estimation unit 52 supplies the first field estimation information indicating the estimated position of the field to be displayed to the label correction unit 53.

[0055] The label correction unit 53 corrects the candidate labels of the feature point information IF based on the first field estimation information supplied from the first field estimation unit 52, the feature point information IF, and the second structure data stored in the second structure data storage unit 23. Here, the label correction unit 53 functionally has a first correction unit 55 and a second correction unit 56.

[0056] The first correction unit 55 detects an error feature extraction position (also called "first label error feature extraction position") that is the position of the correct structural feature point by changing the candidate label to another candidate label that represents a structural feature point of the first structural feature point group, and corrects the candidate label of the first label error feature extraction position. The second correction unit 56 detects an error feature extraction position (also called "second label error feature extraction position") that is the position of the correct structural feature point by changing the candidate label to a candidate label that represents a structural feature point of the second structural feature point group, and corrects the candidate label of the second label error feature extraction position. Then, the label correction unit 53 supplies feature point information IF obtained by correcting the candidate labels of the first label error feature extraction position and the second label error feature extraction position to the second field estimation unit 54. The feature point information IF supplied by the label correction unit 53 to the second field estimation unit 54 is an example of "corrected feature point information".

[0057] The processing of the first correction unit 55 and the processing of the second correction unit 56 may be parallel processing, the processing of the second correction unit 56 may be performed after the processing of the first correction unit 55, or the processing of the first correction unit 55 may be performed after the processing of the second correction unit 56. Furthermore, the first correction unit 55 and the second correction unit 56 may perform mutually dependent processing, and each may take into consideration the detection result of the other's error feature extraction position and finally determine the correction of the candidate label. In this case, for example, when the first correction unit 55 and the second correction unit 56 detect the same error feature extraction position as the first label error feature extraction position and the second label error feature extraction position, they may exchange processing results with each other so as to reflect the correction of either one of them.

[0058] The second field estimation unit 54 generates second field estimation information representing an estimated position of the field to be displayed, based on the feature point information IF reflecting the correction by the label correction unit 53, the first structure data stored in the first structure data storage unit 22, and the second structure data stored in the second structure data storage unit 23. Here, the estimated position of the field to be displayed calculated by the second field estimation unit 54 reflects the respective processing results of the feature point error detection unit 51, the first field estimation unit 52, and the label correction unit 53, and becomes the estimated position of the field to be displayed that is finally output by the field estimation unit 46. Then, based on the second field estimation information output by the second field estimation unit 54, coordinate conversion information Ic is generated.

[0059] (4-2) Minutiae Error Detection 7 is an example of a functional block of the feature point error detection unit 51. Functionally, the feature point error detection unit 51 has an axis adjustment unit 510, a candidate field generation unit 511, an estimated field determination unit 512, and an erroneous label detection unit 513.

[0060] The axis adjustment unit 510 recognizes the vertical direction in the device coordinate system based on the output signal of the acceleration sensor 160. Then, based on the recognized vertical direction, the axis adjustment unit 510 adjusts the height direction axes of the device coordinate system and the field coordinate system so that they are parallel to each other.

[0061] The candidate field generating unit 511 executes a process of generating candidates (also called "candidate fields") representing the field to be displayed in the device coordinate system multiple times using two points out of the feature extraction positions indicated by the feature point information IF. In this case, the candidate field generating unit 511 may generate n candidate fields from n sets (pairs) of feature extraction positions randomly selected n times (n is an integer equal to or greater than 3), or may generate candidate fields from all possible sets of feature extraction positions. Note that the candidate field generating unit 511 identifies, for example, the estimated positions of structural feature points in the device coordinate system for each label and the estimated surface of the field to be displayed in the device coordinate system as the candidate fields.

[0062] Here, a supplementary explanation will be given on the case where a candidate field is generated from all combinations of feature extraction positions. In the case of the first label definition in which the label name and the position of the structural feature point are uniquely determined, if the number of feature extraction positions is N1, the candidate field generation unit 511 generates the feature extraction positions by N1 From a set of two C N1 On the other hand, in the case of a label definition in which the label name and the position of the structural feature point are not uniquely determined, multiple field estimations are required from one pair. For example, in the case of the second label definition example (i.e., a label definition in which point-symmetric labels are made the same), the candidate field generation unit 511 generates a total of N1 Create C2 × 2 candidate fields. In general, in a label definition example where "M" (M is an integer of 3 or more) possible feature point positions are possible for one label, the minimum estimated number of candidate fields when N1 structural feature points are given will differ depending on the symmetry of the label definition, etc. Specifically, if processing efficiency is not taken into consideration, the minimum estimated number of candidate fields is: N1 The result is C2×M×M.

[0063] The estimated field determination unit 512 determines, from among multiple candidate fields, a display target field in the device coordinate system (also referred to as an "estimated field") that is to be finally output as an estimation result. In this case, the estimated field determination unit 512 performs clustering of the candidate fields, and determines an estimated field from the candidate fields that belong to the main cluster to which the largest number of candidate fields belong. In other words, the estimated field determination unit 512 determines the estimated positions of the structural feature points of each label of the display target field in the device coordinate system (and the estimated surface of the display target field).

[0064] The error label detection unit 513 determines the error feature extraction position based on the estimated field determined by the estimated field determination unit 512. In this case, for example, the error label detection unit 513 recognizes the point where each feature extraction position is projected onto the estimated field as each feature extraction position in the device coordinate system based on the shooting position in the device coordinate system specified based on the position and orientation change amount Ap and the parameters (including internal parameters) of the camera 15. Then, the error label detection unit 513 compares the positions of the structural feature points of the first structural feature point group in the estimated field with the feature extraction positions in the device coordinate system for each label of the first structural feature point group, and determines the feature extraction positions that are a predetermined distance or more as the error feature extraction positions. The above-mentioned predetermined distance is, for example, a preset compatibility value stored in the storage unit 14 or the like. Furthermore, the field estimation unit 46 ends the estimation when the number of structural feature points that do not correspond to the error feature extraction positions becomes a predetermined threshold value (for example, 5) or less, and determines that the estimation of the display target field has failed. In this case, the field estimation unit 46 executes a field estimation process based on the newly acquired captured image Im.

[0065] Next, the processing of the axis adjustment unit 510 will be specifically described with reference to FIG.

[0066] Figure 8 shows the positional relationship between the device coordinate system and the field coordinate system after adjustment so that the height axes of the device coordinate system and the field coordinate system are parallel. Here, the field to be displayed is a tennis court, and the field coordinate system has axes in the short and long directions of the tennis court and in the perpendicular directions to these. Hereinafter, the three axes of the device coordinate system are referred to as the x-axis, y-axis, and z-axis, and the three axes of the field coordinate system are referred to as the x^-axis, y^-axis, and z^-axis.

[0067] In this case, the axis adjustment unit 510 moves at least one of the device coordinate system or the field coordinate system so that the y axis of the device coordinate system and the y axis of the field coordinate system are parallel to each other based on the output signal of the acceleration sensor 160. For example, if the y^ axis of the field coordinate system is already parallel to the vertical direction, the axis adjustment unit 510 rotates the device coordinate system so that the y axis of the device coordinate system is parallel to the vertical direction. In another example, if the y^ axis of the field coordinate system is not parallel to the vertical direction, the axis adjustment unit 510 rotates both the device coordinate system and the field coordinate system by an appropriate angle so that the y axis of the device coordinate system and the y^ axis of the field coordinate system are parallel to the vertical direction.

[0068] It should be noted that the axis adjustment unit 510 is not an essential component of the feature point error detection unit 51. For example, instead of the feature point error detection unit 51 having the axis adjustment unit 510, the candidate field generation unit 511 may generate a candidate field from three (or four or more) feature extraction positions.

[0069] Next, the process of generating candidate fields by the candidate field generating section 511 will be specifically described with reference to FIG.

[0070] 9 is a diagram showing the shooting position of the camera 15, the captured image Im, and the display target field in the device coordinate system and the field coordinate system. Here, the y axis of the device coordinate system and the y^ axis of the field coordinate system are adjusted to be parallel, and these axes are shifted by an offset "Y". In this case, the candidate field generating unit 511 can determine the candidate field by calculating the offset Y and the angle between the x axis and the x^ axis (or the angle between the z axis and the z^ axis) from two points of any feature extraction position.

[0071] Here, the process of determining a candidate field from two arbitrary feature extraction positions will be further explained with continued reference to FIG.

[0072] In FIG. 9, the candidate field generating unit 511 generates a candidate field from two feature extraction positions of label number 4 and label number 5 based on the first label definition example shown in FIG. 3(A). The distance between the two feature extraction positions of label number 4 and label number 5 (see arrow 90) can be specified by referring to the first structure data. Then, the candidate field generating unit 511 determines a candidate field based on the shooting position (including the shooting direction) of the camera 15 in the device coordinate system, the parameters of the camera 15, the feature extraction positions in the captured image Im of label number 4 and label number 5, and the distance indicated by the arrow 90. Note that the shooting position (including the shooting direction) of the camera 15 in the device coordinate system is specified based on the position and orientation change amount Ap.

[0073] In addition, the candidate field generating unit 511 needs to execute the process of determining the candidate field from the set of feature extraction positions multiple times, but this process can be calculated in parallel by matrix processing. Therefore, the candidate field generating unit 511 can calculate the candidate field for each of the multiple sets of generated feature extraction positions by parallel processing.

[0074] Next, a supplementary explanation will be given of the clustering process performed by the estimation field determination unit 512.

[0075] The estimated field determination unit 512 performs clustering of the candidate fields by using any clustering method. Such clustering methods include, but are not limited to, the single link method, the complete link method, the group average method, the Ward method, the centroid method, the weighted method, and the median method. In this case, the estimated field determination unit 512 performs the above-mentioned clustering based on, for example, the four-dimensional coordinate value (vector) of the three-dimensional center of gravity coordinate in the device coordinate system of each candidate field and the angle representing the orientation on the xz plane (the orientation in the long direction or the short direction in the case of a tennis court). As the above-mentioned "three-dimensional center of gravity coordinate in the device coordinate system of the candidate field", for example, the center of gravity coordinate of the estimated position in the device coordinate system of the structural feature point of all the labels constituting the candidate field is calculated. Note that, when the axis adjustment based on the vertical direction is not performed, the above-mentioned four-dimensional coordinate value becomes the six-dimensional coordinate value of the three-dimensional center of gravity coordinate in the device coordinate system of each candidate field and the Euler angles (yaw, pitch, roll).

[0076] 10 is a diagram showing the result of clustering the six generated candidate fields "C1" to "C6" in the device coordinate system. In this case, the estimated field determination unit 512 classifies the candidate fields into a first cluster CL1, a second cluster CL2, and a third cluster CL3 by using an arbitrary clustering method. Here, the first cluster CL1 is the main cluster, and the four candidate fields C1, C3, C4, and C6, which are the most numerous, belong to it. Meanwhile, the second cluster CL2 is made up of the candidate field C2, and the third cluster CL3 is made up of the candidate field C5.

[0077] In this case, the estimated field determination unit 512 determines that the candidate field belonging to the first cluster CL1, which is the main cluster, is a correct estimation result of the display target field, and determines the estimated field based on the candidate fields C1, C3, C4, and C6. For example, the estimated field determination unit 512 calculates the center of gravity (average vector) of the above-mentioned four-dimensional coordinate values ​​of each of the candidate fields belonging to the main cluster, and selects the candidate field corresponding to the four-dimensional coordinate value closest to the center of gravity as the estimated field. In another example, the estimated field determination unit 512 may determine the estimated field by integrating the candidate fields belonging to the main cluster based on an average (including a weighted average according to the distance from the above-mentioned center of gravity) of the candidate fields belonging to the main cluster or other statistical processing, or an analysis processing such as the least squares method using the positional relationship of the structural feature points specified from the first structural data (i.e., a model of the display target field). In this case, the estimated field determination unit 512 may determine the estimated field by, for example, setting a model or constraint conditions that represent the positional relationship (distance) between each structural feature point of the first structural feature point group indicated by the first structure data, and performing optimization (including analysis by the least squares method, etc.) to minimize the sum of errors between the structural feature points of the estimated field and the point group of feature extraction positions of the candidate field belonging to the main cluster for each label. In yet another example, the estimated field determination unit 512 may determine, as the estimated field, a candidate field randomly selected from the candidate fields belonging to the main cluster. In this way, the estimated field determination unit 512 determines the estimated field based on the integration or selection of the candidate fields belonging to the main cluster.

[0078] (4-3) First Field Estimation Section Next, the process of the first field estimation section 52 will be described.

[0079] FIG. 11 is a diagram showing, in the device coordinate system, the estimated positions of the display target field obtained by the optimization process based on the feature extraction positions excluding the error feature extraction positions. Here, the display target field has the vertices of each lattice (total of 12 vertices) when the display target field is divided into a lattice shape as the structural feature points of the first structural feature point group. Then, the first field estimation unit 52 estimates the display target field based on the nine feature extraction positions "P2" to "P7", "P10" to "P12" excluding the three error feature extraction positions. In this case, the first field estimation unit 52 specifies the feature extraction positions P2 to P7, P10 to P12 (dashed lines shown in FIG. 11) projected onto the device coordinate system based on the shooting position (including the shooting direction) in the device coordinate system specified based on the position and orientation change amount Ap and the parameters (including the internal parameters) of the camera 15. Then, the candidate field generation unit 511 refers to the first structural data to identify the relative positional relationship (model of the field to be displayed) that indicates the distance between the structural feature points of the first structural feature point group, and determines the feature extraction positions P2 to P7, P10 to P12 and the entire estimated field in the device coordinate system so that the positional relationship between the identified structural feature points is maintained.

[0080] In this way, the first field estimation unit 52 estimates the field to be displayed in the device coordinate system based on at least a portion of the sets of feature extraction positions and candidate labels indicated by the feature point information IF and the first structure data, on the premise that there are no errors in the candidate labels of the feature extraction positions to be used and that distortion caused by the internal parameters of the camera 15, etc., has been adjusted.

[0081] Furthermore, when the structural feature points of the field to be displayed are displaced anisotropically (biased in a certain direction), the first field estimation unit 52 may determine the estimated position of the field to be displayed by performing optimization taking into consideration the anisotropic displacement. Here, an example of a field in which the structural feature points are displaced anisotropically (biased in a certain direction) is the (sports) swimming pool shown in Figures 4(A) and 4(B). A field to be displayed having such properties can be regarded as a quasi-rigid body having properties similar to those of a rigid body.

[0082] In this case, the first field estimation unit 52 first calculates the distances (i.e., errors) in the x-axis direction and the z-axis direction between the estimated positions of the structural feature points of the first structural feature point group to be estimated and the feature extraction position in the device coordinate system for each label, and calculates the sum of the distances in the x-axis direction and the sum of the distances in the z-axis direction. The first field estimation unit 52 then multiplies the calculated sum of the distances in the x-axis direction and the sum of the distances in the z-axis direction by different weighting coefficients (the coefficient corresponding to the x-axis is "ax" and the coefficient corresponding to the y-axis is "ay") to set the sum as an evaluation function. The first field estimation unit 52 then calculates the solution of the estimated position that minimizes the evaluation function, using the relative positional relationship of each structural feature point based on the first structural data as a constraint condition. Note that the above calculation may be performed by weighted least squares method, for example, regarding the constraint condition as a model of the field to be displayed.

[0083] On the other hand, when the structural feature points of the field are not displaced anisotropically, the first field estimation unit 52 performs isotropic optimization. In this case, the first field estimation unit 52 calculates the distance (i.e., error) between the estimated position of the structural feature points of the first structural feature point group and the feature extraction position in the device coordinate system for each label, and calculates the above-mentioned solution of the estimated position that minimizes the evaluation function by using the sum of the distances for each label as an evaluation function. Note that in this case, the first field estimation unit 52 calculates the above-mentioned solution under the constraint that the relative positional relationship of each structural feature point based on the first structure data stored in the first structure data storage unit 22 is satisfied. Note that, for example, the above-mentioned calculation may be performed by the least squares method by regarding the above-mentioned constraint condition as a model of the field to be displayed.

[0084] The above-mentioned optimization method is merely an example, and isotropic or anisotropic optimization may be performed for each coordinate axis based on an arbitrary evaluation function.

[0085] Then, the first field estimation unit 52 supplies the estimation result of the field to be displayed as first field estimation information to the label correction unit 53. Here, the first field estimation information is information indicating, for example, a set of estimated positions and labels of structural feature points of the first structural feature point group in the device coordinate system, and an estimated surface of the field to be displayed in the device coordinate system specified from the estimated positions of these structural feature points (also referred to as a "field estimation surface").

[0086] Note that the process of adjusting the y-axis of the device coordinate system so that it is parallel to the vertical direction (i.e., the normal direction of the field) is not essential, and the first field estimation unit 52 may estimate the field to be displayed by the above-mentioned optimization without performing this process. In this case, the first field estimation unit 52 performs the above-mentioned optimization based on the distance in three-dimensional space between the estimated positions of the structural feature points of the first structural feature point group and the feature extraction positions in the device coordinate system.

[0087] (4-4) Label correction section First, the process of the first correction unit 55 will be specifically described. The first correction unit 55 detects the first label error feature extraction position based on the first field estimation information supplied from the first field estimation unit 52 and the feature point information IF, and corrects the candidate label of the detected first label error feature extraction position. Note that, hereinafter, it is assumed that the distortion caused by the internal parameters of the camera 15, etc., has already been adjusted.

[0088] FIG. 12(A) is a diagram showing feature extraction positions together with candidate labels on a field estimation surface based on the first field estimation information when the field to be displayed is a tennis court. Here, it is assumed that each feature extraction position is converted to a position in the device coordinate system based on the shooting position and parameters of the camera 15 by applying the processing of the first field estimation unit 52 described with reference to FIG. 11 and the like. In FIG. 12(A), as an example, a label based on the first label definition example of the tennis court shown in FIG. 3(A) is assigned to each structural feature point of the first structural feature point group. In FIG. 12(A), the error feature extraction positions are indicated by dashed circles, and the other feature extraction positions are indicated by solid circles. Here, the error feature extraction positions are the feature extraction positions corresponding to the candidate labels 5 and 6.

[0089] FIG. 12B is a diagram showing a set of feature extraction positions and candidate labels after the candidate labels have been corrected by first correction unit 55, on a field estimation plane based on the first field estimation information.

[0090] In this case, the first correction unit 55 compares a set of structural feature points and labels of the first structural feature point group identified based on the first field estimation information with a set of feature extraction positions and candidate labels indicated by the feature point information IF. In this case, the first correction unit 55 calculates, for each feature extraction position indicated by the feature point information IF, a projection position on the field estimation plane indicated by the first field estimation information (for example, feature extraction positions P2 to P7, P10 to P12 in FIG. 11) as each feature extraction position in the device coordinate system. In this case, the first correction unit 55 calculates each feature extraction position in the device coordinate system taking into account distortion caused by internal parameters of the camera 15, etc.

[0091] Then, for labels 0 to 4 and 7 to 9, the first correction unit 55 determines that these feature extraction positions are consistent with the first field estimation information (i.e., match the positions of the structural feature points indicated by the first field estimation information) because the distance between the positions of the structural feature points based on the first field estimation information and the feature extraction positions in the device coordinate system corresponding to the same labels is within a predetermined threshold. The above-mentioned predetermined distance is, for example, a distance that can be regarded as the same position (Euclidean distance), and the matching value is stored in advance in the storage unit 14, etc.

[0092] On the other hand, the first correction unit 55 determines that the distance between the position of the structural feature point based on the first field estimation information and the feature extraction position in the device coordinate system corresponding to the same label is longer than a predetermined threshold for labels 5 and 6. Therefore, in this case, the first correction unit 55 determines that these feature extraction positions are not consistent with the first field estimation information.

[0093] The first correction unit 55 may determine whether or not each feature extraction position is consistent with the first field estimation information based on the detection result of the erroneous feature extraction position by the feature point error detection unit 51. In this case, the first correction unit 55 determines that the erroneous feature extraction position detected by the feature point error detection unit 51 does not match the first field estimation information, and determines that the feature extraction positions other than the erroneous feature extraction position detected by the feature point error detection unit 51 are consistent with the first field estimation information.

[0094] Next, the first correction unit 55 detects, as a first label error feature extraction position, a feature extraction position that is consistent with the first field estimation information by correcting the candidate label to another label of the first structural feature point group among the feature extraction positions that are inconsistent with the first field estimation information (here, the feature extraction positions of candidate labels 5 and 6). In this case, the first correction unit 55 uses the Euclidean distance to detect, as a first label error feature extraction position, a feature extraction position that is inconsistent with the first field estimation information and whose distance to any structural feature point of the first structural feature point group is equal to or smaller than a predetermined threshold. The above-mentioned predetermined threshold may be the same as or different from the threshold used to determine whether each feature extraction position is consistent with the first field estimation information.

[0095] In the example of Figure 12 (B), the first correction unit 55 determines that the feature extraction position of the candidate label 5 is not the first label error feature extraction position, because the distance between the feature extraction position in the device coordinate system of the candidate label 5 and the position of any structural feature point in the first structural feature point group based on the first field estimation information is longer than a predetermined threshold.

[0096] On the other hand, since the distance between the feature extraction position of the candidate label 6 in the device coordinate system and the position of the structural feature point of the label 5 based on the first field estimation information is less than the predetermined threshold, the first correction unit 55 detects the feature extraction position of the candidate label 6 as the first label error feature extraction position. That is, the first correction unit 55 determines that in this case, the error can be corrected by correcting the candidate label to 5 (i.e., it is consistent with the first field estimation information). Therefore, in this case, the first correction unit 55 generates feature point information IF in which the candidate label at the feature extraction position of the candidate label 6 is corrected to 5, as shown in FIG. 12(B). This allows the first correction unit 55 to appropriately correct the error in the feature point information IF.

[0097] In this way, the first correction unit 55 corrects the candidate labels of the first label error feature extraction positions among the feature extraction positions that do not match the first field estimation information. This allows the first correction unit 55 to suitably correct errors in the candidate labels of the feature extraction positions corresponding to the structural feature points of the first structural feature point group.

[0098] Next, the process of the second correction unit 56 will be specifically described. The second correction unit 56 detects the second label error feature extraction position based on the first field estimation information, the feature point information IF, and the second structure data, and corrects the candidate label of the detected second label error feature extraction position. Note that, hereafter, it is assumed that the distortion caused by the internal parameters of the camera 15, etc., has already been adjusted.

[0099] FIG. 13(A) is a diagram showing feature extraction positions, together with candidate labels, on a field estimation surface based on the first field estimation information when the field to be displayed is a tennis court. Here, it is assumed that each feature extraction position has been converted to a position in the device coordinate system based on the shooting position and parameters of the camera 15 by applying the processing of the first field estimation unit 52 described with reference to FIG. 11 and the like. Also, as an example, it is assumed that a label based on the first label definition example of the tennis court shown in FIG. 3(B) is assigned to each structural feature point of the first structural feature point group and the second structural feature point group. In FIG. 13(A), the error feature extraction positions are indicated by dashed circles, and the other feature extraction positions are indicated by solid circles. Here, the error feature extraction positions are the feature extraction positions corresponding to the candidate labels 8 and 9.

[0100] FIG. 13(B) is a diagram showing a set of feature extraction positions and candidate labels after the second correction unit 56 corrects the candidate labels on the field estimation plane based on the first field estimation information. In this case, the second correction unit 56 compares a set of structural feature points and labels of the second structural feature point group specified by the second structure data with a set of feature extraction positions and candidate labels indicated by the feature point information IF. In this case, the second correction unit 56 calculates the projection positions (for example, feature extraction positions P2 to P7, P10 to P12 in FIG. 11) on the field estimation plane indicated by the first field estimation information for each feature extraction position indicated by the feature point information IF as each feature extraction position in the device coordinate system. In this case, the second correction unit 56 calculates each feature extraction position in the device coordinate system taking into account distortion caused by the internal parameters of the camera 15 and the like.

[0101] Then, the second correction unit 56 identifies feature extraction positions that are inconsistent with the first field estimation information (i.e., inconsistent with any of the structural feature points in the first structural feature point group). Here, the second correction unit 56 determines that each feature extraction position of the candidate label 8 and the candidate label 9 is away from any of the structural feature points in the first structural feature point group by a predetermined distance or more and is inconsistent with the first field estimation information. Note that the second correction unit 56 may identify feature extraction positions that are inconsistent with the first field estimation information based on the same processing as the first correction unit 55, or may identify feature extraction positions that are inconsistent with the first field estimation information based on information of the processing result received from the first correction unit 55.

[0102] Next, the second correction unit 56 detects, among the feature extraction positions that are inconsistent with the first field estimation information (here, the feature extraction positions of candidate labels 8 and 9), the feature extraction positions that correspond to any of the structural feature points in the second structural feature point group as second label error feature extraction positions.

[0103] In this case, the second correction unit 56 uses the Euclidean distance to detect a feature extraction position that is inconsistent with the first field estimation information and whose distance to any of the structural feature points of the second structural feature point group is equal to or less than a predetermined threshold as a second label error feature extraction position. The above-mentioned predetermined threshold may be the same as or different from the threshold used to determine whether each feature extraction position is consistent with the first field estimation information. Here, the second correction unit 56 specifies the positions of the structural feature points of the second structural feature point group in the device coordinate system based on the first field estimation information and the second structure data. In this case, the second correction unit 56 specifies the relative positional relationship of the structural feature points of the second structural feature point group to the structural feature points of the first structural feature point group based on the second structure data (and the first structure data).

[0104] In the example of FIG. 13(B), the second correction unit 56 detects the feature extraction position of candidate label 8 as the second label error feature extraction position because the distance between the feature extraction position in the device coordinate system of candidate label 8 and the structural feature point of label o2 in the second structural feature point group is less than or equal to a predetermined threshold. Similarly, the second correction unit 56 detects the feature extraction position of candidate label 9 as the second label error feature extraction position because the distance between the feature extraction position in the device coordinate system of candidate label 9 and the structural feature point of label o3 in the second structural feature point group is less than or equal to a predetermined threshold. Therefore, the second correction unit 56 generates feature point information IF (corrected feature point information) in which the candidate label of the feature extraction position of candidate label 8 is corrected to o2 and the candidate label of the feature extraction position of candidate label 9 is corrected to o3.

[0105] In this way, the second correction unit 56 corrects the candidate labels of the second label error feature extraction positions, among the feature extraction positions that do not match the first field estimation information, where the error can be corrected by changing the candidate labels to labels that represent the structural feature points of the second structural feature point group.

[0106] Here, the effect based on the processing of the second corrector 56 will be additionally explained.

[0107] Each structural feature point of the second structural feature point group is arranged around the first structural feature point group, and there is a possibility that the feature extractor may erroneously extract it as a structural feature point of the first structural feature point group. Even in this case, the second correction unit 56 corrects the candidate label to the label of the second structural feature point group, thereby ensuring a sufficient number of structural feature points used for estimating the position of the display target field in the device coordinate system, and thus enabling highly accurate estimation of the display target field. Even if the specifications of the learning field and the display target field are different, the display target field can be estimated with high accuracy without re-learning the feature extractor. For example, when the learning field shown in FIG. 4(A) and the display target field shown in FIG. 4(B) are used, a feature extractor trained for an 8-lane stadium can be used for a 10-lane stadium without losing accuracy.

[0108] Furthermore, even when the learning field and the field to be displayed have the same specifications, a similar effect can be obtained by defining candidate positions that are erroneously extracted as structural feature points of the first structural feature point group (i.e., frequently occurring erroneous feature extraction positions) as structural feature points of the second structural feature point group.

[0109] FIG. 14 is a diagram showing the correspondence between structural feature points and labels when a pool with a long float in the center is used as a learning field and a display target field. In FIG. 14, the structural feature points (labels 0 to 24) of the first structural feature point group are indicated by solid circles, and the structural feature points (labels o1 to o14) of the second structural feature point group are indicated by dashed circles. In this example, when extracting the center part of a long float as the structural feature points (labels 9 to 15) of the first structural feature point group, the positions of both ends of the float are likely to be erroneously extracted, so the positions of both ends are defined as the structural feature points (labels o1 to o14) of the second structural feature point group. In this way, even if the structural feature points with labels o1 to o14 are erroneously extracted by the feature extractor as the structural feature points with labels 9 to 15, the display device 1 can correct the candidate labels by the second correction unit 56 and use these feature extraction results for field estimation. Therefore, in this case, the display device 1 can ensure a sufficient number of structural feature points to be used for field estimation by correcting the candidate labels to labels corresponding to the structural feature points of the second group of structural feature points, thereby enabling highly accurate field estimation.

[0110] (4-5) Second Field Estimation Unit The second field estimation unit 54 generates the second field estimation information by performing the same processing as the first field estimation unit 52, for example, based on the feature point information IF corrected by the first correction unit 55 and the second correction unit 56. When generating the second field estimation information, the second field estimation unit 54 estimates the field to be displayed based on the feature point information IF in which errors in the candidate labels have been appropriately corrected by the first correction unit 55 and the second correction unit 56, and can therefore suitably generate second field estimation information with higher accuracy than the first field estimation information. In other words, the second field estimation unit 54 can accurately estimate the field to be displayed based on the feature point information IF including information on feature extraction positions that accurately correspond to the structural feature points of the first structural feature point group and the second structural feature point group.

[0111] The field estimation unit 46 may determine whether the field estimation has been successful or unsuccessful based on the amount of error and / or the number of structural feature points used after correction by the label correction unit 53. For example, the field estimation unit 46 stops the field estimation and determines that the field estimation has failed when any of the following (Case A), (Case B), or (Case C) applies.

[0112] (Case A) When the total number “n” of structural feature points used in field estimation by the second field estimation unit 54, including the structural feature points corrected by the label correction unit 53, is equal to or less than a predetermined threshold value, (Case B) When the ratio (n / Np) of the total number of used points n to the number of feature points “Np” obtained by the feature extraction unit 42 (i.e., the number of feature points indicated by the feature point information IF) is equal to or less than a predetermined threshold value, (Case C) When the label of the corrected structural feature and the labels of other structural feature points used for field estimation are collinear in the structural data (e.g., labels 10, 11, 12, and 13 in Figure 4(A)). In other words, when field estimation is performed using only structural feature points that are structurally arranged to be collinear.

[0113] Then, when the field estimation unit 46 corresponds to any one of (Case A), (Case B), and (Case C) and stops the field estimation, it executes a field estimation process based on the newly acquired captured image Im. Note that by stopping the field estimation in the case of (Case C), it is possible to prevent the execution of a field estimation with low accuracy, since the accuracy is significantly lower than other cases when performing field estimation using only structural feature points on the same straight line structurally. Also, the field estimation unit 46 may determine whether or not the estimation process by the second field estimation unit 54 corresponds to (Case A) or (Case B), and determine whether or not the executed field estimation has failed. That is, in this case, similar to the distance comparison with the position of the structural feature point by the label correction unit 53, the field estimation unit 46 may obtain the consistency between the position of the structural feature point obtained by the feature extraction unit 42 and the position of the structural feature point after the field estimation, and determine whether or not the consistent structural feature point corresponds to (Case A) or (Case B).

[0114] (5) Processing Flow FIG. 15 is an example of a flowchart showing an outline of a process related to a display process of a virtual object executed by the control unit 17 in the first embodiment.

[0115] First, the control unit 17 detects the start-up of the display device 1 (step S11). In this case, the control unit 17 sets a device coordinate system based on the attitude and position of the display device 1 at the time of start-up of the display device 1 (step S12). After that, the control unit 17 acquires a captured image Im generated by the camera 15, and acquires a position and orientation change amount Ap based on a detection signal output by the position and orientation detection sensor 16 (step S13). The control unit 17 stores the combination of the captured image Im and the position and orientation change amount Ap acquired in step S13 in the sensor data storage unit 20.

[0116] Then, the control unit 17 determines whether or not there is a request to display a virtual object (step S14). For example, when the virtual object acquisition unit 41 receives distribution information instructing the display of a virtual object from a server device (not shown) managed by the promoter, the virtual object acquisition unit 41 determines that there is a request to display a virtual object. If there is no request to display a virtual object (step S14; No), the control unit 17 continues to acquire the captured image Im and the position and orientation change amount Ap in step S13.

[0117] On the other hand, if there is a request to display a virtual object (step S14; Yes), the control unit 17 executes a calibration process (step S15). The details of the procedure of this calibration process will be described later with reference to FIG.

[0118] Next, the reflection unit 44 of the control unit 17 generates a display signal Sd for displaying the virtual object designated in the display request and the virtual object corresponding to the display position based on the coordinate conversion information Ic obtained by the calibration process in step S15 (step S16). Note that in this case, as in various conventional AR display products, the control unit 17 actually recognizes the space viewed by the user in the device coordinate system by taking into account the user's line of sight and the position and attitude change amount Ap, and generates the display signal Sd so that the virtual object is displayed at the designated position in the space. Then, the light source control unit 45 of the control unit 17 performs emission control of the light source unit 10 based on the display signal Sd (step S17).

[0119] The processing procedure of the flowchart shown in FIG. 15 is an example, and various modifications can be made to this processing procedure.

[0120] For example, the control unit 17 executes the calibration process of step S15 every time a virtual object display request is received, but this is not limited to the above. Alternatively, the control unit 17 may execute the calibration process only when a predetermined time or more has elapsed since the previous calibration process. In this way, the control unit 17 needs to execute the calibration process at least once after the display device 1 is started up.

[0121] Furthermore, the control unit 17 determines the device coordinate system based on the position and orientation of the display device 1 at the time of starting up the display device 1, but this is not limited to the above. Instead of this, for example, the control unit 17 may determine the device coordinate system based on the position and orientation of the display device 1 at the time of the first display request after starting up the display device 1 (i.e., when the calibration process is executed for the first time). In another example, the control unit 17 may reset the device coordinate system based on the position and orientation of the display device 1 at the time of the display request (i.e., when the calibration process is executed) every time a display request is issued. In this case, it is not necessary to use the position and orientation change amount Ap in the process of generating the coordinate conversion information Ic, which will be described later.

[0122] FIG. 16 is an example of a flowchart showing a detailed procedure of the calibration process in step S15 of FIG.

[0123] First, the feature extraction unit 42 of the control unit 17 generates feature point information IF indicating a set of feature extraction positions and candidate labels corresponding to each structural feature point of the first structural feature point group, based on the captured image Im acquired from the sensor data storage unit 20, etc. (Step S21). In this case, the feature extraction unit 42 configures a feature extractor based on parameters acquired from the parameter storage unit 21, and inputs the captured image Im to the feature extractor. Then, the feature extraction unit 42 generates feature point information IF based on information output by the feature extractor.

[0124] Next, the field estimation unit 46 of the coordinate conversion information generation unit 43 executes a feature point error detection process (step S22). Details of the feature point error detection process will be described with reference to FIG.

[0125] Furthermore, the field estimation unit 46 executes a first field estimation process (step S23).

[0126] Then, the field estimation unit 46 executes a first label correction process by the first correction unit 55 and a second label correction process by the second correction unit 56 (step S24). Details of the first label correction process and the second label correction process will be described with reference to Fig. 18 and Fig. 19. The first label correction process and the second label correction process may be performed in any order, and may be performed in parallel.

[0127] Then, the field estimation unit 46 executes a second field estimation process (step S25), thereby generating second field estimation information representing a result of estimating the display target field in the device coordinate system with high accuracy based on the error-corrected feature point information IF.

[0128] Then, the coordinate conversion information generation unit 43 generates coordinate conversion information Ic for conversion from the device coordinate system to the field coordinate system based on the second field estimation information generated by the field estimation unit 46 (step S26). In this case, the coordinate conversion information generation unit 43 matches, for each label of the structural feature point, the detected position in the device coordinate system of each structural feature point indicated by the second field estimation information acquired in step S25 with the position of the structural feature point in the field coordinate system indicated by the registered position information included in the first structural data and the second structural data, and calculates coordinate conversion information Ic such that the matched positions for each label match (i.e., the position error for each label is minimized).

[0129] In this way, the display device 1 compares information extracted from the captured image Im of only structural feature points that have been registered in advance (i.e., labels have already been registered) in the calibration process with information on structural feature points registered in the first structure data and the second structure data. This significantly reduces the amount of calculation required for the comparison process to calculate the coordinate transformation information Ic, and makes it possible to calculate robust coordinate transformation information Ic that is not affected by the extraction of noise contained in the captured image Im (i.e., feature points other than the field to be displayed). Furthermore, in this embodiment, the above-mentioned comparison is performed based on estimated information on structural feature points in which errors in the feature extraction process have been appropriately corrected, making it possible to calculate accurate coordinate transformation information Ic.

[0130] FIG. 17 is an example of a flowchart showing details of the feature point error detection process executed in step S22.

[0131] First, the feature point error detection unit 51 identifies the vertical direction based on the output signal of the acceleration sensor 160, etc., and adjusts the device coordinate system, etc. based on the identified vertical direction so that one axis (y axis and y^ axis) of the device coordinate system and the field coordinate system are parallel (step S31).

[0132] Next, the feature point error detection unit 51 executes a process of generating a candidate field in the device coordinate system based on the first structure data and the selected two feature extraction positions multiple times (step S32). This generates candidate fields the same number of times as the above-mentioned process has been executed. Note that the feature point error detection unit 51 may execute the process of generating candidate fields in parallel.

[0133] Next, the feature point error detection unit 51 performs clustering on the multiple candidate fields generated in step S32 (step S33). In this case, the feature point error detection unit 51 generates one or more clusters for the multiple candidate fields by an arbitrary clustering method.

[0134] Then, the feature point error detection unit 51 determines an estimated field based on the candidate field of the main cluster to which the largest number of candidate fields belong (step S34).The feature point error detection unit 51 then projects each feature extraction position onto the estimated field, and detects an erroneous feature extraction position by comparing the projected feature extraction position with the structural feature point of the estimated field for each label (step S35).

[0135] FIG. 18 is an example of a flowchart showing details of the first label correction process executed in step S24.

[0136] First, the first correction unit 55 specifies a feature extraction position in the device coordinate system based on the first field estimation information generated by the first field estimation unit 52 (step S41). Then, the first correction unit 55 judges whether or not there is a feature extraction position that is inconsistent with the first field estimation information (step S42). In this case, for example, if there is an erroneous feature extraction position detected in step S35, the first correction unit 55 judges that there is a feature extraction position that is inconsistent with the first field estimation information. In another example, the first correction unit 55 judges whether or not there is an erroneous feature extraction position by comparing the position and label of each structural feature point specified from the first field estimation information with the feature extraction position and label in the device coordinate system. Then, if there is no feature extraction position that is inconsistent with the first field estimation information (step S42; No), the first correction unit 55 ends the processing of the flowchart since there is no need to correct the feature point information IF.

[0137] On the other hand, if there is a feature extraction position that is not consistent with the first field estimation information (step S42; Yes), the first correction unit 55 judges whether or not there is a first label error feature extraction position (step S43). Then, if it is judged that there is a first label error feature extraction position (step S43; Yes), the first correction unit 55 corrects the candidate label of the first label error feature extraction position to a label that represents another structural feature point (specifically, the closest structural feature point) of the first structural feature point group (step S44). After that, the first correction unit 55 generates feature point information IF that reflects the correction of the candidate label in step S44. On the other hand, if it is judged that there is no first label error feature extraction position (step S43; No), the first correction unit 55 ends the processing of the flowchart.

[0138] FIG. 19 is an example of a flowchart showing details of the second label correction process executed in step S24.

[0139] First, the second correction unit 56 specifies a feature extraction position in the device coordinate system based on the first field estimation information generated by the first field estimation unit 52 (step S51). Then, the second correction unit 56 judges whether or not there is a feature extraction position that is inconsistent with the first field estimation information (step S52). In this case, for example, if there is an erroneous feature extraction position detected in step S35, the second correction unit 56 judges that there is a feature extraction position that is inconsistent with the first field estimation information. In another example, the second correction unit 56 judges whether or not there is an erroneous feature extraction position by comparing the position and label of each structural feature point specified from the first field estimation information with the feature extraction position and label in the device coordinate system. In yet another example, the second correction unit 56 makes the judgment in step S52 based on the judgment result in step S42 of the first label correction process. If there is no feature extraction position that is inconsistent with the first field estimation information (step S52; No), the second correction unit 56 ends the processing of the flowchart since there is no need to correct the feature point information IF.

[0140] On the other hand, if there is a feature extraction position that is not consistent with the first field estimation information (step S52; Yes), the second correction unit 56 judges whether or not there is a second label error feature extraction position (step S53). If it is judged that there is a second label error feature extraction position (step S53; Yes), the second correction unit 56 corrects the candidate label of the second label error feature extraction position to the label of the corresponding structural feature point (i.e., the closest structural feature point) in the second structural feature point group (step S54). After that, the second correction unit 56 generates feature point information IF that reflects the correction of the candidate label in step S54. On the other hand, if it is judged that there is no second label error feature extraction position (step S53; No), the second correction unit 56 ends the processing of the flowchart.

[0141] If there is a feature extraction position whose candidate label is not corrected in either step S44 or step S54 among the feature extraction positions determined in step S42 or step S52 to be inconsistent with the first field estimation information, the label correction unit 53 considers that the feature extraction position is erroneously estimated. Then, the label correction unit 53 performs processing such as deleting the feature extraction position from the feature point information IF or attaching a flag indicating that it is an erroneous feature extraction position.

[0142] (6) Variations The label correction unit 53 may have only a function corresponding to the second correction unit 56. Even in this case, the label correction unit 53 can ensure a sufficient number of structural feature points used in field estimation by correcting the candidate labels to labels corresponding to the structural feature points of the second structural feature point group, thereby enabling highly accurate field estimation. Furthermore, the field estimation unit 46 does not need to have a function corresponding to the feature point error detection unit 51. In this case, the label correction unit 53 corrects the candidate labels according to the flowchart of FIG. 19 based on first field estimation information indicating the estimation result of the field to be displayed roughly estimated by the first field estimation unit 52.

[0143] <Second embodiment> Fig. 20 shows the configuration of a display system according to the second embodiment. As shown in Fig. 20, the display system according to the second embodiment includes a display device 1A and a server device 2. The second embodiment differs from the first embodiment in that the server device 2 executes a calibration process and the like instead of the display device 1A. Hereinafter, the same components as those in the first embodiment are appropriately designated by the same reference numerals, and the description thereof will be omitted.

[0144] The display device 1A transmits an upload signal "S1" to the server device 2, which is information necessary for the server device 2 to perform a calibration process or the like. In this case, the upload signal S1 includes, for example, a captured image Im generated by the camera 15 and a position and orientation change amount Ap detected based on the output of the position and orientation detection sensor 16. Then, when the display device 1A receives a distribution signal "S2" transmitted from the server device 2, the display device 1A displays a virtual object by controlling the emission of light from the light source unit 10 based on the distribution signal S2. For example, the distribution signal S2 includes information equivalent to the display signal Sd of the first embodiment, and after receiving the distribution signal S2, the display device 1A causes the light source unit 10 to emit light for displaying a virtual object by performing a process similar to that of the light source control unit 45 of the first embodiment.

[0145] Server device 2 is, for example, a server device managed by an exhibitor, and generates distribution signal S2 and distributes distribution signal S2 to display device 1A based on upload signal S1 received from display device 1A. Fig. 21 is a block diagram of server device 2. Server device 2 has input unit 26, control unit 27, communication unit 28, and storage unit 29.

[0146] The storage unit 29 is a non-volatile memory that stores various information required by the control unit 27 to control the server device 2. The storage unit 29 stores a program executed by the control unit 27. The storage unit 29 has a sensor data storage unit 20, a parameter storage unit 21, a first structure data storage unit 22, and a second structure data storage unit 23. The sensor data storage unit 20 stores the captured image Im and the position and orientation change amount Ap included in the upload signal S1 under the control of the control unit 27. The storage unit 29 may be an external storage device such as a hard disk connected to or built into the server device 2, or may be a storage medium such as a flash memory. The storage unit 29 may also be a server device that performs data communication with the server device 2 (i.e., a device that stores information so that it can be referenced from other devices). In this case, the storage unit 29 may be composed of multiple server devices, and the sensor data storage unit 20, the parameter storage unit 21, the first structure data storage unit 22, and the second structure data storage unit 23 may be stored in a distributed manner.

[0147] The control unit 27 has a processor such as a CPU or a GPU, a volatile memory functioning as a working memory, and the like, and performs overall control of the server device 2. The control unit 27 generates information on a virtual object to be displayed as a virtual object and a display position (i.e., information corresponding to the designated display information Id in the first embodiment) based on a user input to the input unit 26, etc. Furthermore, the control unit 27 executes a calibration process by referring to the sensor data storage unit 20, the parameter storage unit 21, the first structure data storage unit 22, and the second structure data storage unit 23, and generates a distribution signal S2. In this way, the control unit 27 includes functions corresponding to the virtual object acquisition unit 41, the feature extraction unit 42, the coordinate conversion information generation unit 43, and the reflection unit 44 shown in FIG. 2.

[0148] FIG. 22 is an example of a flowchart showing a process procedure executed by the control unit 27 of the server device 2 in the second embodiment.

[0149] First, the control unit 27 receives an upload signal S1 including a captured image Im and a position and orientation change amount Ap from the display device 1A via the communication unit 28 (step S61). In this case, the control unit 27 updates the data to be stored in the sensor data storage unit 20 based on the upload signal S1. Then, the control unit 27 determines whether it is time to display a virtual object (step S62). If it is not time to display the virtual object (step S62; No), the control unit 27 continues to receive the upload signal S1 from the display device 1A in step S61.

[0150] On the other hand, if it is the display timing (step S62; Yes), the control unit 27 executes a calibration process based on the latest upload signal S1 received in step S61, etc. In this case, the control unit 27 executes the flowchart shown in FIG. 16. Then, the control unit 27 generates a distribution signal S2 for the display device 1A to display the virtual object based on the coordinate conversion information Ic obtained in the calibration process (step S64). Then, the control unit 27 transmits the generated distribution signal S2 to the display device 1A via the communication unit 28 (step S65). After that, the display device 1A that has received the distribution signal S2 controls the light source unit 10 based on the distribution signal S2 to display the virtual object.

[0151] As described above, according to the second embodiment as well, the display system can accurately calculate the coordinate conversion information Ic required for displaying a virtual object by the display device 1A, and can allow the user to visually recognize the virtual object in a suitable manner.

[0152] In the second embodiment, the calibration process and the like may be performed by the display device 1A instead of by the server device 2. In this case, the display device 1A executes the process of the flowchart shown in Fig. 16 by appropriately receiving information necessary for the calibration process from the server device 2. Even in this mode, the display system can allow the user of the display device 1A to visually recognize the virtual object in a suitable manner.

[0153] <Third embodiment> Fig. 23 shows a schematic configuration of an information processing device 1X in the third embodiment. As shown in Fig. 23, the information processing device 1X mainly includes a structure data acquiring means 46X, a feature point information acquiring means 42X, a label correcting means 53X, and a field estimating means 54X. The information processing device 1X is realized by, for example, the display device 1 or the control unit 17 of the display device 1 in the first embodiment, or the control unit 27 of the server device 2 in the second embodiment. Note that the information processing device 1X may be composed of multiple devices.

[0154] The structure data acquisition means 46X acquires first structure data related to a first feature point group defined as feature points to be extracted for a target field, and second structure data related to a second feature point group other than the first feature point group. The first feature point group can be the first structural feature point group in the first or second embodiment, and the second feature point group can be the second structural feature point group in the first or second embodiment. The structure data acquisition means 46X can be, for example, the field estimation unit 46 in the first or second embodiment.

[0155] The feature point information acquisition means 42X acquires feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence with a feature point of the first feature point group. The "position on the image" may be a feature extraction position in the first or second embodiment. The feature point information acquisition means 42X may receive feature point information generated by a processing block (including a device other than the information processing device 1X) other than the feature point information acquisition means 42X, or the feature point information acquisition means 42X may generate the feature point information. In the latter case, the feature point information acquisition means 42X may be, for example, the feature extraction unit 42 in the first or second embodiment.

[0156] The label correction means 53X specifies a set of positions on the image corresponding to the feature points of the second feature point group and candidate labels based on the second structure data, and generates corrected feature point information by correcting the specified candidate labels to labels indicating the feature points of the second feature point group. The label correction means 53X can be the label correction unit 53 (particularly the second correction unit 56) in the first or second embodiment.

[0157] The field estimation means 54X estimates a field in a coordinate system based on a display device having a camera that captures an image, based on the first structure data, the second structure data, and the corrected feature point information. The field estimation means 54X can be the second field estimation unit 54 in the first or second embodiment.

[0158] FIG. 24 is an example of a flowchart in the third embodiment. The structure data acquisition means 46X acquires first structure data related to a first feature point group defined as feature points to be extracted for a target field, and second structure data related to a second feature point group other than the first feature point group (step S71). The feature point information acquisition means 42X acquires feature point information indicating a pair of a position on an image determined based on an image including at least a part of a field and a candidate label indicating a correspondence between the feature points of the first feature point group (step S72). The label correction means 53X specifies a pair of a position on an image corresponding to a feature point of the second feature point group and a candidate label based on the second structure data, and generates corrected feature point information by correcting the specified candidate label to a label indicating the feature point of the second feature point group (step S73). The field estimation means 54X estimates a field in a coordinate system based on the first structure data, the second structure data, and the corrected feature point information, the coordinate system being determined based on a display device having a camera for capturing an image (step S74).

[0159] According to the third embodiment, the information processing device 1X can appropriately correct feature point information and improve the estimation accuracy of the field in the coordinate system based on the display device.

[0160] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0161] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0162] [Appendix 1] a structure data acquisition means for acquiring first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; a feature point information acquiring means for acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; a label correction means for specifying a set of a position on the image corresponding to a feature point of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the specified candidate label to a label indicating the feature point of the second feature point group; a field estimation means for estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; An information processing device having the above configuration. [Appendix 2] The field estimation means comprises: a first field estimation means for generating first field estimation information indicating a first estimation result of the field in the coordinate system based on the first structure data and the feature point information; a second field estimation means for generating second field estimation information indicating a second estimation result of the field in the coordinate system based on the first structure data, the second structure data, and the corrected feature point information, The information processing device described in Appendix 1, wherein the label correction means identifies a set of positions on the image corresponding to feature points of the second feature point group and candidate labels based on the first field estimation information and the second structure data. [Appendix 3] a feature point error detection means for detecting a pair of a position on the image and the candidate label that does not match any feature point in the first feature point group based on the feature point information and the first structure data; 3. The information processing device according to claim 2, wherein the first field estimation means generates the first field estimation information based on a detection result by the feature point error detection means, the first structure data, and the feature point information. [Appendix 4] The information processing device according to claim 2 or 3, wherein the label correction means corrects the candidate label based on a distance between a projection position of a position on the image projected onto the coordinate system based on the shooting position and parameters of the camera, and a feature point of the second feature point group represented in the coordinate system based on the second structure data and the first field estimation information. [Appendix 5] The information processing device according to any one of Appendices 1 to 4, wherein the field to be displayed and a learning field displayed in an image used for training a feature extractor used to generate the feature point information are fields that have at least the first feature point group as a common feature point, and the field to be displayed is a field that has the first feature point group and the second feature point group as feature points. [Appendix 6] the display target field, which is the field, and a learning field displayed in an image used for learning a feature extractor used to generate the feature point information are fields having the same feature points; The information processing device according to any one of appendix 1 to 4, wherein the second feature point group is a collection of candidate positions that are erroneously extracted as feature points of the first feature point group in generating the feature point information. [Appendix 7] The information processing device according to any one of appendices 1 to 6, further comprising a coordinate conversion information generating means for generating coordinate conversion information regarding coordinate conversion between a first coordinate system, which is the coordinate system, and a second coordinate system, which is a coordinate system adopted in the first structure data and the second structure data, based on a result of the estimation of the field. [Appendix 8] The information processing device is the display device that displays a virtual object superimposed on a landscape, a light source unit that emits display light for displaying the virtual object; an optical element that reflects at least a part of the display light to allow a viewer to visually recognize the virtual object superimposed on a landscape; The information processing device according to any one of claims 1 to 7, further comprising: [Appendix 9] The computer obtaining first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; Control methods. [Appendix 10] obtaining first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; A storage medium that stores a program that causes a computer to execute a process of estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information.

[0163] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0164] 1, 1A display device 1X, 1Y Information processing equipment 2. Server equipment 10 Light source unit 11 Optical elements 12 Communications Department 13 Input section 14 Storage section 15 Camera 16 Position and orientation detection sensor 20 Sensor data storage unit 21 Parameter storage section 22 First structure data storage unit 23 Second structure data storage unit

Claims

1. a structure data acquisition means for acquiring first structure data relating to a first group of feature points defined as feature points to be extracted from a target field, and second structure data relating to a second group of feature points other than the first group of feature points; a feature point information acquiring means for acquiring feature point information indicating a pair of a position on the image, the pair being determined based on an image including at least a part of the field, and a candidate label indicating a correspondence between the position on the image and a feature point of the first feature point group; a label correction means for specifying a set of a position on the image corresponding to a feature point of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the specified candidate label to a label indicating the feature point of the second feature point group; a field estimation means for estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; An information processing device having the above configuration.

2. The field estimation means comprises: a first field estimation means for generating first field estimation information indicating a first estimation result of the field in the coordinate system based on the first structure data and the feature point information; a second field estimation means for generating second field estimation information indicating a second estimation result of the field in the coordinate system based on the first structure data, the second structure data, and the corrected feature point information, The information processing apparatus according to claim 1 , wherein the label correction means identifies a pair of a position on the image corresponding to a feature point of the second group of feature points and a candidate label based on the first field estimation information and the second structure data.

3. a feature point error detection means for detecting a pair of a position on the image that does not match any of the feature points in the first feature point group and the candidate label based on the feature point information and the first structure data; 3 . The information processing apparatus according to claim 2 , wherein the first field estimation means generates the first field estimation information based on a result of detection by the feature point error detection means, the first structure data, and the feature point information.

4. 4. The information processing device according to claim 2, wherein the label correction means corrects the candidate label based on a distance between a projection position of a position on the image projected onto the coordinate system based on the shooting position and parameters of the camera, and a feature point of the second feature point group represented in the coordinate system based on the second structure data and the first field estimation information.

5. 5. The information processing device according to claim 1, wherein the field to be displayed and a learning field displayed in an image used for training a feature extractor used to generate the feature point information are fields that have at least the first feature point group as a common feature point, and the field to be displayed is a field that has the first feature point group and the second feature point group as feature points.

6. the display target field, which is the field, and a learning field displayed in an image used for learning a feature extractor used to generate the feature point information are fields having the same feature points; The information processing apparatus according to claim 1 , wherein the second group of feature points is a collection of candidate positions that are erroneously extracted as feature points of the first group of feature points in generating the feature point information.

7. The information processing device according to any one of claims 1 to 6, further comprising a coordinate conversion information generating means for generating coordinate conversion information regarding coordinate conversion between a first coordinate system, which is the coordinate system, and a second coordinate system, which is a coordinate system adopted in the first structure data and the second structure data, based on the estimation result of the field.

8. The information processing device is the display device that displays a virtual object superimposed on a landscape, a light source unit that emits display light for displaying the virtual object; an optical element that reflects at least a part of the display light to allow a viewer to visually recognize the virtual object superimposed on a landscape; The information processing device according to claim 1 , further comprising:

9. The computer obtaining first structure data relating to a first group of feature points defined as feature points to be extracted for a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information; Control methods.

10. obtaining first structure data relating to a first group of feature points defined as feature points to be extracted for a target field, and second structure data relating to a second group of feature points other than the first group of feature points; acquiring feature point information indicating a pair of a position on the image determined based on an image including at least a part of the field and a candidate label indicating a correspondence between the feature point of the first feature point group; identifying a pair of positions on the image corresponding to the feature points of the second feature point group and a candidate label based on the second structure data, and generating corrected feature point information by correcting the identified candidate label to a label indicating the feature point of the second feature point group; A program that causes a computer to execute a process of estimating the field in a coordinate system based on a display device having a camera that captures the image, based on the first structure data, the second structure data, and the corrected feature point information.

Images