Information processing apparatus, information processing method, and program

Abstract

To efficiently generate a data set to be used for training an estimation model for estimating the positions of feature points of an object.SOLUTION: An information processing apparatus 10 comprises: a three-dimensional position specification unit 153 that specifies three-dimensional positions of a plurality of feature points indicating the feature of an object O, on the basis of first image data on the object O imaged by a first imaging apparatus 1 whose three-dimensional position and imaging direction are specified; an apparatus information acquisition unit 151 that acquires imaging apparatus information indicating the three-dimensional position and the imaging direction of a second imaging apparatus 2; a second image acquisition unit 155 that acquires second image data on the object O imaged by the second imaging apparatus 2; a two-dimensional position specification unit 154 that specifies two-dimensional positions of a plurality of feature points in an image shown by the second image data, on the basis of the three-dimensional position and the imaging direction of the second imaging apparatus 2 and the specified three-dimensional positions of the plurality of feature points; and a generation unit 156 that generates a data set including the second image data and the specified two-dimensional positions of the plurality of feature points.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there is known a technique for estimating the position of an object from image data of the object. For example, Patent Literature 1 discloses a technique for performing machine learning using a dataset including a plurality of learning image data of an object and position information of feature points of the object appearing in each of the images represented by the plurality of learning image data, thereby generating an estimation model for estimating the positions of feature points of the object for input of image data of the object, and estimating the positions of feature points of the object from image data of the object using the estimation model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-077805 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, when generating a dataset for training an estimation model, the operator generating the dataset had to find feature points of an object in image data of the object and manually generate position information of the feature points, which made the task of generating the dataset cumbersome.

[0005] Therefore, the present invention has been made in consideration of these points, and has an object to efficiently generate a data set used for training an estimation model for estimating the positions of feature points of an object. [Means for solving the problem]

[0006] An information processing system according to a first aspect of the present invention includes a three-dimensional position identification unit that identifies three-dimensional positions of each of a plurality of feature points that indicate features of an object, based on first image data that is image data of an object located in a three-dimensional space captured by a first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the first imaging device, the three-dimensional position and imaging direction of the object being captured by the second imaging device, the three-dimensional position identification unit that identifies two-dimensional positions of each of the plurality of feature points in an image represented by the second image data that is generated when the object is captured by the second imaging device, based on the three-dimensional position and imaging direction of the second imaging device indicated by the imaging device information acquired by the device information acquisition unit and the three-dimensional positions of each of the plurality of feature points identified by the three-dimensional position identification unit, and a generation unit that generates a dataset including the second image data acquired by the image acquisition unit and the two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit.

[0007] Markers may be attached to positions corresponding to each of a plurality of feature points of the object, allowing the three-dimensional position identification unit to identify the positions, and the three-dimensional position identification unit may identify three-dimensional positions of the plurality of markers corresponding to each of the plurality of feature points based on the first image data captured by the first imaging device, and identify the three-dimensional position of each of the plurality of feature points based on the identified three-dimensional positions of each of the plurality of markers.

[0008] The three-dimensional position identification unit may display on a display unit a three-dimensional virtual space in which three-dimensional models representing the object and the marker are arranged, and identify the three-dimensional position of the feature point by accepting designation of the feature point corresponding to the marker in the three-dimensional virtual space.

[0009] The generation unit may erase the markers from the second image data, and generate a dataset including the second image data after the markers have been erased and the two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit.

[0010] The device information acquisition unit may acquire the imaging device information corresponding to each of the multiple second imaging devices that capture an image of the object in the three-dimensional space, the image acquisition unit may acquire the second image data captured by each of the multiple second imaging devices, and the two-dimensional position identification unit may identify a two-dimensional position of each of the multiple feature points in an image indicated by the second image data generated when each of the multiple second imaging devices captures the object, based on the three-dimensional position and imaging direction of the second imaging device indicated by the imaging device information corresponding to each of the multiple second imaging devices and the three-dimensional positions of each of the multiple feature points identified by the three-dimensional position identification unit, and the generation unit may generate a dataset including the second image data captured by each of the multiple second imaging devices and the two-dimensional positions of each of the multiple feature points identified for the second image data by the two-dimensional position identification unit.

[0011] The three-dimensional position identification unit may identify three-dimensional positions of the feature points of the object corresponding to each of the multiple orientations based on the first image data corresponding to each of the multiple orientations of the object when the orientation of the object is changed, the image acquisition unit acquires the second image data corresponding to each of the multiple orientations of the object captured by the second imaging device, and the two-dimensional position identification unit may identify two-dimensional positions of each of the multiple feature points in an image indicated by the second image data corresponding to each of the multiple orientations of the object acquired by the image acquisition unit based on the three-dimensional position and imaging direction of the second imaging device indicated by the imaging device information and the three-dimensional positions of each of the multiple feature points corresponding to each of the multiple orientations of the object identified by the three-dimensional position identification unit, and the generation unit may generate a dataset including the second image data corresponding to each of the multiple orientations of the object and the two-dimensional positions of each of the multiple feature points identified for the second image data by the two-dimensional position identification unit.

[0012] An information processing method according to a second aspect of the present invention includes the steps of: identifying three-dimensional positions of each of a plurality of feature points indicating characteristics of an object based on first image data, which is image data of an object located in a three-dimensional space captured by a first imaging device, whose three-dimensional position and imaging direction in the three-dimensional space are specified; acquiring imaging device information indicating the three-dimensional position and imaging direction of a second imaging device that captures the object in the three-dimensional space; acquiring second image data, which is image data of the object captured by the second imaging device; identifying two-dimensional positions of each of the plurality of feature points in an image represented by the second image data generated when the second imaging device captures the object, based on the three-dimensional position and imaging direction of the second imaging device indicated by the acquired imaging device information and the identified three-dimensional positions of each of the plurality of feature points; and generating a dataset including the acquired second image data and the two-dimensional positions of each of the identified plurality of feature points.

[0013] A program according to a third aspect of the present invention causes a computer to function as a three-dimensional position identification unit that identifies three-dimensional positions of each of a plurality of feature points that indicate features of an object, based on first image data, which is image data of an object located in a three-dimensional space captured by a first imaging device, whose three-dimensional position and imaging direction in the three-dimensional space are specified; a device information acquisition unit that acquires imaging device information that indicates the three-dimensional position and imaging direction of a second imaging device that captures the object in the three-dimensional space; an image acquisition unit that acquires second image data, which is image data of the object captured by the second imaging device; a two-dimensional position identification unit that identifies two-dimensional positions of each of the plurality of feature points in an image represented by the second image data generated when the second imaging device captures the object, based on the three-dimensional position and imaging direction of the second imaging device indicated by the imaging device information acquired by the device information acquisition unit and the three-dimensional positions of each of the plurality of feature points specified by the three-dimensional position identification unit; and a generation unit that generates a dataset including the second image data acquired by the image acquisition unit and the two-dimensional positions of each of the plurality of feature points specified by the two-dimensional position identification unit. Effect of the Invention

[0014] According to the present invention, it is possible to efficiently generate a data set used for training an estimation model for estimating the positions of feature points of an object. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3] FIG. 13 is a diagram showing an example in which a marker is added to a position away from a feature point. [Figure 4] 10 is a flowchart illustrating an example of a process flow in an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] [Outline of Information Processing System S] 1 is a diagram showing an overview of an information processing system S. The information processing system S includes a first imaging device 1, a plurality of second imaging devices 2, and an information processing device 10. The information processing system S is a system that generates a data set including image data generated by the second imaging device 2 capturing an image of an object O arranged in a three-dimensional space, and two-dimensional positions in the image data of a plurality of feature points of the object O captured in an image represented by the image data.

[0017] The information processing device 10 is communicatively connected to a first imaging device 1 and a plurality of second imaging devices 2, which are installed so as to be able to capture an image of an object O. Fig. 1 shows an example in which three first imaging devices 1 are provided. The first imaging device 1 is, for example, a motion capture camera that irradiates infrared light and visualizes the infrared light emitted from an object, and the three-dimensional position and imaging direction in the three-dimensional space are specified in advance.

[0018] The second imaging device 2 is, for example, a color camera, a monochrome camera, or a depth sensing camera that generates two-dimensional image data, and its three-dimensional position in three-dimensional space and imaging direction are specified in advance, similar to the first imaging device 1. The depth sensing camera includes a ToF (Time Of Flight) camera and a LiDAR (Light Detection And Ranging) camera.

[0019] The object O is, for example, a vehicle, a robot, a human body, etc., but is not limited thereto and may be any object. Markers M are attached to positions corresponding to each of a plurality of feature points of the object O. The markers M are used by the information processing device 10 to identify the feature points based on the first image data captured by the first imaging device 1.

[0020] The information processing device 10 specifies the three-dimensional position of the marker M appearing in the first image data, which is image data of the object O captured by the first imaging device 1. A method for specifying the three-dimensional position of the marker M will be described later. Then, the information processing device 10 specifies the three-dimensional position of each of a plurality of feature points indicating the features of the object O, based on the specified three-dimensional position of the marker M.

[0021] The information processing device 10 identifies the two-dimensional position of each of the multiple feature points in the image represented by the second image data, which is image data of the object O captured by the second imaging device 2, based on the predetermined three-dimensional position and imaging direction of the second imaging device 2 and the three-dimensional position of each of the identified multiple feature points.

[0022] The information processing device 10 generates a dataset including the second image data and two-dimensional position information indicating the two-dimensional position of each of the multiple feature points. The dataset is used for training an estimation model that estimates the two-dimensional position of each feature point of the object O based on the second image data obtained by capturing an image of the object O. In this way, the information processing device 10 can efficiently generate the dataset used for training the estimation model.

[0023] [Functional configuration of information processing device 10] Next, a detailed description will be given of the configuration of the information processing device 10. Fig. 2 is a diagram showing the functional configuration of the information processing device 10. The information processing device 10 has a communication unit 11, a display unit 12, an operation unit 13, a storage unit 14, and a control unit 15.

[0024] The communication unit 11 is a communication interface for transmitting and receiving data to and from devices such as the first imaging device 1 and the second imaging device 2 via a communication cable, and is, for example, a Universal Serial Bus (USB) (registered trademark) interface.

[0025] The display unit 12 is configured by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The operation unit 13 is an input device such as a mouse or a keyboard that accepts user operations.

[0026] The storage unit 14 is a storage medium that stores various data, and includes a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, etc. The storage unit 14 stores programs executed by the control unit 15. The storage unit 14 stores programs that cause the control unit 15 to function as a device information acquisition unit 151, a first image acquisition unit 152, a three-dimensional position identification unit 153, a two-dimensional position identification unit 154, a second image acquisition unit 155, a generation unit 156, and an output unit 157.

[0027] The control unit 15 is, for example, a CPU (Central Processing Unit). The control unit 15 executes the programs stored in the storage unit 14 to function as a device information acquisition unit 151, a first image acquisition unit 152, a three-dimensional position identification unit 153, a two-dimensional position identification unit 154, a second image acquisition unit 155, a generation unit 156, and an output unit 157.

[0028] The device information acquisition unit 151 acquires imaging device information indicating a three-dimensional position and an imaging direction in a three-dimensional space of each of the second imaging devices 2 that capture an image of the object O in the three-dimensional space. The device information acquisition unit 151 acquires the imaging device information stored in the storage unit 14, for example.

[0029] Any method may be used to create the imaging device information, but the imaging device information may be created, for example, by the following method: A plurality of first imaging devices 1 connected to the information processing device 10 capture an image of a first jig for identifying the relative positions of the plurality of first imaging devices 1, and based on image data generated by these first imaging devices 1, the relative positions of each of these first imaging devices 1 are specified in advance. In addition, a three-dimensional coordinate system is defined for the three-dimensional space, and a three-dimensional position and imaging direction in the three-dimensional coordinate system of each of the plurality of first imaging devices 1 are specified in advance.

[0030] Furthermore, the three-dimensional positions and imaging directions of the second imaging devices 2 in the three-dimensional coordinate system are specified in advance based on image data generated by imaging a third jig for specifying the three-dimensional position and imaging direction of the second imaging devices 2 by the first imaging devices 1 and the second imaging devices 2. Then, imaging device information indicating the three-dimensional position and imaging direction of each of the second imaging devices 2 specified in advance is stored in the storage unit 14.

[0031] The first image acquisition unit 152 acquires first image data, which is image data of an object O arranged in a three-dimensional space and captured by each of a plurality of first imaging devices 1 whose three-dimensional positions and imaging directions in the three-dimensional space are specified in advance. For example, the first image acquisition unit 152 acquires a plurality of first image data generated by the plurality of first imaging devices 1 capturing images simultaneously.

[0032] Here, as shown in Fig. 1, a plurality of markers M are attached to each of a plurality of feature points of the object O. The markers M are, for example, reflective markers that reflect infrared rays, or self-luminous markers that emit infrared rays. The first image acquisition unit 152 acquires first image data, which is infrared image data of the object O to which the plurality of markers M are attached, captured by each of the plurality of first imaging devices 1. It is assumed that the markers M are each captured in an image indicated by the plurality of first image data.

[0033] The three-dimensional position identifying section 153 identifies the three-dimensional position of each of a plurality of feature points indicating features of the object O, based on the first image data acquired by the first image acquiring section 152 from the first imaging device 1. The three-dimensional position identifying section 153 identifies the three-dimensional position of the marker M based on the two-dimensional coordinates of the marker M in each of the plurality of first image data, for example, by using a triangulation method. The three-dimensional position identifying section 153 identifies the three-dimensional position of the feature point from the three-dimensional position of the marker M, based on the relative positions of the marker M and the feature point.

[0034] The three-dimensional position identifying section 153 identifies three-dimensional positions of a plurality of markers M corresponding to the plurality of feature points, respectively, based on the first image data captured by the first imaging device 1. Then, the three-dimensional position identifying section 153 identifies the three-dimensional position of each of the plurality of feature points of the object O, based on the identified three-dimensional position of each of the plurality of markers M.

[0035] For example, when a marker M is attached to a feature point of an object O, the three-dimensional position specifying unit 153 specifies the specified three-dimensional position of the marker M as the three-dimensional position of the feature point. Here, depending on the position of the feature point on the object O, the feature point may not be visible from the first imaging device 1. In such a case, the marker M may be attached to a position away from the feature point, and the three-dimensional position specifying unit 153 may specify the three-dimensional position of the feature point based on the three-dimensional position of the marker M and the relative position of the three-dimensional position of the feature point with respect to the three-dimensional position of the marker M. FIG. 3 is a diagram showing an example in which the marker M is attached to a position away from the feature point. In the example shown in FIG. 3, the marker M is provided at a position different from the multiple feature points F of the object O.

[0036] In order for the three-dimensional position specifying unit 153 to recognize the three-dimensional position of the marker M and the relative positions between the marker M and the three-dimensional positions of the feature points, the three-dimensional position specifying unit 153 may display an image of a three-dimensional model of the object O based on the first image data generated by photographing the object O to which the marker M is attached on the display unit 12, and accept the three-dimensional positions of the feature points in the image. The three-dimensional position specifying unit 153 accepts the designation of the feature point corresponding to the marker M in the three-dimensional virtual model via the operation unit 13, thereby specifying the relative position of the three-dimensional position of the feature point with respect to the three-dimensional position of the marker M. The three-dimensional position specifying unit 153 can specify the three-dimensional position of the feature point based on the three-dimensional position and the relative position of the marker M. In this way, the information processing device 10 can specify the three-dimensional position of the feature point corresponding to the marker M even when the feature point is in a position not captured in the first captured image data and the marker M is in a position captured in the first captured image data.

[0037] In addition, the information processing device 10 may not have a three-dimensional position identification unit 153, and a three-dimensional position identification device that identifies the three-dimensional position of a marker M attached to an object O based on the first image data captured by a plurality of first imaging devices 1 may function as the three-dimensional position identification unit 153.

[0038] The two-dimensional position identification unit 154 identifies the two-dimensional position of each of the multiple feature points of object O in the image indicated by the second image data generated when each of the multiple second imaging devices 2 images object O, based on the three-dimensional position and imaging direction of each of the multiple second imaging devices 2 indicated by the imaging device information acquired by the device information acquisition unit 151 and the three-dimensional position of each of the multiple feature points of object O identified by the three-dimensional position identification unit 153.

[0039] The two-dimensional position specifying unit 154 specifies at what position each of the multiple feature points appears in an image indicated by the second image data generated by the second image capturing device 2 when imaging is performed using the second image capturing device 2 from the three-dimensional position and imaging direction of the second image capturing device 2. For example, for each of the multiple second image capturing devices 2, the two-dimensional position specifying unit 154 specifies, in three-dimensional space, an image plane corresponding to the second image data generated by the second image capturing device 2 capturing an image in the imaging direction of the second image capturing device 2 indicated by the imaging device information from the three-dimensional position of the second image capturing device 2 indicated by the imaging device information.

[0040] Then, for each of the multiple feature points identified by the three-dimensional position identifying section 153, the two-dimensional position identifying section 154 calculates a straight line passing through both the three-dimensional position of the second imaging device 2 and the three-dimensional position of the feature point. The two-dimensional position identifying section 154 identifies the two-dimensional position in the two-dimensional coordinate system corresponding to the image plane of the intersection of each of the straight lines corresponding to each of the multiple feature points with the identified image plane as the two-dimensional position of each of the multiple feature points in the image represented by the second image data. Note that the two-dimensional position identifying section 154 may convert the coordinate position of each of the multiple feature points in the three-dimensional coordinate system into the coordinate position of the two-dimensional coordinate system corresponding to the identified image plane using a known determinant of coordinate transformation.

[0041] The second image acquisition section 155 acquires second image data which is image data of the object O captured by each of the multiple second imaging devices 2. The second image acquisition section 155 acquires second image data which captures the object O in the same state as when the first imaging device 1 generated the first image data. For example, the second image acquisition section 155 acquires second image data which is image data of the object O captured by each of the multiple second imaging devices 2 at the same timing as when the first imaging device 1 captures the object O.

[0042] Although the imaging timing of the first imaging device 1 and the second imaging device 2 is the same, this is not limited to the above. If the state of the object O does not change over time, the imaging timing of the first imaging device 1 and the second imaging device 2 may be different. Also, if the states of multiple objects O do not change over time, the imaging timing of each of the multiple second imaging devices 2 may be different.

[0043] The generating unit 156 generates a data set including the plurality of second image data acquired by the second image acquiring unit 155 and the two-dimensional positions of the plurality of feature points identified for each of the plurality of second image data by the two-dimensional position identifying unit 154. For example, the generating unit 156 erases image data indicating the marker M from the second image data acquired by the second image acquiring unit 155 by performing image processing on the second image data. Then, the generating unit 156 generates a data set including the second image data after erasing the marker M and two-dimensional position information indicating the two-dimensional positions of the plurality of feature points identified by the two-dimensional position identifying unit 154 based on the first image data in which the object O is captured in the same state as the state in which the second image data was generated.

[0044] The output unit 157 outputs the data set generated by the generation unit 156. For example, the output unit 157 causes the storage unit 14 to store the data set generated by the generation unit 156.

[0045] Note that the generating unit 156 generates a data set corresponding to each of the multiple second image data obtained by capturing an image of the object O in one direction by each of the multiple second imaging devices 2, but is not limited to this. For example, the generating unit 156 may generate a data set corresponding to multiple second image data obtained when the object O corresponding to each of the multiple directions is captured by each of the multiple second imaging devices 2.

[0046] In this case, the control unit 15 has an adjustment unit that changes the orientation of the object O after the output unit 157 outputs the data set generated by the generation unit 156. For example, the object O is placed on a turntable that rotates around a predetermined axis. Then, in response to the output unit 157 outputting the data set, the adjustment unit rotates the turntable around the predetermined axis to change the orientation of the object O. Then, the three-dimensional position identification unit 153 identifies three-dimensional positions of feature points of the object O corresponding to each of the multiple orientations based on the first image data corresponding to each of the multiple orientations of the object O when the adjustment unit changes the orientation of the object O.

[0047] Here, the adjustment unit rotates the turntable to change the orientation of the object O, but this is not limiting, and the orientation of the object O may be changed by a device other than the information processing device 10 or manually. In addition, if the object O is an object that can change its orientation by itself, such as an AGV (Automatic Guided Vehicle) or a drone, instruction information may be transmitted to the object O by the adjustment unit, a device other than the information processing device 10, or manually, to cause the object O to change its orientation by itself.

[0048] The second image acquisition unit 155 acquires second image data corresponding to each of the multiple orientations of the object O captured by each of the multiple second imaging devices 2. The two-dimensional position identification unit 154 then identifies the two-dimensional position of each of the multiple feature points in the second image data corresponding to each of the multiple orientations of the object O acquired by the second image acquisition unit 155, based on the three-dimensional positions and imaging directions of the multiple second imaging devices 2 indicated by the imaging device information and the three-dimensional positions of each of the multiple feature points corresponding to each of the multiple orientations of the object O identified by the three-dimensional position identification unit 153 based on the first image data in which the object O is captured in the same state as the state in which the second image data was generated.

[0049] Then, the generating unit 156 generates a dataset including second image data corresponding to each of the multiple orientations of the object O and the two-dimensional positions of each of the multiple feature points identified based on the first image data by the two-dimensional position identifying unit 154. In this manner, the information processing device 10 can efficiently generate a dataset corresponding to the second image data obtained by capturing an image of the object O from various directions.

[0050] [Operation flow] Next, a description will be given of the process flow of the information processing device 10. Fig. 4 is a flowchart showing an example of the process flow in the information processing device 10. In this flowchart, a description will be given of the process flow when an object O is placed on a turntable, and a data set corresponding to each of a plurality of orientations of the object O is generated while rotating the turntable.

[0051] First, the device information acquisition unit 151 acquires imaging device information indicating the three-dimensional position and imaging direction in the three-dimensional space of each of the multiple second imaging devices 2 (S1). Next, the first image acquisition unit 152 acquires first image data which is image data of the object O captured by each of the multiple first imaging devices 1 (S2). Also, the second image acquisition unit 155 acquires second image data which is image data of the object O captured by each of the multiple second imaging devices 2 (S3).

[0052] Next, the three-dimensional position specifying section 153 specifies the three-dimensional position in the three-dimensional space of each of a plurality of feature points indicating the features of the object O, based on the first image data acquired in S2 (S4). Next, the two-dimensional position identification unit 154 identifies the two-dimensional position of each of the multiple feature points of the object O corresponding to the second image data generated when each of the multiple second imaging devices 2 images the object O, based on the three-dimensional position and imaging direction of each of the multiple second imaging devices 2 indicated by the imaging device information acquired in S1 and the three-dimensional position of each of the multiple feature points of the object O identified in S4 (S5).

[0053] Next, the generating unit 156 generates a data set including the multiple second image data acquired in S3 and the two-dimensional positions of the multiple feature points corresponding to each of the multiple second image data identified in S5 (S6). Next, the output unit 157 outputs the data set generated by the generating unit 156 (S7).

[0054] Next, the control unit 15 determines whether or not to end the generation of the data set (S8). For example, the control unit 15 determines whether or not to end the generation of the data set by determining whether or not an instruction to end the process of generating the data set has been received via the operation unit 13. When the control unit 15 determines to end the generation of the data set, it ends the process related to this flowchart.

[0055] Furthermore, when the control unit 15 determines not to end the generation of the data set, it shifts the process to S9 and rotates the turntable to change the orientation of the object O. Thereafter, the control unit 15 shifts the process to S2. This repeats the processes from S2 to S7, and the generation of the data set is repeated. Note that, in this flowchart, the process related to S1 is executed first to obtain the imaging device information, but this is not limiting. The process related to S1 may be executed at any timing before S5, which is a process using the imaging device information.

[0056] [Variations] In the above description, a plurality of second imaging devices 2 are connected to the information processing device 10, and the generation unit 156 generates a data set including second image data captured by each of the plurality of second imaging devices 2 and two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit 154 for the second image data, but this is not limited to the above. A single second imaging device 2 may be connected to the information processing device 10, and the generation unit 156 may generate a data set including second image data captured by the single second imaging device 2 and two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit 154 based on first image data in which the object O is captured in the same state as the state in which the second image data was generated.

[0057] In the above description, multiple first imaging devices 1 are connected to the information processing device 10, and the three-dimensional position specifying unit 153 specifies the three-dimensional position of each of multiple feature points indicating the characteristics of the object O based on the first image data obtained by capturing images of the object O by the multiple first imaging devices 1, but this is not limited to the above. A single first imaging device 1 may be connected to the information processing device 10, and the three-dimensional position of each of multiple feature points indicating the characteristics of the object O may be specified based on the first image data obtained by capturing images of the object O by the single first imaging device 1.

[0058] In this case, a marker with a predetermined pattern is attached to the object O. The predetermined pattern is, for example, a pattern that allows the three-dimensional position specifying unit 153 to detect the inclination and rotation angle of the marker based on the state of the marker in the image represented by the image data captured by the first imaging device 1. The three-dimensional position specifying unit 153 specifies the size of the marker in the image represented by the first image data captured by one first imaging device 1 of the object O, for example, and specifies the inclination and rotation angle of the marker based on the pattern of the marker. Then, the three-dimensional position specifying unit 153 specifies the three-dimensional position of the marker based on the specified size, inclination, and rotation angle of the marker.

[0059] Furthermore, a motion capture camera capable of identifying a three-dimensional position by itself may be adopted as the first imaging device 1. Then, the three-dimensional position identifying unit 153 may identify the three-dimensional position of the marker by acquiring three-dimensional position information indicating the three-dimensional position of the marker measured by the first imaging device 1.

[0060] In the above description, the marker M is attached to the object O, and the three-dimensional position identifying unit 153 identifies the three-dimensional position of the marker M to identify the three-dimensional position of the feature point of the object O, but this is not limited to the above. If the object O is an object having joints such as a human being, and the joints can be identified without providing a marker, it is not necessary to attach a marker to the object O. Then, the three-dimensional position identifying unit 153 may identify the three-dimensional positions of the joints of the object O as the three-dimensional positions of the feature points of the object.

[0061] [Effects of Information Processing System S] As described above, in the information processing system S according to this embodiment, the information processing device 10 specifies the three-dimensional positions of each of the multiple feature points that indicate the characteristics of the object O based on the first image data, which is the image data of the object O captured by the first imaging device 1, whose three-dimensional position and imaging direction in three-dimensional space are specified, and specifies the two-dimensional positions of each of the multiple feature points in the image shown by the second image data, which is the image data of the object O captured by the second imaging device 2, based on the three-dimensional position and imaging direction of the second imaging device 2 that captures the object O and the three-dimensional positions of each of the specified multiple feature points. Then, the information processing device 10 generates a dataset including the second image data captured by the second imaging device and the two-dimensional positions of each of the specified multiple feature points. In this way, the information processing device 10 can efficiently generate a dataset used for learning an estimation model for estimating the positions of the feature points of an object.

[0062] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0063] 1 First imaging device 2 Second imaging device 10. Information processing device 11 Communications Department 12 Display section 13 Control section 14 Storage section 15 Control section 151 Device information acquisition unit 152 First image acquisition unit 153 3D position identification part 154 Two-dimensional position identification part 155 Second Image Acquisition Unit 156 Generation part 157 Output section S Information Processing System

Claims

1. a three-dimensional position specifying unit that specifies three-dimensional positions of each of a plurality of feature points that indicate features of an object based on first image data, which is image data of an object located in a three-dimensional space and captured by a first imaging device, the three-dimensional position and imaging direction of the object being specified in the three-dimensional space; a device information acquisition unit that acquires imaging device information indicating a three-dimensional position and an imaging direction of a second imaging device that captures an image of the object in the three-dimensional space; an image acquisition unit that acquires second image data, which is image data of the object captured by the second imaging device; a two-dimensional position specification unit that specifies a two-dimensional position of each of the plurality of feature points in an image indicated by the second image data generated when the second image capture device captures an image of the object, based on the three-dimensional position and imaging direction of the second image capture device indicated by the imaging device information acquired by the device information acquisition unit and the three-dimensional positions of each of the plurality of feature points specified by the three-dimensional position specification unit; a generation unit that generates a data set including the second image data acquired by the image acquisition unit and the two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit; An information processing system having the above configuration.

2. markers are attached to positions corresponding to a plurality of feature points of the object, the markers being used by the three-dimensional position specifying unit to specify the respective positions; the three-dimensional position identification unit identifies three-dimensional positions of the markers corresponding to the plurality of feature points, based on the first image data captured by the first imaging device, and identifies the three-dimensional positions of the plurality of feature points, based on the identified three-dimensional positions of the markers; The information processing system according to claim 1 .

3. the three-dimensional position specifying unit causes a three-dimensional virtual space in which a three-dimensional model representing the object and the marker is arranged to be displayed on a display unit, and specifies the three-dimensional position of the feature point by accepting designation of the feature point corresponding to the marker in the three-dimensional virtual space; The information processing system according to claim 2 .

4. the generating unit erases the markers from the second image data, and generates a data set including the second image data after the markers have been erased and the two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identifying unit. The information processing system according to claim 2 .

5. the device information acquisition unit acquires the imaging device information corresponding to each of the second imaging devices that capture images of the object in the three-dimensional space; the image acquisition unit acquires the second image data captured by each of the second imaging devices; the two-dimensional position specifying unit specifies a two-dimensional position of each of the plurality of feature points in an image indicated by the second image data generated when each of the plurality of second image capturing devices captures an image of the object, based on the three-dimensional position and imaging direction of the second image capturing device indicated by the imaging device information corresponding to each of the plurality of second image capturing devices and the three-dimensional positions of each of the plurality of feature points specified by the three-dimensional position specifying unit; the generation unit generates a data set including the second image data captured by each of the plurality of second imaging devices and the two-dimensional positions of the plurality of feature points identified for the second image data by the two-dimensional position identification unit. The information processing system according to claim 1 .

6. the three-dimensional position specifying unit specifies, based on the first image data corresponding to each of a plurality of orientations of the object when an orientation of the object is changed, a three-dimensional position of the feature point of the object corresponding to each of the plurality of orientations; the image acquisition unit acquires the second image data corresponding to each of a plurality of orientations of the object captured by the second imaging device; the two-dimensional position specifying unit specifies a two-dimensional position of each of the plurality of feature points in an image indicated by the second image data corresponding to each of a plurality of orientations of the object acquired by the image acquisition unit, based on the three-dimensional position and imaging direction of the second imaging device indicated by the imaging device information and the three-dimensional positions of each of the plurality of feature points corresponding to each of the plurality of orientations of the object identified by the three-dimensional position specifying unit; the generating unit generates a data set including the second image data corresponding to each of a plurality of orientations of the object and the two-dimensional positions of each of the plurality of feature points identified for the second image data by the two-dimensional position identifying unit. The information processing system according to claim 1 .

7. The computer executes A step of identifying three-dimensional positions of each of a plurality of feature points that indicate features of an object based on first image data, which is image data of an object disposed in a three-dimensional space and captured by a first imaging device whose three-dimensional position and imaging direction in the three-dimensional space are identified; acquiring imaging device information indicating a three-dimensional position and an imaging direction of a second imaging device that images the object in the three-dimensional space; acquiring second image data, the second image data being image data of the object captured by the second imaging device; specifying a two-dimensional position of each of the plurality of feature points in an image indicated by the second image data generated when the second image capture device captures an image of the object, based on the three-dimensional position and imaging direction of the second image capture device indicated by the acquired imaging device information and the three-dimensional positions of each of the plurality of feature points that have been specified; generating a data set including the acquired second image data and the two-dimensional position of each of the identified feature points; An information processing method comprising the steps of:

8. Computer, a three-dimensional position specifying unit that specifies three-dimensional positions of each of a plurality of feature points that indicate features of an object based on first image data, which is image data of an object disposed in a three-dimensional space and captured by a first imaging device, the three-dimensional position and imaging direction of the object being specified in the three-dimensional space; a device information acquisition unit that acquires imaging device information indicating a three-dimensional position and an imaging direction of a second imaging device that images the object in the three-dimensional space; an image acquisition unit that acquires second image data, which is image data of the object captured by the second imaging device; a two-dimensional position specification unit that specifies a two-dimensional position of each of the plurality of feature points in an image indicated by the second image data generated when the second image capture device captures an image of the object, based on the three-dimensional position and imaging direction of the second image capture device indicated by the imaging device information acquired by the device information acquisition unit and the three-dimensional positions of each of the plurality of feature points specified by the three-dimensional position specification unit; and a generation unit that generates a data set including the second image data acquired by the image acquisition unit and the two-dimensional positions of each of the plurality of feature points identified by the two-dimensional position identification unit; A program that functions as a

Images