Information processing device, information processing method, and sensor system

Abstract

Provided is an information processing device comprising a position attitude calculation unit that calculates a relative position attitude of another sensor with respect to a reference sensor among a plurality of sensors on the basis of an image obtained by an imaging sensor among a plurality of sensors imaging a marker associated with an imaging sensor to be imaged among the plurality of sensors.

Description

Information processing device, information processing method, and sensor system

[0001] The present technology relates to an information processing device, an information processing method, and a sensor system.

[0002] 2. Description of the Related Art Conventionally, there is a technique for photographing an object with a plurality of cameras and measuring the three-dimensional shape of the object based on the images obtained by the photographing and the principle of triangulation.

[0003] Furthermore, a technology has been proposed in which the positions and orientations of a group of cameras are determined based on a spatial three-dimensional model, and a subject is measured in three dimensions based on the positions and orientations of the group of cameras and corresponding points that exist in common between images (Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2022-55619

[0005] However, the technology of Patent Document 1 uses SfM (Structure from Motion) to estimate the position and orientation of the camera group. However, SfM typically uses the shape of an object present in an image as a feature within the image, which results in a problem of reduced accuracy in estimating the position and orientation when, for example, the accuracy of object detection is poor.

[0006] The present technology has been developed in consideration of such problems, and aims to provide an information processing device, an information processing method, and a sensor system that are capable of calculating the relative position and orientation of a sensor.

[0007] In order to solve the above-mentioned problems, a first technology is an information processing device including a position and orientation calculation unit that calculates the relative position and orientation of one of the multiple sensors with respect to a reference sensor, based on an image obtained by capturing an image of a marker associated with a sensor to be captured, using a capturing sensor among the multiple sensors.

[0008] The second technology is an information processing method that calculates the relative position and orientation of one of the multiple sensors with respect to a reference sensor based on an image obtained by capturing a marker associated with a sensor to be captured by the capturing sensor.

[0009] The third technology is a sensor system including a plurality of sensors, markers corresponding to the plurality of sensors, and an information processing device including a position and orientation calculation unit that calculates the relative position and orientation of other sensors with respect to a reference sensor among the plurality of sensors based on an image obtained by capturing an image of the marker associated with a sensor to be captured among the plurality of sensors using a capturing sensor among the plurality of sensors.

[0010] 1 is a diagram showing the configuration of a sensor system 10 and an information processing device 300. FIG. 2 is an external view of a camera 100 and a marker 200 mounted on a drone 400. FIG. 3 is a flowchart showing processing in the sensor system 10 and the information processing device 300. FIG. 4 is a diagram showing shooting by a plurality of cameras 100. FIG. 5 is a diagram showing an image shot by the camera 100. FIG. 6 is an explanatory diagram showing calculation of a relative position and orientation. FIG. 7 is an external view of a camera 100 according to a modified example. FIG. 8 is a diagram showing a modified example in which the number of cameras 100 is different. FIG. 9 is a diagram showing a modified example in which the number of cameras 100 is different. FIG. 10 is a diagram showing a modified example in which a projector 500 is used. FIG. 11 is a diagram showing a modified example in which a ToF camera 600 and a prism 700 are used. FIG. 12 is a diagram showing a first application example of the sensor system 10. FIG. 13 is a diagram showing a second application example of the sensor system 10.

[0011] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order: <Embodiment> [Configuration of sensor system 10 and information processing device 300] [Processing in sensor system 10 and information processing device 300] <Modification> <Application example>

[0012] <Embodiment> [Configuration of Sensor System 10 and Information Processing Device 300] The configurations of the sensor system 10 and the information processing device 300 will be described with reference to Fig. 1. The sensor system 10 is configured with a plurality of cameras 100, a plurality of markers 200, and an information processing device 300. The cameras 100 and the information processing device 300 have communication functions, and all of the plurality of cameras 100 are connected to the information processing device 300. The connection method may be a wired connection or a wireless connection. Examples of wired connection methods include HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus), and examples of wireless connection methods include Wi-Fi, Bluetooth (registered trademark), LAN (Local Area Network), and NFC (Near Field Communication).

[0013] The camera 100 is an example of a sensor, and is a camera that includes an image sensor, a signal processing circuit, and the like, and is capable of capturing an RGB (Red, Green, Blue) or monochrome image as a sensing result.

[0014] The marker 200 serves as information for calculating the relative position and orientation (hereinafter, may be referred to as the relative position and orientation) of the camera 100. In this embodiment, an LED (Light Emitting Diode), which is a light emitting device, is used as the marker 200, and as shown in FIG. 2 , the marker 200 is attached in association with the camera 100.

[0015] The marker 200 may be configured as a separate device from the camera 100 and attached to the housing of the camera 100, or the camera 100 may be configured to include the marker 200 as an integral part.

[0016] 1, in this embodiment, the sensor system 10 is configured with three cameras 100: camera 100A, camera 100B, and camera 100C. In the following description, the image captured by camera 100A is referred to as image A, the image captured by camera 100B as image B, and the image captured by camera 100C as image C. However, the number of cameras 100 is not limited to three, and may be any number equal to or greater than two.

[0017] Furthermore, in this embodiment, three markers 200 are used: marker 200A, marker 200B, and marker 200C. Marker 200A is associated with camera 100A and provided on the housing of camera 100A, marker 200B is associated with camera 100B and provided on the housing of camera 100B, and marker 200C is associated with camera 100C and provided on the housing of camera 100C.

[0018] Each camera 100 captures an image so that the other cameras 100 and their corresponding markers 200 are included within its angle of view, and generates an image. Then, the camera 100 detects the two-dimensional positions of the markers 200 within the image. Furthermore, each camera 100 identifies which camera 100 corresponds to the marker 200 present in the image it captured. Then, each camera 100 transmits the two-dimensional position information and identification information of the marker 200 together with marker-related information to the information processing device 300.

[0019] As shown in FIG. 2 , the camera 100 is mounted on a drone 400, which is a mobile body, and is configured to be movable. The drone 400 equipped with the camera 100 may be piloted by a person, or may be controlled by automatic pilot using object recognition technology or the like to track subjects and markers 200 present within the angle of view of the camera 100. When piloting the drone 400 by a person, the person needs to manually pilot the drone 400 so that the markers 200 are always within the angle of view of each camera 100. In addition, when piloting by automatic pilot, the movement of the drone 400 needs to be controlled so that the markers 200 are always within the angle of view of each camera 100.

[0020] The information processing device 300 includes a synchronization signal generation unit 301 and a position and orientation calculation unit 302 .

[0021] The synchronization signal generation unit 301 generates a synchronization signal for synchronizing all of the cameras 100 and transmits the synchronization signal to each camera 100. Each camera 100 performs synchronized shooting based on the synchronization signal and generates an image. The reason why all of the cameras 100 perform synchronized shooting in this manner is that if the cameras 100 are movable and the shooting timing of each camera 100 differs, the relationship between the positions and orientations of the multiple cameras 100 will change between one shooting and the next, making it impossible to accurately calculate the relative positions and orientations.

[0022] The position and orientation calculation unit 302 calculates the relative position and orientation of each camera 100 based on marker-related information acquired from an image and transmitted by each camera 100. The position and orientation calculation unit 302 calculates the relative position and orientation of other sensors of the multiple cameras with respect to a reference sensor based on marker-related information obtained from an image obtained by capturing a marker 200 associated with a captured camera that is a camera that is the captured side of the multiple cameras 100, using a capturing camera that is a camera that is the capturing side of the multiple cameras.

[0023] The sensor system 10 and the information processing device 300 are configured as described above. The information processing device 300 may be configured as an electronic device such as a personal computer, smartphone, tablet terminal, or server device, or may be configured in the cloud. When the information processing device 300 is configured as one of these electronic devices, it can communicate with the camera 100 using a communication function provided by the electronic device. The information processing device 300 and the information processing method may also be realized by an electronic device equipped with computer functions executing a program. The program may be pre-installed on the electronic device, or may be distributed by download or storage medium, and installed by a user, etc. The information processing device 300 may also be configured as a standalone device, or one of multiple cameras 100 may have the functionality of the information processing device 300.

[0024] The synchronization signal generation unit 301 and the position and orientation calculation unit 302 may be configured as separate, standalone devices. Alternatively, the synchronization signal generation unit 301 and the position and orientation calculation unit 302 may be configured as separate electronic devices, in which case the electronic devices are connected by wire or wirelessly. Alternatively, one of the multiple cameras 100 may have the functionality of the synchronization signal generation unit 301, and another camera 100 may have the functionality of the position and orientation calculation unit 302.

[0025] 3, the processing in the sensor system 10 and the information processing device 300 will be described. Note that the positional relationship between the camera 100A and the marker 200A, the positional relationship between the camera 100B and the marker 200B, and the positional relationship between the camera 100C and the marker 200C are assumed to be known by prior calibration.

[0026] In this embodiment, as shown in Fig. 4, drone 400 is operated manually or automatically so that markers 200B and 200C are present within the angle of view of camera 100A, markers 200A and 200C are present within the angle of view of camera 100B, and markers 200A and 200B are present within the angle of view of camera 100C. Note that drone 400 is omitted from Fig. 4 for convenience of illustration.

[0027] One camera is defined as a reference camera to indicate which coordinate axes the relative positions and orientations of the cameras 100 to be calculated are based on.

[0028] In this technology, two or more cameras 100 as sensors associated with markers 200 are positioned to face each other and be ready to capture images, thereby calculating the relative positions and orientations of cameras other than the reference camera with respect to the reference camera.

[0029] One of the plurality of cameras 100 is a photographing camera which is a camera on the photographing side, and the other cameras 100 to which the markers 200 are associated are photographed cameras which are cameras on the photographed side, and the markers 200 associated with the photographed cameras are photographed by the photographing cameras to obtain images.

[0030] In this embodiment, all cameras 100 are considered to be shooting cameras, and all images taken by all of the multiple cameras 100 associated with markers 200 are considered to be photographed cameras are used to calculate the relative positions and orientations of cameras other than the reference camera with respect to the reference camera.

[0031] 4, the camera 100A is used as the shooting camera, and the camera 100B associated with the marker 200B and the camera 100C associated with the marker 200C are used as the captured cameras to perform the shooting. Furthermore, the camera 100B is used as the shooting camera, and the camera 100A associated with the marker 200A and the camera 100C associated with the marker 200C are used as the captured cameras to perform the shooting. Furthermore, the camera 100C is used as the shooting camera, and the camera 100A associated with the marker 200A and the camera 100B associated with the marker 200B are used as the captured cameras to perform the shooting. Therefore, all the cameras 100 associated with the markers 200 are able to look at each other and shoot.

[0032] Then, the relative positions and orientations between one reference camera and the other cameras are calculated based on all images that are the shooting results of all combinations, i.e., image A taken by camera 100A, image B taken by camera 100B, and image C taken by camera 100C. For example, if camera 100A is the reference camera, the relative positions and orientations between camera 100A, which is the reference camera, and the other cameras, camera 100B and camera 100C, are calculated.

[0033] First, in step S101 , the synchronization signal generating unit 301 generates a synchronization signal, and the information processing device 300 transmits the synchronization signal to each camera 100 .

[0034] Next, in step S102, the cameras 100 synchronize with each other based on the synchronization signal, perform synchronized photographing at the same timing, and generate images.

[0035] 4, camera 100A takes an image so that marker 200B and marker 200C are within the angle of view. Camera 100B takes an image so that marker 200A and marker 200C are within the angle of view. Camera 100C takes an image so that marker 200A and marker 200B are within the angle of view.

[0036] By having all of the cameras 100 synchronize and capture images at the same time to generate images, it is possible to calculate the relative positions and orientations of the cameras 100, whether all or any of the cameras 100 are moving or are fixed in position. When the cameras 100 move, the relative positions and orientations of the cameras 100 change, and therefore it is not possible to accurately calculate the relative positions and orientations of the cameras 100 from images captured by the cameras 100 at different times.

[0037] Next, in step S103, each camera 100 detects the two-dimensional position of the marker 200 in the image captured by itself. The two-dimensional position of the marker 200 is coordinate information based on a specific position in the image (for example, the upper left corner), and information on the pixel where the marker 200 is located in the image.

[0038] When an LED is used as the marker 200, the marker 200 can be detected in an image based on, for example, the luminance distribution in the image. The marker 200 can also be detected by machine learning, such as a deep neural network (DNN) or a convolutional neural network (CNN), or artificial intelligence.

[0039] As shown in FIG. 4, when camera 100A captures an image so that markers 200B and 200C are within the angle of view, markers 200B and 200C are detected in image A as shown in FIG. 5A.

[0040] Furthermore, when camera 100B captures an image so that marker 200A and marker 200C are within the angle of view, marker 200A and marker 200C are detected in image B as shown in FIG. 5B.

[0041] Furthermore, when camera 100C captures an image so that marker 200A and marker 200B are within the angle of view, as shown in Fig. 5C, marker 200A and marker 200B are detected in image C. These detected positions are the two-dimensional positions of marker 200 in the image.

[0042] Furthermore, in step S104, each camera 100 identifies which camera 100 the marker 200 in the image captured by itself corresponds to.

[0043] When the markers 200 are LEDs, one method of identification is to have the LEDs flash in different flashing patterns and have the camera 100 detect the differences in the flashing patterns to identify the LEDs. Therefore, it is necessary to determine in advance the flashing patterns of the markers 200 corresponding to the cameras 100A, 100B, and 100C, provide the cameras 100 with an identification function, and store the flashing patterns in association with each camera 100. Another method is to use an AR (Augmented Reality) marker 200, in which the wavelength of each LED is changed and the camera 100 detects the difference in wavelength to identify the LEDs.

[0044] Camera 100A needs to distinguish between marker 200B and marker 200C when detected in image A. Camera 100B needs to distinguish between marker 200A and marker 200C when detected in image B. Camera 100C needs to distinguish between marker 200A and marker 200B when detected in image C.

[0045] Note that steps S103 and S104 may be executed in the reverse order, or may be executed simultaneously or almost simultaneously.

[0046] Next, in step S105, each camera 100 transmits the two-dimensional position information and identification information of the marker 200 to the information processing device 300 as marker-related information.

[0047] Note that if the camera 100 is movable, the marker 200 may not be present in the image depending on the position and orientation of the camera 100 at the time of capturing the image. If the marker 200 is not present in the image, it is not possible to detect the two-dimensional position of the marker 200 in the image, nor is it possible to identify the marker 200. In this case, it is not possible to calculate the relative position and orientation of the camera 100.

[0048] Therefore, if the camera 100 does not detect all or any of the markers 200 from the image when detecting the two-dimensional positions or identification information of the markers 200, the camera 100 notifies the information processing device 300 of that fact. Then, when the information processing device 300 receives a notification that the markers 200 have not been detected, it is preferable that the information processing device 300 does not calculate the relative position and orientation, but calculates the relative position and orientation only when the marker 200 is detected in the image, marker-related information is transmitted, and the marker-related information is received.

[0049] If the marker 200 does not exist in the image of the camera 100, the position and orientation of the camera 100 may be estimated by using an inertial sensor and integrating the measurement values ​​of the inertial sensor.

[0050] Next, in step S106, the position and orientation calculation unit 302 calculates the relative positions and orientations of the other cameras with respect to the reference camera based on the marker-related information.

[0051] Here, a method for calculating the relative position and orientation will be described. In calculating the relative position and orientation, a camera model expressed by the following equation 1 is assumed.

[0052] [Formula 1] su ｊｉ =K ｉ (R ｊｉ x ｊ +t ｊｉ )

[0053] i and j represent the type of camera 100. i represents the shooting camera, and j represents the camera being photographed. Therefore, in this embodiment, the relative position and orientation of camera 100 are calculated using images from all three patterns: when i (shooting camera) is camera 100A, when i (shooting camera) is camera 100B, and when i (shooting camera) is camera 100C.

[0054] s is the scale and u ｊｉ is a homogeneous coordinate representation of the position in the image of the marker 200 corresponding to the photographed camera j projected onto the photographing camera i (it is a three-dimensional vector, but the third dimension is always 1). In actual optimization, s will also be solved as one of the variables.

[0055] K ｉ is the camera matrix of the camera i. ｉ It is necessary to obtain it in advance by calibration.

[0056] As shown in FIG. ｊｉ is the rotation matrix of the photographed camera j relative to the photographing camera i. ｊｉ is the position vector of the photographed camera j with respect to the photographing camera i. The relative position and orientation of the camera 100 are expressed by R and t.

[0057] 6, the reference camera is represented as camera 0, and the objects for calculating the relative position and orientation are represented as cameras 1 and 2. Therefore, the rotation matrix of camera 1 with respect to the reference camera (camera 0) is R １０ The position vector of camera 1 relative to the reference camera (camera 0) is t １０ The rotation matrix of camera 2 relative to the reference camera (camera 0) is R ２０ The position vector of camera 2 relative to the reference camera (camera 0) is t ２０ This becomes:

[0058] x ｊ are the three-dimensional coordinates of the marker 200 corresponding to the camera j, which are assumed to be known by calibration.

[0059] Then, by solving the least squares problem of the reprojection error when each marker 200 is projected onto all cameras 100, the relative position and orientation of each camera with respect to the reference camera can be calculated.

[0060] The relative position and orientation group (R) of the reference camera (camera 0) and the other cameras (camera 1, camera 2) when the position and orientation of the reference camera (camera 0) is fixed. ０ｉ ,t ０ｉ (i=1:N)) is found as an unknown variable.

[0061] When j≠0, R,t is the sum of these variables (R ０ｉ ,t ０ｉ (i=1:N)) (i=1:N)), and therefore are not independent variables.

[0062] A set of relative positions and orientations is calculated to minimize the cost function as follows:

[0063] [Formula 2]

[0064] v ｍｊｉ is the two-dimensional position of the marker 200 associated with the photographed camera j observed by the photographing camera i. In Equation 2, since the photographing camera i cannot photograph itself, the case where i and j are the same is excluded by setting i≠j. ｊｉ is expressed as R ｊｉ and ｊｉ It is calculated from

[0065] [Formula 3]

[0066] In equation 3, fh() is (x, y, z) -> (x / z, y / z).

[0067] The least squares problem can be solved by, for example, the Levenberg-Marquardt method, which is a method for solving nonlinear least squares problems. In this way, the information processing device 300 calculates the relative positions and orientations of the cameras 100.

[0068] Processing in the sensor system 10 and the information processing device 300 is carried out in the manner described above.

[0069] According to the present technology, it is possible to calculate the relative position and orientation of the camera 100 even if the camera 100 is moving. All of the cameras 100 are synchronized based on a synchronization signal to capture images at the same timing, so that all of the cameras 100 capture images of the same camera 100 and marker 200, and therefore it is possible to calculate the relative position and orientation of each camera 100 even if the camera 100 is moving.

[0070] This technology is also effective when the position and orientation of the camera 100 is fixed. Conventionally, in order to calculate the relative position and orientation between fixed cameras, it was necessary to prepare a chart and perform calibration based on images obtained by photographing the chart multiple times with each camera. However, according to this technology, even if the position and orientation of the camera 100 is fixed, calibration is not required and the relative position and orientation of the camera 100 can be easily calculated.

[0071] Furthermore, the present technology is effective in cases where the position and orientation of the camera 100 are fixed during image capture, but the position and orientation of the camera 100 are frequently changed. For example, if an image is captured with the camera 100 fixed, and then the position of the camera 100 is changed and fixed again, and this process is repeated many times, conventionally, calibration was required each time the position was changed. However, according to the present technology, it is not necessary to perform calibration while the camera 100 is fixed, and it is possible to easily calculate the relative position and orientation of the camera 100.

[0072] Furthermore, since the present technology uses light-emitting LEDs as the markers 200, it is possible to calculate the relative position and orientation of the camera 100 even in situations or scenes (landscapes) where there is little texture of objects or the like in the real world. Furthermore, since the markers 200 are used, it is possible to calculate the relative position and orientation of the camera 100 with high accuracy regardless of the texture of objects or the situation or state of the scene in the real world.

[0073] Furthermore, by using a light-emitting LED as the marker 200, the relative position and orientation of the camera 100 can be calculated even at dark nighttime or in bad weather.

[0074] In the existing technology of SLAM (Simultaneous Localization and Mapping), which uses markers placed in the real world, it is necessary to place objects that serve as markers in the real world in advance, and the range in which the camera can move is limited by the position of the marker. However, with this technology, it is not necessary to place objects that serve as markers in the real world in advance, and the range in which the camera 100 can move is not limited.

[0075] Furthermore, with existing technologies such as monocular SLAM that uses a monocular camera, the measurement accuracy drops when a moving object is present in a real-world scene (landscape). However, with this technology, the relative position and orientation of camera 100 can be calculated with high accuracy even if a moving object is present in the scene and is captured in an image captured by camera 100.

[0076] Furthermore, this technology can easily prove that someone else has performed the work. For example, this can be proven by checking whether accuracy is reduced by hiding an LED or the like serving as the marker 200. Furthermore, this can be proven by capturing a rapidly changing pattern on an LCD (Liquid Crystal Display) or the like and checking whether the captures by each camera 100 are synchronized.

[0077] <Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0078] The camera 100 is not limited to the one with an appearance shown in Fig. 2, and may be, for example, a camera 100 as shown in Fig. 7. The camera 100 may be any camera that has the functionality of a camera, such as a single-lens reflex camera, a commercial camera, a smartphone, a tablet terminal, or an action camera.

[0079] The moving object is not limited to the drone 400, but may also be a person, an animal, a car, a motorcycle, a bicycle, personal mobility, an airplane, a ship, a robot, construction machinery, agricultural machinery (tractor), etc.

[0080] In the embodiment, an example has been described in which the number of cameras 100 is the same as the number of markers 200, but the number of cameras 100 may differ from the number of markers 200. However, at least three markers 200 are required, and the greater the number of markers 200, the better in terms of improving the calculation accuracy and robustness of the relative position and orientation.

[0081] Fig. 8 shows an example in which there are two cameras 100 and three markers 200. In this case, one of the cameras 100 must be equipped with two markers 200. In the example in Fig. 8, camera 100A is equipped with marker 200A, and camera 100B is equipped with markers 200B and 200C.

[0082] In this case, camera 100A is the shooting camera, camera 100B with which markers 200B and 200C are associated is the photographed camera, and camera 100A photographs markers 200B and 200C. Also, camera 100B is the shooting camera, and camera 100A with which marker 200A is associated is the photographed camera, and camera 100B photographs marker 200A. Then, using all images captured by the shooting cameras, either camera 100A or camera 100B is designated as the reference camera, and the relative position and orientation of the other camera with respect to the reference camera is calculated.

[0083] 9 shows an example in which there are four cameras 100 and three markers 200. In this case, no marker 200 is associated with any one of the cameras 100. In the example of FIG. 9, marker 200A is associated with camera 100A. Marker 200B is associated with camera 100B. Marker 200C is associated with camera 100C. No marker 200 is associated with camera 100D.

[0084] In this case, camera 100A is the shooting camera, camera 100B associated with marker 200B and camera 100C associated with marker 200C are the photographed cameras, and camera 100A photographs marker 200B and marker 200C. Also, camera 100B is the shooting camera, camera 100A associated with marker 200A and camera 100C associated with marker 200C are the photographed cameras, and camera 100B photographs marker 200A and marker 200C. Also, camera 100C is the shooting camera, camera 100A associated with marker 200A and camera 100B associated with marker 200B are the photographed cameras, and camera 100C photographs marker 200A and marker 200B. Furthermore, camera 100D is the shooting camera, and camera 100A associated with marker 200A, camera 100B associated with marker 200B, and camera 100C associated with marker 200C are the photographed cameras, and camera 100D photographs markers 200A, 200B, and 200C. There is no need to photograph camera 100D that is not associated with marker 200.

[0085] Then, using all the images captured by the capturing cameras, one of the cameras 100A, 100B, 100C, and 100D is set as a reference camera, and the relative positions and orientations of the other cameras with respect to the reference camera are calculated.

[0086] Furthermore, the LED serving as the marker 200 may have two or more light-emitting elements. When the LED has two or more light-emitting elements, the accuracy of position and orientation calculation can be improved compared to when there is only one light-emitting element by calculating and integrating the relative position and orientation based on the two-dimensional positions of the respective light-emitting elements in the image. Also, the LED can be identified by the number of light-emitting elements it has.

[0087] When an LED is used as the marker 200, the LED may not be constantly emitting light, but may be emitting light only at the moment of image capture by the camera 100. This can be achieved by transmitting a synchronization signal from the synchronization signal generation unit 301 to the marker 200 in addition to the camera 100, and the marker 200 emitting light in response to the synchronization signal. In this case, the marker 200 may be connected to the information processing device 300 to receive the synchronization signal, or the synchronization signal received by the camera 100 may be supplied to the marker 200.

[0088] The marker 200 is not limited to an LED, and any light-emitting device can be used.

[0089] Furthermore, an image such as a point projected by the projector 500 can also be used as the marker 200 .

[0090] 10, a projector 500 is provided corresponding to each camera 100, and each projector 500 projects an image such as a dot onto an object such as a wall or floor. In Fig. 10, projector 500A corresponds to camera 100A, projector 500B corresponds to camera 100B, and projector 500C corresponds to camera 100C.

[0091] Then, each camera 100 performs synchronized shooting based on the synchronization signal and captures points in the images projected by each projector 500. At that time, each camera 100 must also capture points projected by its corresponding projector 500. Therefore, as shown in FIG. 10 , cameras 100A, 100B, and 100C all capture points projected by projectors 500A, 500B, and 500C. By capturing points projected by their corresponding projectors 500, each camera 100 can determine the three-dimensional position of the points projected by the projectors 500 relative to the camera 100 using stereo vision (structured light method). This results in the same situation as in the embodiment in which LED markers 200 are associated with the cameras. This allows the relative position and orientation of the camera 100 to be calculated in the same manner as in the embodiment. In this case, it is not necessary for other cameras 100 to be within the angle of view of the camera 100.

[0092] As shown in Figure 11, the image projected by the projector 500 may include multiple points. Each projector 500 must project a common image in which the arrangement of multiple points is the same. When the projector 500 projects multiple points, each camera 100 must capture a point at the same position among the multiple points in order to calculate the relative position and orientation. In Figure 11, the projector 500A projects four points PA1, PA2, PA3, and PA4, the projector 500B projects four points PB1, PB2, PB3, and PB4, and the projector 500C projects four points PC1, PC2, PC3, and PC4.

[0093] Therefore, camera 100A captures images so that points PA1, PA2, PA3, and PA4 projected by projector 500A, points PB1, PB2, PB3, and PB4 projected by projector 500B, and points PC1, PC2, PC3, and PC4 projected by projector 500C fit within its angle of view. Camera 100B also captures images so that points PA1, PA2, PA3, and PA4 projected by projector 500A, points PB1, PB2, PB3, and PB4 projected by projector 500B, and points PC1, PC2, PC3, and PC4 projected by projector 500C fit within its angle of view. Furthermore, camera 100C also captures images so that points PA1, PA2, PA3, and PA4 projected by projector 500A, points PB1, PB2, PB3, and PB4 projected by projector 500B, and points PC1, PC2, PC3, and PC4 projected by projector 500C fit within the angle of view. It is desirable for cameras 100A, 100B, and 100C to capture images so that as many points as possible fit within the angle of view at once. However, as long as the number of captured points is sufficient for calculating the relative position and orientation, it is not necessary for all points to fit within the angle of view.

[0094] In this way, the relative positions and orientations of the other cameras with respect to the reference camera are calculated in the same manner as in the embodiment, using images resulting from capturing all points with all cameras 100. By integrating the relative positions and orientations calculated by capturing images of each point, the accuracy of the relative positions and orientations can be improved. Furthermore, the projector 500 has the advantage that it is easier to add points than an LED, since adding points can be achieved simply by changing the image being projected. In this case, it is necessary to know which camera the point captured by the camera 100 corresponds to and which point was projected by the projector 500. The points can be identified, for example, by detecting the blinking pattern or color of the points.

[0095] 12, a ToF (Time of Flight) camera equipped with a laser irradiation function can also be used as a sensor. In that case, it is preferable to provide each ToF camera 600 with a reflective material such as a prism 700 as a marker 200. In FIG. 12, prism 700A corresponds to ToF camera 600A, prism 700B corresponds to ToF camera 600B, and prism 700C corresponds to ToF camera 600C.

[0096] The distance between the ToF cameras 600 can be measured more accurately by irradiating a laser from the ToF camera 600 toward the prism 700 and detecting the laser reflected by the prism 700 with the ToF camera 600. This can improve the accuracy of calculating the position and orientation of the camera 100 compared to a configuration using only the camera 100. In particular, direct distance measurement can improve the accuracy of scale estimation.

[0097] It is preferable to use a retroreflective prism as the prism 700. The retroreflective prism reflects the laser in the direction of irradiation, which has the advantage of improving the accuracy of laser detection by the ToF camera 600, thereby improving the accuracy of distance measurement between the cameras 100 by the ToF camera 600.

[0098] 12 shows a combined use of an embodiment in which an LED is used as the marker 200 and an example in which a ToF camera 600 is used. As in the embodiment, the relative position and orientation are determined based on an image of an LED as the marker 200 captured by the ToF camera 600, and further, the ToF camera 600 detects the distance between the cameras 100. By integrating these results, it is possible to more accurately determine the relative position and orientation of the ToF camera 600.

[0099] The prism 700 can also be used as a marker 200 similar to an LED because it shines when it reflects a laser.

[0100] An event-based vision sensor (EVS) can also be used as the sensor. An EVS is a sensor that detects events by detecting changes in the brightness of each pixel and outputs information on the coordinates and time of the pixel where the event occurred. Since the EVS outputs only information about the event, it has the advantage of reducing the amount of data to be processed and speeding up processing.

[0101] It is also possible to use the camera 100 and the EVS together. In this case, the camera 100 is a camera that captures an object to be photographed, and the EVS functions as a sensor for calculating the relative position and orientation of the camera 100.

[0102] In the present technology, the positions and orientations of one or more of the multiple cameras 100 may be fixed. This makes it possible to calculate the absolute position and orientation based on the single fixed camera 100 without using scene features such as objects in the real world. Furthermore, it is possible to calculate the amount of change in the position and orientation between frames without using scene features.

[0103] Furthermore, by using an inertial sensor or a magnetic sensor as an auxiliary, the accuracy of attitude calculation can be improved by integrating the attitude of the camera 100 detected by these sensors with the attitude estimation result obtained by this technology.

[0104] It is also possible to use an IR camera as a camera for calculating the relative position and orientation, separate from the camera 100 that captures the subject as the image capture target.

[0105] Furthermore, features within a scene can also be used as an auxiliary in addition to the marker 200. For example, when using the marker 200 using IR light in combination with features of a scene within an image, a camera 100 is used that includes an image sensor having IR pixels in addition to RGB pixels.

[0106] In the embodiment, it has been described that the camera 100 detects the two-dimensional position and identification information of the marker 200 in the image, but the camera 100 may transmit the image to the information processing device 300, and the position and orientation calculation unit 302 of the information processing device 300 may detect the two-dimensional position and identification information. Furthermore, the information processing device 300 may include a processing unit different from the position and orientation calculation unit 302 that detects the two-dimensional position and identification information of the marker 200 from the image.

[0107] A two-dimensional code such as a QR code (a registered trademark of Denso Wave) may be used as the marker 200.

[0108] <Application Examples> Next, application examples of the present technology will be described.

[0109] The first application example is a case where multiple cameras 100 equipped with markers 200 are mounted on a drone 400 as a moving body, as shown in Figure 13, and the position and orientation of the camera 100 continue to change by moving the drone 400 in accordance with the movement of the subject to be photographed.

[0110] In the example of Figure 13, a person, who is the subject, walking and moving is photographed by drone 400A equipped with camera 100A and marker 200A while being followed. The image photographed by camera 100A is designated image A. Furthermore, drone 400B equipped with camera 100B and marker 200B photographs the person while following him / her. The image photographed by camera 100B is designated image B. Furthermore, drone 400C equipped with camera 100C and marker 200C photographs the person while following him / her. The image photographed by camera 100C is designated image C. Furthermore, drone 400D equipped with camera 100D and marker 200D photographs the person while following him / her. The image photographed by camera 100D is designated image D.

[0111] In this technology, all cameras 100A to 100D are synchronized based on a synchronization signal to capture images at the same time. As a result, the drones 400A to 400D are moving to track the subject, so the relative positions and orientations of the cameras 100A to 100D can be dynamically calculated even while the cameras 100A to 100D are moving.

[0112] Conventionally, in order to capture images with multiple cameras and generate free-viewpoint video or 3D information, it was necessary to measure the relative positions and orientations of the cameras while they were fixed and perform calibration. However, with this technology, it is possible to generate free-viewpoint video or 3D information even when the camera 100 is constantly moving.

[0113] As long as the cameras 100 continue to move, the above-described processing is performed for each frame, so that the relative positions and orientations of the cameras 100 can be continuously calculated even if the cameras 100 are moving.

[0114] In the second application example, the camera 100 and the markers 200 are attached to multiple parts of the human body, thereby making it possible to calculate the relative positions and orientations of the multiple parts of the human body.

[0115] 14 , for example, a camera 100A equipped with a marker 200A is attached to the left wrist, which is a part of the human body, and a camera 100B equipped with a marker 200B is attached to the right wrist. Then, by executing the processing of the present technology in the same manner as in the embodiment, it is possible to calculate the relative position and orientation of the left wrist and the right wrist. Furthermore, the calculation result of the relative position and orientation of the left wrist and the right wrist can also be used to estimate the pose of the person.

[0116] In addition, in this second application example, by providing a camera 100 and an LED on each of a head-mounted display for VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) and a controller that is held and operated in the hand, and executing the processing of this technology, it is also possible to estimate the relative position and orientation between the head-mounted display and the controller.

[0117] The present technology may also be configured as follows. (1) An information processing device including a position and orientation calculation unit that calculates a relative position and orientation of one of a plurality of sensors with respect to a reference sensor among the plurality of sensors, based on an image obtained by capturing a marker associated with another sensor by an imaging sensor among the plurality of sensors. (2) The information processing device described in (1) that sets one of the plurality of sensors as the reference sensor, and calculates the relative position and orientation of the other sensor with which the marker other than the reference sensor among the plurality of sensors is associated. (3) The information processing device described in (1) or (2) that calculates the relative position and orientation by two or more of the plurality of sensors with which the marker is associated being in a state where they can capture each other's images. (4) The information processing device described in (3) that sets each of the plurality of sensors as the imaging sensor, and sets sensors other than the imaging sensor as imaged sensors, and calculates the relative position and orientation of the other sensor with respect to the reference sensor, based on a plurality of images captured by all of the plurality of sensors as the imaging sensors, capturing the marker associated with the imaged sensor. (5) The information processing device according to any one of (1) to (4), which calculates the relative position and orientation based on marker-related information obtained from the image. (6) The information processing device according to (5), wherein the marker-related information is a two-dimensional position of the marker in the image. (7) The information processing device according to (5) or (6), wherein the marker-related information is identification information indicating which of the sensors the marker corresponds to. (8) The information processing device according to any one of (1) to (7), wherein the plurality of sensors perform sensing in synchronization based on a synchronization signal. (9) The information processing device according to (8), which includes a synchronization signal generation unit that generates the synchronization signal. (10) The information processing device according to any one of (1) to (9), wherein the marker is a light-emitting device. (11) The information processing device according to any one of (1) to (10), wherein the marker is an image projected by a projector. (12) The information processing device according to any one of (1) to (11), wherein the number of markers is at least three.(13) The information processing device according to any one of (1) to (12), wherein the sensor is a camera that outputs an image as a result of sensing. (14) The information processing device according to any one of (1) to (13), wherein the sensor has a plurality of pixels, detects changes in luminance of the pixels, and outputs information related to the pixels whose luminance has changed as a result of sensing. (15) The information processing device according to any one of (1) to (14), wherein the sensor is a Time of Flight camera that outputs distance information. (16) The information processing device according to any one of (1) to (15), wherein the sensor is mounted on a moving body and configured to be movable. (17) The information processing device according to any one of (1) to (16), wherein the sensor is fixed in position. (18) An information processing method, wherein a relative position and orientation of a sensor to be photographed among the plurality of sensors with respect to a reference sensor among the plurality of sensors is calculated based on an image obtained by photographing a marker associated with another sensor among the plurality of sensors with an photographing sensor among the plurality of sensors. (19) A sensor system comprising: a plurality of sensors; markers corresponding to the plurality of sensors; and an information processing device including a position and orientation calculation unit that calculates the relative position and orientation of other sensors with respect to a reference sensor among the plurality of sensors based on an image obtained by capturing an image of the marker associated with a sensor to be captured among the plurality of sensors using an imaging sensor among the plurality of sensors.

[0118] 10... Sensor system 100... Camera 200... Marker 300... Information processing device 301... Synchronization signal generation unit 302... Position and orientation calculation unit 400... Drone

Claims

1. An information processing device comprising a position and orientation calculation unit that calculates the relative position and orientation of other sensors with respect to a reference sensor among the multiple sensors based on an image obtained by capturing an image of a marker associated with a captured sensor among the multiple sensors using an imaging sensor among the multiple sensors.

2. The information processing device according to claim 1, wherein one of the plurality of sensors is set as the reference sensor, and the relative position and orientation of the other sensor to which the marker other than the reference sensor in the plurality of sensors is associated is calculated.

3. The information processing device according to claim 1, wherein the relative position and orientation are calculated when two or more of the plurality of sensors to which the markers are associated are in a state in which they can capture images of each other.

4. The information processing device according to claim 3, wherein each of the plurality of sensors is the photographing sensor, and the sensors other than the photographing sensor are photographed sensors, and all of the sensors are in a state in which they can photograph each other, and the relative position and orientation of the other sensors with respect to the reference sensor is calculated based on a plurality of images captured by all of the plurality of sensors as the photographing sensors of the markers associated with the photographed sensors.

5. The information processing device according to claim 1, wherein the relative position and orientation are calculated based on marker-related information obtained from the image.

6. The information processing device according to claim 5, wherein the marker-related information is a two-dimensional position of the marker within the image.

7. The information processing device according to claim 5, wherein the marker-related information is identification information indicating which of the sensors the marker corresponds to.

8. The information processing device according to claim 1, wherein the plurality of sensors perform sensing in synchronization with a synchronization signal.

9. The information processing device according to claim 8, further comprising a synchronization signal generating unit that generates the synchronization signal.

10. The information processing device according to claim 1, wherein the marker is a light-emitting device.

11. The information processing device according to claim 1, wherein the marker is an image projected by a projector.

12. The information processing device according to claim 1, wherein the number of the markers is at least three.

13. The information processing device according to claim 1, wherein the sensor is a camera that outputs an image as a result of sensing.

14. The information processing device according to claim 1, wherein the sensor is a sensor that has a plurality of pixels, detects a change in luminance of the pixels, and outputs information relating to the pixels whose luminance has changed as a sensing result.

15. The information processing device according to claim 1, wherein the sensor is a ToF camera that outputs distance information.

16. The information processing device according to claim 1, wherein the sensor is mounted on a moving body and configured to be movable.

17. The information processing device according to claim 1, wherein the sensor is fixed in position.

18. An information processing method for calculating the relative position and orientation of another sensor with respect to a reference sensor among a plurality of sensors based on an image obtained by capturing an image of a marker associated with a captured sensor among the plurality of sensors using an capturing sensor among the plurality of sensors.

19. A sensor system comprising: a plurality of sensors; markers corresponding to the plurality of sensors; and an information processing device having a position and orientation calculation unit that calculates the relative position and orientation of other sensors with respect to a reference sensor among the plurality of sensors based on an image obtained by capturing an image of the marker corresponding to a captured sensor among the plurality of sensors with an capturing sensor among the plurality of sensors.

Images