Information processing device, information processing method, and program

Abstract

[Problem] To propose a novel and improved information processing device, information processing method, and program which can more simply detect whether a parameter deviation occurs between camera parameters in a plurality of cameras. [Solution] The information processing device comprising: a calculation unit which calculates a three-dimensional position of a feature point included in each of images on the basis of the images obtained by a plurality of cameras imaging a subject at a first timing and external parameters of the plurality of cameras; a first estimation unit which estimates, on the basis of one image included in images obtained by the plurality of cameras imaging the subject at a second timing and the three-dimensional position of the feature point, a first external parameter that is the external parameter of a camera imaging the one image among the plurality of cameras at the second timing; a second estimation unit which estimates, on the basis of the first external parameter estimated by the first estimation unit, a second external parameter that is an external parameter of any one among the plurality of cameras at the first timing; and a determination unit which determines whether the deviation pertaining to the external parameter occurs in the plurality of cameras on the basis of the second external parameter estimated by the second estimation unit and a previous external parameter of the one among the plurality of cameras.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, technologies have been developed that use images captured by multiple cameras to estimate the distance from each camera to an object contained in each image. In order to estimate such distances with high accuracy, the camera parameters of the multiple cameras must be accurate.

[0003] The camera parameters include, for example, external parameters that depend on the positions and orientations of the multiple cameras. Therefore, if deviations occur in the positions and orientations of the multiple cameras due to disturbances such as changes in environmental temperature or vibration, the accuracy of the external parameters may be reduced, and the accuracy of the estimated distance values ​​may also be reduced. Therefore, techniques for detecting whether deviations have occurred in the positions and orientations of the cameras have been developed. For example, Patent Document 1 discloses a technique for estimating camera parameters using travel distance information and multiple images of a mobile object equipped with multiple cameras. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-107938 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 requires that the moving object be equipped with multiple cameras and a sensor capable of acquiring travel distance information, making it difficult to apply this technology to moving objects that do not have sensors capable of acquiring travel distance information.

[0006] Therefore, the present disclosure proposes a new and improved information processing device, information processing method, and program that can more simply detect whether or not deviations in camera parameters have occurred among multiple cameras. [Means for solving the problem]

[0007] According to the present disclosure, there is provided an information processing device comprising: a calculation unit that calculates, based on each image obtained by a plurality of cameras photographing an object at a first timing and extrinsic parameters of the plurality of cameras, a three-dimensional position of a feature point included in each of the images; a first estimation unit that estimates, based on one image included in each image obtained by the plurality of cameras photographing the object at a second timing and the three-dimensional position of the feature point, a first extrinsic parameter that is an extrinsic parameter of a camera among the plurality of cameras that photographed the one image at the second timing, based on the first extrinsic parameter estimated by the first estimation unit; and a determination unit that determines whether a deviation related to the extrinsic parameter has occurred in the plurality of cameras, based on the second extrinsic parameter of the one camera among the plurality of cameras estimated by the second estimation unit and the previous extrinsic parameter of the camera.

[0008] Furthermore, according to the present disclosure, there is provided an information processing method executed by a computer, including: calculating, based on each image obtained by a plurality of cameras photographing an object at a first timing and extrinsic parameters of the plurality of cameras, a three-dimensional position of a feature point included in each of the images; estimating, based on one image included in each image obtained by the plurality of cameras photographing the object at a second timing and the three-dimensional position of the feature point, a first extrinsic parameter that is an extrinsic parameter of a camera among the plurality of cameras that photographed the one image at the second timing, based on the estimated first extrinsic parameter; and determining, based on the estimated second extrinsic parameter of the one camera among the plurality of cameras and the previous extrinsic parameter of the camera, whether a deviation related to the extrinsic parameter has occurred in the plurality of cameras.

[0009] Furthermore, according to the present disclosure, there is provided a program that causes a computer to implement the following: a calculation function that calculates, based on each image obtained by multiple cameras photographing an object at a first timing and extrinsic parameters of the multiple cameras, a three-dimensional position of a feature point included in each image; a first estimation function that estimates, based on one image included in each image obtained by the multiple cameras photographing the object at a second timing and the three-dimensional position of the feature point, a first extrinsic parameter that is an extrinsic parameter of a camera among the multiple cameras that photographed the one image at the second timing, based on the first extrinsic parameter estimated by the first estimation function; and a determination function that determines whether a deviation related to the extrinsic parameter has occurred in the multiple cameras, based on the second extrinsic parameter of the one camera among the multiple cameras estimated by the second estimation function and the previous extrinsic parameter of the camera. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of an information processing system according to the present disclosure. [Figure 2] 2 is an explanatory diagram illustrating an example of a functional configuration of an information processing device 30 according to the present disclosure. FIG. [Figure 3] FIG. 10 is an explanatory diagram illustrating an example of distance measurement processing using the principle of triangulation. [Figure 4] 3 is an explanatory diagram for explaining an example of operation processing of the information processing device 30 according to the present disclosure. FIG. [Figure 5] 1 is an explanatory diagram for explaining an example of image capture by a stereo camera 10 according to the present disclosure. [Figure 6] 10A and 10B are explanatory diagrams for explaining an example of operation processing related to image processing according to the present disclosure. [Figure 7] FIG. 1 is an explanatory diagram illustrating an example of a feature point detection method according to the Harris method. [Figure 8] 10A and 10B are explanatory diagrams for explaining an example of an operation process related to determining a set of feature points according to the present disclosure. [Figure 9] FIG. 10 is an explanatory diagram for explaining a specific example of a correlation calculation according to the present disclosure. [Figure 10] 10A and 10B are explanatory diagrams for explaining an example of an operation process related to determining whether an image group is suitable for misalignment determination according to the present disclosure. [Figure 11A] 10 is an explanatory diagram for explaining a first example of a process of a method for provisionally determining whether or not a deviation has occurred in the stereo camera 10 according to the present disclosure. FIG. [Figure 11B] 10 is an explanatory diagram for explaining a second example of a process of a method for provisionally determining whether or not a deviation has occurred in the stereo camera 10 according to the present disclosure. FIG. [Figure 11C] 10 is an explanatory diagram for explaining a third example of a process of a method for provisionally determining whether or not a deviation has occurred in the stereo camera 10 according to the present disclosure. FIG. [Figure 12] FIG. 10 is an explanatory diagram for explaining an example of a final determination of whether or not a deviation has occurred according to the present disclosure. [Figure 13]10 is an explanatory diagram illustrating an example of notification information generated by a notification information generating unit 351. FIG. [Figure 14] FIG. 2 is a block diagram showing the hardware configuration of an information processing device 30. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The "Mode for Carrying Out the Invention" will be described in the following order. 1. Overview of the information processing system 2. Example of functional configuration of information processing device 30 3. Identifying issues 4. Example of operation processing 4.1. Overall Operation 4.2.Example of photography 4.3. Image processing example 4.4. Harris's method for detecting feature points 4.5. Identifying feature point pairs 4.6. Determine whether the image group is suitable for judgment processing 4.7. Camera Misalignment Detection 5. Examples of effects 6. Hardware configuration example 7. Supplementary Information

[0013] <<1. Overview of the information processing system>> As an embodiment of the present disclosure, a mechanism for detecting whether or not deviations have occurred in camera parameters of a plurality of cameras based on images captured by the plurality of cameras will be described.

[0014] 1 is an explanatory diagram illustrating an example of an information processing system according to the present disclosure. The information processing system according to the present disclosure includes a network 1, a moving object 5, a stereo camera 10, an information processing device 30, and an information terminal TB.

[0015] (Network 1) Network 1 is a wired or wireless transmission path for information transmitted from devices connected to network 1. For example, network 1 may include public network such as the Internet, a telephone network, or a satellite communication network, or various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). Network 1 may also include dedicated network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0016] The information terminal TB and the information processing device 30 are connected via a network 1.

[0017] (Mobile 5) The mobile object 5 is a device that moves autonomously or by user operation. The mobile object 5 may be, for example, a drone as shown in Fig. 1. The mobile object 5 may also be a car, a ship, or an aircraft.

[0018] {Stereo Camera 10} The stereo camera 10 photographs a subject and acquires an image. The stereo camera 10 acquires depth information of the subject by mounting two cameras side by side. In the following description, of the two cameras mounted on the stereo camera 10, the camera mounted on the left side as viewed from the subject is referred to as the left camera 15A, and the camera mounted on the right side as viewed from the subject is referred to as the right camera 15B. In the following description, the left camera 15A and the right camera 15B may be collectively referred to as the stereo camera 10 unless a particular distinction is required.

[0019] Furthermore, in this specification, two cameras, left camera 15A and right camera 15B, are described as the multiple cameras mounted on moving body 5, but the number of cameras mounted on moving body 5 is not limited to this example. For example, the number of cameras mounted on moving body 5 may be three or more.

[0020] {Information processing device 30} The information processing device 30 estimates the extrinsic parameters of the left camera 15A or the right camera 15B based on each image obtained by photographing a subject at multiple times using the stereo camera 10. Furthermore, the information processing device 30 determines whether or not a deviation has occurred in the extrinsic parameters of the left camera 15A or the right camera 15B based on the estimated extrinsic parameters and the extrinsic parameters set previously. That is, it determines whether or not a deviation has occurred between the actual installation position or orientation of the left camera 15A or the right camera 15B and the installation position or orientation corresponding to the set extrinsic parameters.

[0021] (Information terminal TB) The information terminal TB is a terminal used by the user OP. The information terminal TB may be, for example, a tablet terminal as shown in Fig. 1, or may be various devices such as a smartphone or a PC (Personal Computer).

[0022] For example, the display provided on the information terminal TB displays images captured by the stereo camera 10. The information terminal TB also remotely controls the moving object 5 based on operations by the user OP.

[0023] The information processing system according to the present disclosure has been outlined above. Next, an example of the functional configuration of the information processing device 30 according to the present disclosure will be described with reference to FIG.

[0024] <<2. Example of Functional Configuration of Information Processing Device 30>> Fig. 2 is an explanatory diagram for explaining an example of the functional configuration of the information processing device 30 according to the present disclosure. As shown in Fig. 2, the moving object 5 includes a stereo camera 10, an operating device 20, and the information processing device 30. The functional configuration of the stereo camera 10 has been explained with reference to Fig. 1, and therefore will not be explained in Fig. 2.

[0025] (Operating device 20) The operating device 20 is a device that operates under the control of an operation control unit 355, which will be described later. The operating device 20 includes, for example, an engine, a braking device, etc. Specific examples of the operation of the operating device 20 will be described later.

[0026] (Information processing device 30) As shown in FIG. 2, the information processing device 30 according to the present disclosure includes a communication unit 310, a storage unit 320, and a control unit 330.

[0027] {Communications Department 310} The communication unit 310 performs various communications with the information terminal TB. For example, the communication unit 310 receives operation information of the mobile object 5 from the information terminal TB. The communication unit 310 also transmits notification information generated by a notification information generation unit 351 (described later) to the information terminal TB.

[0028] {Storage section 320} The storage unit 320 holds software and various data. For example, the storage unit 320 stores provisional determination results determined by the deviation detection unit 347. Furthermore, when the number of stored provisional determination results exceeds a predetermined number, the storage unit 320 may delete the oldest determination results in order.

[0029] {Control unit 330} The control unit 330 controls the overall operation of the information processing device 30 according to the present disclosure. As shown in FIG. 2 , the control unit 330 includes an image processing unit 331, a feature point detection unit 335, a pair determination unit 339, an estimation unit 343, a deviation detection unit 347, a notification information generation unit 351, an operation control unit 355, a distance measurement unit 359, and a distance measurement data utilization unit 363.

[0030] The image processing unit 331 performs image processing on each image acquired by the stereo camera 10. For example, the image processing unit 331 performs various types of image processing, such as shading correction and noise reduction, on each image.

[0031] The image processing unit 331 also performs various processes such as removing lens distortion, parallelization, and cropping.

[0032] The feature point detection unit 335 detects feature points from each image obtained by photographing a subject at a certain timing using the stereo camera 10.

[0033] For example, the feature point detection unit 335 may detect feature points from each image using known techniques such as the Harris method or SIFT (Scale Invariant Feature Transform). Note that in this specification, a feature point detection method using the Harris method will be described later as an example of a method by which the feature point detection unit 335 detects feature points from an image.

[0034] The pair discrimination unit 339 is an example of a discrimination unit, and discriminates, as a pair of feature points, feature points included in each image obtained by photographing a subject at a certain timing with feature points included in another image that have the highest degree of correlation with each other.

[0035] The estimation unit 343 is an example of a calculation unit, and calculates the three-dimensional positions of feature points included in each image based on each image obtained by photographing the subject with the stereo camera 10 at a certain timing and the external parameters of the multiple cameras.

[0036] The estimation unit 343 is an example of a first estimation unit, and estimates, as first external parameters, the extrinsic parameters of the camera that captured one of the images of the stereo camera 10 at another timing, based on one of the images obtained by the stereo camera 10 capturing an image of the subject at the second timing and the calculated three-dimensional position of the feature point.

[0037] In addition, the estimation unit 343 is an example of a second estimation unit, and estimates the external parameters of the left camera 15A or the right camera 15B at the above-mentioned first timing as second external parameters based on the estimated first external parameters.

[0038] The deviation detection unit 347 is an example of a judgment unit, and judges whether or not a deviation related to the external parameters of the left camera 15A or the right camera 15B has occurred based on the estimated external parameters of the left camera 15A or the right camera 15B and the external parameters of the camera at the time of the previous setting.

[0039] The notification information generation unit 351 is an example of a notification unit, and generates notification information related to the deviation when the deviation detection unit 347 determines that a deviation related to an external parameter has occurred in the left camera 15A or the right camera 15B. The notification information generation unit 351 also causes the communication unit 310 to transmit the generated notification information to the information terminal TB of the user OP. Specific examples of the notification information will be described later.

[0040] When the deviation detection unit 347 determines that a deviation related to an external parameter has occurred in the left camera 15A or the right camera 15B, the operation control unit 355 controls a predetermined operation of the moving object 5 on which the stereo camera 10 is mounted. For example, when the deviation detection unit 347 determines that a deviation related to an external parameter has occurred in the left camera 15A or the right camera 15B, the operation control unit 355 may control the operating device 20, such as the engine, to impose a speed limit on the moving object.

[0041] Furthermore, the operation control unit 159 may control the operation device 20 via a control device that controls the overall operation of the moving object 5.

[0042] Furthermore, the operation control unit 159 may control the operation device 20 based on various information obtained by the ranging data utilization unit 363. For example, when the ranging data utilization unit 363 determines that there is a high possibility of a collision with an object, the operation control unit 159 may control a braking device to stop the moving object 5.

[0043] The distance measurement unit 359 executes distance measurement processing to calculate the distance from the stereo camera 10 to the subject based on the images and camera parameters obtained by the stereo camera 10. The distance measurement processing according to the present disclosure may use known techniques such as the principle of triangulation.

[0044] The ranging data utilization unit 363 utilizes the ranging information calculated by the ranging unit 359. For example, the ranging data utilization unit 363 may determine the possibility of a collision between the moving body 5 equipped with the stereo camera 10 and an object, based on the calculated ranging information.

[0045] An example of the functional configuration of the information processing device 30 according to the present disclosure has been described above. Next, the problem associated with the present disclosure will be described in detail.

[0046] <<3. Identifying the issues>> Various operation policies can be applied to the moving body 5, such as autonomous movement or movement by user OP operation. In any operation policy, it is desirable to estimate the distance from the moving body 5 to an object in order to reduce the possibility of the moving body 5 colliding with an object such as an obstacle or an animal while traveling.

[0047] Here, with reference to FIG. 3, a method using the principle of triangulation will be described as an example of estimating the distance from the moving body 5 to an object.

[0048] 3 is an explanatory diagram for explaining an example of distance measurement processing using the principle of triangulation. In FIG. 3, when a subject P is photographed by the left camera 15A and the right camera 15B, the image pickup surface S of the left camera 15A is L and the imaging surface S of the right camera 15B. R 10 shows the imaging positions of the respective subjects P in the image plane.

[0049] Imaging position P L is the image pickup surface S of the left camera 15A when the left camera 15A photographs the subject P. L The object P is imaged at the imaging position P R is the image of the image pickup surface S of the right camera 15B when the right camera 15B photographs the subject P at the same timing as the left camera 15A. R is the position where the subject P was imaged.

[0050] In addition, the imaging position P of the subject P in the left camera 15A Land the imaging position P of the subject P in the right camera 15B. R The difference between these is called the parallax S. R Upper imaging position P L is illustrated as an aid to explain the parallax S.

[0051] Here, the distance DS is expressed by the following equation (Equation 1) using the base line length B, the focal length F, and the parallax S.

[0052]

number

[0053] As shown in the above equation (Equation 1), the distance DS is calculated using the parallax S. For example, the parallax S decreases as the distance from the stereo camera 10 to the subject P increases, and increases as the distance decreases.

[0054] The parallax S is the image pickup position P of the subject P in the left camera 15A. L and the imaging position P of the subject P in the right camera 15B. R and the imaging position P L and imaging position P R depends on external parameters set based on the positions and orientations of the left camera 15A and the right camera 15B.

[0055] On the other hand, there is no guarantee that external parameters set at a certain time in the past are still accurate at the present time. For example, the position or orientation of left camera 15A or right camera 15B may change due to disturbances such as vibrations or temperature changes. When the position or orientation changes, the accuracy of the external parameters set based on the original position or orientation of left camera 15A or right camera 15B also decreases.

[0056] As a result of the reduction in accuracy of the external parameters, the value of the disparity S becomes inaccurate, which may reduce the accuracy of estimating the distance DS by the distance measuring unit 359. Therefore, it is desirable to periodically detect whether a deviation related to the external parameters has occurred in the left camera 15A or the right camera 15B.

[0057] Furthermore, if the direction of the line connecting the centers of left camera 15A and right camera 15B is defined as the x direction and the direction perpendicular to the x direction is defined as the y direction, changes in the positions and attitudes of left camera 15A and right camera 15B, for example, deviation in the y direction can be easily detected by parallelization processing described later, whereas deviation in the x direction has been difficult to detect from images alone.

[0058] Therefore, the information processing device 30 according to the present disclosure determines whether or not a deviation in an external parameter has occurred in the left camera 15A or the right camera 15B, based on a group of images obtained by photographing a subject. Note that the deviation in an external parameter includes, for example, a deviation in the mounting angle or the mounting position of the left camera 15A or the right camera 15B.

[0059] Hereinafter, with reference to FIGS. 4 to 13, an example of operation processing in which the information processing device 30 according to the present disclosure determines whether or not a deviation has occurred in the external parameters of the stereo camera 10 will be sequentially described in detail.

[0060] <<4. Operational Processing Example>> <4.1. Overall Operation> 4 is an explanatory diagram for explaining an example of operation processing of the information processing device 30 according to the present disclosure. First, the stereo camera 10 captures an image of a subject and acquires an image group including a plurality of images (S101).

[0061] Next, the image processing unit 331 performs image processing on each image captured by the stereo camera 110 (S105).

[0062] Next, the feature point detection unit 335 detects feature points from each image (S109).

[0063] Then, the pair determination unit 339 determines pairs of feature points that have a high degree of correlation between each of the feature points detected from one image and each of the feature points detected from another image (S113).

[0064] Then, the estimation unit 343 determines whether or not the group of images captured by the stereo camera 10 are suitable for determining a deviation (S117).

[0065] Next, the misalignment detection unit 347 executes a provisional misalignment determination process using a group of images suitable for misalignment determination (S121).

[0066] Then, the storage unit 320 stores the provisional determination result of the deviation (S125).

[0067] Then, the control unit 330 determines whether or not the provisional misalignment determination process has been performed using a predetermined number of image groups (S129). If the provisional misalignment determination process has been performed using the predetermined number of image groups (S129 / Yes), the process proceeds to S133, and if the provisional misalignment determination process has not been performed using the predetermined number of image groups (S129 / No), the process proceeds to S101.

[0068] When a provisional determination process for misalignment is performed using a predetermined number of image groups (S129 / Yes), the misalignment detection unit 347 determines whether or not a misalignment related to an external parameter has occurred in the left camera 15A or the right camera 15B based on the provisional determination results for each misalignment (S133), and the information processing device 30 according to the present disclosure terminates the process.

[0069] An example of the overall operation processing of the information processing device 30 according to the present disclosure has been described above. Next, specific examples of the operation processing relating to S101 to S133 will be described in order. First, an example of image capture by the stereo camera 10 according to the present disclosure will be described with reference to FIG.

[0070] <4.2. Photo examples> FIG. 5 is an explanatory diagram illustrating an example of imaging by the stereo camera 10 according to the present disclosure. The stereo camera 10 according to the present disclosure captures images at intervals of a time width T1. For example, the time width T1 may be set to a value that allows for detection of the amount of movement of feature points in each image while including overlapping of subjects (e.g., T1=0.2 seconds). Images captured at such intervals of the time width T1 may be referred to as an image group PG. Note that in FIG. 5, the stereo camera 10 captures images twice at an interval of the time width T1, and the images obtained by the two captures are considered to be one image group PG. However, the stereo camera 10 may capture images three or more times at an interval of the time width T1. In this case, the images obtained corresponding to the number of captures are considered to be one image group PG.

[0071] Furthermore, the stereo camera 10 may acquire an image group PG by capturing images at intervals of a time span T2. ​​For example, the stereo camera 10 may acquire images while changing the subject at intervals of a time span T2 (e.g., 10 to 60 seconds). As will be described in detail later, an image group PG is acquired for each time span T2, and a provisional determination process for the misalignment is performed using each of the image groups PG acquired for each time span T2.

[0072] Furthermore, although the present specification mainly describes an example in which the moving object 5 is equipped with one stereo camera 10, the moving object 5 may be equipped with multiple stereo cameras 10. When the moving object 5 is equipped with multiple stereo cameras 10, the control unit 330 may control the timing of capturing images of the multiple stereo cameras 10 using, for example, a round robin method.

[0073] An example of imaging by the stereo camera 10 according to the present disclosure has been described above. Next, referring to FIG. 6, An example of operation processing related to image processing according to the present disclosure will be described.

[0074] <4.3. Image processing examples> 6 is an explanatory diagram for explaining an example of operation processing related to image processing according to the present disclosure. First, the image processing unit 331 performs lens distortion removal on each image obtained by photographing with the stereo camera 10 using the camera parameters of the stereo camera 10 (S201).

[0075] Next, the image processing unit 331 performs a parallelization process on each image obtained by photographing with the stereo camera 10 (S205). The parallelization process is a process of matching the imaging position of a certain subject in the y direction for each image obtained by photographing the subject with the stereo camera 10. The direction of a straight line connecting the centers of the left camera 15A and the right camera 15B is defined as the x direction, and the direction perpendicular to the x direction is defined as the y direction.

[0076] Then, the image processing unit 331 performs cropping processing on the image that has undergone lens distortion removal and parallelization processing to cut it out to a desired image size (S209), and the image processing unit 331 according to the present disclosure ends the processing related to the image processing.

[0077] The above describes an example of the image processing operation according to the present disclosure. Next, an example of the feature point detection operation according to the present disclosure will be described with reference to FIG.

[0078] 4.4. Feature point detection using Harris's method 7 is an explanatory diagram for explaining an example of a method for detecting feature points by the Harris method. First, the feature point detection unit 335 generates a differential image in the x direction from each input image (S301).

[0079] The feature point detection unit 335 also generates a differential image in the y direction from each of the input images (S305). Note that the feature point detection unit 335 may generate a differential image in the x direction and a differential image in the y direction by, for example, applying a Sobel filter according to the x direction and the y direction to each of the input images.

[0080] Then, the feature point detection unit 335 calculates the matrix M(x, y) using the pixel values ​​at the same pixel positions of the differential images in each direction and the following equation (Equation 2) (S309).

[0081]

number

[0082] Note that g(u, v) is a weighting coefficient, and may be, for example, a Gaussian function with x and y as the origin. x is the pixel value of the differential image in the x direction, and I y is the pixel value of the differential image in the y direction.

[0083] Next, the feature point detection unit 335 calculates the feature amount R(x, y) of the pixel (x, y) using the matrix M(x, y) and the following equation (Equation 3) (S313).

[0084]

number

[0085] Here, detM(x, y) is the value of the determinant of the matrix M(x, y), and trM is the trace of the matrix M(x, y). Furthermore, k is a parameter designated by the user, and is designated within the range of 0.04 to 0.06, for example.

[0086] Then, the feature point detection unit 335 performs the processes of S309 to S313 on all pixels of the input image (S317). Therefore, if the processes of S309 and S313 have not been performed on all pixels of the input image (S317 / No), the process returns to S309, and if the processes of S309 and S313 have been performed on all pixels of the input image (S317 / Yes), the process proceeds to S321.

[0087] Then, the feature point detection unit 335 detects feature points based on the feature amounts R(x, y) of all the pixels (S321).

[0088] For example, the feature point detection unit 335 detects a pixel position where the feature amount R(x, y) is a maximum and is equal to or greater than a threshold value designated by the user as a feature point (for example, a corner point) of the image.

[0089] An example of a method for detecting feature points using the Harris method has been described above. Next, an example of a method for determining a set of feature amounts according to the present disclosure will be described with reference to FIG.

[0090] <4.5. Identifying pairs of feature points> 8 is an explanatory diagram for explaining an example of an operation process related to determining a set of feature points according to the present disclosure. In FIG. 8, an example of a method in which the pair determination unit 339 determines a set of feature points from each feature point included in each of two images is explained. In the following explanation, of the two images, the image obtained by photographing with the left camera 15A may be referred to as the Left image, and the image obtained by photographing with the right camera 15B may be referred to as the Right image.

[0091] First, the pair determination unit 339 acquires one feature point on the left image side (S401).

[0092] Next, the pair determination unit 339 acquires the u×v image blocks that are set as feature points from the Left image (S405).

[0093] Then, the pair determination unit 339 acquires one feature point on the right image side (S409).

[0094] Then, the pair determination unit 339 acquires a u×v image block centered on the feature point from the right image (S413).

[0095] Next, the pair determination unit 339 performs a correlation calculation between the image block on the light side and the image block on the right side (S417).

[0096] For example, the pair discrimination unit 339 may calculate the degree of correlation between each feature point using a known calculation method for the correlation calculation according to the present disclosure, or may calculate the degree of correlation between each feature point using, for example, any of the following equations (Equation 4) to (Equation 7).

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101] Note that AVE in (Equation 7) represents an average value. Here, a specific example of correlation calculation according to the present disclosure will be described with reference to FIG.

[0102] 9 is an explanatory diagram for explaining a specific example of correlation calculation according to the present disclosure. The pair determination unit 339 acquires pixel values ​​within a u×v block range with a certain feature point in the Left image as the origin.

[0103] Then, the pair determination unit 339 similarly obtains the range of the u×v block on the right image side, with a certain feature point as the origin.

[0104] Then, the pair discriminator 339 calculates the pixel values ​​I of the Left image in the range of u×v. l and each pixel value I of the right image R The correlation degree of the feature points is calculated by applying the above-mentioned formulas 4 to 7 to the above.

[0105] A specific example of correlation calculation according to the present disclosure has been described above. Referring again to FIG. 8, the remaining example of the operational processing relating to determining a set of feature points will be described.

[0106] After a correlation calculation is performed between the feature points on the left image side and the feature points on the right image side (S417), the pair determination unit 339 leaves the feature points on the right image side that have the highest degree of correlation with the feature points on the left image side acquired in S401 as candidates for feature point pairs (S421).

[0107] Then, the pair determination unit 339 executes the processes of S409 to S421 between the feature points on the left image side acquired in S401 and all feature points on the right image side (S425). Therefore, if not all feature points on the right image side have been checked for the feature points on the left image side acquired in S401 (S425 / No), the process returns to S409, and if all feature points on the right image side have been checked (S425 / Yes), the process proceeds to S429.

[0108] If all feature points on the right image side have been checked (S425 / Yes), the pair determination unit 339 determines whether the correlation value between the feature points on the left image side acquired in S401 and the feature points on the right image side finally remaining in S421 is equal to or greater than a predetermined value (S429). If the correlation value is less than the predetermined value (S429 / No), the process proceeds to S437, and if the correlation value is equal to or greater than the predetermined value (S429 / Yes), the process proceeds to S433.

[0109] If the correlation value is less than the predetermined value (S429 / No), the pair determination unit 339 determines that there is no feature point on the right image side that corresponds to the pair of feature points on the left image side acquired in S401 (S437).

[0110] If the correlation value is greater than or equal to a predetermined value (S429 / Yes), the pair determination unit 339 determines that the feature points on the left image side acquired in S401 and the feature points on the right image side finally remaining in S421 are a pair of feature points (S433).

[0111] Then, the pair determination unit 339 executes the processing of S401 to S437 related to determining pairs of feature points for all feature points on the left image side (S441). Therefore, if the processing related to determining pairs of feature points has not been executed for all feature points on the left image side (S441 / No), the processing returns to S401 again, and if the processing related to matching has been executed for all feature points on the left image side (S441 / Yes), the pair determination unit 339 according to the present disclosure ends the processing.

[0112] The above describes an example of the operation process related to determining a set of feature points according to the present disclosure. Next, with reference to FIG. 10 , an example of the operation process in which the estimation unit 343 determines whether an image group is suitable for misalignment determination based on feature points and sets of feature points will be described.

[0113] <4.6. Determine whether the image group is suitable for judgment processing> 10 is an explanatory diagram illustrating an example of an operational process for determining whether an image group is suitable for misalignment determination according to the present disclosure. To improve the accuracy of the misalignment determination process, pairs of feature points determined from each image must indicate the same target position. Meanwhile, the pair determination unit 339 does not necessarily determine that feature points located at the same target position are a pair of feature points. Therefore, the greater the number of feature points detected by the feature point detection unit 335 and the greater the number of pairs of feature points determined by the pair determination unit 339 (i.e., the greater the number of samples), the more likely it is that the influence of misidentification of pairs of feature points can be reduced.

[0114] 9 is an explanatory diagram for explaining an example of an operation process for determining whether an image group is suitable for the misalignment determination process according to the present disclosure. First, the feature point detection unit 335 detects feature points from each image (S109).

[0115] Next, the estimation unit 343 determines whether the number of detected feature points is equal to or greater than a predetermined value (S501). If the number of feature points is equal to or greater than the predetermined value (S501 / Yes), the process proceeds to S113. If the number of feature points is less than the predetermined value (S501 / No), the process proceeds to S525.

[0116] If the number of feature points is equal to or greater than the predetermined value (S501 / Yes), the pair determination unit 339 determines pairs of feature points from each image (S113).

[0117] If the number of sets of feature points is equal to or greater than the predetermined value (S509 / Yes), the estimation unit 343 proceeds to S513, and if the number of sets of feature points is less than the predetermined value (S509 / No), the estimation unit 343 proceeds to S525.

[0118] If the number of pairs of feature points is equal to or greater than the predetermined value (S509 / Yes), the estimation unit 343 calculates the amount of change in the feature point positions between the images (S513).

[0119] Then, the estimation unit 343 determines whether the calculated amount of change in the feature point position is equal to or greater than a predetermined value (S517). If the amount of change in the feature point position is equal to or greater than the predetermined value (S517 / Yes), the process proceeds to S521. If the amount of change in the feature point position is less than the predetermined value (S517 / No), the process proceeds to S525. Note that if the moving object 5 is equipped with a sensor that acquires the amount of movement, the amount of change in the feature point position may be estimated based on movement information of the moving object 5 acquired by the sensor.

[0120] If the amount of change in the feature point position is greater than or equal to a predetermined value (S517 / Yes), the estimation unit 343 determines that the image group is suitable for the deviation determination process (S521), and the information processing device 30 according to the present disclosure ends the process.

[0121] If the number of feature points is less than a predetermined value (S501 / No), if the number of sets of feature points is less than a predetermined value (S509 / No), or if the amount of change in feature point positions is less than a predetermined value (S517 / No), the estimation unit 343 determines that the image group is not suitable for the deviation determination process (S525), and the information processing device 30 according to the present disclosure terminates the process.

[0122] The above describes an example of an operational process for determining whether an image is suitable for the calibration process according to the present disclosure, but the method for determining whether an image is suitable for the calibration process according to the present disclosure is not limited to this example.

[0123] For example, the estimation unit 343 may determine whether or not the image group is suitable for the displacement determination process by combining the process of 1 or 2, without executing all of the processes of S501, S509, and S517.

[0124] Furthermore, the estimation unit 343 may divide each image (for example, into four areas) and perform the process of S501, S509, or S517 on each divided area.

[0125] <4.7. Determining Camera Misalignment> The deviation determination process according to the present disclosure includes two steps: a provisional determination process and a final determination process. Details of the provisional determination and the final determination will be explained below.

[0126] (Provisional determination of whether a discrepancy has occurred) An example of a method for provisionally determining whether or not a deviation has occurred according to the present disclosure will be described with reference to FIGS. 11A to 11C.

[0127] {1st step} 11A is an explanatory diagram illustrating an example of a first step of a method for provisionally determining whether or not misalignment has occurred in the stereo camera 10 according to the present disclosure. First, the estimation unit 343 calculates the three-dimensional positions of feature points included in each image based on the images captured by the left camera 15A and the right camera 15B at time T, and the external parameters P1 of the left camera 15A and the external parameters P2 of the right camera 15B. Note that the feature points whose three-dimensional positions are calculated are the feature points that have been determined by the pair determination unit 339 to be a pair of feature points.

[0128] Then, the estimation unit 343 estimates the external parameter P4' of the right camera 15B at time T-T1 as the first external parameter based on the three-dimensional position of the calculated feature point and the feature point of the image captured by the right camera 15B at time T-T1.

[0129] {Second step} 11B is an explanatory diagram for explaining an example of a second step of the method for provisionally determining whether or not misalignment has occurred in the stereo camera 10 according to the present disclosure. Following the first step, the estimation unit 343 calculates three-dimensional positions of feature points included in each image based on the image captured by the left camera 15A at time T and the image captured by the right camera 15B at time T-T1, as well as the external parameter P1 of the left camera 15A and the external parameter P4' of the right camera 15B.

[0130] Then, the estimation unit 343 estimates the external parameter P3' of the left camera 15A at time T-T1 as the first external parameter based on the three-dimensional position of the calculated feature point and the feature point of the image captured by the left camera 15A at time T-T1.

[0131] {3rd step} 11C is an explanatory diagram illustrating an example of a third step of the method for provisionally determining whether or not a misalignment has occurred in the stereo camera 10 according to the present disclosure. Following the second step, the estimation unit 343 calculates the three-dimensional positions of feature points included in each image based on the images captured by the left camera 15A and the right camera 15B at time T-1 and the external parameters P3' and P4' estimated as the first external parameters.

[0132] Then, the estimation unit 343 estimates the external parameter P1' of the left camera 15A at time T as the second external parameter based on the calculated three-dimensional position of the feature point and the feature point of the image captured by the left camera 15A at time T.

[0133] {Tentative judgment} The deviation detection unit 347 provisionally determines whether a deviation related to an external parameter has occurred in either the left camera 15A or the right camera 15B, based on the external parameter P1' obtained in the third step and the external parameter P1 at the time of the previous setting. For example, when the difference between the external parameter P1' and the external parameter P1 is equal to or greater than a predetermined value, the deviation detection unit 347 provisionally determines that a deviation related to an external parameter has occurred in either the left camera 15A or the right camera 15B. Note that deviations related to external parameters include, for example, deviations related to the mounting angle or mounting position of either the left camera 15A or the right camera 15B.

[0134] Although an example of a method for provisionally determining whether a deviation has occurred according to the present disclosure has been described above, the provisional deviation determination process according to the present disclosure is not limited to this example. For example, the estimation unit 343 may estimate the external parameter P4' in the first step described above. Subsequently, the estimation unit 343 may estimate the external parameter P1' based on the external parameter P2 and the external parameter P4'. This makes it possible to omit the second step, thereby simplifying the process.

[0135] Furthermore, the estimation unit 343 estimates the external parameters P3' and P4' through the first and second steps. Subsequently, the deviation detection unit 347 may compare the external parameters P1 and P2 from the previous setting with the external parameters P3' and P4' estimated by the estimation unit 343, and determine whether a deviation related to the external parameters has occurred in either the left camera 15A or the right camera 15B. This makes it possible to omit the third step, thereby simplifying the processing.

[0136] 11A to 11C illustrate an example in which the estimation unit 343 estimates the external parameter P1' of the left camera 15A through the first to third steps, but the estimation unit 343 may estimate the external parameter P2' of the right camera 15B. In this case, the deviation detection unit 347 may provisionally determine whether a deviation related to the external parameter has occurred in either the left camera 15A or the right camera 15B, based on the external parameter P2' obtained by the estimation unit 343 and the external parameter P2 at the time of the previous setting.

[0137] Furthermore, instead of making a binary determination as to whether or not a deviation has occurred, the deviation detection unit 347 may make a determination regarding the deviation by dividing the determination into multiple stages such as the degree of danger, or by making a determination regarding the deviation as a continuous value such as the probability of deviation occurring. In this case, the notification information generation unit 351 and the operation control unit 355 may generate notification information and control the operation device 20 according to the degree of danger.

[0138] 11A to 11C, an example has been described in which a provisional determination process for misalignment is performed from a group of images obtained by photographing with the stereo camera 10 at two timings, t=T and t=T-T1, but a provisional determination process for misalignment may also be performed from a group of images obtained by photographing with the stereo camera 10 at three or more timings.

[0139] Furthermore, the deviation detection unit 347 may determine the result of one provisional determination process as the final determination result, or may estimate the final determination result based on the results of multiple provisional determination processes. Next, an example of the final determination of whether or not a deviation has occurred according to the present disclosure will be described.

[0140] (Final decision on whether or not a discrepancy has occurred) 12 is an explanatory diagram illustrating an example of a final determination of whether or not a deviation has occurred according to the present disclosure. First, the deviation detection unit 347 determines whether or not the majority of the provisional deviation determination results executed the predetermined number of times indicate that there is a deviation (S601). If the majority of the determination results indicate that there is a deviation (S601 / No), the process proceeds to S605. If the majority of the determination results indicate that there is a deviation (S601 / Yes), the process proceeds to S609.

[0141] If the majority of the judgment results indicate that there is a misalignment (S601 / No), the misalignment detection unit 347 determines that there is no misalignment related to the external parameters of the stereo camera 10, and the information processing device 30 according to the present disclosure terminates the processing.

[0142] If the majority of the determination results indicate that there is a deviation (S601 / Yes), the deviation detection unit 347 determines that there is a deviation related to the external parameters of the stereo camera 10 (S609).

[0143] Then, the notification information generating unit 351 generates notification information relating to the deviation, and causes the communication unit 310 to transmit the information to the information terminal TB (S613).

[0144] Then, the operation control unit 355 executes control related to a predetermined operation of the moving object 5 (for example, limiting the speed of the moving object 5) (S617), and the information processing device 30 according to the present disclosure ends the processing.

[0145] 13 is an explanatory diagram for explaining an example of notification information generated by the notification information generating unit 351. The display D of the information terminal TB may display notification information N relating to the deviation in addition to the images acquired by the stereo camera, for example, as shown in FIG.

[0146] The notification information N may be video notification information as shown in FIG. 13, or audio notification information.

[0147] Furthermore, the notification information generation unit 351 may generate notification information for obtaining permission as to whether or not control related to a predetermined operation may be executed by the operation control unit 355. For example, when the user selects that control related to a predetermined operation may be executed, the operation control unit 355 may execute control related to the predetermined operation of the moving object 5.

[0148] 10 that are determined to be unsuitable for the misalignment determination process may be included in the provisional misalignment determination results as indistinguishable tickets. For example, if the indistinguishable tickets are the most prevalent among the provisional misalignment determination results performed a predetermined number of times, the misalignment detection unit 347 may ultimately determine that misalignment has occurred as "indistinguishable."

[0149] The above has described an example of the operation process of the information processing device 30 according to the present disclosure. Next, an example of the effects according to the present disclosure will be described.

[0150] <5. Examples of effects> According to the present disclosure described above, various operational effects can be obtained. For example, it is possible to determine whether a deviation related to an external parameter has occurred in left camera 15A or right camera 15B based on a group of images captured by stereo camera 10. Therefore, it is possible to detect deviation in the horizontal direction of the image plane (the above-mentioned y direction) as well as the vertical direction of the image plane (the above-mentioned x direction) of left camera 15A or right camera 15B without using sensing information from another sensor.

[0151] Furthermore, the estimation unit 343 according to the present disclosure determines whether or not an image group is suitable for misalignment determination, which makes it possible to exclude an image group that is inappropriate for misalignment determination from the determination process, and the misalignment detection unit 347 can detect misalignment occurring in the left camera 15A or the right camera 15B with higher accuracy.

[0152] Furthermore, misalignment detection unit 347 according to the present disclosure provisionally determines whether misalignment has occurred multiple times, and when the number of times it has been provisionally determined that misalignment has occurred meets a predetermined criterion, determines that misalignment related to the external parameters has occurred in left camera 15A or right camera 15B. This allows misalignment detection unit 347 to reduce the influence of erroneous determination that may occur with a single determination result, and can determine whether misalignment has occurred with higher accuracy.

[0153] <6. Hardware configuration example> The above has described an embodiment of the present disclosure. The various information processes described above are realized by cooperation between software and the hardware of the information processing device 30 described below. Note that the hardware configuration described below can also be applied to the information terminal TB.

[0154] 14 is a block diagram showing the hardware configuration of an information processing device 30. The information processing device 30 includes a CPU (Central Processing Unit) 3001, a ROM (Read Only Memory) 3002, a RAM (Random Access Memory) 3003, and a host bus 3004. The information processing device 30 also includes a bridge 3005, an external bus 3006, an interface 3007, an input device 3008, an output device 3010, a storage device (HDD) 3011, a drive 3012, and a communication device 3015.

[0155] The CPU 3001 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 30 in accordance with various programs. The CPU 3001 may also be a microprocessor. The ROM 3002 stores programs used by the CPU 3001, calculation parameters, etc. The RAM 3003 temporarily stores programs used in the execution of the CPU 3001, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 3004 that includes a CPU bus, etc. Cooperation between the CPU 3001, ROM 3002, RAM 3003, and software can realize functions such as the estimation unit 343 and deviation detection unit 347 described with reference to FIG. 2.

[0156] The host bus 3004 is connected to an external bus 3006 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 3005. It is not necessary to configure the host bus 3004, bridge 3005, and external bus 3006 separately, and these functions may be implemented on a single bus.

[0157] The input device 3008 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 3001. By operating the input device 3008, the user of the information processing device 30 can input various data to the information processing device 30 and instruct the information processing device 30 to perform processing operations.

[0158] The output device 3010 includes, for example, a display device such as a liquid crystal display device, an OLED device, and a lamp. Furthermore, the output device 3010 includes an audio output device such as a speaker and a headphone. The output device 3010 outputs, for example, reproduced content. Specifically, the display device displays various information such as reproduced video data as text or images. Meanwhile, the audio output device converts reproduced audio data and the like into audio and outputs the audio.

[0159] The storage device 3011 is a device for storing data. The storage device 3011 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. The storage device 3011 is configured, for example, with an HDD (Hard Disk Drive). This storage device 3011 drives a hard disk and stores programs executed by the CPU 3001 and various data.

[0160] The drive 3012 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 30. The drive 3012 reads information recorded on a removable storage medium 35, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 3003. The drive 3012 can also write information to the removable storage medium 35.

[0161] The communication device 3015 is, for example, a communication interface configured with a communication device for connecting to the network 1. The communication device 3015 may be a wireless LAN compatible communication device, a LTE (Long Term Evolution) compatible communication device, or a wired communication device that performs wired communication.

[0162] The hardware configuration example according to the present disclosure has been described above. Next, supplementary information according to the present disclosure will be described.

[0163] <7. Supplementary Information> Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art to which the present disclosure pertains can conceive of various modifications or alterations within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0164] For example, although the present specification has mainly described an example in which the information processing device 30 is mounted on the moving object 5, the functions of the information processing device 30 may be realized by an information terminal TB. For example, the stereo camera 10 transmits a group of images obtained by photographing a subject to the information terminal TB. Then, the information terminal TB may execute various processes related to determining whether or not a deviation related to an external parameter occurs in the left camera 15A or the right camera 15B based on the received group of images.

[0165] Furthermore, it may be desirable that the image in which feature points are detected by the feature point detection unit 335 is free of blur. Therefore, the feature point detection unit 335 may detect the amount of blur from the image in which feature points are to be detected, and detect feature points from images in which the amount of blur is less than a predetermined value. Furthermore, if the moving object 5 is equipped with a sensor that acquires motion information such as an IMU (Inertial Measurement Unit), the feature point detection unit 335 may estimate the amount of blur based on the motion information acquired by the sensor.

[0166] Furthermore, the steps in the processing of the information processing device 30 in this specification do not necessarily have to be processed in chronological order according to the order described in the flowchart. For example, the steps in the processing of the information processing device 30 may be processed in an order different from the order described in the flowchart or in parallel.

[0167] It is also possible to create a computer program that causes the hardware, such as the CPU, ROM, and RAM, built into the information processing device 30 and the information terminal TB to perform functions equivalent to those of the above-described information processing device 30 and the information terminal TB. A storage medium storing the computer program is also provided.

[0168] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0169] The following configurations also fall within the technical scope of the present disclosure. (1) a calculation unit that calculates three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing, based on each of the images and extrinsic parameters of the plurality of cameras; a first estimation unit that estimates a first extrinsic parameter, which is an extrinsic parameter of a camera that captured the one image among the plurality of cameras at the second timing, based on one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional position of the feature point; a second estimation unit that estimates a second extrinsic parameter, which is an extrinsic parameter of any one of the plurality of cameras at the first timing, based on the first extrinsic parameter estimated by the first estimation unit; and a determination unit that determines whether or not a deviation related to the extrinsic parameters of any one of the plurality of cameras has occurred based on the second extrinsic parameters of the one camera among the plurality of cameras estimated by the second estimation unit and the previous extrinsic parameters of the one camera; An information processing device comprising: (2) The calculation unit when the number of each feature point included in the group of images obtained by the plurality of cameras photographing the subject at the first timing and the second timing is equal to or greater than a predetermined number, calculate a three-dimensional position of the feature point included in each of the images based on each image obtained by photographing the subject at the first timing and the external parameters of the plurality of cameras. The information processing device according to (1) above. (3) a discrimination unit that discriminates, as a set of feature points, feature points included in one of the images that have the highest degree of correlation with feature points included in the other image; Further provided with the calculation unit calculates three-dimensional positions of the feature points included in each of the images based on each of the feature points determined as a set of feature points among the feature points included in each of the images and the external parameters of the plurality of cameras. The information processing device according to (1) or (2). (4) The calculation unit when the number of sets of feature points included in each image obtained by the plurality of cameras photographing the subject at a first timing satisfies a predetermined condition, calculate three-dimensional positions of the feature points included in each image based on each of the feature points determined as a set of feature points and the extrinsic parameters of the plurality of cameras; The information processing device according to (3) above. (5) The predetermined condition includes a case where the number of sets of feature points is equal to or greater than a predetermined number. The information processing device according to (4) above. (6) The calculation unit calculating a three-dimensional position of the feature point included in each of the images when a change amount of the imaging position of the feature point between the first timing and the second timing is equal to or greater than a predetermined value; The information processing device according to any one of (1) to (5). (7) The first estimation unit estimating extrinsic parameters of a camera among the plurality of cameras that captured the other image at the second timing based on another image different from the one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional positions of the feature points; The information processing device according to any one of (1) to (6). (8) The second estimation unit estimating the second extrinsic parameters of any one of the plurality of cameras at the first timing based on the first extrinsic parameters of the plurality of cameras at the second timing estimated by the first estimation unit; The information processing device according to (7) above. (9) a notification unit that, when it is determined by the determination unit that a deviation related to the external parameter has occurred in the plurality of cameras, notifies a user of the plurality of cameras of the deviation; The information processing device according to any one of (1) to (8), further comprising: (10) an operation control unit that executes control related to a predetermined operation of a moving body equipped with the plurality of cameras when the determination unit determines that a deviation related to the external parameter has occurred in the plurality of cameras; The information processing device according to any one of (1) to (9), further comprising: (11) The determination unit using a plurality of image groups, provisionally determining whether or not deviations related to the external parameters have occurred in the plurality of cameras a plurality of times, and determining that deviations related to the external parameters have occurred in the plurality of cameras when the number of times it has been provisionally determined that deviations related to the external parameters have occurred satisfies a predetermined criterion; The information processing device according to (9) or (10). (12) the predetermined criteria include a case where the number of times it is provisionally determined that a deviation has occurred in the posture information of the plurality of cameras is equal to or greater than the number of times it is provisionally determined that no deviation has occurred. The information processing device according to (11) above. (13) calculating three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing and based on extrinsic parameters of the plurality of cameras; estimating a first extrinsic parameter, which is an extrinsic parameter of a camera among the plurality of cameras that captured the one image at the second timing, based on one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional position of the feature point; estimating a second extrinsic parameter, which is an extrinsic parameter of any one of the plurality of cameras at the first timing, based on the estimated first extrinsic parameter; determining whether or not a deviation related to the extrinsic parameters of any one of the plurality of cameras has occurred based on the estimated second extrinsic parameters of the one camera and the previous extrinsic parameters of the camera; 2. A computer-implemented information processing method, comprising: (14) a calculation function that calculates three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing, based on each of the images and external parameters of the plurality of cameras; a first estimation function that estimates a first external parameter, which is an external parameter of a camera among the plurality of cameras that captured the one image at the second timing, based on an image included in each image obtained by the plurality of cameras capturing the subject at the second timing and a three-dimensional position of the feature point; a second estimation function that estimates a second external parameter, which is an external parameter of any one of the plurality of cameras at the first timing, based on the first external parameter estimated by the first estimation function; and a determination function that determines whether or not a deviation related to the external parameters of any one of the plurality of cameras has occurred based on the second external parameters of the one camera among the plurality of cameras estimated by the second estimation function and the previous external parameters of the one camera; A program that enables a computer to achieve this. [Explanation of symbols]

[0170] 1 Network 5. Mobile 10 Stereo Camera 15A Left camera 15B Right Camera 20 Operating device 30 Information processing equipment 310 Communications Department 320 Storage section 330 Control Unit 331 Image Processing Unit 335 Feature Point Detection Unit 339 Pair Discrimination Unit 343 Estimation Department 347 Misalignment detection unit 351 Notification information generation unit 355 Motion control section 359 Ranging section 363 Ranging Data Utilization Department

Claims

1. a calculation unit that calculates three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing, based on each of the images and external parameters of the plurality of cameras; a first estimation unit that estimates a first extrinsic parameter, which is an extrinsic parameter of a camera that captured the one image among the plurality of cameras at the second timing, based on one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional position of the feature point; a second estimation unit that estimates a second external parameter, which is an external parameter of any one of the plurality of cameras at the first timing, based on the first external parameter estimated by the first estimation unit; and a determination unit that determines whether or not a deviation related to the extrinsic parameters of any one of the plurality of cameras has occurred based on the second extrinsic parameters of the one camera among the plurality of cameras estimated by the second estimation unit and the previous extrinsic parameters of the one camera; An information processing device comprising:

2. The calculation unit when the number of each feature point included in the images obtained by the plurality of cameras photographing the subject at the first timing and the second timing is equal to or greater than a predetermined number, calculate a three-dimensional position of the feature point included in each of the images based on each image obtained by photographing the subject at the first timing and the external parameters of the plurality of cameras. The information processing device according to claim 1 .

3. a discrimination unit that discriminates, as a set of feature points, feature points included in one of the images that have the highest degree of correlation with feature points included in the other image; Further provided with the calculation unit calculates three-dimensional positions of the feature points included in each of the images based on each of the feature points determined as a set of feature points among the feature points included in each of the images and the external parameters of the plurality of cameras. The information processing device according to claim 2 .

4. The calculation unit when the number of sets of feature points included in each image obtained by the plurality of cameras photographing the subject at a first timing satisfies a predetermined condition, calculate three-dimensional positions of the feature points included in each image based on each of the feature points determined as a set of feature points and the extrinsic parameters of the plurality of cameras; The information processing device according to claim 3 .

5. The predetermined condition includes a case where the number of sets of feature points is equal to or greater than a predetermined number. The information processing device according to claim 4 .

6. The calculation unit calculating a three-dimensional position of the feature point included in each of the images when a change amount of the imaging position of the feature point between the first timing and the second timing is equal to or greater than a predetermined value; The information processing device according to claim 5 .

7. The first estimation unit estimating extrinsic parameters of a camera among the plurality of cameras that captured the other image at the second timing based on another image different from the one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional positions of the feature points; The information processing device according to claim 6 .

8. The second estimation unit estimating the second extrinsic parameters of any one of the plurality of cameras at the first timing based on the first extrinsic parameters of the plurality of cameras at the second timing estimated by the first estimation unit; The information processing device according to claim 7 .

9. a notification unit that, when it is determined by the determination unit that a deviation related to the external parameter has occurred in the plurality of cameras, notifies a user of the plurality of cameras of the deviation; The information processing device according to claim 8 , further comprising:

10. an operation control unit that executes control related to a predetermined operation of a moving body equipped with the plurality of cameras when the determination unit determines that a deviation related to the external parameter has occurred in the plurality of cameras; The information processing device according to claim 9 , further comprising:

11. The determination unit using a plurality of image groups, provisionally determining whether or not deviations related to the external parameters have occurred in the plurality of cameras a plurality of times, and determining that deviations related to the external parameters have occurred in the plurality of cameras when the number of times it has been provisionally determined that deviations related to the external parameters have occurred satisfies a predetermined criterion; The information processing device according to claim 10.

12. the predetermined criteria include a case where the number of times it is provisionally determined that a deviation has occurred in the posture information of the plurality of cameras is equal to or greater than the number of times it is provisionally determined that no deviation has occurred. The information processing device according to claim 11.

13. calculating three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing and based on extrinsic parameters of the plurality of cameras; estimating a first extrinsic parameter, which is an extrinsic parameter of a camera among the plurality of cameras that captured the one image at the second timing, based on one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional position of the feature point; estimating a second extrinsic parameter, which is an extrinsic parameter of any one of the plurality of cameras at the first timing, based on the estimated first extrinsic parameter; determining whether or not a deviation related to the extrinsic parameters of any one of the plurality of cameras has occurred based on the estimated second extrinsic parameters of the one camera and the previous extrinsic parameters of the one camera; 2. A computer-implemented information processing method, comprising:

14. a calculation function that calculates three-dimensional positions of feature points included in each of the images obtained by a plurality of cameras photographing a subject at a first timing, based on each of the images and external parameters of the plurality of cameras; a first estimation function that estimates a first external parameter, which is an external parameter of a camera among the plurality of cameras that captured the one image at the second timing, based on one image included in each image obtained by the plurality of cameras capturing the subject at the second timing and the three-dimensional position of the feature point; a second estimation function that estimates a second external parameter, which is an external parameter of any one of the plurality of cameras at the first timing, based on the first external parameter estimated by the first estimation function; a determination function that determines whether or not a deviation related to the external parameters of any one of the plurality of cameras has occurred based on the second external parameters of the one camera among the plurality of cameras estimated by the second estimation function and the previous external parameters of the one camera; A program that enables a computer to achieve this.

Images