Information processing device, information processing method, and program

Abstract

The present invention further reduces the effect of positional deviation of a sensor when positional deviation occurs in a sensor. This information processing device comprises a calculation unit that calculates a positional deviation amount for each of a plurality of sensors of different types, a recognition unit that integrates sensing data acquired by the plurality of sensors to execute a recognition task, and a processing unit that masks a partial region of a recognition result of the recognition task on the basis of the positional deviation amount.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] In recent years, systems have been developed that use multiple sensors mounted on a moving object such as a vehicle to recognize the environment around the moving object in three dimensions. By recognizing the environment in three dimensions, the moving object can move autonomously or automatically drive.

[0003] On the other hand, in order to integrate and handle sensing data acquired by a plurality of sensors, it is important to determine the relative positions of these sensors with high accuracy.

[0004] For example, Patent Document 1 listed below discloses a stereo camera that calculates the distance to an object based on outputs from a left camera and a right camera that are arranged so that their fields of view are different. Patent Document 1 also discloses that, in order to calibrate the positional deviation of the left and right cameras due to external factors, the coordinates of feature points of a planar object whose size is known in advance are used.

[0005] JP 2014-74632 A

[0006] As disclosed in Patent Literature 1, in order to calibrate the positional relationships of multiple sensors including cameras, specialized knowledge is generally required in addition to dedicated tools, so users must take their vehicles to a dealership or the like to have a specialist calibrate the positional relationships of the multiple sensors.

[0007] However, frequently bringing the vehicle to a dealer or the like to calibrate the positional relationships of multiple sensors reduces convenience for the user.

[0008] Therefore, when a positional deviation occurs in the sensor, it is desired to reduce the influence of the positional deviation of the sensor.

[0009] According to the present disclosure, there is provided an information processing device including: a calculation unit that calculates a positional deviation amount for each of a plurality of sensors of different types; a recognition unit that integrates sensing data acquired by the plurality of sensors to execute a recognition task; and a processing unit that masks a partial area of ​​the recognition result of the recognition task based on the positional deviation amount.

[0010] Furthermore, according to the present disclosure, there is provided an information processing method by a computer, including calculating a positional deviation amount for each of a plurality of sensors of different types, integrating sensing data acquired by the plurality of sensors to execute a recognition task, and masking a partial area of ​​the recognition result of the recognition task based on the positional deviation amount.

[0011] Furthermore, according to the present disclosure, there is provided a program for causing a computer to function as a calculation unit that calculates the amount of positional deviation of each of a plurality of sensors of different types, a recognition unit that integrates sensing data acquired by the plurality of sensors to execute a recognition task, and a processing unit that masks a partial area of ​​the recognition result of the recognition task based on the amount of positional deviation.

[0012] 1 is an explanatory diagram illustrating a moving body to which the technology according to the present disclosure is applied. FIG. 1 is a block diagram illustrating a functional configuration of an information processing device according to a first embodiment. FIG. 2 is a graph diagram illustrating an example of a criterion for determining whether or not there is misalignment of a sensor. FIG. 3 is a flowchart illustrating the operation flow of an information processing device according to a first embodiment. FIG. 4 is a block diagram illustrating the functional configuration of an information processing device according to a second embodiment. FIG. 5 is a flowchart illustrating the operation flow of an information processing device according to a second embodiment. FIG. 6 is a block diagram illustrating the functional configuration of an information processing device according to a third embodiment. FIG. 7 is a schematic diagram illustrating an example of a criterion for determining whether or not there is misalignment of a sensor. FIG. 8 is a flowchart illustrating the operation flow of an information processing device according to a third embodiment. FIG. 9 is a block diagram illustrating the functional configuration of an information processing device according to a fourth embodiment. FIG. 10 is a flowchart illustrating the operation flow of an information processing device according to the fourth embodiment. FIG. 11 is an explanatory diagram illustrating detection of a skeletal posture of a user in a modified example of the fourth embodiment. FIG. 12 is a block diagram illustrating the functional configuration of an information processing device according to a fifth embodiment. FIG. 13 is a flowchart illustrating the operation flow of an information processing device according to the fifth embodiment. FIG. 14 is a block diagram illustrating the functional configuration of an information processing device according to a sixth embodiment. FIG. 15 is a flowchart illustrating the operation flow of an information processing device according to the sixth embodiment. FIG. 16 is a block diagram illustrating the functional configuration of an information processing device according to a seventh embodiment. FIG. 17 is a flowchart illustrating the operation flow of an information processing device according to the seventh embodiment. FIG. 18 is a block diagram illustrating the functional configuration of an information processing device according to an eighth embodiment. FIG. 19 is a flowchart illustrating a first operation flow of an information processing device according to the eighth embodiment. FIG. 16 is a flowchart showing a second operation flow of the information processing device according to the eighth embodiment. FIG. 17 is a flowchart showing a third operation flow of the information processing device according to the eighth embodiment. FIG. 18 is a block diagram showing the functional configuration of the information processing device according to the ninth embodiment. FIG. 19 is a graph diagram schematically showing the relationship between the amount of positional deviation of the sensor and the threshold value of the masking process. FIG. 19 is a block diagram showing the flow of various processes by the information processing device according to the ninth embodiment. FIG. 19 is a flowchart showing the operation flow of the information processing device according to the ninth embodiment. FIG. 20 is a block diagram showing the functional configuration of the information processing device according to the tenth embodiment.It is an explanatory diagram showing an example of an image notified to a user from an output unit.It is an explanatory diagram showing an example of an image notified to a user from an output unit.It is a block diagram showing an example of the configuration of a computer that realizes an information processing device according to each embodiment.

[0013] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0014] The description will be given in the following order: 1. Overview 2. Embodiments 2.1. First embodiment 2.2. Second embodiment 2.3. Third embodiment 2.4. Fourth embodiment 2.5. Fifth embodiment 2.6. Sixth embodiment 2.7. Seventh embodiment 2.8. Eighth embodiment 2.9. Ninth embodiment 3. Hardware configuration example

[0015] 1. System Overview First, an overview of the technology according to the present disclosure will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram illustrating a moving body 1 to which the technology according to the present disclosure is applied.

[0016] 1 , a mobile object 1 is equipped with a plurality of different types of sensors 10. The mobile object 1 is, for example, an automobile. The mobile object 1 can perform autonomous driving or driving assistance by recognizing the surrounding environment using sensing data acquired by the plurality of sensors 10 equipped therein. The sensors 10 equipped on the mobile object 1 may be, for example, a light detection and ranging (LiDAR), radar, an ultrasonic sensor, or a camera.

[0017] A mobile body 1 equipped with multiple sensors 10 can recognize the surrounding environment with higher accuracy by integrating the sensing data acquired by each of the multiple sensors 10. For example, the mobile body 1 can improve the accuracy of object recognition and tracking by combining depth information to an object acquired by LiDAR and color information of the object acquired by a camera at the same coordinates. In order to integrate the sensing data acquired by each of the multiple sensors 10, it is important to unify the coordinate system of the sensing data acquired by each of the multiple sensors 10 based on the relative positions of the multiple sensors 10.

[0018] Generally, the relative positional relationships of the multiple sensors 10 mounted on the mobile object 1 are specified by performing calibration when the mobile object 1 is shipped from a factory. This allows the mobile object 1 to unify the coordinate system of sensing data among the multiple sensors 10 whose relative positional relationships have been specified.

[0019] However, after the moving body 1 is shipped from the factory, the positions of the sensors 10 may change due to aging of parts, vibrations during driving, etc. In such cases, the relative positions of the multiple sensors 10 may change from known values, which may reduce the accuracy of integrating the sensing data acquired by each of the multiple sensors 10.

[0020] The technology according to the present disclosure has been conceived in consideration of the above circumstances. According to an information processing device according to an embodiment of the present disclosure, when the relative position of the sensor 10 mounted on the moving object 1 changes from a known value (i.e., when a positional shift of the sensor 10 occurs), the influence of the positional shift on the recognition result can be reduced. Furthermore, according to an information processing device according to another embodiment of the present disclosure, the occurrence of a positional shift of the sensor 10 mounted on the moving object 1 can be detected, the amount of the positional shift can be calculated, and the positional shift can be corrected. Below, an information processing device that realizes the technology according to the present disclosure will be described in more detail.

[0021] Although the mobile object 1 has been described above as an automobile, the technology according to the present disclosure is not limited to the above example. The mobile object 1 may be a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, or the like, as long as it is capable of moving on land, in the air, or on water.

[0022] 2. Embodiments (2.1. First Embodiment) First, a first embodiment of the present disclosure will be described with reference to Figs. 2 to 4. Fig. 2 is a block diagram showing the functional configuration of an information processing device 100 according to the first embodiment. Fig. 3 is a graph showing an example of a criterion for determining whether or not there is a positional deviation of the sensor 10. Fig. 4 is a flowchart showing the flow of operations of the information processing device 100 according to the first embodiment.

[0023] 2, the information processing device 100 according to the first embodiment includes a difference detection unit 110, a storage unit 120, a projection processing unit 130, and a determination unit 140. In the first embodiment, the multiple sensors 10 may be different types of sensors. The multiple sensors 10 may include, for example, at least a LiDAR and a camera.

[0024] The information processing device 100 can determine the presence or absence of positional deviation between the multiple sensors 10, using a stationary object placed in the environment as a reference. Note that the presence of positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0025] The difference detection unit 110 detects a stationary object area as a difference from the sensing data of each of the multiple sensors 10. Specifically, each of the multiple sensors 10 acquires sensing data of the environment before the stationary object is placed and sensing data of the environment after the stationary object is placed. As a result, the difference detection unit 110 can detect the stationary object area by detecting the difference between the sensing data of the environment before the stationary object is placed and the sensing data of the environment after the stationary object is placed.

[0026] A stationary object is an object that can be detected by each of the multiple sensors 10. The stationary object is placed in the environment by, for example, a user, etc. The stationary object may be a corner reflector, a chessboard chart, a calibration chart, or the like.

[0027] The storage unit 120 is a storage unit that temporarily stores sensing data of the environment acquired by the multiple sensors 10. Specifically, the storage unit 120 temporarily stores sensing data of the environment before the stationary object is placed, which is acquired by each of the multiple sensors 10.

[0028] The projection processing unit 130 projects a region of a stationary object in sensing data acquired by a certain sensor 10 onto sensing data acquired by another sensor 10. Specifically, the projection processing unit 130 projects a region of a stationary object in sensing data of a certain sensor 10 onto a corresponding region in sensing data of another sensor 10, based on a known positional relationship between the sensors 10. For example, the projection processing unit 130 may project an image generated from a point cloud corresponding to a stationary object in point cloud data acquired by a LiDAR (certain sensor 10) onto a corresponding region in an image captured by a camera (another sensor 10).

[0029] The determination unit 140 determines whether or not there is a positional deviation of each of the multiple sensors 10 by comparing the areas of the stationary object in the sensing data acquired by each of the multiple sensors 10. Specifically, the determination unit 140 may determine whether or not there is a positional deviation of each of the multiple sensors 10 based on the degree of overlap between the area of ​​the stationary object projected onto the sensing data of another sensor 10 by the projection processing unit 130 and the area of ​​the stationary object in the sensing data of another sensor 10.

[0030] For example, the determination unit 140 may determine whether or not there is a positional misalignment between the sensors 10 based on the overlap ratio between the area of ​​a stationary object in an image captured by a camera and the area of ​​a stationary object in the point cloud data acquired by LiDAR.

[0031] 3 is a graph in which the horizontal axis represents the number of points included in a stationary object region in the point cloud data acquired by LiDAR, and the vertical axis represents the overlap ratio between the stationary object region in the image captured by the camera and the stationary object region in the point cloud data acquired by LiDAR. As shown in FIG. 3 , the determination unit 140 may determine that the multiple sensors 10 are not deviated from the known positional relationship (are in a normal state) if the overlap ratio between the stationary object regions is high. Furthermore, the determination unit 140 may determine that the multiple sensors 10 are deviated from the known positional relationship (are in a state of deviation) if the overlap ratio between the stationary object regions is low.

[0032] However, if the number of points in the point cloud included in the stationary object region is small, the reliability of the boundary of the stationary object region in the point cloud data acquired by LiDAR may be low. Therefore, the determination unit 140 may lower the threshold for the degree of overlap for determining a normal state as the number of points in the point cloud included in the stationary object region in the point cloud data decreases. This reduces the possibility that the determination unit 140 may erroneously determine that a positional deviation has occurred in the sensor 10 when the reliability of the boundary of the stationary object region in the point cloud data is low. Furthermore, if the number of points in the point cloud included in the stationary object region is extremely small, the determination unit 140 may consider it difficult to determine a positional deviation and determine a normal state regardless of the degree of overlap between the stationary object regions.

[0033] Next, the flow of operations of the information processing apparatus 100 according to the first embodiment will be described with reference to FIG.

[0034] 4, first, the sensors 10 each acquire sensing data of an environment where no stationary object is placed (S101). Next, after a stationary object is placed in the environment by a user or the like, the sensors 10 again acquire sensing data of the environment where the stationary object is placed (S103).

[0035] Next, the difference detection unit 110 detects the area of ​​the stationary object by detecting the difference between the sensing data of the environment before the stationary object is placed and the sensing data of the environment after the stationary object is placed, for each of the multiple sensors 10 (S105). Next, the projection processing unit 130 projects the area of ​​the stationary object between the sensing data based on the known positional relationship between the sensors 10 (S107). Thereafter, the determination unit 140 determines whether or not there is a positional deviation for each of the multiple sensors 10 based on the overlap rate between the projected area of ​​the stationary object and the area of ​​the stationary object in the sensing data to which it is projected (S109).

[0036] According to the above configuration, the information processing device 100 of the first embodiment can determine whether or not there is a positional deviation between the multiple sensors 10 by comparing the areas of stationary objects sensed by each of the multiple sensors 10 with each other.

[0037] (2.2. Second embodiment) Next, a second embodiment of the present disclosure will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a block diagram showing the functional configuration of an information processing device 200 according to the second embodiment. Fig. 6 is a flowchart showing the flow of operations of the information processing device 200 according to the second embodiment.

[0038] 5 , the information processing device 200 according to the second embodiment includes a projection processing unit 210, a line detection unit 220, a determination unit 240, and a calibration unit 250. In the second embodiment, the multiple sensors 10 may be different types of sensors. The multiple sensors 10 may include, for example, a LiDAR and a camera.

[0039] The information processing device 200 detects straight lines from sensing data acquired by the multiple sensors 10 and matches the detected straight lines with each other, thereby determining whether or not there is a positional deviation between the multiple sensors 10. Note that the presence of a positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0040] The projection processing unit 210 projects sensing data acquired by a certain sensor 10 onto sensing data acquired by another sensor 10. Specifically, the projection processing unit 210 projects an image generated from sensing data of a certain sensor 10 onto an image generated from sensing data of another sensor 10 so that the angles of view correspond to each other, based on a known positional relationship between the sensors 10. For example, the projection processing unit 210 may project an image generated from point cloud data acquired by a LiDAR (certain sensor 10) onto an image captured by a camera (another sensor 10) so that the angles of view correspond to each other.

[0041] The line detection unit 220 detects a line from each of the images generated from the sensing data acquired by the multiple sensors 10. For example, the line detection unit 220 may detect a line from each of the images using a Hough transform, a line segment detector (LSD), or a known line detection algorithm.

[0042] The determination unit 240 matches straight lines in an image generated from sensing data acquired by each of the multiple sensors 10, and compares the matched straight lines to determine whether or not there is positional misalignment among the multiple sensors 10. Specifically, the determination unit 240 first matches a straight line in an image generated from sensing data from another sensor 10 with a straight line in an image generated from sensing data from one sensor 10 and projected so that the angle of view corresponds to the sensing data of the other sensor 10. Next, the determination unit 240 determines whether or not there is positional misalignment among the multiple sensors 10 based on the amount of misalignment between the matched straight lines.

[0043] For example, the determination unit 240 may match a straight line in an image captured by a camera with a straight line in an image corresponding to the camera image generated from point cloud data acquired by LiDAR. In this way, the determination unit 240 can determine whether or not there is positional deviation of each of the multiple sensors 10 based on the amount of deviation between the matched straight lines.

[0044] The calibration unit 250 calibrates the positional relationship between the multiple sensors 10 based on the amount of deviation between the straight lines matched by the determination unit 240. Specifically, the calibration unit 250 calculates the amount of deviation for each combination of matched straight lines, and calculates a correction amount for the positional deviation so that the sum of the calculated amounts of deviation is minimized. The calibration unit 250 can calibrate the positional relationship between the multiple sensors 10 by correcting the known positional relationship between the multiple sensors 10 with the calculated amount of correction for the positional deviation.

[0045] Next, the flow of operations of the information processing apparatus 200 according to the second embodiment will be described with reference to FIG.

[0046] 6 , first, the multiple sensors 10 each acquire sensing data (S201). Next, the projection processing unit 210 projects an image between the sensing data based on the known positional relationship between the sensors 10 (S203). Specifically, the projection processing unit 210 projects an image generated from one of the sensing data onto an image generated from the other sensing data based on the known positional relationship between the sensors 10 so that the angle of view corresponds to that of the image generated from the other sensing data.

[0047] Next, the line detection unit 220 detects lines in the images generated from each of the sensing data (S205). Furthermore, the determination unit 240 matches lines between the images generated from each of the sensing data (S207) and calculates the amount of deviation between the matched lines to determine whether or not there is positional deviation between the multiple sensors 10 (S209). Next, the calibration unit 250 calibrates the positional relationship between the multiple sensors 10 based on the amount of deviation between the matched lines (S211).

[0048] According to the above configuration, the information processing device 200 according to the second embodiment can determine whether or not there is a positional deviation between the multiple sensors 10 by matching straight lines in images generated from the sensing data of the multiple sensors 10. Furthermore, the information processing device 200 can calibrate the positional relationship between the multiple sensors 10 by calculating the amount of deviation between the matched straight lines.

[0049] (2.3. Third Embodiment) Next, a third embodiment of the present disclosure will be described with reference to Figs. 7 to 9. Fig. 7 is a block diagram showing the functional configuration of an information processing device 300 according to the third embodiment. Figs. 8A and 8B are schematic diagrams showing an example of a criterion for determining whether or not the sensor 10 is misaligned. Fig. 9 is a flowchart showing the flow of operations of the information processing device 300 according to the third embodiment.

[0050] 7, an information processing device 300 according to the third embodiment includes a projection processing unit 310 and a determination unit 340. In the third embodiment, the plurality of sensors 10 may be, for example, a plurality of cameras.

[0051] The information processing device 300 projects sensing data acquired by the multiple sensors 10 so as to connect seamlessly into the projection space, and can determine the presence or absence of positional deviation between the multiple sensors 10 based on the connection deviation between the projected sensing data. Note that the presence of positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0052] The projection processing unit 310 virtually projects sensing data acquired by the multiple sensors 10 into the projection space. Specifically, the projection processing unit 310 virtually projects sensing data, such as images acquired by the multiple sensors 10, into the projection space to generate an image in which the sensing data are connected to one another. For example, the projection processing unit 310 may virtually project images captured by four sensors 10 attached to the front, rear, left, and right sides of the mobile object 1 into the projection space to generate an omnidirectional image overlooking the 360° surroundings of the mobile object 1 from above.

[0053] The determination unit 340 determines whether or not there is a positional deviation of each of the plurality of sensors 10 by detecting a connection deviation between the sensing data projected into the projection space.

[0054] Here, as shown in Figures 8A and 8B, the judgment criteria of the judgment unit 340 will be explained using the example of a case where an image S1 sensed by a first sensor 10A and an image S2 sensed by a second sensor 10B are projected into a projection space PS.

[0055] For example, when the positional relationship between the first sensor 10A and the second sensor 10B remains the same, the images S1 and S2 are seamlessly connected to each other in the projection space PS, as shown in FIG. 8A. However, when the positional relationship between the first sensor 10A and the second sensor 10B deviates from the known positional relationship, the images S1 and S2 are not seamlessly connected to each other in the projection space PS, as shown in FIG. 8B. A connection misalignment area EL is, for example, an area where the images S1 and S2 overlap, or a gap area where the images S1 and S2 are separated and not connected, and is generated due to a misalignment of the relative positions of the first sensor 10A and the second sensor 10B.

[0056] The determination unit 340 can determine whether or not there is a positional deviation among the plurality of sensors 10 by detecting the above-described connection deviation region EL when sensing data acquired by the plurality of sensors 10 is projected into the projection space. That is, if the connection deviation region EL is detected, the determination unit 340 can determine that a positional deviation has occurred among the plurality of sensors 10. On the other hand, if the connection deviation region EL is not detected, the determination unit 340 can determine that there is no positional deviation among the plurality of sensors 10.

[0057] Next, the flow of operations of the information processing device 300 according to the third embodiment will be described with reference to FIG.

[0058] 9 , first, the multiple sensors 10 each acquire sensing data (S301). Next, the projection processing unit 310 projects the acquired sensing data into the projection space (S303). Subsequently, the projection processing unit 310 corrects differences between the projected sensing data due to sensing conditions (S305). For example, if the sensing data is an image, the projection processing unit 310 corrects differences in brightness, color, etc. between the images by adjusting the white balance, contrast, etc. of the image.

[0059] Thereafter, the determination unit 340 detects a linear misalignment area EL that may occur between the sensing data projected into the projection space (S307). This allows the determination unit 340 to determine whether or not there is a positional misalignment of each of the multiple sensors 10 based on the presence or absence of the misalignment area EL between the sensing data (S309).

[0060] According to the above configuration, the information processing device 300 of the third embodiment can determine whether or not there is a positional shift between multiple sensors 10 based on whether or not a connection shift occurs when the sensing data of each of the multiple sensors 10 is connected.

[0061] (2.4. Fourth Embodiment) Next, a fourth embodiment of the present disclosure will be described with reference to Figs. 10 to 12. Fig. 10 is a block diagram showing the functional configuration of an information processing device 400 according to the fourth embodiment. Fig. 11 is a flowchart showing the flow of operations of the information processing device 400 according to the fourth embodiment. Fig. 12 is an explanatory diagram for explaining detection of a skeletal posture of a user in a modified example of the fourth embodiment.

[0062] 10 , an information processing device 400 according to the fourth embodiment includes a determination unit 440, a parameter storage unit 420, a notification unit 410, and a calibration unit 450. In the fourth embodiment, the plurality of sensors 10 may be, for example, any of LiDAR, radar, ultrasonic sensors, or cameras.

[0063] When a positional deviation occurs in the sensors 10, the information processing device 400 notifies the user of an operation to calibrate the positional relationship between the multiple sensors 10. This allows the information processing device 400 to calibrate the positional relationship between the multiple sensors 10 using sensing data obtained by an operation performed by the user in accordance with the notification from the information processing device 400. Note that the presence of a positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0064] The determination unit 440 determines whether or not there is misalignment of each of the plurality of sensors 10. For example, the determination unit 440 may determine whether or not there is misalignment of each of the plurality of sensors 10 using the method described in other embodiments or a known method.

[0065] The parameter storage unit 420 stores various parameters including external parameters that represent known positional relationships among the multiple sensors 10. For example, the parameter storage unit 420 may store internal parameters of cameras included in the multiple sensors 10 and external parameters that represent known positional relationships among the multiple sensors 10. The determination unit 440 can determine whether or not a positional deviation has occurred in each of the multiple sensors 10 by referring to the external parameters stored in the parameter storage unit 420.

[0066] The notification unit 410 notifies the user using an image, text, audio, or a combination thereof. Specifically, when the determination unit 440 determines that a positional deviation has occurred in the sensor 10, the notification unit 410 may first notify the user that a positional deviation has occurred in the sensor 10. The notification unit 410 may also notify the user of the content of an operation for calibrating the positional relationships of the multiple sensors 10. For example, as the operation for calibrating the positional relationships of the multiple sensors 10, the notification unit 410 may notify the user to present a calibration chart at a predetermined position and to move the calibration chart along a predetermined trajectory. The notification unit 410 may notify the user of the content directly or via a terminal device (e.g., a smartphone) carried by the user.

[0067] The calibration unit 450 calibrates the positional relationship of the multiple sensors 10 using sensing data acquired by the multiple sensors 10. Specifically, the calibration unit 450 calibrates the positional relationship of the multiple sensors 10 using sensing data obtained by sensing the content of an operation performed by a user in response to a notification from the notification unit 410. For example, the calibration unit 450 may calibrate the positional relationship of the multiple sensors 10 using coordinates in the sensing data of a calibration chart that the user presents at a predetermined position and moves along a predetermined trajectory. The positional relationship of the multiple sensors 10 calibrated by the calibration unit 450 is stored in the parameter storage unit 420 as an external parameter representing the positional relationship of the multiple sensors 10.

[0068] Furthermore, the calibration unit 450 may determine whether sensing data usable for calibration has been acquired after notifying the user of the calibration work via the notification unit 410. If sensing data usable for calibration has not been acquired, the calibration unit 450 may cause the notification unit 410 to again notify the user of the work to calibrate the positional relationships of the multiple sensors 10. Note that sensing data that cannot be used for calibration is, for example, data in which the calibration chart cannot be recognized, or data in which the calibration chart is not present at a predetermined position or on a predetermined trajectory.

[0069] Next, the flow of operations of the information processing apparatus 400 according to the fourth embodiment will be described with reference to FIG.

[0070] 11 , the multiple sensors 10 each acquire sensing data (S401). Next, the determination unit 440 determines whether or not there is misalignment of each of the multiple sensors 10 (S403). Subsequently, if it is determined that there is misalignment of the sensor 10, the notification unit 410 notifies the user of the occurrence of the misalignment of the sensor 10 (S405) and further notifies the user of the contents of the work to calibrate the positional relationships of the multiple sensors 10 (S407). Next, after the user performs the work to calibrate the positional relationships of the multiple sensors 10, the multiple sensors 10 each acquire sensing data again (S409).

[0071] Here, the calibration unit 450 determines whether the sensing data acquired by the multiple sensors 10 can be used to calibrate the positional relationships of the multiple sensors 10 (S411). If the sensing data cannot be used for calibration (S411 / NO), the multiple sensors 10 each acquire sensing data again (S409). On the other hand, if the sensing data can be used for calibration (S411 / YES), the calibration unit 450 calibrates the positional relationships of the multiple sensors 10 (S413). Thereafter, the positional relationships of the multiple sensors 10 derived by the calibration by the calibration unit 450 are stored in the parameter storage unit 420. As a result, the results of the calibration by the calibration unit 450 are reflected in calculation processes such as environment recognition (S415).

[0072] According to the above configuration, when a positional deviation occurs in the sensor 10, the information processing device 400 according to the fourth embodiment can notify the user of the positional deviation of the sensor 10 and the work to calibrate the positional relationships of the sensors 10. Furthermore, the information processing device 400 can calibrate the positional relationships of the multiple sensors 10 by sensing a calibration chart presented by the user with the multiple sensors 10.

[0073] (Modification) A modification of the fourth embodiment will be described below. The calibration work notified to the user by the notification unit 410 is not limited to presenting the calibration chart at a predetermined position and moving it along a predetermined trajectory.

[0074] For example, the calibration task may involve having the user assume a plurality of predetermined poses, which allows the calibration unit 450 to calibrate the positional relationships of the plurality of sensors 10 by using the position coordinates of the joint points extracted from the user's poses included in the sensing data.

[0075] Specifically, as shown in FIG. 12 , first, sensing data IM of a user assuming a predetermined pose is acquired by each of the multiple sensors 10. The calibration unit 450 can generate skeletal posture data FM of the user by extracting the user's joint points (or feature points such as eyes) from the sensing data IM of the user assuming the predetermined pose. The three-dimensional coordinates of the joint points in the generated skeletal posture data FM are the same among the multiple sensors 10 that sense the same user pose. Therefore, the calibration unit 450 can calibrate the positional relationship of the multiple sensors 10 by matching the three-dimensional coordinates of the joint points in the skeletal posture data FM among the multiple sensors 10. Furthermore, by performing the above process for multiple poses of the user, the calibration unit 450 can calibrate the positional relationship of the multiple sensors 10 with higher accuracy.

[0076] (2.5. Fifth Embodiment) Next, a fifth embodiment of the present disclosure will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a block diagram showing the functional configuration of an information processing device 500 according to the fifth embodiment. Fig. 14 is a flowchart showing the flow of operations of the information processing device 500 according to the fifth embodiment.

[0077] 13 , an information processing device 500 according to the fifth embodiment includes a synchronization control unit 520 and a light emission control unit 510 provided for each sensor 10. In the fifth embodiment, the plurality of sensors 10 may be, for example, LiDAR, radar, ultrasonic sensors, or cameras. The light emission unit 20 may be, for example, a lamp, an array, or a display configured with light-emitting diodes (LEDs) capable of blinking at high speeds.

[0078] The information processing device 500 can synchronize the sensing timing of the multiple sensors 10 by using the imaging unit 531 to capture the light emission status of the multiple light emitting units 20 that emit light in synchronization with the sensing timing of the multiple sensors 10.

[0079] The light emission control unit 510 controls the light emission timing of the light emitting unit 20 so as to synchronize with the sensing timing of the sensor 10. For example, the light emission control unit 510 may cause the light emitting unit 20 to emit light at the same timing as the sensing timing of the sensor 10. In this way, the synchronization determination unit 532 can determine whether the sensing timing of the multiple sensors 10 is synchronized based on whether the light emission timings of the multiple light emitting units 20 match by capturing images of the light emission states of the multiple light emitting units 20 using the imaging unit 531.

[0080] Furthermore, the light emission control unit 510 may change the light emission pattern of the light emitting unit 20 for each light emission timing. In this way, the synchronization determination unit 532 can determine the degree of deviation between the sensing timings of the multiple sensors 10 by comparing the light emission patterns of the multiple light emitting units 20 captured by the imaging unit 531 with each other.

[0081] For example, if the multiple sensors 10 are each a camera, the light emission control unit 510 may cause the light emitting unit 20 to emit light in synchronization with the imaging frames of the cameras. In this case, the light emission control unit 510 may control the light emission pattern or light emission color of the light emitting unit 20 so that the frame number of the synchronized imaging frame can be determined. For example, the light emission control unit 510 may control an LED array or LED display constituting the light emitting unit 20 so that the light emission pattern corresponds to the frame number of the synchronized imaging frame. Furthermore, the light emission control unit 510 may control an LED lamp constituting the light emitting unit 20 so that the light emission color corresponds to a color previously associated with the frame number of the imaging frame.

[0082] The synchronization control unit 520 controls the sensing timing of the multiple sensors 10 based on the determination of the synchronization determination unit 532. For example, if the synchronization determination unit 532 derives the amount of deviation in the sensing timing of the multiple sensors 10, the synchronization control unit 520 may control the sensing timing of the multiple sensors 10 based on the derived amount of deviation in the sensing timing. Furthermore, if the synchronization determination unit 532 determines only the deviation in the sensing timing of the multiple sensors 10, the synchronization control unit 520 may finely vary the sensing timing of the multiple sensors 10.

[0083] Next, the flow of operations of the information processing apparatus 500 according to the fifth embodiment will be described with reference to FIG.

[0084] 14 , first, each light-emission control unit 510 controls the light emission of the light-emitting unit 20 so as to synchronize with the sensing timing of the corresponding sensor 10 (S501). Next, the imaging unit 531 simultaneously captures images of the light-emission states of the multiple light-emitting units 20 (S503). Next, the synchronization determination unit 532 derives the amount of deviation in the sensing timing of the multiple sensors 10 based on the light-emission states of the multiple light-emitting units 20 in the captured images (S505). Thereafter, the synchronization control unit 520 controls the sensing timing of the multiple sensors 10 based on the derived amount of deviation in the sensing timing of the multiple sensors 10 (S507).

[0085] According to the above configuration, the information processing device 500 of the fifth embodiment can synchronize the sensing timing of multiple sensors 10 by simultaneously capturing images of multiple light-emitting units 20 that emit light in synchronization with each of the multiple sensors 10 using the imaging unit 531.

[0086] (2.6. Sixth Embodiment) Next, a sixth embodiment of the present disclosure will be described with reference to Fig. 15 and Fig. 16. Fig. 15 is a block diagram showing the functional configuration of an information processing device 600 according to the sixth embodiment. Fig. 16 is a flowchart showing the flow of operations of the information processing device 600 according to the sixth embodiment.

[0087] 15 , an information processing device 600 according to the sixth embodiment includes a position estimation unit 610, a trajectory storage unit 630, a calibration unit 650, and a parameter storage unit 620, each of which is provided for each of the plurality of sensors 10. The plurality of sensors 10 may be, for example, a position sensor that acquires information about the position of a moving object 1, such as a Global Navigation Satellite System (GNSS) sensor or an Inertial Measurement Unit (IMU), or may be a sensor that acquires information about the environment, such as a LiDAR, a radar, an ultrasonic sensor, or a camera.

[0088] The information processing device 600 can calibrate the positional relationship of the multiple sensors 10 by comparing the travel trajectories of the moving object 1 estimated from the sensing data acquired by each of the multiple sensors 10 with each other.

[0089] The position estimation unit 610 estimates the self-position of the moving object 1 using sensing data acquired by the sensor 10. For example, if the sensor 10 is a GNSS sensor, the position estimation unit 610 can estimate the self-position of the moving object 1 based on position information acquired by the GNSS sensor. Furthermore, if the sensor 10 is an IMU, the position estimation unit 610 can estimate the self-position of the moving object 1 by detecting the movement of the moving object 1 based on information regarding three-dimensional inertial motion (acceleration and angular velocity on three orthogonal axes) acquired by the IMU. Furthermore, if the sensor 10 is a camera, the position estimation unit 610 can estimate the self-position of the moving object 1 from an image captured by the camera by using Visual-SLAM (Simultaneous Localization and Mapping). Furthermore, if the sensor 10 is a LiDAR, radar, or ultrasonic sensor, the position estimation unit 610 can estimate the self-position of the moving object 1 from depth information acquired by the sensor 10.

[0090] The trajectory storage unit 630 stores the traveling trajectory of the moving object 1 by storing, in chronological order, the self-position of the moving object 1 estimated from the sensing data acquired by each of the multiple sensors 10. For example, the trajectory storage unit 630 may store, in chronological order, the most recent estimated self-position of the moving object 1 for each sensor 10, thereby storing the most recent traveling trajectory of the moving object 1.

[0091] The calibration unit 650 calibrates the positional relationship between the multiple sensors 10 by comparing the travel trajectories of the moving object 1 estimated from the sensing data acquired by each of the multiple sensors 10. Specifically, the calibration unit 650 calibrates the positional relationship between the multiple sensors 10 by comparing the travel trajectories of the moving object 1 stored in the trajectory storage unit 630 for each sensor 10. For example, it is considered that travel trajectories of the moving object 1 over the same period will approximately match each other regardless of the sensor 10 used for the estimation. Therefore, the calibration unit 650 can estimate the amount of positional deviation between the multiple sensors 10 based on the deviation of the travel trajectories of the moving object 1 estimated for each sensor 10. The positional relationship between the multiple sensors 10 calibrated by the calibration unit 650 is stored in the parameter storage unit 620 as an external parameter representing the positional relationship between the multiple sensors 10.

[0092] The parameter storage unit 620 stores various parameters of the plurality of sensors 10. For example, the parameter storage unit 620 may store the positional relationship of the plurality of sensors 10 calibrated by the calibration unit 650 as an external parameter.

[0093] Next, the flow of operations of the information processing device 600 according to the sixth embodiment will be described with reference to FIG.

[0094] 16 , the plurality of sensors 10 each acquire sensing data (S601). Next, the position estimation unit 610 estimates the self-position of the moving object 1 for each sensor 10 based on the sensing data acquired by each of the plurality of sensors 10 (S603). Subsequently, the self-position of the moving object 1 estimated for each sensor 10 is stored in the trajectory storage unit 630 (S605).

[0095] Here, when a determination unit (not shown) determines that a positional deviation has occurred among the plurality of sensors 10, the calibration unit 650 calibrates the positional relationship among the plurality of sensors 10. Specifically, the calibration unit 650 reads out the estimated self-position of the moving object 1 for the same period from the trajectory storage unit 630 and compares the travel trajectories of the moving object 1 among the plurality of sensors 10 (S607). As a result, the calibration unit 650 can calibrate the positional relationship among the plurality of sensors 10 based on the difference between the travel trajectories of the moving object 1 estimated for each sensor 10 (S609). Thereafter, the positional relationship among the plurality of sensors 10 derived by the calibration by the calibration unit 650 is stored in the parameter storage unit 620. As a result, the result of the calibration by the calibration unit 650 is reflected in arithmetic processing such as environment recognition (S611).

[0096] According to the above configuration, the information processing device 600 of the sixth embodiment can calibrate the positional relationship between multiple sensors 10 by using the self-position and driving trajectory of the moving body 1 estimated for each sensor 10.

[0097] (2.7. Seventh Embodiment) Next, a seventh embodiment of the present disclosure will be described with reference to Fig. 17 and Fig. 18. Fig. 17 is a block diagram showing the functional configuration of an information processing device 700 according to the seventh embodiment. Fig. 18 is a flowchart showing the flow of operations of the information processing device 700 according to the seventh embodiment.

[0098] 17 , an information processing device 700 according to the seventh embodiment includes a rest determination unit 710, a moving object detection unit 720, a moving object exclusion unit 730, and a determination unit 740. In the seventh embodiment, the plurality of sensors 10 may be, for example, LiDAR, radar, an ultrasonic sensor, or a camera. The sensor 12 is a sensor capable of acquiring Doppler velocity, and may be, for example, radar or FMCW (Frequency Modulated Continuous Wave)-LiDAR.

[0099] The information processing device 700 determines whether the moving object 1 is stationary based on information about the position acquired by the position sensor 11, and detects moving objects present in the environment based on the Doppler velocity acquired by the sensor 12. This allows the information processing device 700 to exclude moving objects from the sensing data acquired by the sensor 10, and therefore to determine whether or not there is a positional deviation between the multiple sensors 10 based on the stationary objects in the sensing data. Note that the presence of a positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0100] The position sensor 11 is a sensor that acquires information about the position of the mobile object 1, such as a Global Navigation Satellite System (GNSS) sensor or an Inertial Measurement Unit (IMU).

[0101] The rest-state determination unit 710 determines whether the moving body 1 is at rest based on information about the position of the moving body 1 acquired by the position sensor 11. Specifically, the rest-state determination unit 710 may determine whether the moving body 1 is at rest based on the amount of change in the position of the moving body 1 acquired by the position sensor 11, or the speed or acceleration of the moving body 1.

[0102] The sensor 12 measures the distance or direction to an object by measuring the reflected waves of light or radio waves emitted toward the object, and measures the relative speed or displacement of the object by observing the shift in frequency of the reflected waves due to the Doppler effect. For example, the sensor 12 can acquire point cloud data of the environment to which Doppler velocity, which represents the speed or displacement relative to the moving object 1, is added. As described above, the sensor 12 may be a radar or an FMCW-LiDAR.

[0103] The moving object detection unit 720 detects moving objects in the environment based on the Doppler velocity acquired by the sensor 12. Specifically, the moving object detection unit 720 detects points having a velocity relative to the moving object 1 from the point cloud data acquired by the sensor 12 based on the Doppler velocity. If the rest determination unit 710 determines that the moving object 1 is resting, the points having a velocity relative to the moving object 1 can be considered to be points moving relative to the environment. Therefore, the moving object detection unit 720 can detect a collection of points having a velocity relative to the moving object 1 as moving objects in the environment.

[0104] The moving object exclusion unit 730 excludes the area of ​​the moving object detected by the moving object detection unit 720 from the sensing data acquired by each of the multiple sensors 10. Specifically, the moving object exclusion unit 730 excludes the area of ​​a collection of points having a velocity relative to the moving object 1 detected by the moving object detection unit 720 from the sensing data acquired by each of the multiple sensors 10. As a result, the sensing data from which the area of ​​the collection of points detected by the moving object detection unit 720 has been excluded excludes moving objects in the environment, and becomes data that includes only stationary objects.

[0105] The determination unit 740 determines whether or not there is a positional deviation between the multiple sensors 10 based on the sensing data from which moving objects have been excluded. Specifically, the determination unit 740 determines whether or not there is a positional deviation between the multiple sensors 10 by comparing feature points in the sensing data from which moving objects (a collection of points having a velocity relative to the moving object 1) have been excluded after being acquired by each of the multiple sensors 10. The feature points in the sensing data from which moving objects have been excluded are stationary feature points extracted from stationary objects in the environment. Therefore, the determination unit 740 can determine whether or not there is a positional deviation between the multiple sensors 10 by comparing the three-dimensional coordinates of the stationary feature points between the multiple sensors 10.

[0106] Next, the flow of operations of the information processing device 700 according to the seventh embodiment will be described with reference to FIG.

[0107] 18 , first, the rest-state determination unit 710 determines whether the moving object 1 is at rest based on the sensing data acquired by the position sensor 11 (S701). Next, the sensor 12 acquires point cloud data with Doppler velocity added as sensing data (S703). Next, the moving object detection unit 720 detects, as a moving object, a group of points having a velocity relative to the moving object 1 from the point cloud data acquired by the sensor 12 based on the Doppler velocity acquired by the sensor 12 (S705).

[0108] Thereafter, the multiple sensors 10 each acquire sensing data (S707). Furthermore, the moving object exclusion unit 730 excludes the area of ​​the moving object detected by the moving object detection unit 720 from the sensing data acquired by each of the multiple sensors 10 (S709). Next, the determination unit 740 determines whether or not there is a positional deviation between the multiple sensors 10 based on the sensing data from which the area corresponding to the moving object has been excluded (S711). Specifically, the determination unit 740 can determine whether or not there is a positional deviation between the multiple sensors 10 by comparing, between the multiple sensors 10, the three-dimensional coordinates of feature points extracted from the sensing data from which the area of ​​the moving object has been excluded.

[0109] According to the above configuration, the information processing device 700 of the seventh embodiment can determine the positional deviation between multiple sensors 10 from the sensing data of multiple sensors 10 by determining whether the moving body 1 is stationary and detecting moving bodies in the environment.

[0110] (2.8. Eighth Embodiment) Next, an eighth embodiment of the present disclosure will be described with reference to Figs. 19 to 22. Fig. 19 is a block diagram showing the functional configuration of an information processing device 800 according to the eighth embodiment. Fig. 20 is a flowchart showing a first operation flow of the information processing device 800 according to the eighth embodiment. Fig. 21 is a flowchart showing a second operation flow of the information processing device 800 according to the eighth embodiment. Fig. 22 is a flowchart showing a third operation flow of the information processing device 800 according to the eighth embodiment.

[0111] 19 , an information processing device 800 according to the eighth embodiment includes a determination unit 840, a calculation unit 860, and a generation unit 810. The plurality of sensors 10 may be, for example, a LiDAR, a radar, an ultrasonic sensor, or a camera.

[0112] When a positional deviation occurs in the sensor 10, the information processing device 800 corrects the positional deviation by correcting the sensing data based on the amount of positional deviation of the sensor 10, thereby generating corrected data in which the viewpoint before the positional deviation is reproduced. Note that the positional deviation of the sensor 10 means that the position of the sensor 10 is deviated from a known position.

[0113] The determination unit 840 determines whether or not there is misalignment of each of the plurality of sensors 10. For example, the determination unit 840 may determine whether or not there is misalignment of each of the plurality of sensors 10 using the method described in other embodiments or a known method.

[0114] The calculation unit 860 calculates the amount of positional deviation of the sensor 10 in which a positional deviation has occurred. For example, the calculation unit 860 may calculate the amount of positional deviation of the sensor 10 in which a positional deviation has occurred using the method described in other embodiments or a known method.

[0115] The generating unit 810 generates correction data by correcting the positional deviation from the sensing data acquired by the sensor 10, based on the positional deviation amount of the sensor 10 calculated by the calculating unit 860. For example, the generating unit 810 may generate the correction data by converting the viewpoint of the sensing data to the viewpoint before the positional deviation, based on the positional deviation amount of the sensor 10.

[0116] For example, if the sensor 10 is a LiDAR and can acquire sensing data including depth information of the environment, the generation unit 810 may project the sensing data onto the angle of view of the viewpoint before the positional shift based on the depth information of the environment and the calculated amount of positional shift. For example, the generation unit 810 may use a technique such as warping to project the sensing data including the depth information onto the angle of view of the viewpoint before the positional shift.

[0117] Furthermore, if machine learning has been performed in advance using the amount of positional deviation of the sensor 10 and the sensing data after the positional deviation as inputs and the sensing data before the positional deviation as training data, the generation unit 810 may generate correction data using the machine-learned generative model. For example, the generation unit 810 may generate the sensing data before the positional deviation by using a neural network that changes the viewpoint of the sensing data after the positional deviation based on the amount of positional deviation. Alternatively, the generation unit 810 may generate the sensing data before the positional deviation from the sensing data after the positional deviation by using another machine learning model that changes the viewpoint, such as Neural Radiance Fields (NeRF).

[0118] Furthermore, when an around-view image or an overhead image is generated by viewpoint conversion of sensing data acquired by sensor 10, generation unit 810 may change the viewpoint conversion position when generating the around-view image or overhead image, based on the amount of positional deviation of sensor 10. For example, generation unit 810 may generate an around-view image or overhead image in which the positional deviation of sensor 10 has been corrected by correcting the viewpoint conversion position when generating the around-view image or overhead image from the sensing data, using the amount of positional deviation of sensor 10.

[0119] Next, first to third operational flows of the information processing device 800 according to the eighth embodiment will be described with reference to FIGS.

[0120] 20 , first, the determination unit 840 determines whether or not there is a positional deviation of the sensor 10 using a method described in another embodiment or a known method (S811). Next, if there is a positional deviation of the sensor 10, the calculation unit 860 calculates the amount of positional deviation of the sensor 10 using a method described in another embodiment or a known method (S813).

[0121] Next, the sensor 10 acquires sensing data (S815). At this time, the acquired sensing data includes depth information of the environment. Therefore, the generation unit 810 acquires the depth information of the environment from the sensing data (S817). As a result, the generation unit 810 can generate, as corrected data, sensing data before the positional deviation, obtained by correcting the positional deviation from the sensing data after the positional deviation, based on the depth information and the amount of positional deviation (S819).

[0122] 21 , first, as in the first operation example, the determination unit 840 determines whether or not the sensor 10 is misaligned using the method described in another embodiment or a known method (S821). Next, if the sensor 10 is misaligned, the calculation unit 860 calculates the amount of misalignment of the sensor 10 using the method described in another embodiment or a known method (S823).

[0123] Next, the sensor 10 acquires sensing data (S825). The generation unit 810 can generate, as corrected data, sensing data in which the amount of positional deviation has been corrected by using the acquired sensing data and a trained neural network that changes the viewpoint based on the amount of positional deviation (S827).

[0124] 22 , first, the determination unit 840 determines whether or not the sensor 10 is misaligned, using the method described in another embodiment or a known method, as in the first operation example (S831). Next, if the sensor 10 is misaligned, the calculation unit 860 calculates the amount of misalignment of the sensor 10, using the method described in another embodiment or a known method (S833). Here, the generation unit 810 corrects the viewpoint conversion position when generating an around-view image or an overhead image from the sensing data, based on the calculated amount of misalignment (S835).

[0125] Next, the sensor 10 acquires sensing data (S837). The generation unit 810 converts the viewpoint of the sensing data using the corrected viewpoint conversion position, thereby generating an around-view image or an overhead image similar to that before the position shift (S839).

[0126] According to the above configuration, when a positional deviation occurs in the sensor 10, the information processing device 800 according to the eighth embodiment can generate sensing data before the positional deviation by correcting, by the amount of positional deviation, sensing data acquired by the sensor 10 after the positional deviation. This makes it possible for the information processing device 800 to prevent a change in the viewpoint of the sensing data due to a positional deviation of the sensor 10, and a decrease in the recognition performance of the sensing data in a subsequent recognition task.

[0127] (2.9. Ninth Embodiment) Next, a ninth embodiment of the present disclosure will be described with reference to FIGS. 23 to 26. FIG. 23 is a block diagram showing the functional configuration of an information processing device 900 according to the ninth embodiment. FIG. 24 is a graph diagram schematically showing the relationship between the amount of positional deviation of the sensor 10 and the threshold value of the masking process. FIG. 25 is a block diagram showing the flow of various processes performed by the information processing device 900. FIG. 26 is a flowchart diagram showing the flow of operations of the information processing device 900 according to the ninth embodiment.

[0128] As shown in FIG. 23, an information processing apparatus 900 according to the ninth embodiment includes a threshold setting unit 910 , a recognition unit 920 , and a processing unit 930 .

[0129] When a positional deviation occurs in the sensor 10, the information processing device 900 performs mask processing on a partial area determined to have low reliability among the recognition results of a recognition task performed on sensing data acquired by the sensor 10. In this way, the information processing device 900 can invalidate the area whose reliability has decreased due to the positional deviation of the sensor 10 by mask processing, thereby suppressing the influence of the positional deviation of the sensor 10 on a judgment using the recognition results in a subsequent stage. Therefore, the information processing device 900 can reduce the influence of the positional deviation of the sensor 10. Note that mask processing means invalidating data present in the masked area so that it will not be used in subsequent processing.

[0130] The recognition unit 920 performs various recognition tasks using sensing data acquired by the multiple sensors 10. Specifically, the recognition unit 920 integrates each piece of sensing data acquired by the multiple sensors 10 and performs various recognition tasks on the integrated data. For example, the recognition unit 920 may perform recognition tasks such as object detection, segmentation, depth estimation, optical flow estimation, or object motion estimation. Furthermore, the recognition unit 920 may perform preprocessing, such as distortion removal, on the sensing data acquired by the multiple sensors 10 before performing various recognition tasks.

[0131] The recognition unit 920 executes various recognition tasks using the sensing data, and thereby derives the recognition results and reliability of the recognition tasks. The reliability of the recognition tasks is derived, for example, for each pixel or region of the sensing data.

[0132] The threshold setting unit 910 sets a threshold for masking the recognition result of the recognition task based at least on the amount of positional deviation when a positional deviation occurs in the sensor 10. Specifically, the threshold setting unit 910 includes a determination unit 940, a calculation unit 960, and a threshold calculation unit 911.

[0133] The determination unit 940 determines whether or not there is misalignment of each of the plurality of sensors 10. For example, the determination unit 940 may determine whether or not there is misalignment of each of the plurality of sensors 10 using the method described in other embodiments or a known method.

[0134] The calculation unit 960 calculates the amount of positional deviation of the sensor 10 in which a positional deviation has occurred. For example, the calculation unit 960 may calculate the amount of positional deviation of the sensor 10 in which a positional deviation has occurred using the method described in the other embodiments or a known method.

[0135] The threshold calculation unit 911 calculates a reliability threshold for masking the recognition results of the recognition task executed on the sensing data acquired by the sensor 10 in which a positional deviation has occurred, based on the amount of positional deviation of the sensor 10. For example, as shown in FIG. 24 , the threshold calculation unit 911 may calculate the reliability threshold for masking so as to be a linear function with a negative slope with respect to the amount of positional deviation of the sensor 10. That is, the threshold calculation unit 911 may set a lower reliability threshold for masking as the amount of positional deviation of the sensor 10 increases. In this way, the threshold calculation unit 911 can lower the reliability threshold for masking so that a wider range of the recognition results is masked as the amount of positional deviation of the sensor 10 increases.

[0136] Furthermore, the threshold calculation unit 911 may calculate a reliability threshold for masking the recognition result based on the type of sensor 10 in which a positional shift has occurred and the type of recognition task executed on the sensing data. Specifically, the threshold calculation unit 911 may calculate a reliability threshold for masking based on the degree of influence that sensing data acquired by the sensor 10 in which a positional shift has occurred has on the recognition result of the recognition task.

[0137] For example, sensing data obtained by LiDAR has a high contribution to the recognition task of depth estimation. Therefore, when a positional shift occurs in the LiDAR, the threshold calculation unit 911 may lower the reliability threshold for masking so that a wider range of the recognition results of depth estimation is masked. Furthermore, sensing data obtained by radar has a high contribution to the recognition task of object motion estimation. Therefore, when a positional shift occurs in the radar, the threshold calculation unit 911 may lower the threshold for masking so that a wider range of the recognition results of object motion estimation is masked.

[0138] The processing unit 930 masks a partial region of the recognition result of the recognition task based on the reliability of the recognition task executed by the recognition unit 920 and the reliability threshold set by the threshold setting unit 910. Specifically, the processing unit 930 masks a partial region of the recognition result of the recognition task where the reliability of the recognition task is lower than the reliability threshold. As described above, the reliability of the recognition task is derived for each pixel or region of the sensing data. Therefore, the processing unit 930 can mask the recognition result of the recognition task for each pixel or region.

[0139] Furthermore, the processing unit 930 may limit the partial region to be masked to a region where the sensing data acquired by each of the multiple sensors 10 overlaps. Specifically, the processing unit 930 may mask a partial region, among the regions where the sensing data acquired by each of the multiple sensors 10 overlap, where the reliability of the recognition task is lower than a reliability threshold. This is because, while the influence of overlapping misalignment due to misalignment between the multiple sensors 10 is significant in regions where the sensing data overlaps, the influence of overlapping misalignment between the multiple sensors 10 is almost eliminated in regions where the sensing data do not overlap. Therefore, the processing unit 930 may omit masking in regions where the sensing data acquired by each of the multiple sensors 10 do not overlap.

[0140] Next, the flow of operations of the information processing device 900 according to the ninth embodiment will be described with reference to FIGS.

[0141] 25 and 26 , for example, sensing data is acquired by a camera, LiDAR, and radar (S901). The sensing data acquired by the camera, LiDAR, and radar are each preprocessed and then integrated, and various recognition tasks (tasks A, B, and C in FIG. 25 ) are executed by the recognition unit 920 (S903).

[0142] Specifically, images sensed by a camera may be preprocessed, such as by removing lens distortion. Point cloud data sensed by LiDAR may be preprocessed, such as by motion compensation and distortion removal. Sensing data sensed by radar may be preprocessed, such as by generating point cloud data and calculating motion information based on Doppler velocity. The sensing data integrated after preprocessing is subjected to various recognition tasks in the recognition unit 920 through recognition inference processing using a trained multimodal or multitask network.

[0143] Next, the threshold setting unit 910 calculates the amount of misalignment between the camera, LiDAR, and radar (S905). Specifically, the determination unit 940 determines whether or not there is misalignment between two of the camera, LiDAR, and radar. Thereafter, the calculation unit 960 calculates the amount of misalignment between each of the camera, LiDAR, and radar that has been determined to be misaligned.

[0144] Furthermore, the threshold setting unit 910 calculates a reliability threshold for each of the various recognition tasks based on the amount of positional shift (S907). Specifically, the threshold calculation unit 911 calculates a reliability threshold for masking the recognition results of each of the various recognition tasks based on the amount of positional shift between the camera, the LiDAR, and the radar.

[0145] Thereafter, the processing unit 930 masks some regions of the recognition results of various recognition tasks whose reliability is lower than a threshold (S909). Specifically, the processing unit 930 compares the reliability derived for each pixel, point in the point cloud, or region of the recognition task with the reliability threshold set for each recognition task by the threshold setting unit 910, and masks pixels, points in the point cloud, or regions whose reliability is lower than the threshold.

[0146] As a result, the information processing device 900 can output, for each recognition task, a recognition result in which a partial region has been masked based on the reliability (S911). Since data present in the masked pixels, points of the point cloud, or regions is invalidated, the information processing device 900 can prevent recognition results with reduced reliability due to misalignment of the sensor 10 from affecting subsequent judgments.

[0147] According to the above configuration, the information processing device 900 of the ninth embodiment can invalidate recognition results whose reliability has been reduced due to a positional shift of the sensor 10 by masking processing, thereby preventing the positional shift of the sensor 10 from affecting subsequent judgments.

[0148] (2.10. Tenth Embodiment) Next, a tenth embodiment of the present disclosure will be described with reference to Figs. 27 to 29. Fig. 27 is a block diagram showing the functional configuration of an information processing device 1000 according to the tenth embodiment. Fig. 28 is an explanatory diagram showing an example of an image P1 notified to a user from the output unit 1020. Fig. 29 is an explanatory diagram showing an example of an image P2 notified to a user from the output unit 1020.

[0149] 27 , an information processing device 1000 according to the tenth embodiment includes a determination unit 1040, a calibration unit 1050, a calibration instruction unit 1010, and an output unit 1020. The multiple sensors 10 may be, for example, LiDAR, radar, an ultrasonic sensor, or a camera.

[0150] When a positional deviation occurs among the sensors 10, the information processing device 1000 instructs the user to perform an operation to calibrate the positional relationship between the multiple sensors 10. This allows the information processing device 1000 to calibrate the positional relationship between the multiple sensors 10 using sensing data obtained by sensing an operation performed by the user. Note that a positional deviation between the multiple sensors 10 means that the positional relationship between the multiple sensors 10 deviates from a known positional relationship.

[0151] The determination unit 1040 determines whether or not there is misalignment of each of the plurality of sensors 10. For example, the determination unit 1040 may determine whether or not there is misalignment of each of the plurality of sensors 10 using the method described in other embodiments or a known method.

[0152] The calibration instruction unit 1010 generates an image, text, sound, or a combination thereof that instructs the user to perform an operation to calibrate the multiple sensors 10. For example, the calibration instruction unit 1010 may generate an image that instructs the user to present a calibration chart in a predetermined position. For example, the calibration instruction unit 1010 may generate the image shown in FIG. 28 or FIG. 29.

[0153] As shown in FIG. 28, the calibration instruction unit 1010 may generate an image P1 including an overhead image RV and a camera image SV sensed by a sensor 10 mounted on the moving object 1.

[0154] The overhead image RV is an image in which sensing data from a sensor 10 such as a LiDAR is projected so that the positional relationship between the moving object 1 and the user U is viewed from above. The calibration instruction unit 1010 may superimpose a target point GP to which the user U carrying the calibration chart C is heading on the overhead image RV. The camera image SV is an image of the user U present in front of the moving object 1 captured by a sensor 10 such as a camera. The calibration instruction unit 1010 may superimpose a target area GC in which the calibration chart C is presented to the sensor 10 on the camera image SV.

[0155] The calibration instruction unit 1010 can instruct the user U on the position at which to present the calibration chart C by instructing the three-dimensional position at which to present the calibration chart C from the sky direction and the horizontal direction using the overhead image RV and the camera image SV. The information processing device 1000 can calibrate the positional relationship between the multiple sensors 10 by having the user present such a calibration chart C multiple times.

[0156] On the other hand, if it is difficult for the user to perform the work of calibrating multiple sensors 10, the calibration instruction unit 1010 may generate an image P2 instructing the user to contact a service center, as shown in Figure 29.

[0157] The output unit 1020 presents to the user the details of the work for calibrating the multiple sensors 10 by outputting the image, text, or sound generated by the calibration instruction unit 1010. The output unit 1020 may be, for example, a display device that outputs the image or text, a sound output device that outputs the sound, or a communication device that transmits the image, text, or sound to a terminal device carried by the user.

[0158] The calibration unit 1050 calibrates the positional relationship of the plurality of sensors 10 using sensing data acquired by the plurality of sensors 10. Specifically, the calibration unit 1050 calibrates the positional relationship of the plurality of sensors 10 using sensing data of the content of an operation performed by a user in accordance with instructions output by the output unit 1020. For example, the calibration unit 1050 may calibrate the positional relationship of the plurality of sensors 10 using three-dimensional coordinates in the sensing data of a calibration chart presented by the user at a predetermined position.

[0159] According to the above configuration, when a positional shift occurs in the sensor 10, the information processing device 1000 according to the tenth embodiment can notify the user of the occurrence of the positional shift in the sensor 10 and the details of the work required to calibrate the positional relationship of the sensor 10.

[0160] 3. Hardware Configuration Example> Furthermore, a configuration example of a computer 2000 that realizes the information processing devices 100, 200, 300, 400, 500, 600, 700, 800, 900, and 1000 according to the embodiments of the present disclosure will be described with reference to Fig. 30. Fig. 30 is a block diagram showing the configuration example of the computer 2000.

[0161] The functions of the computer 2000 may be realized by cooperation of software and the hardware described below. Each function of detection, calculation, judgment, control, or processing in each embodiment of the present disclosure may be executed by, for example, the CPU 2001.

[0162] As shown in FIG. 30, a computer 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, and a RAM (Random Access Memory) 2003.

[0163] The computer 2000 may further include a host bus 2004a, a bridge 2004, an external bus 2004b, an interface 2005, an input device 2006, an output device 2007, a storage device 2008, a drive 2009, a connection port 2010, or a communication device 2011. The computer 2000 may include a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 2001.

[0164] The CPU 2001 functions as an arithmetic processing unit or a control unit, and controls the operations within the computer 2000 in accordance with various programs recorded in the ROM 2002, the RAM 2003, the storage device 2008, or a removable recording medium attached to the drive 2009. The ROM 2002 stores programs used by the CPU 2001, calculation parameters, etc. The RAM 2003 temporarily stores programs used in the execution of the CPU 2001, and parameters used during the execution of the programs.

[0165] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004a capable of high-speed data transmission. The host bus 2004a is connected to an external bus 2004b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 2004, and the external bus 2004b is connected to various components via an interface 2005.

[0166] The input device 2006 is a device that accepts input from a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 2006 may also be a microphone that detects the user's voice. The input device 2006 may also be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device that supports operation of the computer 2000.

[0167] The input device 2006 further includes an input control circuit that outputs an input signal generated based on information input by the user to the CPU 2001. By operating the input device 2006, the user can input various data to the computer 2000 or instruct the computer 2000 to perform processing operations.

[0168] The output device 2007 is a device capable of visually or audibly presenting information acquired or generated by the computer 2000 to a user. The output device 2007 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an OLED (Organic Light Emitting Diode) display, a hologram, or a projector, an audio output device such as a speaker or headphones, or a printing device such as a printer. The output device 2007 can output information acquired by processing by the computer 2000 as video such as text or an image, or sound such as voice or audio.

[0169] The storage device 2008 is a data storage device configured as an example of a storage unit of the computer 2000. The storage device 2008 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 2008 can store programs executed by the CPU 2001, various data, various data acquired from the outside, and the like.

[0170] The drive 2009 is a device for reading or writing data from or to a removable recording medium such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is either built into or externally attached to the computer 2000. For example, the drive 2009 can read information recorded on an attached removable recording medium and output the information to the RAM 2003. The drive 2009 can also write data to an attached removable recording medium.

[0171] The connection port 2010 is a port for directly connecting an external device to the computer 2000. The connection port 2010 may be, for example, a Universal Serial Bus (USB) port, an IEEE 13204 port, or a Small Computer System Interface (SCSI) port. The connection port 2010 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting the connection port 2010 to an external device, various types of data can be sent and received between the computer 2000 and the external device.

[0172] The communication device 2011 is, for example, a communication interface configured with a communication device for connecting to the communication network 2020. The communication device 2011 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 2011 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0173] The communication device 2011 can transmit and receive signals, for example, via the Internet or with other communication devices using a predetermined protocol such as TCP / IP. The communication network 2020 connected to the communication device 2011 is a wired or wireless network, and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0174] It is also possible to create a program for causing hardware such as the CPU 2001, ROM 2002, and RAM 2003 built into the computer 2000 to perform functions equivalent to those of the information processing devices 100, 200, 300, 400, 500, 600, 700, 800, 900, and 1000. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0175] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0176] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0177] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing device comprising: a calculation unit that calculates a positional deviation amount for each of a plurality of sensors of different types; a recognition unit that executes a recognition task by integrating sensing data acquired by the plurality of sensors; and a processing unit that masks a partial area of ​​a recognition result of the recognition task based on the positional deviation amount. (2) The information processing device according to (1), wherein the processing unit masks an area where the reliability of the recognition result is lower than a threshold based on the positional deviation amount. (3) The information processing device according to (2), wherein the threshold is set further based on the type of the recognition task and the type of the sensor for which the positional deviation amount is calculated. (4) The information processing device according to any one of (1) to (3), further comprising a generation unit that generates correction data from the sensing data by correcting the positional deviation amount based on the positional deviation amount. (5) The information processing device according to (4), wherein the generation unit generates the correction data by performing viewpoint conversion on the sensing data based on the positional deviation amount. (6) The information processing device according to any one of (1) to (5), further comprising a determination unit that determines whether or not there is misalignment of the plurality of sensors. (7) The information processing device according to (6), wherein the determination unit determines whether or not there is misalignment of the plurality of sensors by comparing the sensing data from which an area corresponding to a moving object has been excluded. (8) The information processing device according to (7), wherein the area corresponding to the moving object is determined based on a Doppler velocity detected by radar. (9) The information processing device according to (6), wherein the determination unit determines whether or not there is misalignment of the plurality of sensors by comparing areas of stationary objects included in the sensing data. (10) The information processing device according to (6), wherein the determination unit determines whether or not there is misalignment of the plurality of sensors by matching straight lines detected from the sensing data. (11) The information processing device according to (6), wherein the determination unit determines whether or not there is misalignment of the plurality of sensors based on a connection error when the sensing data is continuously spliced ​​together.(12) The information processing device according to any one of (6) to (11), further comprising a calibration unit that calibrates a positional relationship between the plurality of sensors. (13) The information processing device according to (12), wherein the calibration unit calibrates the positional relationship between the plurality of sensors by using coordinates of joint points of a person included in the sensing data. (14) The information processing device according to (12), wherein the plurality of sensors are mounted on a moving body, and the calibration unit calibrates the positional relationship between the plurality of sensors based on a travel trajectory of the moving body estimated from the sensing data. (15) The information processing device according to any one of (6) to (14), further comprising a notification unit that notifies a user of information regarding positional deviations of the plurality of sensors. (16) The information processing device according to (1), further comprising a synchronization control unit that adjusts sensing timings of the plurality of sensors based on captured images of a plurality of light-emitting units that emit light in synchronization with the sensing timings of the plurality of sensors. (17) An information processing method by a computer, comprising: calculating an amount of positional deviation for each of a plurality of sensors of different types; integrating sensing data acquired by the plurality of sensors to execute a recognition task; and masking a partial area of ​​a recognition result of the recognition task based on the amount of positional deviation. (18) A program for causing a computer to function as: a calculation unit that calculates an amount of positional deviation for each of a plurality of sensors of different types; a recognition unit that integrates sensing data acquired by the plurality of sensors to execute a recognition task; and a processing unit that masks a partial area of ​​a recognition result of the recognition task based on the amount of positional deviation.

[0178] REFERENCE SIGNS LIST 1 Mobile object 10 Sensor 20 Light emitting unit 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000 Information processing device 110 Difference detection unit 120 Storage unit 130, 210, 310 Projection processing unit 220 Line detection unit 410 Notification unit 420, 620 Parameter storage unit 510 Light emission control unit 520 Synchronization control unit 610 Position estimation unit 630 Trajectory storage unit 710 Stationary determination unit 720 Moving object detection unit 730 Moving object exclusion unit 810 Generation unit 860 Calculation unit 910 Threshold setting unit 911 Threshold calculation unit 920 Recognition unit 930 Processing unit 960 Calculation unit 140,240,340,440,740,840,940,1040 Judgment section 250,450,650,1050 Calibration section 1010 Calibration instruction section 1020 Output section

Claims

1. An information processing device comprising: a calculation unit that calculates an amount of positional deviation for each of a plurality of sensors of different types; a recognition unit that integrates sensing data acquired by the plurality of sensors to execute a recognition task; and a processing unit that masks a partial area of ​​the recognition result of the recognition task based on the amount of positional deviation.

2. The information processing device according to claim 1, wherein the processing section performs mask processing on an area where the reliability of the recognition result is lower than a threshold based on the amount of positional deviation.

3. The information processing device according to claim 2, wherein the threshold value is set further based on the type of the recognition task and the type of the sensor for which the position deviation amount is calculated.

4. The information processing device according to claim 1, further comprising a generating unit that generates correction data by correcting the amount of positional deviation from the sensing data based on the amount of positional deviation.

5. The information processing device according to claim 4, wherein the generation unit generates the correction data by performing viewpoint conversion on the sensing data based on the amount of positional deviation.

6. The information processing device according to claim 1, further comprising a determination unit that determines whether or not there is misalignment of the plurality of sensors.

7. The information processing device according to claim 6, wherein the determination unit determines the presence or absence of positional deviation of the plurality of sensors by comparing the sensing data from which an area corresponding to a moving object has been excluded with each other.

8. The information processing device according to claim 7, wherein the region corresponding to the moving object is determined based on a Doppler velocity detected by a radar.

9. The information processing device according to claim 6, wherein the determination unit determines whether or not there is a positional deviation of the plurality of sensors by comparing areas of stationary objects included in the sensing data with each other.

10. The information processing device according to claim 6, wherein the determination unit determines whether or not there is misalignment of the plurality of sensors by matching straight lines detected from the sensing data.

11. The information processing device according to claim 6, wherein the determination unit determines whether or not there is a positional deviation of the plurality of sensors based on a connection deviation when the sensing data is continuously pieced together.

12. The information processing device according to claim 6, further comprising a calibration unit that calibrates the positional relationship of the plurality of sensors.

13. The information processing device according to claim 12, wherein the calibration unit calibrates the positional relationship of the plurality of sensors by using coordinates of joint points of the person included in the sensing data.

14. The information processing device according to claim 12, wherein the plurality of sensors are mounted on a moving body, and the calibration unit calibrates the positional relationship of the plurality of sensors based on a travel trajectory of the moving body estimated from the sensing data.

15. The information processing device according to claim 6, further comprising a notification unit that notifies a user of information regarding positional deviations of the plurality of sensors.

16. The information processing device according to claim 1, further comprising a synchronization control unit that adjusts the sensing timing of the plurality of sensors based on captured images of a plurality of light-emitting elements that emit light in synchronization with the sensing timing of the plurality of sensors, respectively.

17. An information processing method by a computer, comprising: calculating an amount of positional deviation for each of a plurality of sensors of different types; integrating sensing data acquired by the plurality of sensors to execute a recognition task; and masking a partial area of ​​the recognition result of the recognition task based on the amount of positional deviation.

18. A program for causing a computer to function as: a calculation unit that calculates the amount of positional deviation of each of multiple sensors of different types; a recognition unit that integrates sensing data acquired by the multiple sensors to execute a recognition task; and a processing unit that masks a partial area of ​​the recognition result of the recognition task based on the amount of positional deviation.

Images