Information processing device, information processing method, and program

Abstract

To provide a novel and improved technique capable of detecting a change in an installation condition of a sensor and highly accurately calibrating the sensor.SOLUTION: An information processing device comprises: an acquisition unit which acquires first point group data indicative of a shape of a first object; a conversion unit which converts coordinates of the acquired first point group data on the basis of conversion parameters; a detection unit which detects a position of the first object on the basis of the first point group data after the coordinate conversion; an object tracking unit which generates a movement locus of the first object by tracking the position of the first object; a determination unit which determines whether prescribed conditions based on the movement locus are met; and a correction unit which estimates a movement direction of a second object from the second point group data obtained by the sensor on the basis of the determination that the movement locus meets the prescribed conditions, calculates a matrix in which the movement direction is made to match a prescribed direction, and corrects the conversion parameters on the basis of the matrix.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, a technology has been known that uses a sensor to estimate the position of an object around the sensor. For example, Patent Document 1 discloses an apparatus that identifies the state of a plurality of devices, such as their positions. Position information of an object estimated using such a technology can be utilized for controlling the automatic operation of a moving body such as a vehicle or a robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-066679 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described techniques, if there is a change in the installation conditions of the sensor, such as a shift in the installation angle of the sensor, there is a possibility that the sensing results of the sensor will deviate.

[0005] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a new and improved technology that is capable of detecting changes in sensor installation conditions and calibrating the sensor with high accuracy. [Means for solving the problem]

[0006] In order to solve the above problem, according to one aspect of the present invention, there is provided an information processing device comprising: an acquisition unit that acquires first point cloud data obtained by a sensor and indicating a shape of a first object; a conversion unit that performs processing to convert coordinates of the acquired first point cloud data based on a transformation parameter; a detection unit that detects a position of the first object based on the first point cloud data after coordinate transformation; an object tracking unit that generates a movement trajectory of the first object by tracking the position of the first object; a determination unit that determines whether a predetermined condition based on the movement trajectory is satisfied; and a correction unit that estimates a movement direction of a second object from second point cloud data obtained by the sensor based on the determination that the movement trajectory satisfies the predetermined condition, calculates a matrix that matches the movement direction with a predetermined direction, and corrects the transformation parameter based on the matrix.

[0007] The predetermined direction may be a first coordinate axis direction in a sensor coordinate system.

[0008] The specified condition may be that the difference in coordinates of two different positions of the first object included in the movement trajectory in a second coordinate axis direction perpendicular to the first coordinate axis direction in the sensor coordinate system exceeds a threshold value.

[0009] The two points may include, among the positions of the first object, a point whose coordinate in the first coordinate axis direction is maximum, and a point whose coordinate in the first coordinate axis direction is minimum.

[0010] The transformation parameters may include a rotation matrix for rotating coordinates of the first point cloud data.

[0011] The information processing device may further include a notification unit that performs a zone determination process to determine a zone to which coordinates in a second coordinate axis direction orthogonal to the first coordinate axis direction of the position of the first object belong.

[0012] The notification unit may notify another device of information indicating the section to which the coordinate of the first object in the second coordinate axis direction belongs, based on a result of the section determination process.

[0013] The correction unit may estimate, based on a distribution of points corresponding to the detected second object among the points included in the second point cloud data, a direction in which the variance of the distribution is maximum as the movement direction of the second object.

[0014] The correction unit may calculate a covariance matrix based on a distribution of coordinates of each point included in the second point cloud data, an eigenvalue and an eigenvector of the covariance matrix, and calculate the matrix that matches the movement direction of the second object to the predetermined direction based on the eigenvalue and the eigenvector.

[0015] The information processing device may further include a screen generation unit that generates an output screen including a display of a trajectory point cloud based on points corresponding to the second object among the points included in the second point cloud data.

[0016] The screen generation unit may generate an output screen that displays, in comparison, the trajectory point cloud after the points corresponding to the second object have been coordinate transformed based on the transformation parameters before modification by the modification unit and the trajectory point cloud after the points corresponding to the second object have been coordinate transformed based on the transformation parameters after modification.

[0017] The screen generation unit may generate an output screen that superimposes and displays the trajectory point cloud after the points corresponding to the second object have been coordinate transformed based on the transformation parameters before modification and the trajectory point cloud after the points corresponding to the second object have been coordinate transformed based on the transformation parameters after modification.

[0018] The object tracking unit may generate the movement trajectory of the first object by matching a position of the first object detected in a frame with a position of the first object detected in a frame preceding the frame in the point cloud data obtained by the sensor.

[0019] The second point cloud data may include point cloud data in a frame different from a frame included in the first point cloud data.

[0020] The second point cloud data may include point cloud data in the same frames as frames included in the first point cloud data.

[0021] The acquisition unit may acquire the first point cloud data and the second point cloud data by combining point cloud data acquired by a plurality of the sensors.

[0022] The first object may include a mobile object traveling by land, air, or sea.

[0023] The moving body may include at least one or both of a manned moving body that moves by being operated by a passenger and an unmanned moving body that can move autonomously.

[0024] In addition, according to another aspect of the present invention, in order to solve the above-mentioned problem, there is provided an information processing method executed by a computer, including a processor acquiring first point cloud data indicating a shape of a first object, obtained by a sensor, performing a process of converting coordinates of the acquired first point cloud data based on a transformation parameter, detecting a position of the first object based on the first point cloud data after coordinate transformation, generating a movement trajectory of the first object by tracking the position of the first object, determining whether a predetermined condition based on the movement trajectory is satisfied, estimating a movement direction of a second object from second point cloud data obtained by the sensor based on the determination that the movement trajectory satisfies the predetermined condition, calculating a matrix that matches the movement direction with a predetermined direction, and modifying the transformation parameter based on the matrix.

[0025] In addition, according to another aspect of the present invention, in order to solve the above problem, a program is provided for causing a computer to function as an information processing device, comprising: an acquisition unit that acquires first point cloud data indicating a shape of a first object obtained by a sensor; a conversion unit that performs processing to convert coordinates of the acquired first point cloud data based on a transformation parameter; a detection unit that detects a position of the first object based on the first point cloud data after coordinate transformation; an object tracking unit that generates a movement trajectory of the first object by tracking the position of the first object; a determination unit that determines whether a predetermined condition based on the movement trajectory is satisfied; and a correction unit that estimates a movement direction of a second object from second point cloud data obtained by the sensor based on the determination that the movement trajectory satisfies the predetermined condition, calculates a matrix that matches the movement direction to a predetermined direction, and corrects the transformation parameter based on the matrix. Effect of the Invention

[0026] As described above, the present invention provides a new and improved technique that is capable of detecting changes in sensor installation conditions and calibrating the sensor with high accuracy. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is an explanatory diagram showing an application example of an information processing system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram for explaining the correspondence relationship between roads in the real world and the position and moving direction of a vehicle in an operational coordinate system before the installation conditions of the sensor device 10 are changed. [Diagram 3] 1 is a diagram for explaining the correspondence between roads in the real world and the movement trajectory of a vehicle in an operational coordinate system before the installation conditions of the sensor device 10 are changed. FIG. [Figure 4] 1 is a diagram for explaining the correspondence relationship between roads in the real world and the position and moving direction of a vehicle in an operational coordinate system after the installation conditions of the sensor device 10 have changed. [Diagram 5] 10 is a diagram for explaining the correspondence between roads in the real world and the movement trajectory of a vehicle in an operational coordinate system after the installation conditions of the sensor device 10 have changed. FIG. [Figure 6] 1 is a block diagram showing an example of the configuration of an information processing device 20 according to the present embodiment. [Figure 7] 13 is a diagram for explaining a determination made by a determining unit 236 as to whether or not the installation conditions of the sensor device 10 have changed. FIG. [Figure 8] 2 is an explanatory diagram showing an example of a trajectory point cloud of a vehicle 40 extracted by an extraction unit 237. FIG. [Figure 9] 13A to 13C are diagrams illustrating estimation of a moving direction based on a trajectory point group by a correction unit 238 according to the present embodiment. [Figure 10] 13 is a diagram showing an example of an output screen generated by a screen generating unit 239. FIG. [Figure 11] 13 is a diagram showing another example of the output screen generated by the screen generating unit 239. FIG. [Figure 12] 10 is a flowchart illustrating an example of the operation of the information processing system according to the present embodiment. [Figure 13] 1 is a diagram illustrating an example of a hardware configuration of an information processing device 20 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and duplicated explanations will be omitted.

[0029] In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numeral. However, when there is no need to particularly distinguish each of multiple components having substantially the same functional configuration, each of the multiple components is given only the same reference numeral.

[0030] <1. Overview> An embodiment of the present invention relates to a technology that can automatically detect a change in installation conditions of a sensor used to estimate the position of an object and calibrate the sensor with higher accuracy.

[0031] Fig. 1 is an explanatory diagram showing an application example of an information processing system according to an embodiment of the present invention. The example shown in Fig. 1 is an example in which the embodiment of the present invention detects information such as the position of a traveling vehicle based on three-dimensional point cloud data acquired by a sensor installed on the roadside of a highway. The detected information such as the position of the traveling vehicle is distributed as traffic support information to each vehicle traveling on the highway. In addition, the detected position of the traveling vehicle can be used, for example, for controlling an autonomous vehicle.

[0032] It should be noted that the application of the information processing system shown in FIG. 1 is merely an example, and suitable applications of the information processing system are not limited to sensors installed on expressways.

[0033] <1-1. System configuration example> 1, an information processing system according to an embodiment of the present invention includes a sensor device 10 and an information processing device 20. The sensor device 10 and the information processing device 20 are connected to each other via a network so as to be able to communicate with each other.

[0034] (Sensor device 10) The sensor device 10 according to the present embodiment is installed on the roadside of the expressway R1 and has a function of acquiring point cloud data showing the shapes of each of the vehicles 40A to 40D traveling on the expressway R1. In the following description, the vehicles 40A to 40D may be simply referred to as "vehicle 40" without distinction. The sensor device 10 is realized by, for example, a LiDAR (Light Detection and Ranging) sensor that emits laser light and measures the distance to a surrounding object based on the reflected light of the laser light, thereby outputting point cloud data showing the three-dimensional shape of the object. The LiDAR sensor may also be referred to as a "laser sensor." Note that various sensors other than the LiDAR sensor (for example, a distance measuring sensor, etc.) may be used as the sensor device 10. For example, the sensor device 10 may be a radar (for example, a millimeter wave radar, etc.) that uses electromagnetic waves instead of laser light. Alternatively, the sensor device 10 may be a sonar that uses sound waves instead of laser light.

[0035] 1, the information processing system includes two sensor devices, a sensor device 10A and a sensor device 10B. The sensor device 10A transmits point cloud data of objects included in a detection range S1 to the information processing device 20. Similarly, the sensor device 10B transmits point cloud data of objects included in a detection range S2 to the information processing device 20.

[0036] The number of sensor devices 10 included in the present information processing system is not particularly limited. The present information processing system may include one sensor device 10. Alternatively, the present information processing system may include three or more sensor devices 10.

[0037] (Information processing device 20) The information processing device 20 has a function of receiving the point cloud data from the sensor device 10 and detecting the position of a vehicle 40 traveling on a highway based on the point cloud data.

[0038] In addition, the information processing device 20 has a function of automatically detecting a change in the installation conditions of the sensor device 10 based on the point cloud data acquired from the sensor device 10.

[0039] In this specification, a change in the installation conditions of the sensor device 10 may include, for example, a shift in the installation angle of the sensor device 10 caused by a physical tilt of the sensor device 10. In addition, a change in the installation conditions of the sensor device 10 may also include a change in the roads included in the detection range of the sensor device 10 due to a change in the surrounding environment in which the sensor device 10 is installed, such as distortion of a road due to an earthquake.

[0040] When the installation conditions of the sensor device 10 change, the detection range S of the sensor device 10 may also change. When the detection range S of the sensor device 10 changes in this way, there is a possibility that a deviation occurs in the position of an object detected based on the point cloud data acquired by the sensor device 10.

[0041] Therefore, when the information processing device 20 according to this embodiment detects a change in the installation conditions of the sensor device 10, it calibrates the sensor device 10 based on the point cloud data obtained by the sensor device 10. This allows the information processing device 20 to maintain the accuracy of detecting the position of the vehicle 40 based on the point cloud data obtained from the sensor device 10 even if the installation conditions of the sensor device 10 change.

[0042] In addition, the information processing device 20 transmits to the communication device 30 various pieces of information related to the position of the vehicle 40 detected based on the point cloud data.

[0043] 1 is installed on an expressway R1 and has a function of distributing various types of traffic support information to vehicles 40 traveling on the expressway R1. Each vehicle 40 can obtain, from the traffic support information received from the communication device 30, for example, road congestion status, accident occurrence status, or driving support information used for autonomous driving.

[0044] In the example shown in FIG. 1, the vehicle 41 is assumed to be an autonomous vehicle. The vehicle 41 is about to merge from the connecting road to the main road of the expressway R1. In such a situation, in order for the vehicle 41 to merge safely onto the main road, information on the positions of the vehicles 40A to 40D traveling on the main road is required. For example, the information on the position may include information such as the position of the vehicle and the detection time of the position, the inter-vehicle time or inter-vehicle distance between the vehicle and a preceding vehicle traveling in front of the vehicle, the estimated speed of the vehicle, and which lane the vehicle 40 is traveling in. Therefore, the information processing device 20 estimates information on the positions of the vehicles 40A to 40D from the point cloud data obtained by the sensor device 10A and the sensor device 10B, and distributes the information to each vehicle via the communication device 30. The vehicle 41 can control the operation when merging onto the main road based on the information received from the communication device 30A and the communication device 30B installed on the roadside of the connecting road.

[0045] The information regarding the position of the vehicle 40 given above is merely an example, and the information processing device 20 may transmit various other pieces of information to the communication device 30 as traffic assistance information.

[0046] The display device 50 shown in FIG. 1 is a display device capable of outputting various pieces of information received from the information processing device 20 on its screen.

[0047] For example, the information processing device 20 may control the display device 50 to display information about the position of the vehicle 40 estimated based on the point cloud data acquired from the sensor device 10, or information such as the movement trajectory of the vehicle 40. The operator of the information processing device 20 can check the calibration results by the information processing device 20 according to the various information displayed on the display device 50.

[0048] In the example shown in FIG. 1, the information processing device 20 and the display device 50 are realized as separate devices, but the display device 50 may be configured integrally with the information processing device 20 as one of the components of the information processing device 20.

[0049] <1-2. Identifying issues> In recent years, the realization of self-driving cars is expected to help resolve social issues such as reducing traffic accidents and resolving labor shortages. As a fundamental technology for the operation of self-driving cars, for example, in Japan, progress is being made in the practical application of infrastructure-cooperative ITS (Intelligent Transport Systems), in which self-driving cars and infrastructure such as transportation systems cooperate to provide traffic support for self-driving cars.

[0050] One example of infrastructure-cooperative ITS is a technology that supports autonomous vehicles merging from a connecting road onto a main road on an expressway. On a connecting road with poor visibility, it is difficult for autonomous vehicles to grasp traffic information on the main road. For this reason, there is a risk that autonomous vehicles will disrupt traffic flow at the merging section. As described above with reference to FIG. 1, a merging support system is provided in which a sensor such as a sensor device 10 is installed on the roadside of the expressway to extract traffic information on the main road in a planar manner, and traffic support information is distributed to each vehicle 40 via a communication device 30 (also called an ITS spot in Japan) installed on the roadside. This allows autonomous vehicles to safely and smoothly merge onto the expressway.

[0051] Here, in order for the information processing device 20 to estimate information related to the position of the vehicle 40 with high accuracy from the point cloud data obtained by the sensor device 10, a process (calibration) is performed to calculate conversion parameters for converting the coordinates of each point of the point cloud data from the coordinates of a sensor coordinate system (hereinafter also referred to as a "reference coordinate system") that is the reference for the sensor device 10 to coordinates of a sensor coordinate system (hereinafter also referred to as an "operation coordinate system") suitable for analysis. Note that in the following description, it is assumed that the sensor coordinate system is a three-dimensional orthogonal coordinate system with three axes, the X-axis, the Y-axis, and the Z-axis, as its coordinate axes.

[0052] Fig. 2 is a diagram for explaining the correspondence relationship between roads in the real space and the position and movement direction of a vehicle in the operational coordinate system before a change in the installation conditions of the sensor device 10. A space SP1 on the left side of Fig. 2 shows the state of the real space included in the detection range S of the sensor device 10. In the example shown in Fig. 2, the space SP1 includes a vehicle 40 traveling on a lane L1 of a road R.

[0053] On a straight road such as an expressway, the vehicle 40 travels along a course that follows the lane markings LM in principle. In this case, the moving direction of the vehicle 40 is estimated to be a straight line that generally follows the lane markings LM. Therefore, in the initial setting of the transformation parameters, the transformation parameters are set so that the Y-axis in the operational coordinate system of the sensor device 10 follows the lane markings LM. Then, until the installation conditions of the sensor device 10 change, the estimation of information related to the vehicle position in such an operational coordinate system of the sensor device 10 continues.

[0054] The operational coordinate system SC1 shown on the right side of Figure 2 is the operational coordinate system of the sensor device 10 before the installation conditions of the sensor device 10 are changed, and the operational coordinate system SC1 shows the position and movement direction of the vehicle 40 detected by the sensor device 10 from the space SP1 and transformed using initially set transformation parameters.

[0055] As shown in Fig. 2, after the conversion parameters are initially set and before the installation conditions of the sensor device 10 are changed, the Y axis of the operational coordinate system SC1 of the sensor device 10 is oriented along the lane line LM and is generally parallel to the direction of movement of the vehicle 40. Note that in Fig. 2, the Z axis of the operational coordinate system SC1 is omitted and only the X axis and Y axis are shown. The same applies to Figs. 3 to 5 below.

[0056] Fig. 3 is a diagram for explaining the correspondence between roads in the real world and the movement trajectory of the vehicle in the operation coordinate system before the installation conditions of the sensor device 10 change. The movement trajectory TR1 shown in Fig. 3 indicates the transition of the position of the vehicle 40 in the operation coordinate system before the installation conditions of the sensor device 10 change. In the example shown in Fig. 3, the vehicle 40 moves in the direction indicated by the dashed arrow D1.

[0057] When a coordinate axis (e.g., the Y axis) of the operational coordinate system is generally parallel to the moving direction of the vehicle 40, as in the operational coordinate system SC1 before the installation conditions of the sensor device 10 are changed, the information processing device 20 can estimate the moving distance of the vehicle 40 from the Y coordinate value of the position of the vehicle 40 in the operational coordinate system. Also, the information processing device 20 can estimate the lane L in which the vehicle 40 is traveling from the X coordinate value of the position of the vehicle 40 in the operational coordinate system.

[0058] In this way, if the coordinates of the point cloud data can be converted so that the coordinate axes and the direction of the lane lines are parallel in the operational coordinate system of the sensor device 10, it will be convenient for the information processing device 20 to detect information regarding the position of the vehicle 40.

[0059] The transformation parameters may be, for example, a rotation matrix that rotates the point cloud data.

[0060] However, even if the above-mentioned highly accurate matrix is ​​obtained when the sensor device 10 is first installed, the installation angle of the sensor device 10 may gradually shift as the merging assistance system is operated. For example, the installation fixture of the sensor device 10 may gradually loosen. Or, the ground on which the sensor is installed may tilt due to the occurrence of an earthquake. Also, on the roadside of a highway, it is possible that minute deviations in the installation angle may accumulate over the long term due to vibrations caused by large trucks passing by the sensor device 10.

[0061] If the installation conditions of the sensor device 10 change from the initial installation conditions, even if coordinate conversion is performed based on the initial conversion parameters, the coordinate axis direction after the coordinate conversion process may deviate from a direction parallel to the division line.

[0062] 4 is a diagram for explaining the correspondence relationship between roads in the real world and the position and moving direction of a vehicle in the operational coordinate system after a change in the installation conditions of the sensor device 10. As shown in FIG. 4, after a change in the installation conditions of the sensor device 10, the Y-axis of the operational coordinate system SC2 of the sensor device 10 is not parallel to the moving direction of the vehicle 40.

[0063] 5 is a diagram for explaining the correspondence between roads in the real world and the movement trajectory of the vehicle in the operation coordinate system after the installation conditions of the sensor device 10 are changed. The movement trajectory TR2 shown in FIG. 5 shows the transition of the position of the vehicle 40 in the operation coordinate system after the installation conditions of the sensor device 10 are changed. Although the vehicle 40 is actually traveling on the lane L1, as shown in FIG. 5, in the movement trajectory TR2, the movement direction of the vehicle 40 extends in a direction straddling the lane L1 to the lane L2 as indicated by the dashed arrow D2.

[0064] In this way, when the installation conditions of the sensor device 10 change, a discrepancy occurs between the actual position of the vehicle 40 on the expressway and the vehicle position on the coordinate axis of the point cloud data. As a result, the information processing device 20 may erroneously calculate lane information indicating which lane the vehicle 40 is in, for example. If erroneous traffic support information is distributed via the communication device 30, there is a risk that the autonomous vehicle will not be able to safely merge onto the main lane.

[0065] 5, the X-coordinate position of the minimum distance point OP_Min2 belongs to lane L1, while the X-coordinate positions of each point move away from lane L1 as the vehicle 40 moves in the direction of dashed arrow D2. Thus, the longer the detection distance of the sensor device 10, the greater the error in the position detection result may be caused by a slight deviation in the installation angle of the sensor device 10. For this reason, high accuracy is required for the conversion parameters used in the coordinate conversion process for rotating the point cloud data obtained by the sensor device 10.

[0066] As described above, when the installation conditions of the sensor device 10 change, it is desirable to be able to calculate transformation parameters as quickly and accurately as possible so that the coordinate axis direction in the operational coordinate system of the sensor device 10 becomes parallel to the direction of the demarcation line.

[0067] A method of recalculating the conversion parameters with high accuracy when the installation conditions of the sensor device 10 change can be, for example, a method of using RTK-GNSS (Real Time Kinematic-Global Navigation Satellite System), which is a centimeter-level high-accuracy positioning method. For example, a highly accurate trajectory point cloud can be obtained by mounting an RTK mobile station on a test vehicle for positioning and running on a highway. At the same time, the position of the test vehicle is also detected from the point cloud data obtained by the sensor device 10. By comparing the trajectory point cloud obtained by the RTK-GNSS as a true value with the detection result of the point cloud data obtained by the sensor device 10, it is possible to correct deviations due to changes in the installation conditions of the sensor device 10 with high accuracy.

[0068] However, with this method, every time the installation conditions of the sensor device 10 change, the mobile station needs to be mounted on a test vehicle and the test vehicle needs to be driven, which requires a lot of time and effort to recalculate the conversion parameters.

[0069] Therefore, an embodiment of the present invention has been conceived with the above-mentioned circumstances in mind, and provides a technology that can automatically detect changes in the installation conditions of the sensor device 10 and quickly recalculate highly accurate conversion parameters in accordance with the changed installation conditions.

[0070] More specifically, the information processing device 20 according to this embodiment determines whether or not a predetermined condition is satisfied based on the movement trajectory of the vehicle 40 extracted from the point cloud data acquired by the sensor device 10. The predetermined condition may be a condition that the difference in coordinates of two different points among the positions of the first vehicle included in the movement trajectory in the direction of a second coordinate axis (e.g., X-axis) perpendicular to the direction of a first coordinate axis (e.g., Y-axis) in the operational coordinate system of the sensor device 10 exceeds a threshold value.

[0071] In addition, the two points may include a point among the positions of the first vehicle whose coordinate in the direction of a first coordinate axis (e.g., the Y axis) is maximum, and a point whose coordinate in the direction of the first coordinate axis (e.g., the Y axis) is minimum.

[0072] By carrying out the above-mentioned processing, the information processing device 20 can automatically detect a change in the installation conditions of the sensor device 10.

[0073] Furthermore, the information processing device 20 according to this embodiment estimates the moving direction of the second vehicle in the reference coordinate system of the sensor device 10 from the second point cloud data obtained by the sensor device 10, based on the determination that the moving trajectory satisfies a predetermined condition. Note that the second vehicle may be the same vehicle as the first vehicle, or may be a different vehicle.

[0074] Furthermore, the information processing device 20 according to the present embodiment calculates a matrix that makes the estimated moving direction coincide with a predetermined direction. The predetermined direction may be the direction of a first coordinate axis (e.g., the Y axis) in the reference coordinate system of the sensor device 10.

[0075] By performing the above-mentioned processing, when the information processing device 20 detects a change in the installation conditions of the sensor device 10, it can perform calibration based on the point cloud data acquired by the sensor device 10. Therefore, even if the installation conditions of the sensor device 10 change, it is possible to correct the deviation of the point cloud data after coordinate conversion quickly and with reduced effort.

[0076] In addition, the information processing device 20 according to this embodiment may estimate, based on the distribution of points corresponding to the detected second vehicle among the points included in the second point cloud data, the direction in which the variance of the distribution is maximum as the direction of movement of the second vehicle.

[0077] Furthermore, the information processing device 20 according to the present embodiment may calculate a covariance matrix based on the distribution of coordinates of each point included in the second point cloud data, and an eigenvalue and an eigenvector of the covariance matrix. The information processing device 20 may calculate a matrix that matches the moving direction of the second vehicle with a predetermined direction based on the calculated eigenvalue and eigenvector.

[0078] By performing the above-mentioned processing, the information processing device 20 can more accurately calculate a matrix that matches the moving direction of the second vehicle with the direction of the first coordinate axis (e.g., Y-axis) in the operational coordinate system of the sensor device 10. Therefore, the information processing device 20 can perform calibration with higher accuracy.

[0079] An example of the functional configuration of the sensor device 10 and the information processing device 20 that constitute the present information processing system for achieving the above will be described in detail below.

[0080] <2. Configuration example> <2-1. Sensor device 10>

[0081] The sensor device 10 has a function of acquiring a three-dimensional point cloud representing the shape of an object around the sensor device 10. The sensor device 10 is realized by, for example, a LiDAR sensor. The sensor device 10 according to this embodiment is installed on the roadside of a highway and acquires the shape of a vehicle 40 traveling on the highway.

[0082] The sensor device 10 according to this embodiment has a function of emitting laser light and measuring the distance and position of an object based on the emitted light reflected by the object.

[0083] The sensor device 10 according to the present embodiment generates point cloud data representing the shape of an object based on the measured distances to the distance measurement points on the surface of the object and the positions of the measured distances to the distance measurement points. For example, the sensor device 10 generates point cloud data representing a vehicle 40 traveling on the highway R1.

[0084] The sensor device 10 according to the present embodiment also has a communication function for communicating with other devices. For example, the sensor device 10 transmits the generated point cloud data to the information processing device 20.

[0085] An example of the configuration of the sensor device 10 has been described above.

[0086] <2-2. Information processing device 20> Fig. 6 is a block diagram showing an example of the configuration of the information processing device 20 according to this embodiment. As shown in Fig. 6, the information processing device 20 according to this embodiment has a communication unit 210, a control unit 230, a conversion parameter storage unit 240, an object information storage unit 250, and a trajectory point cloud data storage unit 260.

[0087] (Communication unit 210) The communication unit 210 according to this embodiment has a function of communicating with other devices. For example, the communication unit 210 receives point cloud data acquired by the sensor device 10 from the sensor device 10.

[0088] Furthermore, the communication unit 210, under the control of the control unit 230, transmits various pieces of information related to the position of the vehicle 40, analyzed based on the point cloud data acquired from the sensor device 10, to the communication device 30. For example, the communication unit 210 may transmit lane information, which is information indicating the lane (section) to which the vehicle 40 detected by the control unit 230 belongs, to the communication device 30.

[0089] Further, the communication unit 210 may transmit data of an output screen generated by a screen generating unit 239 (described later) to the display device 50 under the control of the control unit 230.

[0090] (Control unit 230) The control unit 230 controls the overall operation of the information processing device 20. For example, the control unit 230 controls communication between the communication unit 210 and other devices.

[0091] Such a control unit 230 has functions as an acquisition unit 231, a conversion unit 232, a detection unit 233, an object tracking unit 234, a notification unit 235, a determination unit 236, an extraction unit 237, a correction unit 238, and a screen generation unit 239.

[0092] The acquisition unit 231 acquires the point cloud data received by the communication unit 210 from the sensor device 10. When the communication unit 210 receives point cloud data from a plurality of sensor devices 10, the acquisition unit 231 may perform a process of synthesizing the point cloud data.

[0093] For example, there are cases where the detection ranges S of a plurality of sensor devices 10 are adjacent to each other, such as the sensor device 10A and the sensor device 10B shown in Fig. 1. In this case, the detection unit 233 may perform synthesis so as to link the point cloud data of the detection range S1 obtained by the sensor device 10A with the point cloud data of the detection range S2 obtained by the sensor device 10B. This makes it possible to obtain point cloud data of a wider range using a plurality of sensor devices 10.

[0094] The conversion unit 232 performs a process of converting the coordinates of the point cloud data acquired by the acquisition unit 231. The conversion unit 232 performs this process by using the conversion parameters stored in the conversion parameter storage unit 240.

[0095] The transformation parameters may be, for example, a rotation matrix that rotates the coordinates of the point cloud data obtained by the sensor device 10.

[0096] The transformation parameters are initially set so that the travel direction of the vehicle 40 (for example, the direction along the lane line LM) is aligned with the first coordinate axis direction in the operational coordinate system after the reference coordinate system of the sensor device 10 is transformed based on the transformation parameters. Hereinafter, an example in which the first coordinate axis direction is the Y-axis direction will be described.

[0097] The conversion unit 232 uses the conversion parameters to convert the coordinates of the point cloud data acquired by the sensor device 10. As a result, the coordinates of the point cloud data are converted from coordinates in the reference coordinate system to coordinates in the operational coordinate system.

[0098] The detection unit 233 has a function of detecting the position of an object included in the point cloud data based on the point cloud data after the coordinate conversion by the conversion unit 232. For example, the detection unit 233 detects the position of the vehicle 40 from the point cloud data obtained by the sensor device 10.

[0099] The detection unit 233 may detect the vehicle 40 from the point cloud data after the coordinate transformation using a known object detection method. For example, the detection unit 233 may detect the vehicle 40 from the point cloud data after the coordinate transformation using a background subtraction technique or a clustering technique.

[0100] The detection unit 233 stores information about objects detected in each frame of the point cloud data after coordinate transformation in the object information storage unit 250.

[0101] The object tracking unit 234 has a function of generating a movement trajectory of an object detected by the detection unit 233 by tracking the position of the object.

[0102] More specifically, the object tracking unit 234 generates the movement trajectory of an object as the movement trajectory of the vehicle 40 by matching the position of an object detected in a frame with the position of an object detected in the previous frame in the point cloud data after coordinate transformation.

[0103] For example, the position of object O detected in frame F_k containing point cloud data after coordinate transformation is assumed to be (X_O, Y_O, Z_O). The position of object O here may be a representative value such as the center of gravity of the point cloud corresponding to the detected object O, or an estimated value calculated from the moving speed of object O, etc. The object tracking unit 234 identifies and tracks the object that is closest to object O detected in frame F_k among the object group detected in the past frame F_(k-1) as the same object between frames F_K and F_(k-1). The distance here is assumed to be a three-dimensional Euclidean distance.

[0104] The object tracking unit 234 uses information on the detected object in each frame stored in the object information storage unit 250 to perform a process of generating a movement trajectory of the vehicle 40 as described above.

[0105] The notification unit 235 has a function of generating information on the position of the detected vehicle 40 based on the point cloud data after coordinate transformation. The generated various information on the vehicle 40 is transmitted from the communication unit 210 to the communication device 30 and is used as traffic support information for each vehicle 40 traveling on the expressway.

[0106] For example, the notification unit 235 may generate lane information indicating which lane of a highway the detected vehicle 40 is traveling in.

[0107] For this purpose, the notification unit 235 may perform a section determination process to determine the section to which the coordinates of the detected position of the vehicle 40 in the X-axis direction (second coordinate axis direction) perpendicular to the Y-axis direction belong in the point cloud data after coordinate transformation.

[0108] 3, before the installation conditions of the sensor device 10 are changed, coordinate conversion should be performed in the operational coordinate system of the sensor device 10 so that the Y-axis direction is parallel to the dividing lines of the expressway. Therefore, when an X-coordinate section is assigned to each lane, the notification unit 235 can generate lane information of the vehicle 40 based on the section to which the coordinate of the position of the vehicle 40 in the X-axis direction perpendicular to the Y-axis direction belongs.

[0109] The lane information generated by the notification unit 235 may be distributed to each vehicle 40 traveling on the expressway via the communication device 30, and may be used for driving control of autonomous vehicles among the vehicles 40.

[0110] Furthermore, in addition to the lane information, the notification unit 235 may generate other examples of information related to the vehicles 40, such as information indicating how many vehicles 40 are in each lane, information on the inter-vehicle time or inter-vehicle distance of each vehicle 40, and information on the predicted time for each vehicle 40 to reach a predetermined position on the expressway. Such information can be used to assist autonomous vehicles in merging from a connecting road to an expressway onto a main road.

[0111] The determination unit 236 has a function of automatically determining, based on the point cloud data after the coordinate transformation, a change in the installation conditions of the sensor device 10 that acquired the point cloud data.

[0112] More specifically, the determination unit 236 determines whether a predetermined condition based on the movement trajectory generated by the object tracking unit 234 is satisfied, thereby determining whether the installation conditions of the sensor device 10 have changed.

[0113] The specified condition may be that the difference in coordinates in the X-axis direction (second coordinate axis direction) perpendicular to the Y-axis direction (first coordinate axis direction) in the operational coordinate system of the sensor device 10 between two different points among the detected positions of the vehicle 40 included in the movement trajectory exceeds a threshold value.

[0114] Fig. 7 is a diagram for explaining the determination of whether or not the installation conditions of the sensor device 10 have changed by the determination unit 236. The movement trajectory TR shown in Fig. 7 is an example of a movement trajectory generated by the object tracking unit 234 based on the point cloud data after coordinate transformation.

[0115] In the example shown in Figure 7, before the sensor installation angle shifted, the movement trajectory of the vehicle 40 generated from the point cloud data after coordinate transformation would have continued to travel on lane L1. However, because the installation angle of the sensor device 10 has shifted from the initial installation angle, the movement trajectory of the vehicle 40 generated from the point cloud data after coordinate transformation has shifted from lane L1 to lane L2.

[0116] Each point included in the movement trajectory TR shown in Fig. 7 indicates the position of the detected vehicle 40. Note that, in Fig. 7, similarly to Figs. 2 to 5, only the X-axis and Y-axis of the operational coordinate system of the sensor device 10 are illustrated, and the Z-axis is not illustrated.

[0117] 7, it is assumed that the Y-axis direction is not parallel to the direction of the lane markings in the operational coordinate system due to a deviation in the installation angle of the sensor device 10. In this case, even if the vehicle 40 is actually traveling on lane L1, the position of the vehicle 40 changes from lane L1 to lane L2 on the movement trajectory TR. In addition, the farther the vehicle 40 moves, the more the X-coordinate value of the position of the vehicle 40 included in the movement trajectory TR deviates from the position on lane L1.

[0118] Taking advantage of this, the determination unit 236 can detect a change in the installation conditions of the sensor device 10 by selecting two different points from among the positions of the vehicle 40 included in the movement trajectory TR and comparing the difference in the X coordinate between the selected two points with a threshold value.

[0119] For example, the judgment unit 236 may select, as two different points, a maximum distance point OP_Max, which is the point at which the coordinate in the Y-axis direction is maximum, and a minimum distance point OP_Min, which is the point at which the coordinate in the Y-axis direction is minimum, among the positions of the vehicle 40 included in the movement trajectory TR.

[0120] The X and Y coordinates of the minimum distance point OP_Min are (x_min, y_min). The X and Y coordinates of the maximum distance point OP_Max are (x_max, y_max). The notification unit 235 calculates an absolute value Dif_x of the difference between x_min and x_max. The notification unit 235 may determine that the installation conditions of the sensor device 10 have changed when Dif_x exceeds a threshold value Th_x.

[0121] By the determination unit 236 making such a determination, the information processing device 20 can automatically detect a change in the installation conditions of the sensor device 10.

[0122] When the determining unit 236 determines that the above-mentioned predetermined condition is satisfied, the extracting unit 237 and the modifying unit 238 perform calibration by modifying the conversion parameters used in the coordinate conversion process of the converting unit 232.

[0123] More specifically, when the determination unit 236 determines that the difference in X-coordinate between two different points included in the movement trajectory TR exceeds a threshold value, the extraction unit 237 has a function of extracting a trajectory point cloud indicating points corresponding to the same vehicle 40 detected by the object tracking unit 234 from among the points included in the point cloud data obtained by the sensor device 10.

[0124] 8 is an explanatory diagram showing an example of a trajectory point cloud of the vehicle 40 extracted by the extraction unit 237. The extraction unit 237 may identify a time period in which the number of detected objects is 1 based on the object detection result in the operation coordinate system of the sensor device 10 by the detection unit 233, acquire a point cloud of the identified time period from a point cloud in the reference coordinate system of the sensor device 10 (i.e., point cloud data acquired from the sensor device 10), extract a point cloud corresponding to the vehicle 40 from the point cloud of the time period (for example, by using a background subtraction technique or a clustering technique), and generate a trajectory point cloud PC1 in the reference coordinate system of the sensor device 10 by superimposing the point cloud corresponding to the vehicle 40.

[0125] In the above, it has been described that the object tracking unit 234 uses each point, such as the center of gravity position of the vehicle 40 or an estimated position estimated from the moving speed of the vehicle 40, as the position of the vehicle 40 when generating a movement trajectory. Meanwhile, unlike when the object tracking unit 234 generates a movement trajectory, the extraction unit 237 extracts all the point clouds corresponding to the vehicle 40 from the point cloud data acquired from the sensor device 10 as a trajectory point cloud.

[0126] However, the point cloud data used by the extraction unit 237 to extract the trajectory point cloud may include point cloud data in a frame different from the frame of the point cloud data used to generate the movement trajectory for determining whether or not the installation conditions of the sensor device 10 have changed, or may include point cloud data in the same frame. The point cloud data used to generate the movement trajectory for determining whether or not the installation conditions of the sensor device 10 have changed is an example of first point cloud data. The point cloud data used by the extraction unit 237 to extract the trajectory point cloud is an example of second point cloud data.

[0127] Also, as shown in FIG. 8, the extraction unit 237 may generate a trajectory point group PC1 by combining a trajectory point group PC_S1 extracted from the point cloud data acquired by the sensor device 10A shown in FIG. 1 and a trajectory point group PC_S2 extracted from the point cloud data acquired by the sensor device 10B.

[0128] The correction unit 238 has a function of estimating the moving direction of the vehicle 40 corresponding to the trajectory point group generated by the extraction unit 237 from the trajectory point group. The correction unit 238 calculates a matrix indicating a transformation such that the estimated moving direction of the vehicle 40 coincides with a predetermined direction in the reference coordinate system of the sensor device 10. The predetermined direction may be a first coordinate axis direction (e.g., Y-axis direction) in the reference coordinate system of the sensor device 10.

[0129] Furthermore, the modifying unit 238 modifies the transformation parameters based on the calculated matrix.

[0130] 8, each point of the trajectory point cloud generated by the extraction unit 237 is distributed along the detected moving direction of the vehicle 40. Therefore, based on the distribution of each point included in the trajectory point cloud, the correction unit 238 can estimate the moving direction of the object of the vehicle 40 from the direction in which the variance of the distribution is maximum.

[0131] Therefore, the correction unit 238 according to the present embodiment estimates a vector of the moving direction of the vehicle 40 on the trajectory point cloud by applying a principal component analysis technique to the distribution of the trajectory point cloud. In addition, the correction unit 238 calculates a matrix for rotating the trajectory point cloud so that the estimated moving direction of the vehicle 40 is aligned with the Y coordinate axis direction.

[0132] More specifically, the correction unit 238 may estimate a vector of the moving direction of the vehicle 40 based on the trajectory point cloud by performing the following calculation.

[0133] Fig. 9 is a diagram for explaining estimation of a moving direction based on a trajectory point group by the correction unit 238 according to the present embodiment. The coordinate system shown in Fig. 9 is a reference coordinate system of the sensor device 10, and the X-axis, Y-axis, and Z-axis shown in Fig. 9 are coordinate axes in the reference coordinate system of the sensor device 10.

[0134] PC_traj shown in FIG. 9 is an example of a trajectory point group generated by the extraction unit 237. The total number of points included in PC_traj is set to i. A vector representing the coordinates of each point included in PC_traj is expressed as vector Pn=(x n ,y n ,z n ) T (1≦n≦i).

[0135] Using the data components of each point included in PC_traj, a matrix A is defined as shown in the following formula (1).

[0136]

number

[0137] Here, matrix A and transpose matrix A T By multiplying with , the covariance matrix Σ of the three variables, the X component, Y component, and Z component of the coordinates of each point included in PC_traj, can be expressed as the following equation.

[0138]

number

[0139] Suppose there is a three-dimensional coordinate system V that is different from the reference coordinate system of the sensor device 10. Let u be the orthogonal basis vector of V. n (n=1~3).

[0140] Each point Pi included in PC_traj is expressed as u n When projected in the direction of , each point of PC_traj is n It can be converted into one-dimensional data on a vector in the direction of u n The projected values ​​in the direction of are Pi and u n Since it can be calculated by the inner product of PC_traj, n The variance var of the values ​​projected onto the direction can be expressed by the following formula.

[0141]

number

[0142] Furthermore, by expanding and rearranging the above equation (3), the above variance var can be expressed as the following equation (4) using the above-described equation (2).

[0143]

number

[0144] Here, we consider the u that maximizes the variance var expressed by Eq. (4). n Think about it. nSince is an orthonormal basis vector, there is a constraint that the norm is 1. Using the Lagrange multiplier method, we can find a solution of u that maximizes the variance var. n When we search for u n are eigenvectors corresponding to the eigenvalues ​​of the covariance matrix Σ.

[0145] The modification unit 238 calculates eigenvalues ​​and eigenvectors of the covariance matrix Σ of PC_traj, which is expressed by Equation (2). Since the covariance matrix Σ is a three-variate covariance matrix with the X, Y, and Z coordinates of each point of PC_traj as variables, three eigenvalues ​​and three eigenvectors of the covariance matrix Σ are calculated.

[0146] The eigenvectors are u1, u2, and u3 in descending order of eigenvalue. The direction in which the distribution of the points included in PC_traj is maximum is the moving direction of the vehicle 40 whose trajectory is represented by PC_traj. Therefore, the correction unit 238 can estimate the direction of the eigenvector u1 corresponding to the maximum eigenvalue as the moving direction of the vehicle 40. Furthermore, the direction in which the distribution of the points included in PC_traj is second largest may be the width direction of the lane. Therefore, the correction unit 238 can estimate the direction of the eigenvector u2 corresponding to the second largest eigenvalue as the width direction of the lane.

[0147] Therefore, it is necessary to calculate a rotation matrix v to transform u1 into a unit vector (0,1,0) on the Y coordinate axis of the reference coordinate system, u2 into a unit vector (1,0,0) on the X coordinate axis of the reference coordinate system, and u3 into a unit vector (0,0,1) on the Z coordinate axis of the reference coordinate system, and to update the transformation parameters using the calculated rotation matrix v.

[0148] To find the rotation matrix v, let u be the matrix used to transform the unit vector (0,1,0) on the Y coordinate axis of the reference coordinate system to u1, the unit vector (1,0,0) on the X coordinate axis of the reference coordinate system to u2, and the unit vector (0,0,1) on the Z coordinate axis of the reference coordinate system to u3, then u = (u2, u1, u3). Therefore, the desired rotation matrix v can be expressed as the inverse matrix of matrix u as follows:

[0149]

number

[0150] Therefore, the modification unit 238 may calculate the rotation matrix v expressed by the above formula (5) based on the matrix u, and modify the transformation parameters using the rotation matrix v. Note that modifying the transformation parameters using the rotation matrix v may mean updating the transformation parameters stored in the transformation parameter storage unit 240 to the rotation matrix v. As a result, the transformation parameters used in the coordinate transformation by the transformation unit 232 may be changed to the rotation matrix v.

[0151] The correction unit 238 performs the calculations using the above-mentioned formulas (1) to (5), so that the information processing device 20 can perform calibration based on the point cloud data obtained by the sensor device 10.

[0152] Further, the correction unit 238 does not perform principal component analysis on a movement trajectory that is information in which each point corresponding to the vehicle 40 included in the point cloud data obtained by the sensor device 10 is aggregated at the center of gravity position or the like, but performs principal component analysis on a trajectory point cloud generated based on all of the points corresponding to the vehicle 40. Then, the correction unit 238 corrects the conversion parameters based on the result of the principal component analysis.

[0153] This allows the correction unit 238 to more accurately calculate a matrix such that the moving direction of the vehicle 40 detected on the point cloud data is parallel to the first coordinate axis direction of the sensor coordinate system. Therefore, the information processing device 20 can perform calibration with higher accuracy.

[0154] The screen generation unit 239 has a function of generating an output screen for various information to be displayed on the display device 50. For example, when the determination unit 236 determines that a predetermined condition is satisfied, the screen generation unit 239 may generate a notification screen indicating that a change in the installation condition of the sensor device 10 has been detected. The screen generation unit 239 may cause the display device 50 to display the screen.

[0155] Furthermore, the screen generation unit 239 may generate an output screen including a display of the trajectory point group extracted by the extraction unit 237 .

[0156] Fig. 10 is a diagram showing an example of an output screen generated by the screen generating unit 239. As in the screen example G1 shown in Fig. 10, the screen generating unit 239 may generate an output screen that displays, in comparison with each other, a pre-correction trajectory point cloud BPC1 which is a trajectory point cloud after the trajectory point cloud PC1 (Fig. 8) is converted based on the conversion parameters before the correction by the correction unit 238, and a post-correction trajectory point cloud RPC1 which is a trajectory point cloud after the trajectory point cloud PC1 (Fig. 8) is converted based on the conversion parameters after the correction by the correction unit 238.

[0157] 10, the screen example G1 includes a pre-correction trajectory point cloud BPC1 and a post-correction trajectory point cloud RPC1. The operator of the information processing device 20 can check, by using the screen example G1, whether the trajectory point cloud after conversion using the conversion parameters after the correction by the correction unit 238 is an appropriate coordinate conversion result.

[0158] 10, the screen example G1 includes a message M1 for confirming whether the operator of the information processing device 20 should confirm the modification of the conversion parameters. Furthermore, the screen example G1 includes a confirmation button B1 and a cancel button B2.

[0159] An operator of the information processing device 20 can confirm the transformation parameters after correction by the correction unit 238 by pressing a confirmation button B1 on an input device (e.g., a touch panel, etc.) not shown. That is, when a pressing operation of the confirmation button B1 is detected, the correction unit 238 may correct the transformation parameters stored in the transformation parameter storage unit 240 by using a matrix used for the coordinate conversion from the trajectory point group PC1 (FIG. 8) to the corrected trajectory point group RPC1.

[0160] This allows the operator of the information processing device 20 to visually determine whether the point cloud data has been appropriately corrected by the correction of the transformation parameters by the correction unit 238. Furthermore, the operator of the information processing device 20 can have the correction unit 238 correct the transformation parameters stored in the transformation parameter storage unit 240 after checking the correction contents on the screen.

[0161] Moreover, when a pressing operation of the cancel button B2 is detected, the correction unit 238 may cancel the process of correcting the transformation parameters by the matrix used in the coordinate transformation from the trajectory point cloud PC1 (FIG. 8) to the corrected trajectory point cloud RPC1. Furthermore, the correction unit 238 may perform the matrix calculation process again based on the point cloud data acquired by the acquisition unit 231 from the sensor device 10.

[0162] Thereby, when the operator of the information processing device 20 checks the screen example G1 and determines that the correction content is not appropriate, the operator can cause the correction unit 238 to recalculate the conversion parameters and redo the correction.

[0163] FIG. 11 is a diagram showing another example of an output screen generated by the screen generating unit 239. As in the screen example G2 shown in FIG. 11, the screen generating unit 239 may generate an output screen that displays a pre-correction trajectory point group BPC1, which is a trajectory point group after the trajectory point group PC1 (FIG. 8) is converted based on the conversion parameters before the correction, and a post-correction trajectory point group RPC1, which is a trajectory point group after the trajectory point group PC1 (FIG. 8) is converted based on the conversion parameters after the correction, in a superimposed manner. Note that the display mode of the pre-correction trajectory point group BPC1 and the display mode of the post-correction trajectory point group RPC1 may be different. This allows the pre-correction trajectory point group BPC1 and the post-correction trajectory point group RPC1 to be visually distinguished from each other. For example, the difference in display mode may be a difference in color.

[0164] The message M1, the confirmation button B1, and the cancel button B2 shown in Fig. 11 are as described above with reference to Fig. 10. In the example screen G2, the pre-correction trajectory point cloud BPC1 and the post-correction trajectory point cloud RPC1 are displayed superimposed on each other. This makes it easier to visually understand the difference in the moving direction of the vehicle 40 indicated by the trajectory point clouds before and after the correction of the conversion parameters. Therefore, it becomes easier for the operator of the information processing device 20 to determine whether the correction of the conversion parameters by the matrix calculated by the correction unit 238 is appropriate.

[0165] (Conversion parameter storage unit 240) The transformation parameter storage unit 240 stores information on the transformation parameters used in the coordinate transformation process of the point cloud data by the transformation unit 232. In addition, the transformation parameters stored in the transformation parameter storage unit 240 are modified by the modification unit 238.

[0166] (Object information storage unit 250) The object information storage unit 250 stores information on objects detected in each frame of the point cloud data by the detection unit 233. For example, the object information storage unit 250 may store the coordinates of the position of the object detected in each frame under the control of the detection unit 233.

[0167] The object tracking unit 234 can generate a movement trajectory of the detected vehicle 40 based on the information stored in the object information storage unit 250 .

[0168] (Trajectory point cloud data storage unit 260) The trajectory point cloud data storage unit 260 stores the data of the trajectory point cloud extracted by the extraction unit 237. The screen generation unit 239 may perform processing to generate an output screen using the data of the trajectory point cloud stored in the trajectory point cloud data storage unit 260.

[0169] An example of the functional configuration of the information processing device 20 according to the present embodiment has been described above in detail with reference to Fig. 6. Note that the above configuration described with reference to Fig. 6 is merely an example, and the configuration of the information processing device 20 according to the present embodiment is not limited to this example.

[0170] The information processing device 20 according to the present embodiment may further include, for example, an input unit that accepts information input by a user. The information processing device 20 may be configured integrally with the display device 50.

[0171] <3. Example of operation> Next, an operation example of the information processing system according to one embodiment of the present invention will be described with reference to Fig. 12. Fig. 12 is a flowchart illustrating an operation example of the information processing system according to this embodiment.

[0172] First, the acquisition unit 231 of the information processing device 20 starts acquiring point cloud data from the sensor device 10 via the communication unit 210. The acquisition unit 231 may continuously acquire the point cloud data in real time (S101).

[0173] Next, the conversion unit 232 performs a process of converting the coordinates of the point cloud data acquired by the acquisition unit 231 based on the conversion parameters (S103).

[0174] The detection unit 233 performs a process of detecting an object from the point cloud data after the coordinate transformation (S105).

[0175] The object tracking unit 234 generates a movement trajectory of an object by associating an object detected from the current frame of the point cloud data with an object detected from the frame preceding the current frame (S107).

[0176] The determination unit 236 determines whether or not a predetermined condition based on the movement trajectory generated in S107 is satisfied, thereby detecting the presence or absence of a change (deviation) in the installation conditions of the sensor device 10 (S109).

[0177] In this operation example, the determination unit 236 performs the determination based on a predetermined condition that the absolute value of the difference in the X coordinate values ​​between the point with the smallest Y coordinate and the point with the largest Y coordinate among the points included in the movement trajectory exceeds a threshold value.

[0178] When the determination unit 236 determines that the predetermined condition based on the movement trajectory is not satisfied (S109 / NO), the information processing system repeats the processes of S101 to S107.

[0179] When the judgment unit 236 judges that the predetermined condition based on the movement trajectory is satisfied (S109 / YES), the screen generation unit 239 generates an output screen notifying the occurrence of a change (deviation) in the installation conditions of the sensor device 10, and displays the output screen on the display device 50 (S111).

[0180] Next, the extraction unit 237 extracts a trajectory point cloud corresponding to the detected object from the point cloud data acquired by the sensor device 10 (S113).

[0181] The correction unit 238 estimates the movement direction of the detected object in the trajectory point cloud based on the distribution of the points of the extracted trajectory point cloud. The correction unit 238 calculates a rotation matrix for the trajectory point cloud such that the estimated movement direction is parallel to the Y axis in the reference coordinate system of the sensor device 10 (S115).

[0182] The screen generation unit 239 generates an output screen including a display of the trajectory point cloud after being transformed by the transformation parameters before being modified using the matrix calculated by the modification unit 238, and the trajectory point cloud after being transformed by the transformation parameters after being modified using the matrix calculated by the modification unit 238. The screen generation unit 239 causes the output screen to be displayed on the display device 50 (S117).

[0183] When the operator of the information processing device 20 performs an operation to cancel the correction content on the output screen displayed on the display device 50 (S119 / NO), the information processing device 20 repeats the processes of S113 to S117. As a result, the correction unit 238 calculates the matrix again.

[0184] When the operator of the information processing device 20 performs an operation to confirm the correction content on the output screen displayed on the display device 50 (S119 / YES), the correction unit 238 corrects the conversion parameters stored in the conversion parameter memory unit 240 using the matrix calculated in S115 (S121).

[0185] The information processing device 20 repeats the processes of S101 to S121 until an operation to end the series of processes is performed (S123 / NO). When an operation to end the series of processes is performed, the information processing device 20 ends the processes (S123 / YES).

[0186] An example of the operation of the information processing system according to this embodiment has been described above with reference to FIG.

[0187] <4. Hardware configuration example> An embodiment of the present invention has been described above. The above-mentioned process of determining whether or not a predetermined condition based on a movement trajectory is satisfied by the information processing device 20, recalculating and correcting the conversion parameters when the predetermined condition is satisfied, and generating an output screen of the trajectory point group before and after correction are realized by cooperation between software and the hardware of the information processing device 20. An example of the hardware configuration of the information processing device 20 according to an embodiment of the present invention will be described below.

[0188] Note that the hardware configuration example of the information processing device 20 described below is merely one example of the hardware configuration of the information processing device 20. Therefore, in the hardware configuration of the information processing device 20 described below, unnecessary components may be deleted, or new components may be added.

[0189] 13 is a diagram showing an example of a hardware configuration of an information processing device 20 according to an embodiment of the present invention. The information processing device 20 can include a processor 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an internal bus 1004, an input / output interface 1010, a display device 1011, an input device 1012, an audio output unit 1013, a storage device 1014, a drive 1015, a network interface 1016, and an external interface 1017.

[0190] The processor 1001 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 20 in accordance with various programs. The processor 1001 cooperates with a ROM 1002, a RAM 1003, and software described below to realize, for example, the functions of the control unit 230.

[0191] The ROM 1002 stores programs and computation parameters used by the processor 1001. The RAM 1003 temporarily stores programs used during execution by the processor 1001, as well as parameters that change as appropriate during the execution of the programs.

[0192] The processor 1001, ROM 1002, and RAM 1003 are connected to each other by an internal bus 1004, and are further connected via an input / output interface 1010 to a display device 1011, an input device 1012, an audio output unit 1013, a storage device 1014, a drive 1015, a network interface 1016, and an external interface 1017, which will be described later.

[0193] The display device 1011 is, for example, a display device such as a CRT display device, a liquid crystal display (LCD), or an OLED device, and converts video data into video and outputs the video. The input device 1012 may be composed of a mouse, a keyboard, a touch panel, a button, a microphone, a sensor, a switch, a control circuit, etc. The audio output unit 1013 is an audio output device such as a speaker and headphones, and converts audio data and the like into audio and outputs the audio.

[0194] The storage device 1014 is a data storage device configured as an example of the conversion parameter storage unit 240, the object information storage unit 250, and the trajectory point cloud data storage unit 260 according to this embodiment. The storage device 1014 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage device 1014 is configured, for example, by a hard disk drive (HDD) or a solid state drive (SSD), or a memory having an equivalent function. The storage device 1014 drives the storage and stores programs executed by the processor 1001 or various data.

[0195] The drive 1015 is a reader / writer for a storage medium, and is built into or externally attached to the information processing device 20. The drive 1015 reads out information stored in a removable storage medium, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 1003. The drive 1015 is also capable of writing information to the removable storage medium.

[0196] The network interface 1016 is a communication interface configured with a device for connecting to a communication network such as the Internet, etc. The network interface 1016 may be a communication device compatible with a wireless LAN (Local Area Network) and / or a wired LAN.

[0197] The external interface 1017 is a connection interface composed of connection ports for connecting external devices, such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, an RS-232C port, or an optical audio terminal.

[0198] <5. Summary> Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

[0199] For example, a vehicle has been described as an example of an object detected by the detection unit 233. However, the object detected by the detection unit 233 may be an object other than a vehicle. For example, the object detected by the detection unit 233 may be a moving body other than a vehicle. The moving body may be a manned moving body that moves by being operated by a passenger, or an unmanned moving body that can move autonomously. The vehicle is an example of a moving body that moves on land.

[0200] In the above embodiment, an example of a suitable application of the embodiment of the present invention has been described in which a vehicle traveling on a highway is detected by a sensor installed on the roadside. However, the application of the embodiment of the present invention is not limited to such an example. For example, another suitable application of the embodiment of the present invention is an example of a sensor installed on a train platform detecting a person on the platform. A person is an example of a moving object that moves on land.

[0201] In recent years, sensors installed on station platforms where passengers board and disembark trains have been used to detect the positions of passengers coming and going on the platform. This makes it possible to detect the occurrence of abnormalities, such as passengers falling from the platform onto the tracks or falling into the gap between the train and the platform. If an abnormality is detected, a notification can be sent before the train enters the platform, making it possible to take safety measures such as stopping the train.

[0202] The embodiment of the present invention can also be applied to automatic detection of changes in installation conditions of sensors installed on such station platforms, and calibration in response to the changes in the installation conditions.

[0203] In general, the platform of a station where passengers board and disembark trains is often formed in a straight line. Trains enter stations along a straight platform. Therefore, based on the train's movement trajectory detected in point cloud data obtained by a sensor installed in a station, it is possible to automatically detect changes in the installation conditions of the sensor. A train is an example of a mobile object that travels on land.

[0204] Depending on the structure or location of the platform, passengers walking along the edge of the platform may move in a generally straight line along the edge of the platform. In such cases, it is possible to automatically detect changes in the sensor installation conditions by extracting the passenger movement trajectory from the 3D point cloud obtained by the sensor and using the movement trajectory to perform threshold judgment.

[0205] Another example of a suitable application of the embodiment of the present invention is a ship sailing on a sea route, or an aircraft running on a runway. In this case, too, in a scene where the ship or aircraft moves in a generally straight line along a predefined course, it is possible to automatically determine changes in the installation conditions of the sensor and perform calibration using the movement trajectory of the ship or aircraft. A ship is an example of a moving body that moves on a sea route, and an aircraft is an example of a moving body that moves on an air route.

[0206] In the above embodiment, an example has been described in which the initial values ​​of the preset conversion parameters are stored in advance in the conversion parameter storage unit 240. However, the information processing device 20 according to the embodiment of the present invention may calculate a matrix using a trajectory point cloud of an object detected in point cloud data obtained by a sensor at the time of initial installation of the sensor. The information processing device 20 may set the calculated matrix as the initial value of the conversion parameter.

[0207] In the above embodiment, when the determination unit 236 determines that a predetermined condition based on the movement trajectory of the detected object is satisfied, the modification unit 238 performs the modification process of the transformation parameters. However, the embodiment of the present invention is not limited to this example.

[0208] For example, the information processing device 20 may accumulate the determination result by the determination unit 236 as to whether or not a predetermined condition is satisfied. Furthermore, the information processing device 20 may perform a process of estimating the time interval during which the installation deviation of the sensor device 10 occurs and the amount of deviation based on the accumulated determination result. Furthermore, when the estimated time interval has elapsed, the correction unit 238 may recalculate and correct the conversion parameters without performing the determination process by the determination unit 236.

[0209] As a result, for example, in a case where deviations in the sensor device 10 occur frequently at short time intervals, the information processing device 20 can quickly correct the conversion parameters without having the determination unit 236 perform a determination process.

[0210] In the above embodiment, an example has been described in which the correction unit 238 estimates the moving direction of the vehicle based on the trajectory point group in the reference coordinate system of the sensor device 10. At this time, the correction unit 238 calculates a matrix that matches the estimated moving direction with a predetermined direction in the reference coordinate system, and updates the transformation parameters with the inverse matrix of the calculated matrix.

[0211] However, the modification unit 238 may estimate the moving direction of the vehicle based on the trajectory point cloud in the operational coordinate system after the reference coordinate system is transformed using the initially set transformation parameters. In this case, the modification unit 238 may calculate a matrix that matches the estimated moving direction with a predetermined direction in the operational coordinate system, calculate a matrix that expresses a composite transformation that performs a transformation using the calculated matrix after the transformation using the initially set transformation parameters, and update the transformation parameters with the inverse matrix of the matrix that expresses the composite transformation.

[0212] In the above embodiment, when the determination unit 236 determines that a predetermined condition is satisfied, the screen generation unit 239 generates a notification screen indicating that a change in the installation conditions of the sensor device 10 has been detected. In addition, an example has been described in which the screen generation unit 239 displays the screen on the display device 50. However, the embodiment of the present invention is not limited to the example. For example, the control unit 230 of the information processing device 20 may cause a sound output device such as a speaker to output a sound (including voice) notifying that a change in the installation conditions of the sensor device 10 has occurred. The sound output device may be configured integrally with the information processing device 20. Alternatively, the sound output device may be a device separate from the information processing device 20.

[0213] Alternatively, the control unit 230 may control a light emitting device having a light source such as an LED (Light Emitting Diode) to emit light when a change occurs in the installation conditions of the sensor device 10. The light emitting device may be configured as an integral part of the information processing device 20, or may be configured as a separate unit. This allows the operator of the information processing device 20 to be notified by sound or light that a change has occurred in the installation conditions of the sensor device 10.

[0214] Furthermore, the steps in the processing of the operation of the sensor device 10 and the information processing device 20 according to the present embodiment do not necessarily need to be processed in chronological order according to the order described in the explanatory diagram. For example, the steps in the processing of the operation of the sensor device 10 and the information processing device 20 may be processed in an order different from the order described in the explanatory diagram, or may be processed in parallel.

[0215] Also, one or more computer programs can be created to cause hardware such as a processor, ROM, and RAM built into the above-mentioned sensor device 10 and information processing device 20 to perform the functions of the information processing system according to this embodiment. Also provided is a computer-readable storage medium storing the one or more computer programs.

[0216] In addition, the effects described in this specification are merely explanatory or exemplary and are not limiting. In other words, the technology according to the present invention may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. [Explanation of symbols]

[0217] 10 Sensor device 20 Information processing device 210 Communications Department 230 Control Unit 231 Acquisition Department 232 Conversion unit 233 Detector 234 Object Tracking Unit 235 Notification Department 236 Judgment section 237 Extraction part 238 Correction section 239 Screen generation section 240 Conversion parameter storage unit 250 Object information storage unit 260 Trajectory point cloud data storage unit 50 Display device

Claims

1. an acquisition unit that acquires first point cloud data indicating a shape of a first object obtained by a sensor; A conversion unit that converts coordinates of the acquired first point cloud data based on a conversion parameter; a detection unit that detects a position of a first object based on the first point cloud data after coordinate transformation; an object tracking unit that generates a movement trajectory of the first object by tracking a position of the first object; a determination unit that determines whether or not a predetermined condition based on the movement trajectory is satisfied; estimating a moving direction of a second object from second point cloud data acquired by the sensor based on the determination that the movement trajectory satisfies the predetermined condition; Calculating a matrix that aligns the movement direction with a predetermined direction; a modification unit that modifies the transformation parameters based on the matrix; An information processing device comprising:

2. The predetermined direction is a first coordinate axis direction in a sensor coordinate system. The information processing device according to claim 1 .

3. the predetermined condition is that a difference between coordinates of two different points of the positions of the first object included in the movement trajectory in a second coordinate axis direction orthogonal to the first coordinate axis direction in the sensor coordinate system exceeds a threshold value; The information processing device according to claim 2 .

4. the two points include a point at which the coordinate in the first coordinate axis direction is maximum and a point at which the coordinate in the first coordinate axis direction is minimum among the positions of the first object; The information processing device according to claim 3 .

5. the transformation parameters include a rotation matrix that rotates coordinates of the first point cloud data; The information processing device according to claim 1 .

6. a notification unit that performs a section determination process to determine a section to which a coordinate in a second coordinate axis direction orthogonal to the first coordinate axis direction belongs among the position of the first object; The information processing device according to claim 3 .

7. The notification unit notifies another device of information indicating the section to which the coordinate of the first object in the second coordinate axis direction belongs based on a result of the section determination process. The information processing device according to claim 6.

8. the correction unit estimates, based on a distribution of points corresponding to the detected second object among the points included in the second point cloud data, a direction in which the variance of the distribution is maximum as a moving direction of the second object; The information processing device according to claim 2 .

9. the correction unit calculates a covariance matrix based on a distribution of coordinates of each point included in the second point cloud data, an eigenvalue of the covariance matrix, and an eigenvector; calculating the matrix that causes the moving direction of the second object to coincide with the predetermined direction based on the eigenvalues ​​and the eigenvectors; The information processing device according to claim 8.

10. a screen generating unit that generates an output screen including a display of a trajectory point cloud based on points corresponding to the second object among the points included in the second point cloud data; The information processing device according to claim 1 , further comprising:

11. the screen generation unit generates an output screen for comparatively displaying the trajectory point cloud after the points corresponding to the second object have been coordinate-transformed based on the transformation parameters before modification by the modification unit and the trajectory point cloud after the points corresponding to the second object have been coordinate-transformed based on the transformation parameters after modification. The information processing device according to claim 10.

12. the screen generation unit generates an output screen for superimposing and displaying the trajectory point cloud after the points corresponding to the second object have been coordinate-transformed based on the transformation parameters before the modification and the trajectory point cloud after the points corresponding to the second object have been coordinate-transformed based on the transformation parameters after the modification. The information processing device according to claim 10.

13. the object tracking unit generates the movement trajectory of the first object by associating a position of the first object detected in a certain frame with a position of the first object detected in a frame preceding the certain frame in the point cloud data obtained by the sensor; The information processing device according to claim 1 .

14. The second point cloud data includes point cloud data in a frame different from a frame included in the first point cloud data. The information processing device according to claim 1 .

15. The second point cloud data includes point cloud data in the same frames as frames included in the first point cloud data. The information processing device according to claim 1 .

16. the acquisition unit acquires the first point cloud data and the second point cloud data by synthesizing point cloud data acquired by the plurality of sensors; The information processing device according to claim 1 .

17. The first object includes a moving object moving by land, air, or sea; The information processing device according to any one of claims 1 to 16.

18. The moving body includes at least one or both of a manned moving body that moves by being operated by a passenger and an unmanned moving body that can move autonomously. The information processing device according to claim 17.

19. The processor: acquiring first point cloud data obtained by a sensor, the first point cloud data indicating a shape of a first object; performing a process of converting coordinates of the acquired first point cloud data based on a conversion parameter; Detecting a position of a first object based on the first point cloud data after coordinate transformation; generating a trajectory of the first object by tracking a position of the first object; determining whether a predetermined condition based on the movement trajectory is satisfied; estimating a movement direction of a second object from second point cloud data acquired by the sensor based on the determination that the movement trajectory satisfies the predetermined condition; calculating a matrix that causes the movement direction to coincide with a predetermined direction; modifying the transformation parameters based on the matrix; 2. An information processing method implemented by a computer, comprising:

20. Computer, an acquisition unit that acquires first point cloud data indicating a shape of a first object obtained by a sensor; A conversion unit that converts coordinates of the acquired first point cloud data based on a conversion parameter; a detection unit that detects a position of a first object based on the first point cloud data after coordinate transformation; an object tracking unit that generates a movement trajectory of the first object by tracking a position of the first object; a determination unit that determines whether or not a predetermined condition based on the movement trajectory is satisfied; estimating a moving direction of a second object from second point cloud data acquired by the sensor based on the determination that the movement trajectory satisfies the predetermined condition; Calculating a matrix that aligns the movement direction with a predetermined direction; a modification unit that modifies the transformation parameters based on the matrix; A program for causing the information processing device to function as an information processing device.

Images