Mobile monitoring device

Abstract

To provide a movable body monitoring device which can improve the versatility of calibration of an infrastructure sensor.SOLUTION: A movable body monitoring device includes: a measurement part 21; a static information extraction part 202; a calibration reference storage part 301; and an attitude estimation part 305. The measurement part 21 measures the surrounding environment and outputs measurement information of a measurement point. The static information extraction part 202 extracts only the measurement point with no positional change for a prescribed time as static information based on the measurement information. The calibration reference storage part 301 stores the calibration reference (survey information) measured in a reference coordinate system. The attitude estimation part 305 estimates the attitude of the measurement part 21 in the reference coordinate system based on the calibration reference and static information, and outputs the attitude information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a moving object monitoring device. [Background Art]

[0002] In recent years, monitoring devices that detect vehicles and pedestrians using infrastructure sensors installed alongside roads, such as roadside cameras and roadside LiDAR (light detection and ranging), have been developed.

[0003] These monitoring devices can improve the safety of autonomous driving by, for example, detecting the positions of vehicles and pedestrians with infrastructure sensors, and instructing the autonomous driving vehicle to decelerate when a high-risk situation such as the presence of a pedestrian in the vicinity of the autonomous driving vehicle is detected.

[0004] Infrastructure sensors can also detect the position of the autonomous driving vehicle itself and distribute it via wireless communication. For public-road autonomous driving, the vehicle position is detected by comparing the measurement information of on-board sensors with on-board map data, so autonomous driving cannot be performed in areas not covered by on-board map data such as private land. However, by detecting the vehicle position using a combination of infrastructure sensors, autonomous driving becomes possible even in areas not covered by on-board map data. Furthermore, by detecting the vehicle position using only infrastructure sensors, autonomous driving becomes possible even if the vehicle is not equipped with the on-board sensors or on-board map required for detecting the vehicle position.

[0005] The device described in Patent Document 1 is one example thereof; it detects the position of a moving object using an infrastructure sensor fixed to a building or the like, transmits the position to the moving object via wireless communication, and uses this for guiding the moving object.

[0006] Since infrastructure sensors detect the relative position of the object being detected, accurately determining the installation orientation of the infrastructure sensor itself is necessary to detect the absolute position of the object. In particular, any error in the rotational orientation of the infrastructure sensor will result in an error in the detection position proportional to the distance between the infrastructure sensor and the object being detected. However, precisely installing infrastructure sensors according to a predetermined orientation requires a great deal of effort. Furthermore, the installation orientation of infrastructure sensors may change over time. Therefore, a means of calibrating the installation orientation of infrastructure sensors is necessary.

[0007] The apparatus described in Patent Document 2 is one example, which calibrates the installation orientation of the infrastructure sensor by comparing the position of the road sign detected by the infrastructure sensor's road sign recognition with the position of the road sign stored in the infrastructure sensor's memory. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] International Publication No. WO2020230325 [Patent Document 2] Special Publication No. 2020-535572 [Overview of the project] [Problems that the invention aims to solve]

[0009] However, when calibration is based on object recognition such as road sign recognition, there is a problem in that the infrastructure sensor must have objects that it can recognize within its field of view, which reduces its versatility. For example, since road signs differ from country to country and region to region, road sign recognition must also be prepared for each country and region. In addition, depending on the installation location of the infrastructure sensor, road signs may not be present within the sensor's field of view, making calibration impossible.

[0010] The object of the present invention is to provide a mobile monitoring device that can improve the versatility of calibration of infrastructure sensors. [Means for solving the problem]

[0011] To achieve the above objective, the mobile monitoring device of the present invention includes a measurement unit that measures the surrounding environment and outputs measurement information of measurement points, a static information extraction unit that extracts only the measurement points that have not changed position for a predetermined time as static information based on the measurement information, a calibration reference storage unit that stores calibration references measured in a reference coordinate system, and the calibration references which are 6-dimensional survey information including 3D position information and 3D twist information measured at a first time point. 、 At the second time Based on the measured information Extracted static information and A posture estimation unit that estimates the posture of the measurement unit in the reference coordinate system based on the above and outputs posture information for the second time, and a static information storage unit that stores the static information and outputs stored static information indicating the stored static information, Measured at a third time, different from the second time mentioned above. A dynamic information extraction unit extracts only the measurement points that are not included in the stored static information as dynamic information based on the measurement information, and at the second time point Based on the measured information The extracted static information and , the above 3rd time Measured The system includes a posture change estimation unit that calculates posture change information for the third time point by comparing the measurement information, and a coordinate transformation unit that calculates transformed dynamic information by transforming the dynamic information to the reference coordinate system based on the posture information for the second time point and the posture change information for the third time point. The system receives static information from the static information extraction unit and notifies the user that the static information has been received. [Effects of the Invention]

[0012] According to the present invention, the versatility of infrastructure sensor calibration can be improved. Other problems, configurations, and effects will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram showing the schematic configuration of the mobile monitoring device in the first embodiment. [Figure 2]FIG. 1 is a diagram showing a schematic configuration of a vehicle in a first embodiment. [Figure 3] FIG. 1 is a diagram showing a schematic configuration of an infrastructure sensor in a first embodiment. [Figure 4] FIG. 1 is a diagram showing a schematic configuration of a calibration server in a first embodiment. [Figure 5] FIG. 1 is a diagram showing a schematic configuration of a control server in a first embodiment. [Figure 6] FIG. 1 is a block diagram of a moving object monitoring device in a first embodiment. [Figure 7] FIG. 1 is a flowchart showing an operation of a vehicle calculation unit in a first embodiment. [Figure 8] FIG. 1 is a flowchart showing an operation of an infrastructure sensor calculation unit in a first embodiment. [Figure 9] FIG. 1 is a flowchart showing an operation of a calibration server calculation unit in a first embodiment. [Figure 10] FIG. 1 is a flowchart showing an operation of a control server calculation unit in a first embodiment. [Figure 11] FIG. 1 is a diagram for explaining an operation of a static information extraction unit in a first embodiment. [Figure 12] FIG. 1 is a diagram for explaining an operation of a dynamic information extraction unit in a first embodiment. [Figure 13] FIG. 1 is a block diagram of an infrastructure sensor in a second embodiment. [Figure 14] FIG. 1 is a block diagram of a calibration server in a third embodiment. [Figure 15] FIG. 1 is a block diagram of a calibration server in a fourth embodiment. MODE FOR CARRYING OUT THE INVENTION

[0014] Hereinafter, modes for carrying out the present invention will be described with reference to the drawings.

[0015] [First Embodiment] In the embodiments described below, the present invention is explained using the example of its application to a control system for automobiles, but the present invention can also be applied to control systems for transport vehicles that travel within warehouses, construction vehicles that travel on construction sites, and the like.

[0016] Figure 1 shows a schematic configuration of a mobile monitoring device in the first embodiment. The dashed arrows in Figure 1 indicate the signal flow.

[0017] The mobile monitoring system comprises a vehicle 1 that operates autonomously in a limited driving environment, an infrastructure sensor 2 fixed to the driving environment such as buildings and pillars and detecting pedestrians and other vehicles present in the driving environment, a calibration server 3 installed outside the infrastructure sensor 2, and a control server 4 installed outside the vehicle 1.

[0018] Figure 2 shows a schematic configuration of vehicle 1 in the first embodiment. The dashed arrows in Figure 2 indicate the signal flow.

[0019] Vehicle 1 includes driven wheels 11 located on the left and right rear sides, drive wheels 12 located on the left and right front sides, a motor 13 that drives vehicle 1, brakes 14 that brake vehicle 1, a reduction gear 15 that reduces the driving force generated by the motor 13, a steering mechanism 16 that changes the angle of the drive wheels 12, a position sensor 17 that measures the position of vehicle 1, a vehicle communication unit 18 that communicates wirelessly with a control server 4, and a vehicle calculation unit 19 that commands the motor 13, brakes 14, and steering mechanism 16 to perform actions.

[0020] The power generated by the motor 13 through the conversion of electrical energy is transmitted to the reduction gear 15, where it is reduced by a gear-type reduction mechanism inside the reduction gear 15, and then transmitted to the left and right drive wheels 12, becoming the driving force that propels the vehicle 1.

[0021] Brakes 14 are provided near the driven wheels 11 and the drive wheels 12 to generate braking force for the vehicle 1. The brakes 14 generate frictional force by pressing brake pads against a disc rotor using hydraulic pressure. This converts kinetic energy into thermal energy, thereby braking the vehicle 1.

[0022] The vehicle 1 can be turned by changing the angle of the drive wheels 12 using the linkage mechanism provided in the steering mechanism 16.

[0023] The position sensor 17 can measure the position of vehicle 1 using a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System) and transmit the position information to the vehicle calculation unit 19.

[0024] The vehicle communication unit 18 can communicate with the control server 4 using wireless communication methods such as local LTE (Long Term Evolution) or local 5G (Generation).

[0025] The vehicle calculation unit 19 consists of a CPU (Central Processing Unit) and memory, and executes a vehicle control program. Based on the automatic driving stop request received by the vehicle communication unit 18 from the control server 4 and the position information of vehicle 1 obtained from the position sensor 17, it calculates command values ​​for the motor 13, brake 14, and steering mechanism 16. This allows for the control of the acceleration, deceleration, and turning of vehicle 1, and the guidance of vehicle 1 to any desired location.

[0026] Figure 3 shows a schematic configuration of the infrastructure sensor 2 in the first embodiment. The dashed arrows in Figure 3 indicate the signal flow.

[0027] The infrastructure sensor 2 includes a measurement unit 21 that measures the surrounding conditions, an infrastructure sensor calculation unit 22 that processes the output of the measurement unit 21, and an infrastructure sensor communication unit 23 that communicates wirelessly with the calibration server 3 and the control server 4.

[0028] The measurement unit 21 consists of a mechanically rotating LiDAR and other components, and can measure the distance to surrounding objects from the time it takes for laser light emitted into the surroundings to reflect back and output the shape as a point cloud.

[0029] The infrastructure sensor calculation unit 22 consists of a CPU, GPU (Graphics Processing Unit), memory, hard disk drive, etc., and executes the infrastructure sensor control program to process the output of the measurement unit 21.

[0030] The infrastructure sensor communication unit 23 can communicate with the calibration server 3 and the control server 4 using wireless communication methods such as local LTE or local 5G.

[0031] Figure 4 shows a schematic configuration of the calibration server 3 in the first embodiment. The dashed arrows in Figure 4 indicate the signal flow.

[0032] The calibration server 3 comprises a calibration server calculation unit 31, a calibration server communication unit 32, and a calibration server terminal unit 33.

[0033] The calibration server calculation unit 31 consists of a CPU, GPU, memory, hard disk drive, etc., and executes a calibration server control program to calculate the information necessary for processing by the infrastructure sensor calculation unit 22.

[0034] The calibration server communication unit 32 can communicate with the infrastructure sensor communication unit 23 using wireless communication methods such as local LTE or local 5G.

[0035] The calibration server terminal unit 33 consists of a CPU, memory, hard disk drive, display, mouse, keyboard, etc., and assists the administrator (user) in inputting necessary information to the calibration server calculation unit 31, and can also transmit the processing results of the calibration server calculation unit 31 to the administrator.

[0036] Figure 5 shows a schematic configuration of the control server 4 in the first embodiment. The dashed arrows in Figure 5 indicate the signal flow.

[0037] The control server 4 comprises a control server calculation unit 41 and a control server communication unit 42.

[0038] The control server calculation unit 41 consists of a CPU, GPU, memory, hard disk drive, etc., and executes the control server control program to calculate the information that the vehicle calculation unit 19 uses to make decisions regarding autonomous driving.

[0039] The control server communication unit 42 can communicate with the vehicle communication unit 18 and the infrastructure sensor communication unit 23 using wireless communication methods such as local LTE or local 5G.

[0040] Figure 6 is a block diagram of the mobile monitoring device in the first embodiment. Figure 7 is a flowchart of the operation of the vehicle calculation unit 19 in the first embodiment. Figure 8 is a flowchart of the operation of the infrastructure sensor calculation unit 22 in the first embodiment. Figure 9 is a flowchart of the operation of the calibration server calculation unit 31 in the first embodiment. Figure 10 is a flowchart of the operation of the control server calculation unit 41 in the first embodiment.

[0041] The control operation of the first embodiment will be explained below using Figures 6, 7, 8, 9, and 10.

[0042] The CPU of the vehicle calculation unit 19 constitutes a part of the control block shown in Figure 6 in the form of microcomputer software, and while the power supply (not shown) of the vehicle 1 is on, it repeatedly executes the operation (vehicle control program) shown in Figure 7.

[0043] The CPU of the infrastructure sensor calculation unit 22 constitutes a part of the control block shown in Figure 6 in the form of microcomputer software, and while the power supply (not shown) of the infrastructure sensor 2 is turned on, it repeatedly executes the operation (infrastructure sensor control program) shown in Figure 8.

[0044] The CPU of the calibration server calculation unit 31 constitutes a part of the control block shown in Figure 6 in the form of microcomputer software, and while the power supply (not shown) of the calibration server 3 is turned on, it repeatedly executes the operation (calibration server control program) shown in Figure 9.

[0045] The CPU of the control server calculation unit 41 constitutes a part of the control block shown in Figure 6 in the form of microcomputer software, and while the power supply (not shown) of the control server 4 is turned on, it repeatedly executes the operation (control server control program) shown in Figure 10.

[0046] The vehicle calculation unit 19 is equipped with a position information conversion unit 101 and a stopping unit 102.

[0047] The infrastructure sensor calculation unit 22 is equipped with an anomaly determination unit 201, a static information extraction unit 202, a static information storage unit 203, a posture change estimation unit 204, a dynamic information extraction unit 205, and a coordinate transformation unit 206.

[0048] The calibration server calculation unit 31 is equipped with a calibration reference storage unit 301, a reception notification unit 302, a difficulty determination unit 303, a transmission information generation unit 304, and a posture estimation unit 305.

[0049] The control server calculation unit 41 is provided with a control unit 401.

[0050] The operation of each part is explained below.

[0051] In step S201 of Figure 9, the calibration reference storage unit 301 determines whether it has received survey information from the calibration server terminal unit 33. If it determines that it has received survey information, it proceeds to step S302; if it determines that it has not received survey information, it proceeds to step S203.

[0052] The administrator can input survey information via the calibration server terminal unit 33.

[0053] The survey information is three-dimensional shape information of the driving environment, and may also be point cloud information of the driving environment measured by an MMS (Mobile Mapping System).

[0054] Administrators may input survey data via a Web API (Application Programming Interface). This allows survey data to be entered from any point.

[0055] In step S202 of Figure 9, the calibration reference storage unit 301 stores the survey information input from the calibration server terminal unit 33 as a calibration reference, and proceeds to step S203.

[0056] In step S101 of Figure 8, the abnormality determination unit 201 determines whether there are any abnormalities in the measurement information input from the measurement unit 21, transmits the presence or absence of abnormalities to the transmission information generation unit 304, and proceeds to step S102.

[0057] When using LiDAR as the measurement unit 21, if dirt or ice adheres to the surface, it becomes impossible to obtain measurement information correctly, making it difficult to calibrate the infrastructure sensor 2. If such deposits are present, a large number of measurement points in the measurement information will be distributed near the measurement unit 21. Therefore, an abnormality may be determined if more than a predetermined number of measurement points in the measurement information are distributed within a predetermined distance from the measurement unit 21. This allows for the determination of an abnormality if dirt or ice adheres to the surface of the measurement unit 21.

[0058] When using LiDAR as the measurement unit 21, measurement points distributed within a predetermined distance from the measurement unit 21 may be filtered out depending on the specifications of the LiDAR. Therefore, an abnormality may be determined if the number of measurement points in the measurement information falls below a predetermined level. This allows for the determination of an abnormality if dirt or ice adheres to the surface of the measurement unit 21 and causes it to be filtered out.

[0059] In step S102 of Figure 8, the static information extraction unit 202 determines whether the calibration of the infrastructure sensor 2 is complete. If it is determined that the calibration is not complete, the process proceeds to step S103; if it is determined that the configuration is complete, the process proceeds to step S105.

[0060] In step S103 of Figure 8, the static information extraction unit 202 extracts only measurement points that do not change for a predetermined time from the measurement information input from the measurement unit 21 as static information, transmits the static information to the static information storage unit 203, the reception notification unit 302, the difficulty determination unit 303, and the posture estimation unit 305, and proceeds to step 104. Since the infrastructure sensor 2 is fixed to a building or support column, by extracting only measurement points that do not change for a predetermined time, it is possible to remove measurement points of pedestrians and cars that move quickly and extract only measurement points of buildings and road surfaces.

[0061] Figure 11 is a diagram illustrating the operation of the static information extraction unit 202 in the first embodiment.

[0062] Alternatively, measurement points of measurement information can be stored in voxels representing the three-dimensional space of the driving environment for a predetermined time, and only voxels containing a predetermined number or more measurement points can be extracted as static information. This allows for the extraction of voxels containing measurement points of buildings, road surfaces, etc., as static information.

[0063] In step S104 of Figure 8, the static information storage unit 203 stores the static information input from the static information extraction unit 202 as stored static information and terminates the infrastructure sensor control program.

[0064] In step S105 of Figure 8, the posture change estimation unit 204 compares the measurement information input from the measurement unit 21 with the stored static information stored in the static information storage unit 203 to estimate the posture change of the measurement unit 21, transmits the posture change information to the coordinate transformation unit 206, and proceeds to step 106. By estimating the posture change from the stored static information to the measurement information, it is possible to estimate the posture change of the measurement unit 21 that has occurred since the static information extraction unit 202 extracted the static information.

[0065] It is also possible to estimate the change in posture from measured information to stored static information using ICP (Iterative Closest Point) as posture change information.

[0066] In step S106 of Figure 8, the dynamic information extraction unit 205 extracts only the measurement points from the measurement information input from the measurement unit 21 that are not included in the stored static information stored in the static information storage unit 203 as dynamic information, transmits the dynamic information to the coordinate transformation unit 206, and proceeds to step S107. Since the infrastructure sensor 2 is fixed to a building or support column, by extracting only the measurement points that are not included in the stored static information, it is possible to extract measurement points of pedestrians, cars, etc., that move quickly.

[0067] Figure 12 is a diagram illustrating the operation of the dynamic information extraction unit 205 in the first embodiment.

[0068] Alternatively, measurement points of measurement information can be stored in voxels representing the three-dimensional space of the driving environment, and only voxels that do not contain stored static information can be extracted as dynamic information. This makes it possible to extract only the measurement points of pedestrians, vehicles, and other objects that move quickly.

[0069] In step S203 of Figure 9, the reception notification unit 302 determines whether it has received static information from the static information extraction unit 202. If it determines that static information has been received, it proceeds to S204; if it determines that it has not been received, it proceeds to S207.

[0070] In step S204 of Figure 9, the reception notification unit 302 transmits the reception notification to the transmission information generation unit 304, and proceeds to step S205.

[0071] In step S205 of Figure 9, the difficulty determination unit 303 determines whether it is difficult for the posture estimation unit 305 to estimate the posture based on the static information input from the static information extraction unit 202, transmits the presence or absence of difficulty to the transmission information generation unit 304, and proceeds to step S206.

[0072] If the static information does not include measurement points on buildings or road surfaces with distinctive shapes, or if it does not contain enough information to determine the orientation of the infrastructure sensor 2, then calibration of the infrastructure sensor 2 becomes difficult. In such cases, the difficulty determination unit 303 determines that it is difficult for the orientation estimation unit 305 to estimate the orientation using the static information input from the static information extraction unit 202.

[0073] If the shape feature quantity calculated from the static information input from the static information extraction unit 202 is below a predetermined value, it may be determined that the process is difficult.

[0074] Shape features may also be information about the planes, lines, and angles that make up buildings and road surfaces, calculated from static information.

[0075] In step S206 of Figure 9, the attitude estimation unit 305 compares the static information input from the static information extraction unit 202 with the calibration reference stored in the calibration reference storage unit 301 to estimate the attitude of the measurement unit 21 in the coordinate system of the calibration reference, i.e., the reference coordinate system, transmits the attitude information to the coordinate transformation unit 206, and proceeds to step S207.

[0076] Survey information, which contains information about the three-dimensional shape of the driving environment, is often much wider than the measurement range of the infrastructure sensor 2. Therefore, it is possible to first sample multiple points and orientations included in the survey information, and then calculate the final attitude information using ICP with the point and orientation that receives the highest evaluation as the initial estimation designation. This allows the attitude of the measurement unit 21 to be estimated in the reference coordinate system and the infrastructure sensor 2 to be calibrated.

[0077] The multiple points to be sampled may be sampled at equal intervals within the range included in the survey information. This allows for a comprehensive evaluation of the range included in the survey information.

[0078] The multiple points to be sampled may be selected with a focus on locations such as intersections estimated from survey data. This allows for a focused evaluation of locations such as intersections where infrastructure sensors 2 are likely to be installed.

[0079] In step S207 of Figure 9, the transmission information generation unit 304 generates transmission information to be transmitted to the administrator from the presence or absence of an abnormality input from the abnormality determination unit 201, the reception notification input from the reception notification unit 302, and the presence or absence of a difficulty input from the difficulty determination unit 303, transmits the transmission information to the calibration server terminal unit 33, and terminates the control server program.

[0080] If the abnormality detection unit 201 determines that there is an abnormality, the information transmission generation unit 304 transmits information to the administrator via the calibration server terminal unit 33 that the measurement information is abnormal. This makes the administrator aware that the measurement information is abnormal.

[0081] If the reception notification unit 302 notifies that static information has been received, the transmission information generation unit 304 transmits to the administrator via the calibration server terminal unit 33 that static information has been received as transmission information. This prevents the administrator from continuing to wait for calibration to complete without realizing that static information has not been transmitted due to a failure of the infrastructure sensor 2 or the like.

[0082] If the difficulty determination unit 303 determines that calibration is difficult, the transmission information generation unit 304 notifies the administrator via the calibration server terminal unit 33 that calibration is difficult. This makes the administrator aware that calibration is difficult.

[0083] The information can also be sent directly to the administrator's smartphone or other device. This allows the administrator to be notified of the information early.

[0084] The transmitted information could be made available via a Web API. This would allow administrators to check the transmitted information even when they are at the construction site of the infrastructure sensor 2.

[0085] In step S107 of Figure 8, the coordinate transformation unit 206 calculates transformed dynamic information by transforming the dynamic information into a reference coordinate system from the attitude change information input from the attitude change estimation unit 204, the dynamic information input from the dynamic information extraction unit 205, and the attitude information input from the attitude estimation unit 305, transmits the transformed dynamic information to the control unit 401, and terminates the infrastructure sensor control program. The attitude information is the attitude of the measurement unit 21 in the reference coordinate system at the time the static information extraction unit 202 extracted the static information, and the attitude change information is the change in the attitude of the measurement unit 21 that occurred after the static information extraction unit 202 extracted the static information. Therefore, the coordinate transformation unit 206 can transform the dynamic information into the reference coordinate system by taking into account the change in the attitude of the measurement unit 21 that occurred after the static information was extracted.

[0086] In step S001 of Figure 7, the position information conversion unit 101 converts the position information input from the position sensor 17 into a reference coordinate system, transmits the converted position information to the control unit 401, and proceeds to step S002.

[0087] In step S301 of Figure 10, the control unit 401 compares the transformed dynamic information input from the coordinate transformation unit 206 with the transformed position information input from the position information transformation unit 101 to determine whether a dangerous situation has occurred, such as the presence of a pedestrian near vehicle 1. If it is determined that a dangerous situation has occurred, the process proceeds to S302; if it is determined that no dangerous situation has occurred, the control server program terminates.

[0088] In step S302 of Figure 10, the control unit 401 transmits a stop request to the stop unit 102.

[0089] In step S002 of Figure 7, the stopping unit 102 determines whether it has received a stop request from the control unit 401. If it determines that it has received a request, it proceeds to step S003; if it determines that it has not received a request, it terminates the vehicle control program.

[0090] In step S003 of Figure 7, the stopping unit 102 transmits a hydraulic command to the brake 14 to stop vehicle 1. This allows vehicle 1 to be stopped in dangerous situations, such as when there are pedestrians near vehicle 1, thereby improving the safety of autonomous driving.

[0091] The main features of the first embodiment can also be summarized as follows:

[0092] The measurement unit 21 (e.g., LiDAR) shown in Figure 6 measures the surrounding environment and outputs measurement information for the measurement points. In this example, the measurement information is three-dimensional position information. The static information extraction unit 202 extracts only measurement points that have not changed position for a predetermined time as static information based on the measurement information. The calibration reference storage unit 301 stores calibration references (survey information) measured in the reference coordinate system. In this example, the calibration reference is six-dimensional information including three-dimensional position information and three-dimensional torsion (rotation on three axes) information. The attitude estimation unit 305 estimates the attitude of the measurement unit 21 in the reference coordinate system based on the calibration references and static information and outputs attitude information (position and torsion).

[0093] Since the orientation of the measurement unit 21 in the reference coordinate system is estimated based on calibration criteria (survey information) and static information (measurement points that do not change position over a predetermined period of time), there is no need to prepare algorithms for recognizing predetermined objects such as road signs, which differ from country to country or region to region. Furthermore, calibration of the infrastructure sensor 2 can be performed even if the predetermined object is not within the detection range of the infrastructure sensor 2. As a result, the versatility of the calibration of the infrastructure sensor 2 can be improved.

[0094] The mobile monitoring device receives static information from the static information extraction unit 202 (reception notification unit 302), and when static information is received, it notifies the user (administrator) that static information has been received (transmission information generation unit 304). This allows the user to know that the static information has been successfully extracted. As a result, it is possible to prevent the user from having to wait indefinitely for calibration to complete.

[0095] The mobile monitoring device determines whether there are any abnormalities in the measurement information due to substances adhering to the measurement unit 21 (abnormality determination unit 201), and if an abnormality is determined, it issues a warning to the user (administrator) (transmission information generation unit 304). This allows the user to be aware of any abnormalities in the measurement information.

[0096] For example, the mobile object monitoring device determines that there is an abnormality if the number of measurement points distributed within a predetermined distance from the measurement unit 21 is equal to or greater than a first threshold (abnormality determination unit 201). This allows the device to determine abnormalities in measurement information due to substances adhering to the measurement unit 21 based on the distance and number of measurement points. The mobile object monitoring device also determines that there is an abnormality if the number of measurement points is equal to or less than a second threshold (abnormality determination unit 201). This allows the device to determine abnormalities in measurement information due to substances adhering to the measurement unit 21 based on the number of measurement points.

[0097] The mobile monitoring device determines whether it is difficult for the attitude estimation unit 305 to estimate the attitude of the measurement unit 21 (difficulty determination unit 303), and if it determines that it is difficult to estimate the attitude of the measurement unit 21, it issues a warning to the user (administrator) (transmission information generation unit 304). This allows the user to know that it is difficult to estimate the attitude of the measurement unit 21. In other words, the user can know that it is difficult to calibrate the infrastructure sensor 2.

[0098] For example, the mobile object monitoring device determines that it is difficult to estimate the orientation of the measurement unit 21 if the amount of shape features included in the static information is less than a predetermined amount (difficulty determination unit 303). This makes it possible to determine whether it is difficult to estimate the orientation of the measurement unit 21 from the amount of shape features included in the static information.

[0099] The static information storage unit 203 stores static information and outputs stored static information indicating the stored static information. The dynamic information extraction unit 205 extracts only the measurement points not included in the stored static information as dynamic information based on the measurement information. The coordinate transformation unit 206 calculates transformed dynamic information by transforming the dynamic information to a reference coordinate system based on attitude information, as an example.

[0100] This allows dynamic information to be converted into a reference coordinate system from attitude information obtained by comparing the calibration standard (survey information) at the first time point (survey timing) with the static information at the second time point (static information extraction timing). As a result, even if the position or orientation of the infrastructure sensor 2 shifts between the first and second time points, the infrastructure sensor 2 is calibrated, thereby improving the accuracy of dynamic information (measurement points that move within a predetermined time: for example, measurement points of people, vehicles, etc.).

[0101] In this embodiment, the posture change estimation unit 204 calculates posture change information by comparing the measured information with the stored static information. The coordinate transformation unit 206 calculates transformed dynamic information based on the posture information and posture change information.

[0102] This process converts dynamic information into a reference coordinate system from attitude information obtained by comparing the calibration standard (survey information) at the first time point (survey timing) with the static information at the second time point (static information extraction timing), and attitude change information obtained by comparing the static information at the second time point (static information extraction timing) with the measurement information at the third time point (attitude change estimation timing). As a result, even if the position or orientation of the infrastructure sensor 2 shifts between the first and third time points, the infrastructure sensor 2 is calibrated, further improving the accuracy of the dynamic information. Furthermore, even if the infrastructure sensor 2 is shaken between the second and third time points, for example, by wind vibrations or traffic vibrations, the infrastructure sensor 2 is calibrated, ensuring the accuracy of the dynamic information.

[0103] The measurement unit 21 consists of a sensor (e.g., LiDAR). The static information extraction unit 202, the calibration reference storage unit 301, and the attitude estimation unit 305 each consist of at least a processor (CPU, etc.). The calibration reference storage unit 301 (CPU, etc.) stores the calibration reference (survey information) in a storage device such as memory or a hard disk drive. This allows the static information extraction unit 202, the calibration reference storage unit 301, and the attitude estimation unit 305 to be implemented in software.

[0104] In detail, the calibration reference storage unit 301 and the attitude estimation unit 305 are composed of at least a first processor (such as the CPU of the calibration server 3), and the static information extraction unit 202 is composed of at least a second processor (such as the CPU of the infrastructure sensor 2).

[0105] Since large data-sized calibration criteria (survey information) are not transmitted from the first processor (such as the CPU of calibration server 3) to the second processor (such as the CPU of infrastructure sensor 2), the communication load can be reduced.

[0106] The mobile object monitoring device includes an infrastructure sensor 2, a first server (calibration server 3), and a second server (control server 4). The first server (calibration server 3) includes a calibration reference storage unit 301 and an attitude estimation unit 305. The infrastructure sensor 2 includes a measurement unit 21, a static information extraction unit 202, a static information storage unit 203, a dynamic information extraction unit 205, and a coordinate transformation unit 206. The second server (control server 4) generates a control command (stop request) to the mobile object from the position (transformed position information) and transformed dynamic information of the mobile object (vehicle 1) in the reference coordinate system, and transmits the control command to the mobile object (control unit 401).

[0107] Since the converted dynamic information is highly accurate, the reliability of the mobile body control, which is based on the distance between the moving object (person, vehicle, etc.) detected by the infrastructure sensor 2 and the mobile body (vehicle 1), is improved. In this example embodiment, the drive source for the mobile body (vehicle 1) is a motor, but it may also be an engine (internal combustion engine), or a combination of a motor and an engine; the drive source is arbitrary.

[0108] [Second Embodiment] In the embodiments described below, the present invention is explained using the example of its application to a control system for automobiles, but the present invention can also be applied to control systems for transport vehicles that travel within warehouses, construction vehicles that travel on construction sites, and the like.

[0109] The second embodiment modifies some of the configurations of the first embodiment described above. Elements identical to those shown in Figures 1 to 12 are denoted by the same reference numerals, and the differences will be explained below.

[0110] Figure 13 is a block diagram of the infrastructure sensor 2 in the second embodiment.

[0111] In the second embodiment, the infrastructure sensor 2 includes an imaging unit 24.

[0112] In the second embodiment, the infrastructure sensor calculation unit 22 includes a shape restoration unit 207.

[0113] In the second embodiment, the imaging unit 24 is composed of a CMOS (complementary metal-oxide semiconductor) image sensor or the like, and transmits image information to the shape restoration unit 207.

[0114] In the second embodiment, the shape restoration unit 207 restores the three-dimensional shape from the image information and transmits the restored three-dimensional shape as measurement information to the anomaly determination unit 201, the static information extraction unit 202, the posture change estimation unit 204, and the dynamic information extraction unit 205. The shape restoration unit 207 may also use a neural network to restore the three-dimensional shape from the image information. This allows the posture of the imaging unit 24 to be estimated in a reference coordinate system and the infrastructure sensor 2 to be calibrated.

[0115] The main features of the second embodiment can also be summarized as follows:

[0116] The shape restoration unit 207 shown in Figure 13 restores a three-dimensional shape from image information and outputs the restored three-dimensional shape as measurement information. This improves the versatility of calibration for the infrastructure sensor 2, which has an imaging unit 24 (camera) as the measurement unit. The imaging unit 24 is, for example, a stereo camera, but it may also be a monocular camera, and the number of image sensors is arbitrary. In the case of a monocular camera, for example, the three-dimensional shape is restored using deep learning (machine learning).

[0117] [Third Embodiment] In the embodiments described below, the present invention is explained using the example of its application to a control system for automobiles, but the present invention can also be applied to control systems for transport vehicles that travel within warehouses, construction vehicles that travel on construction sites, and the like.

[0118] The third embodiment modifies some of the configurations of the first embodiment described above. Elements identical to those shown in Figures 1 to 12 are denoted by the same reference numerals, and the differences will be described below.

[0119] Figure 14 is a block diagram of the calibration server calculation unit 31 in the third embodiment.

[0120] The calibration server calculation unit 31 in the third embodiment includes a wide-area information storage unit 306 and a survey information extraction unit 307.

[0121] In the third embodiment, the wide-area information storage unit 306 stores wide-area survey information that is broader than the driving environment.

[0122] In the third embodiment, the survey information extraction unit 307 stores the area range corresponding to the area identifier, extracts the area range corresponding to the area identifier input from the calibration server terminal unit 33 from the wide-area survey information stored in the wide-area information storage unit 306, and transmits it to the calibration reference storage unit 301 as survey information.

[0123] This allows the infrastructure sensor 2 to be calibrated even when the wide-area survey information is large in scale.

[0124] The main features of the third embodiment can also be summarized as follows:

[0125] The wide-area information storage unit 306 shown in Figure 14 stores wide-area survey information. The survey information extraction unit 307 extracts the range (survey information) corresponding to the area identifier from the wide-area survey information. This reduces the processing load on the attitude estimation unit 305, for example, and makes attitude estimation easier.

[0126] [Fourth Embodiment] In the embodiments described below, the present invention is explained using the example of its application to a control system for automobiles, but the present invention can also be applied to control systems for transport vehicles that travel within warehouses, construction vehicles that travel on construction sites, and the like.

[0127] The fourth embodiment is a modification of some of the configurations of the first embodiment described above. Elements identical to those shown in Figures 1 to 12 are denoted by the same reference numerals, and the differences will be described below.

[0128] Figure 15 is a block diagram of the calibration server calculation unit 31 in the fourth embodiment.

[0129] The calibration server calculation unit 31 in the fourth embodiment includes a posture candidate estimation unit 308 and a posture determination unit 309.

[0130] In the fourth embodiment, the attitude candidate estimation unit 308 compares the static information input from the static information extraction unit 202 with the calibration criteria stored in the calibration criteria storage unit 301, estimates a plurality of attitude candidates for the measurement unit 21 in a reference coordinate system, and transmits them to the attitude determination unit 309.

[0131] Multiple points and orientations included in the survey information may be sampled, and the points and orientations with high evaluations may be used as initial estimation designations to calculate multiple final orientation candidates using ICP. This makes it possible to estimate multiple plausible orientations of the measurement unit 21.

[0132] In the fourth embodiment, if the posture determination unit 309 receives posture candidates from the posture candidate estimation unit 308 and includes multiple postures with high evaluations, it sends a presentation of posture candidates to the calibration server terminal unit 33 to notify the administrator that multiple plausible postures exist. This prompts the administrator to select a posture from the posture candidates. Furthermore, the posture determination unit 309 determines posture information from the posture candidates based on the posture selected by the administrator via the calibration server terminal unit 33, i.e., the selected posture, and transmits it to the coordinate transformation unit 206. This allows the infrastructure sensor 2 to be calibrated even when multiple plausible posture candidates exist.

[0133] The main features of the fourth embodiment can also be summarized as follows:

[0134] The mobile object monitoring device estimates multiple posture candidates for the measurement unit 21 in a reference coordinate system based on calibration standards and static information, and outputs posture candidates (posture candidate estimation unit 308, Figure 15). If the posture candidates include multiple postures (most likely postures) with an accuracy (evaluation) higher than a predetermined value, the mobile object monitoring device notifies the user (administrator) and outputs the posture selected by the user as posture information (posture determination unit 309, Figure 15).

[0135] This allows the infrastructure sensor 2 to be calibrated even when there are multiple possible orientations, such as when there are several locations similar to the installation position of the infrastructure sensor 2. The mobile object monitoring device (attitude determination unit 309) may also present the orientation candidates to the user in order of accuracy (evaluation value) (for example, by displaying them on a screen).

[0136] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0137] Furthermore, some or all of the above configurations and functions may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, the above configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0138] Furthermore, embodiments of the present invention may also be as follows.

[0139] (1) A mobile body monitoring device comprising: a measurement unit that measures the surrounding environment and outputs measurement information; a static information extraction unit that extracts only measurement points that do not change for a predetermined time from the measurement information as static information; a calibration reference storage unit that stores calibration references measured in a reference coordinate system; and a posture estimation unit that estimates the posture of the measurement unit in a reference coordinate system based on the calibration references and the static information and outputs posture information.

[0140] (2) A mobile monitoring device according to (1), comprising a reception notification unit that notifies of the reception of static information, wherein the reception notification unit notifies the administrator when it receives the static information.

[0141] (3) A mobile monitoring device as described in (1), comprising an abnormality determination unit that determines an abnormality in the measurement unit, and the abnormality determination unit issues a warning to the administrator when it determines an abnormality.

[0142] (4)(3) A mobile object monitoring device, characterized in that the abnormality determination unit determines an abnormality when a predetermined number or more of the measurement points of the measurement information are distributed within a predetermined distance from the measurement unit.

[0143] (5)(3) A mobile object monitoring device, characterized in that the abnormality determination unit determines an abnormality when the number of measurement points of the measurement information is less than or equal to a predetermined value.

[0144] (6) A mobile body monitoring device as described in (1), comprising a difficulty determination unit that determines whether it is difficult for the posture estimation unit to estimate the posture, and the difficulty determination unit issues a warning to the administrator when it determines that it is difficult.

[0145] (7)(6) A mobile object monitoring device, characterized in that the difficulty determination unit determines that it is difficult when the shape feature quantity included in the static information is less than or equal to a predetermined value.

[0146] (8) A mobile body monitoring device as described in (1), comprising: a posture candidate estimation unit that estimates a plurality of posture candidates for the measurement unit in a reference coordinate system based on the calibration standard and the static information and outputs posture candidates; and a posture determination unit that notifies the administrator when the posture candidates include a plurality of plausible postures and outputs the posture selected by the administrator as posture information.

[0147] (9) A mobile body monitoring device as described in (1), comprising: a static information storage unit that stores the static information and outputs the stored static information; a dynamic information extraction unit that extracts only the measurement points from the measurement information that are not included in the stored static information as dynamic information; and a coordinate transformation unit that calculates transformed dynamic information obtained by transforming the dynamic information to the reference coordinate system based on the attitude information and transmits it to a control unit.

[0148] A mobile body monitoring device according to (10).(9), comprising a posture change estimation unit that calculates posture change information by comparing the measurement information with the stored static information, and a coordinate transformation unit that calculates the transformed dynamic information based on the posture information and the posture change information.

[0149] (11). A mobile monitoring device as described in (1), characterized by comprising: a wide-area information storage unit for storing wide-area survey information; and a survey information extraction unit for extracting a range corresponding to an area identifier from the wide-area survey information.

[0150] (12). A mobile object monitoring device as described in (1), characterized in that it comprises a shape restoration unit that restores a three-dimensional shape from image information. [Explanation of Symbols]

[0151] 1: Vehicle 2: Infrastructure Sensors 3: Calibration Server 4: Control Server 11: Driven wheel 12: Drive wheels 13: Motor 14: Brakes 15: Reducer 16: Steering mechanism 17: Position sensor 18: Vehicle Communications Department 19: Vehicle Calculation Unit 21: Measurement Unit 22: Infrastructure Sensor Calculation Unit 23: Infrastructure Sensor Communication Unit 24: Imaging Department 31: Calibration Server Calculation Unit 32: Calibration Server Communication Unit 33: Calibration Server Terminal Unit 41: Control Server Computing Unit 42: Control Server Communications Unit 101: Location Information Conversion Unit 102: Stop part 201: Abnormality determination section 202: Static information extraction section 203: Static information storage section 204: Posture Change Estimation Unit 205: Dynamic Information Extraction Unit 206: Coordinate Transformation Unit 207: Shape Restoration Section 301: Calibration Reference Memory Unit 302: Receipt Notification Section 303: Difficulty judgment part 304: Transmission Information Generation Unit 305: Posture estimation section 306: Wide-area information storage unit 307: Survey information extraction part 308: Posture candidate estimation unit 309: Posture determining part 401:Control Department

Claims

1. A measurement unit that measures the surrounding environment and outputs measurement information for the measurement point, A static information extraction unit extracts only the measurement points that have not changed position for a predetermined time based on the measurement information, as static information. A calibration reference storage unit that stores calibration references measured in a reference coordinate system, A posture estimation unit estimates the posture of the measurement unit in the reference coordinate system and outputs posture information for the second time, based on the calibration standard, which is six-dimensional survey information including three-dimensional position information and three-dimensional torsion information measured at the first time, and the static information extracted based on the measurement information measured at the second time. A static information storage unit that stores the static information and outputs the stored static information, A dynamic information extraction unit extracts only the measurement points not included in the stored static information as dynamic information based on the measurement information measured at a third time different from the second time; A posture change estimation unit calculates posture change information for the third time by comparing the static information extracted based on the measurement information measured at the second time with the measurement information measured at the third time, The system includes a coordinate transformation unit that calculates transformed dynamic information by transforming the dynamic information into the reference coordinate system based on the posture information at the second time and the posture change information at the third time, A mobile monitoring device that receives static information from the static information extraction unit and notifies the user that the static information has been received.

2. A mobile monitoring device according to claim 1, The mobile monitoring device determines whether it is difficult for the attitude estimation unit to estimate the attitude of the measurement unit, and issues a warning to the user if it determines that it is difficult to estimate the attitude of the measurement unit. A mobile monitoring device characterized by the following features.

3. A mobile monitoring device according to claim 2, The mobile monitoring device determines that it is difficult to estimate the posture of the measurement unit when the amount of shape features included in the static information is less than or equal to a predetermined amount. A mobile monitoring device characterized by the following features.

4. A mobile monitoring device according to claim 1, The aforementioned mobile monitoring device is Based on the calibration criteria and the static information, the measurement unit estimates multiple candidate postures in a reference coordinate system and outputs the candidate postures. If the candidate postures include multiple postures with a probability higher than a predetermined value, the user is notified, and the posture selected by the user is output as posture information. A mobile monitoring device characterized by the following features.

5. A mobile monitoring device according to claim 1, A wide-area information storage unit that stores wide-area survey information, A survey information extraction unit extracts the range corresponding to the area identifier from the aforementioned wide-area survey information, A mobile monitoring device characterized by comprising the following features.

6. A mobile monitoring device according to claim 1, The system includes a shape restoration unit that restores a three-dimensional shape from image information and outputs the restored three-dimensional shape as measurement information. A mobile monitoring device characterized by the following features.

7. A mobile monitoring device according to claim 1, The aforementioned measurement unit is composed of sensors, The static information extraction unit, the calibration reference storage unit, and the attitude estimation unit are comprised of at least a processor. A mobile monitoring device characterized by the following features.

8. A mobile monitoring device according to claim 7, The calibration reference storage unit and the attitude estimation unit are comprised of at least a first processor. The static information extraction unit is composed of at least a second processor. A mobile monitoring device characterized by the following features.

9. A mobile monitoring device according to claim 1, The mobile monitoring device includes an infrastructure sensor, a first server, and a second server. The first server comprises the calibration reference storage unit and the attitude estimation unit, The infrastructure sensor comprises the measurement unit, the static information extraction unit, the static information storage unit, the dynamic information extraction unit, and the coordinate transformation unit. The second server generates a control command to the mobile body from the position of the mobile body in the reference coordinate system, which is obtained by converting the position of the mobile body measured by the mobile body's position sensor, and the converted dynamic information calculated by the coordinate conversion unit of the infrastructure sensor, and transmits the control command to the mobile body. A mobile monitoring device characterized by the following features.

Images