Observation device and observation method

Abstract

This observation device comprises a control unit. The control unit carries out calibration on the basis of an interest point set for a first region among the first region, a second region, and a third region, and information pertaining to the position of the interest point in a real space. The control unit acquires information pertaining to the position in the real space of a prescribed location in an observation region of a first sensor or a second sensor. The first region corresponds to the observation regions of both the first sensor and the second sensor. The second region corresponds to the observation region of the first sensor and does not overlap with the first region. The third region corresponds to the observation region of the second sensor, and does not overlap with the first region.

Description

Observation device and observation method Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2022-054582, filed on March 29, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an observation device and an observation method.

[0003] In recent years, observation devices using multiple sensors have been widely used. For example, observation devices installed on roadsides use multiple sensors to detect objects such as vehicles and pedestrians over a wide area.

[0004] In an observation device using multiple sensors, it is necessary to perform calibration of each of the multiple sensors in order to improve the detection accuracy of each of the multiple sensors. Therefore, a method for performing calibration of each of the multiple sensors is known (for example, Patent Document 1).

[0005] JP 2010-156609 A

[0006] An observation device according to an embodiment of the present disclosure includes a control unit that performs calibration based on feature points set for the first region and information relating to positions of the feature points in real space among a first region corresponding to the observation regions of both a first sensor and a second sensor, a second region corresponding to the observation region of the first sensor and not overlapping with the first region, and a third region corresponding to the observation region of the second sensor and not overlapping with the first region, and acquires information relating to positions in real space of predetermined points in the observation region of the first sensor or the second sensor.

[0007] An observation method according to one embodiment of the present disclosure includes: performing calibration based on feature points set for a first region corresponding to the observation regions of both a first sensor and a second sensor; a second region corresponding to the observation region of the first sensor and not overlapping with the first region; and a third region corresponding to the observation region of the second sensor and not overlapping with the first region, and information regarding positions of the feature points in real space; and acquiring information regarding positions in real space of predetermined points in the observation regions of the first sensor or the second sensor.

[0008] FIG. 2 is a block diagram of an observation system according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an installation state of the observation device shown in FIG. 1. FIG. 4 is a diagram showing a first image and a second image. FIG. 5 is a diagram showing a first region, a second region, and a third region. FIG. 6 is a diagram for explaining setting of feature points. FIG. 7 is a diagram for explaining a first input and a second input. FIG. 8 is a flowchart showing the flow of an observation method for the observation system shown in FIG. 1. FIG. 9 is a diagram for explaining calibration according to a comparative example.

[0009] Conventional techniques for performing calibration have room for improvement. For example, sensor fusion may be performed in an observation device using multiple sensors. Sensor fusion is a technique for integrating and processing detection results from multiple sensors. Performing calibration for each of the multiple sensors may result in a decrease in detection accuracy achieved by sensor fusion. According to an embodiment of the present disclosure, it is possible to provide a technique for performing improved calibration.

[0010] The observation device of the present disclosure observes an observation area using multiple sensors. The observation device of the present disclosure may be used for any purpose. For example, the observation device of the present disclosure may be a roadside device, a surveillance camera device, or the like.

[0011] Hereinafter, an embodiment of an observation device to which the present disclosure is applied will be described with reference to the drawings. In the following embodiment, the observation device is assumed to be a roadside device.

[0012] (Configuration of Observation System) As shown in Fig. 1 , the observation system 1 includes an observation device 10. The observation system 1 further includes a terminal device 20 when performing calibration of the observation device 10. The observation device 10 and the terminal device 20 are communicatively connected via a communication line. The communication line includes at least either a wired or wireless communication line.

[0013] The observation device 10 is a roadside device. The observation device 10 is provided in a communication system. The communication system including the observation device 10 is, for example, a safe driving support communication system of an intelligent transport system (ITS). The safe driving support communication system is also called a safe driving support system or a safe driving support wireless system.

[0014] The observation device 10 is placed, for example, on a roadside strip at an intersection or the like. The observation device 10 detects objects present in the observation area. The observation area is, for example, a road surface 2 such as a road as shown in FIG. 2. The observation device 10 is fixed to a structure 3 so as to be able to observe the road surface 2, which is the observation area, as shown in FIG. 2. The structure 3 is, for example, a traffic light, a utility pole, or a street light. The observation device 10 detects objects such as vehicles and pedestrians present in the observation area. The observation device 10 notifies surrounding moving objects, etc. of the detection results.

[0015] The user of the terminal device 20 is an operator. When performing calibration of the observation device 10, the operator connects the terminal device 20 to the observation device 10. The operator inputs information used for calibration of the observation device 10 from the terminal device 20.

[0016] (Configuration of Observation Device) As shown in Fig. 1, the observation device 10 includes a first sensor 11, a second sensor 12, a communication unit 13, a storage unit 14, and a control unit 15. The observation device 10 shown in Fig. 1 includes two sensors. However, the number of sensors included in the observation device 10 is not limited to two. The observation device 10 may include three or more sensors depending on, for example, the application.

[0017] The first sensor 11 captures an image of the observation area to generate a first image 30 as shown in FIG. 3 . The second sensor 12 captures an image of the observation area to generate a second image 31 as shown in FIG. 3 . A portion of the observation area of ​​the first sensor 11 overlaps with a portion of the observation area of ​​the second sensor 12. By overlapping a portion of the observation area of ​​the first sensor 11 with a portion of the observation area of ​​the second sensor 12, sensor fusion can be performed, combining the detection results of the first sensor 11 and the second sensor 12. However, another portion of the observation area of ​​the first sensor 11 does not have to overlap with another portion of the observation area of ​​the second sensor 12. In FIG. 3 , the positional relationship between the first image 30 and the second image 31 is shown corresponding to the positional relationship between the observation area of ​​the first sensor 11 and the observation area of ​​the second sensor 12. 3, the overlapping area between the first image 30 and the second image 31 corresponds to the overlapping area between the observation area of ​​the first sensor 11 and the observation area of ​​the second sensor 12, i.e., the observation areas of both the first sensor 11 and the second sensor 12. Hereinafter, when there is no particular distinction between the first image 30 and the second image 31, they are also collectively referred to as "image 32."

[0018] The first sensor 11 and the second sensor 12 are, for example, monocular cameras. The monocular camera may be, for example, an imaging device such as a visible light camera or an FIR (Far Infrared Rays) camera. The first sensor 11 and the second sensor 12 may be image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) that constitute a visible light camera or an FIR camera, respectively. The first sensor 11 and the second sensor 12 may be the same type of sensor or different types of sensors. For example, one of the first sensor 11 and the second sensor 12 may be a visible light camera and the other an FIR camera. Each of the first sensor 11 and the second sensor 12 may generate images 32 at a predetermined frame rate. However, the first sensor 11 and the second sensor 12 may be any sensor other than a monocular camera or an image sensor. For example, the first sensor 11 and the second sensor 12 may be a LiDAR (Light Detection and Ranging) sensor. In this case, the control unit 15 may generate an image 32 of the observation area from point cloud data obtained by measurement using the LiDAR. Alternatively, one of the first sensor 11 and the second sensor 12 may be a visible light camera or an FIR camera, and the other may be a LiDAR.

[0019] The communication unit 13 includes at least one communication module capable of communicating with moving objects on the road surface 2. The communication module is, for example, a communication module that complies with a standard for road-to-vehicle communication. The communication unit 13 may use this communication module to perform wireless communication with moving objects on the road surface 2, for example, in the 700 MHz band allocated to ITS.

[0020] The communication unit 13 includes at least one communication module capable of communicating with the terminal device 20 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The standard of the communication line is, for example, a wired communication standard or a short-range wireless communication standard such as Bluetooth (registered trademark), infrared, or NFC (Near Field Communication).

[0021] The storage unit 14 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 14 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores data used in the operation of the observation device 10 and data obtained by the operation of the observation device 10.

[0022] The control unit 15 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 15 executes processing related to the operation of the observation device 10 while controlling each part of the observation device 10.

[0023] <Distance Measurement Process> The control unit 15 acquires data of the image 32 from the first sensor 11 or the second sensor 12. The control unit 15 executes image recognition processing on the data of the image 32 to detect an object from the data of the image 32. The image recognition processing is, for example, processing such as pattern matching or machine learning such as deep learning.

[0024] When the control unit 15 detects an object from the data of the image 32, it acquires information about the position of the object in real space. The information about the position of the object in real space may be any information that can identify the position of the object in real space. As an example, the information about the position of the object in real space may be position information of the object in real space, or information about the distance from the observation device 10 to the object in real space. The position information of the object in real space may be given by coordinates in a world coordinate system. The world coordinate system is a coordinate system set in three-dimensional real space. In the following description, the information about the position of the object in real space will be described as information about the distance from the observation device 10 to the object in real space.

[0025] The control unit 15 acquires information about the distance from the observation device 10 to an object in real space using the correspondence data. The correspondence data is generated during calibration, which will be described later. The correspondence data associates coordinates in an image 32, as shown in FIG. 3, with data about the distance in real space from the observation device 10 to an object depicted at the pixel at that coordinate in the image 32. The coordinates in the image 32 are given using a first coordinate system, a second coordinate system, or a third coordinate system, as will be described later. For example, assume that an object depicted at a pixel at coordinate p1 in the image 32, as shown in FIG. 3, is located at position p2 on the road surface 2, as shown in FIG. 2. In this case, the correspondence data associates the coordinate p1 in the image 32 with data about the distance d1 from the observation device 10 to position p2. The position p2 on the road surface 2 is calculated from a virtual plane 4 during calibration, which will be described later. The virtual plane 4 is formed during calibration, for example, by treating the road surface 2 in real space, which is the observation area, as a single plane. The virtual surface 4 may also be formed by treating the road surface 2 in real space as a combination of multiple planes. In this embodiment, the virtual surface 4 is formed by treating it as a single plane. The control unit 15 identifies the coordinates of the pixel in the image 32 where the object appears, and obtains distance data associated with the identified coordinates from the correspondence data, thereby obtaining information on the distance from the observation device 10 to the object in real space.

[0026] <Calibration Processing> The control unit 15 receives information on feature points used for calibration from the terminal device 20 via the communication unit 13. Feature points are also called calibration points. Feature points are arbitrary points (arbitrary points) appropriately selected by an operator from structures, etc. on the road surface 2 in real space. The operator operates the terminal device 20 and appropriately selects, from among structures, etc. on the road surface 2, structures, etc. that appear in the image 32 as feature points.

[0027] The feature points are set by the operator separately for a first region 40, a second region 41, and a third region 42, as shown in FIG. 4 . A feature point set for the first region 40 is also referred to as a “feature point P1.” A feature point set for the second region 41 is also referred to as a “feature point P2.” A feature point set for the third region 42 is also referred to as a “feature point P3.” At least some of the multiple feature points P1 may be set on the boundary between the first region 40 and the second region 41. The feature point P1 set on the boundary between the first region 40 and the second region 41 may also be the feature point P2. At least some of the multiple feature points P1 may be set on the boundary between the first region 40 and the third region 42. The feature point P1 set on the boundary between the first region 40 and the third region 42 may also be the feature point P3.

[0028] The first region 40 is a region corresponding to the observation regions of both the first sensor 11 and the second sensor 12. In Fig. 4 , as in Fig. 3 , the positional relationship between the first image 30 and the second image 31 is shown corresponding to the positional relationship between the observation region of the first sensor 11 and the observation region of the second sensor 12. In other words, in Fig. 4 , as in Fig. 3 , the overlapping region between the first image 30 and the second image 31 corresponds to the observation regions of both the first sensor 11 and the second sensor 12. Therefore, in Fig. 4 , the first region 40 is shown as the overlapping region between the first image 30 and the second image 31.

[0029] The second region 41 is a region that corresponds to the observation region of the first sensor 11 and does not overlap with the first region 40. The first image 30 includes the first region 40 and the second region 41.

[0030] The third region 42 is a region that corresponds to the observation region of the second sensor and does not overlap with the first region 40. The second image 31 includes the first region 40 and the third region 42.

[0031] The information on the feature points used for calibration includes information on the positions of the feature points in real space. The information on the positions of the feature points in real space may be any information that can identify the positions of the feature points in real space. As an example, the information on the positions of the feature points in real space may be position information of the feature points in real space, or may be information on the distance from the observation device 10 to the feature points in real space. Hereinafter, the information on the positions of the feature points in real space is assumed to be position information of the feature points in real space. The position information of the feature points in real space may be position information identified by a Global Navigation Satellite System (GNSS). The position information of the feature points in real space may be given by coordinates in a world coordinate system.

[0032] The information on the feature points used for calibration includes position information of the feature points in the first region 40, the second region 41, or the third region 42. The first region 40, the second region 41, and the third region 42 are partial regions of the first image 30 or the second image 31. Therefore, the first region 40, the second region 41, and the third region 42 include pixels. The position information of the feature points in the first region 40, the second region 41, or the third region 42 is position information of pixels in the first region 40, the second region 41, or the third region 42 in which the feature points are captured. The position information of the feature points in the first region 40, the second region 41, and the third region 42 may be any information capable of identifying the position of the feature points in each of the first region 40, the second region 41, and the third region 42.

[0033] The control unit 15 sets a first coordinate system based on the position information of the feature point P1 in the first region 40. The first coordinate system is set based on the first region 40. The first coordinate system may be a two-dimensional coordinate system. Note that the control unit 15 sets a first coordinate system when an existing first coordinate system does not exist. For example, when a worker newly places the observation device 10 on a roadside strip at an intersection or the like, an existing first coordinate system does not exist. Therefore, for example, when a worker newly places the observation device 10 on a roadside strip at an intersection or the like, the control unit 15 sets a new first coordinate system. Furthermore, when an existing first coordinate system exists, the control unit 15 corrects the existing first coordinate system based on the position information of the feature point P1 in the first region 40. For example, when a worker performs maintenance work on an observation device 10 that has already been installed, an existing first coordinate system exists. Therefore, for example, when an operator performs maintenance work on an observation device 10 that has already been installed, the control unit 15 corrects the existing first coordinate system using the position information of the characteristic point P1 in the first region 40.

[0034] The control unit 15 sets a second coordinate system based on the position information of the feature point P2 in the second region 41. The second coordinate system is set based on the second region 41. The second coordinate system is independent of the first coordinate system. The second coordinate system may be a two-dimensional coordinate system. The control unit 15 sets a second coordinate system when an existing second coordinate system does not exist. For example, when a worker newly places the observation device 10 on a roadside strip at an intersection, etc., an existing second coordinate system does not exist. Therefore, the control unit 15 newly sets a second coordinate system when a worker newly places the observation device 10 on a roadside strip at an intersection, etc. Furthermore, when an existing second coordinate system exists, the control unit 15 corrects the existing second coordinate system based on the position information of the feature point P2 in the second region 41. For example, when a worker performs maintenance work on an observation device 10 that has already been installed, an existing second coordinate system exists. Therefore, for example, when an operator performs maintenance work on an observation device 10 that has already been installed, the control unit 15 corrects the existing second coordinate system using the position information of the feature point P2 in the second region 41.

[0035] The control unit 15 sets a third coordinate system based on the position information of the feature point P3 in the third region 42. The third coordinate system is set based on the third region 42. The third coordinate system is independent of the first and second coordinate systems. The third coordinate system may be a two-dimensional coordinate system. The control unit 15 sets a third coordinate system when an existing third coordinate system does not exist. For example, when a worker newly places the observation device 10 on a roadside strip at an intersection, etc., an existing third coordinate system does not exist. Therefore, the control unit 15 newly sets a third coordinate system when a worker newly places the observation device 10 on a roadside strip at an intersection, etc. Furthermore, when an existing third coordinate system exists, the control unit 15 corrects the existing third coordinate system based on the position information of the feature point P3 in the third region 42. For example, when a worker performs maintenance work on an observation device 10 that has already been installed, an existing third coordinate system exists. Therefore, for example, when an operator performs maintenance work on an observation device 10 that has already been installed, the control unit 15 corrects the existing third coordinate system using the position information of the characteristic point P3 in the third region 42.

[0036] The control unit 15 regards the road surface 2 in real space as a single plane and forms a virtual plane 4 based on the positional information of each of the feature points P1, P2, and P3 in real space. The control unit 15 generates correspondence data using the formed virtual plane 4 and the first coordinate system, second coordinate system, and third coordinate system after setting or correction. For example, the control unit 15 calculates the distance from the observation device 10 to an object depicted in a pixel at each coordinate in the first coordinate system, the second coordinate system, and the third coordinate system using the virtual plane 4. The control unit 15 generates the correspondence data by associating the calculated distance data with each coordinate in the first coordinate system, the second coordinate system, and the third coordinate system from the observation device 10.

[0037] However, the control unit 15 may form the virtual surface 4 based only on the position information of the feature point P1 in real space. The control unit 15 may generate correspondence data based on the formed virtual surface 4 and the set or corrected first coordinate system.

[0038] Furthermore, the control unit 15 may form the virtual plane 4 based on the position information of the feature point P1 in the real space and the position information of the feature point P2 in the real space. The control unit 15 may generate correspondence data based on the formed virtual plane 4 and the first coordinate system and the second coordinate system after the setting or correction.

[0039] Furthermore, the control unit 15 may form the virtual plane 4 based on the position information of the feature point P1 in the real space and the position information of the feature point P3 in the real space. The control unit 15 may generate correspondence data based on the formed virtual plane 4 and the first coordinate system and the third coordinate system after the setting or correction.

[0040] (Configuration of Terminal Device) As shown in FIG. 1 , the terminal device 20 includes a communication unit 21 , an input unit 22 , a display unit 23 , a storage unit 24 , and a control unit 25 .

[0041] The communication unit 21 includes at least one communication module capable of communicating with the observation device 10 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The standard of the communication line is, for example, a wired communication standard or a short-range wireless communication standard such as Bluetooth (registered trademark), infrared, or NFC.

[0042] The input unit 22 can accept input from a user. The input unit 22 is configured to include at least one input interface that can accept input from a user. The input interface is, for example, a physical key, a capacitance key, a pointing device, a touch screen that is integrated with a display, or a microphone. However, the input interface is not limited to these.

[0043] The display unit 23 includes a display. The display is, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display may be configured as a touch panel display or a touch screen display together with a touch screen included in the input unit 22.

[0044] The memory unit 24 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The memory unit 24 may function as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 24 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20.

[0045] The control unit 25 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or ASIC. The control unit 25 executes processes related to the operation of the terminal device 20 while controlling each part of the terminal device 20.

[0046] The control unit 25 may receive data of the image 32 from the observation device 10 via the communication unit 21 based on the user input accepted by the input unit 22, and may display the received image 32 on the display unit 23, as shown in Fig. 5. In Fig. 5, a first image 30 is displayed on the display unit 23.

[0047] The control unit 25 receives, via the input unit 22, a setting input of feature points for at least one of the first region 40, the second region 41, and the third region 42. When the control unit 25 displays an image 32 on the display unit 23 as shown in FIG. 5 , the control unit 25 may receive points in the image 32 specified via the input unit 22 as feature points. For example, in FIG. 5 , the control unit 25 receives points Q11 to Q24 in the first image 30 specified via the input unit 22 as feature points P11 to P24. The worker operates, for example, a pointing device or the like of the input unit 22 to specify points Q11 to Q24 from the first image 30 displayed on the display unit 23. The control unit 25 receives, via the input unit 22, the setting input of the received feature points P11 to P24, as will be described later with reference to FIG. 6 .

[0048] The control unit 25 may receive a first input via the input unit 22 as a setting input for a feature point. The first input indicates which of the first, second, and third coordinate systems the received feature point is to be used to set or correct. For example, the control unit 25 may cause the display unit 23 to display information such as that shown in FIG. 6. In FIG. 6, columns for the first, second, and third coordinate systems associated with feature points P11 to P24 are displayed. The feature points P11 to P24 are feature points received from the user as described above with reference to FIG. 5. In FIG. 6, the first input is to check the column for the coordinate system to be used to set or correct the feature point among the columns for the first, second, and third coordinate systems associated with the feature point. That is, the operator operates a pointing device or the like of the input unit 22 to check the column for the coordinate system to be used to set or correct the feature point among the columns for the first, second, and third coordinate systems associated with the feature point, as the first input. 5, the feature points P16, P17, and P18 are set on the boundary between the first region 40 and the second region 41. In other words, the feature points P16, P17, and P18 can be used to set or correct both the first coordinate system and the second coordinate system. Therefore, for the feature points P16, P17, and P18, check boxes for both the first coordinate system and the second coordinate system are checked.

[0049] Here, the first coordinate system cannot be set or corrected by feature points set only for the second region 41 or the third region 42. Therefore, when the first input indicates that a feature point set only for the second region 41 or the third region 42 is to be used to set or correct the first coordinate system, the control unit 25 may cause the display unit 23 to display a warning message. If the terminal device 20 is equipped with a speaker, the control unit 25 may cause the speaker to output a buzzer sound instead of displaying a warning message.

[0050] Furthermore, the second coordinate system cannot be set or corrected by feature points set only for the first region 40 or the third region 42. Therefore, when the first input indicates that a feature point set only for the first region 40 or the third region 42 is to be used to set or correct the second coordinate system, the control unit 25 may cause the display unit 23 to display a warning message. If the terminal device 20 is equipped with a speaker, the control unit 25 may cause the speaker to output a buzzer sound instead of displaying a warning message.

[0051] Furthermore, the third coordinate system cannot be set or corrected by feature points set only for the first region 40 or the second region 41. Therefore, when the first input indicates that a feature point set only for the first region 40 or the second region 41 is to be used to set or correct the third coordinate system, the control unit 25 may cause the display unit 23 to display a warning message. If the terminal device 20 is equipped with a speaker, the control unit 25 may cause the speaker to output a buzzer sound instead of displaying a warning message.

[0052] The control unit 25 may receive a second input via the input unit 22 as a setting input for the feature point. The second input is an input indicating information related to the position of the feature point in real space. In this embodiment, the second input is an input indicating the position information of the feature point in real space. However, the second input may be an input indicating any information that can identify the position of the feature point in real space. As another example, the second input may be information related to the distance from the observation device 10 to the feature point in real space.

[0053] The control unit 25 may cause the display unit 23 to display, in association with each other, information on the first input and information on the second input for each feature point. For example, as shown in Fig. 6, the control unit 25 may cause the display unit 23 to display, in association with each other, columns for the first coordinate system, the second coordinate system, and the third coordinate system for each of feature points P11 to P24 and position information of the feature point in real space.

[0054] The control unit 25 may receive, via the input unit 22, a third input indicating the selection of feature points not to be used in calibration from among the plurality of feature points for which the first and second inputs have been received. In FIG. 6 , the third input is to uncheck all of the boxes for the first coordinate system, the second coordinate system, and the third coordinate system associated with the feature points selected not to be used in calibration. For example, the worker refers to the information shown in FIG. 6 and selects not to use feature points whose position information in real space has low measurement accuracy for calibration. The worker also selects not to use feature points located high above the road surface 2 for calibration. In FIG. 6 , the worker operates a pointing device or the like of the input unit 22 to uncheck all of the boxes for the first coordinate system, the second coordinate system, and the third coordinate system associated with the feature points selected not to be used in calibration. For example, if the worker selects not to use feature points P12 and P14 for calibration, the worker unchecks the boxes for the second coordinate system associated with each of feature points P12 and P14.

[0055] The control unit 25 receives an input of an instruction to transmit information about feature points via the input unit 22. This input is input by the operator via the input unit 22 after the setting input of the feature points is completed. Upon receiving the input of the transmission instruction, the control unit 25 acquires information about feature points to be used for calibration. The feature points to be used for calibration are, for example, feature points for which a check mark is placed in at least one of the first coordinate system, second coordinate system, and third coordinate system columns as shown in FIG. 6 . As described above, the information about feature points to be used for calibration includes position information of the feature points in real space and position information of the feature points in the first region 40, the second region 41, or the third region 42.

[0056] The control unit 25 acquires position information in real space of the feature points used for calibration from the information of the second input.

[0057] The control unit 25 acquires position information of the feature points used for calibration in the first area 40, the second area 41, or the third area 42 from the information of the first input and the position information of the feature points on the screen of the display unit 23 as shown in Figure 5.

[0058] For example, the first coordinate system is set or corrected by position information in the first region 40 of a feature point set with respect to the first region 40. When the first input indicates that a feature point is to be used for setting or correcting the first coordinate system, the control unit 25 acquires the position information of the feature point in the first region 40 by converting the position information of the feature point on the screen of the display unit 23 into position information within the first region 40.

[0059] For example, the second coordinate system is set or corrected by position information in the second region 41 of a feature point set with respect to the second region 41. When the first input indicates that a feature point is to be used for setting or correcting the second coordinate system, the control unit 25 acquires the position information of the feature point in the second region 41 by converting the position information of the feature point on the screen of the display unit 23 into position information in the second region 41.

[0060] For example, the third coordinate system is set or corrected by position information in the third region 42 of a feature point set with respect to the third region 42. When the first input indicates that a feature point is to be used for setting or correcting the third coordinate system, the control unit 25 acquires the position information of the feature point in the third region 42 by converting the position information of the feature point on the screen of the display unit 23 into position information in the third region 42.

[0061] When the control unit 25 acquires the information on the feature points used for calibration, the control unit 25 transmits the information on the feature points used for calibration to the observation device 10 via the communication unit 21 .

[0062] (Operation of Observation System) Figure 7 is a flowchart showing the flow of an observation method of the observation system 1 shown in Figure 1. The processes of steps S1, S2, and S3 correspond to an example of an operation method of the terminal device 20 according to this embodiment. For example, when the terminal device 20 is connected to the observation device 10, the observation system 1 starts the process of step S1.

[0063] In the terminal device 20, the control unit 25 receives, via the input unit 22, an input for setting feature points for at least one of the first area 40, the second area 41, and the third area 42 (step S1). The control unit 25 receives, via the input unit 22, an input for an instruction to transmit information about the feature points (step S2). Upon receiving the input of the transmission instruction, the control unit 25 acquires information about the feature points to be used for calibration and transmits the acquired information about the feature points to the observation device 10 via the communication unit 21 (step S3).

[0064] In the observation device 10, the control unit 15 receives information about feature points from the terminal device 20 via the communication unit 13 (step S4). The control unit 15 sets or corrects a first coordinate system, a second coordinate system, and a third coordinate system based on the position information of the feature points in the image 32 (step S5). The control unit 15 forms a virtual plane 4 based on the position information of the feature points in real space (step S6). The control unit 15 generates correspondence data based on the formed virtual plane 4 and the set or corrected first coordinate system, second coordinate system, and third coordinate system (step S7). The control unit 15 acquires information about the real-space position of a specific point in the observation area of ​​the first sensor 11 or the second sensor 12 based on the correspondence data (step S8). For example, when the control unit 15 detects an object from the image 32 data, it acquires information about the object's real-space position as information about the real-space position of the specific point.

[0065] As described above, in the observation device 10, the control unit 15 performs calibration based on the feature point P1 set for the first region 40 among the first region 40, the second region 41, and the third region 42 shown in FIG. 4 , and on the position information of the feature point P1 in real space. Furthermore, the control unit 15 acquires information on the position in real space of a predetermined point in the observation region of the first sensor 11 or the second sensor 12. For example, the control unit 15 performs calibration using the information on the feature point P1 and generates correspondence data. Furthermore, the control unit 15 acquires information on the position in real space of the predetermined point in the observation region of the first sensor 11 or the second sensor 12 using the correspondence data.

[0066] As a comparative example, consider the case where feature points Px1 and Px2 are set as shown in FIG. 8 . Note that while the case where a coordinate system is corrected is described here as a comparative example, the same or similar situation may occur when a coordinate system is set. The feature point Px1 is set with respect to the first image 30. In this comparative example, the coordinate system of the first image 30 is corrected based on position information of the feature point Px1 in the first image 30. The feature point Px2 is set with respect to the second image 31. In this comparative example, the coordinate system of the second image 31 is corrected based on position information of the feature point Px2 in the second image 31. Here, the feature point Px1 is set without taking into account the observation areas of the second image 31 and the second sensor 12. Furthermore, the feature point Px2 is set without taking into account the observation areas of the first image 30 and the first sensor 11. Therefore, in this comparative example, the coordinate system of the first image 30 corrected by the feature point Px1 may be misaligned with the coordinate system of the second image 31 corrected by the feature point Px2. If the coordinate system of the first image 30 and the coordinate system of the second image 31 are misaligned, the detection accuracy of the sensor fusion will decrease when sensor fusion is performed by combining the detection results of the first sensor 11 and the second sensor 12. For example, for an object present in the observation areas of both the first sensor 11 and the second sensor 12, the distance from the observation device 10 to the object measured using the first image 30 may differ from the distance from the observation device 10 to the object measured using the second image 31. Furthermore, if the coordinate system of the first image 30 and the coordinate system of the second image 31 are misaligned, it may be necessary to repeatedly perform calibration until the misalignment between the coordinate systems of the first image 30 and the second image 31 is reduced to a certain extent.

[0067] In contrast to this comparative example, in the observation device 10 according to this embodiment, the feature point P1 is set relative to the first region 40. A first coordinate system based on the first region 40 can be set or corrected based on the position information of the feature point P1 in the first region 40. The first region 40 is included in both the first image 30 and the second image 31. That is, the first coordinate system is a common coordinate system corresponding to a partial region of the first image 30 and a partial region of the second image 31. By using such a first coordinate system, it is possible to avoid a situation in which the coordinate system of the first image 30 and the coordinate system of the second image 31 are misaligned, as in the comparative example. With this configuration, this embodiment can suppress a decrease in detection accuracy due to sensor fusion. Furthermore, in this embodiment, it is not necessary to repeatedly perform calibration until the misalignment between the coordinate system of the first image 30 and the coordinate system of the second image 31 is reduced to a certain extent, as in the comparative example. Therefore, this embodiment can provide a technique for performing improved calibration.

[0068] Furthermore, in the observation device 10, the control unit 15 may acquire information about the position of a predetermined point based on another feature point P2 or P3 different from the feature point P1 and information about the position of the other feature point P2 or P3 in real space. The feature point P2 can be used to set or correct a second coordinate system that is set for a second region 41 that does not overlap with the first region 40 included in the first image 30. This configuration can reduce the deviation between the first coordinate system and the second coordinate system. The feature point P3 can be used to set or correct a third coordinate system that is set for a third region 42 that does not overlap with the first region 40 included in the second image 31. This configuration can reduce the deviation between the first coordinate system and the third coordinate system.

[0069] Furthermore, as shown in FIG. 4 , at least some of the multiple feature points P1 may be set at the boundary between the first region 40 and the second region 41. The first coordinate system and the second coordinate system can be set or corrected using the feature point P1, i.e., feature point P2, set at the boundary between the first region 40 and the second region 41. With this configuration, the first coordinate system after setting or correcting and the second coordinate system after setting or correcting are continuous at the boundary between the first region 40 and the second region 41. The continuity between the first coordinate system after setting or correcting and the second coordinate system after setting or correcting reduces the possibility of a discrepancy between the detection results of the first sensor 11 and the detection results of the second sensor 12 at the boundary between the first region 40 and the second region 41. For example, the possibility of a discrepancy between the distance from the observation device 10 to an object measured using the first image 30 and the distance from the observation device 10 to the object measured using the second image 31 at the boundary between the first region 40 and the second region 41 can be reduced.

[0070] Furthermore, as shown in FIG. 4 , at least some of the multiple feature points P1 may be set on the boundary between the first region 40 and the third region 42. The first coordinate system and the third coordinate system can be set or corrected using the feature point P1, i.e., feature point P3, set on the boundary between the first region 40 and the third region 42. With this configuration, the first coordinate system after the setting or correction and the third coordinate system after the setting or correction are continuous at the boundary between the first region 40 and the third region 42. The continuity between the first coordinate system after the setting or correction and the third coordinate system after the setting or correction reduces the possibility of a discrepancy between the detection results of the first sensor 11 and the detection results of the second sensor 12 at the boundary between the first region 40 and the third region 42. For example, the possibility of a discrepancy between the distance from the observation device 10 to an object measured using the first image 30 and the distance from the observation device 10 to the object measured using the second image 31 at the boundary between the first region 40 and the third region 42 can be reduced.

[0071] Furthermore, in the terminal device 20, the control unit 25 accepts, via the input unit 22, input of feature points for at least one of the first region 40, the second region 41, and the third region 42. This configuration reduces the burden on the worker of setting and inputting feature points compared to when the worker inputs feature points Px1 and Px2 for each of the first image 30 and the second image 31 as shown in FIG. 8 . For example, for the first region 40 included in both the first image 30 and the second image 31, the worker only needs to input the feature points for the first region 40 once, and does not need to input and set feature points for each of the first image 30 and the second image 31. Therefore, according to this embodiment, the burden on the worker can be reduced.

[0072] Furthermore, in the terminal device 20, the control unit 25 may cause the display unit 23 to display first input information and second input information associated with each feature point, as shown in FIG. 6 . The control unit 25 may also use the input unit 22 to receive a third input indicating the selection of feature points not to be used for calibration from among the plurality of feature points for which the first input and second input have been received. By displaying the first input information and the second input information associated with each feature point on the display unit 23, the worker can easily select feature points not to be used for calibration. For example, the worker can easily select feature points whose position information in real space is low in measurement accuracy not to be used for calibration, or feature points located high above the road surface 2 not to be used for calibration. This improves the worker's efficiency in performing calibration. Furthermore, when the terminal device 20 receives the third input, it can transmit information, such as feature points whose position information in real space is high in measurement accuracy selected by the worker, to the observation device 10 as information on feature points to be used for calibration. This allows the observation device 10 to perform calibration with high accuracy.

[0073] While the present disclosure has been described based on various drawings and examples, it should be noted that those skilled in the art can easily make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the functions included in each functional unit can be rearranged so as not to cause logical inconsistencies. Multiple functional units can be combined into one or divided. The above-described embodiments of the present disclosure are not limited to faithful implementation of each of the described embodiments, but can be implemented by combining features or omitting some features as appropriate. In other words, those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, these modifications and alterations are within the scope of the present disclosure. For example, in each embodiment, each functional unit, means, step, etc. can be added to other embodiments so as not to cause logical inconsistencies, or can be replaced with each functional unit, means, step, etc. of other embodiments. Furthermore, in each embodiment, multiple functional units, means, steps, etc. can be combined into one or divided into two or more. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but can also be implemented by combining each feature or omitting some of them as appropriate.

[0074] For example, in the above-described embodiment, the information regarding the position of the feature point in real space has been described as being position information of the feature point in real space. However, as described above, the information regarding the position of the feature point in real space may also be information regarding the distance from the observation device 10 to the feature point in real space. In this case, the second input may be information regarding the distance from the observation device 10 to the feature point in real space. In the terminal device 20, the control unit 25 may transmit, via the communication unit 21, to the observation device 10, information regarding the distance from the observation device 10 to the feature point in real space, which information is received as the second input by the input unit 22. In the observation device 10, the control unit 15 may form the virtual plane 4 based on the information regarding the distance from the observation device 10 to the feature point in real space, which information is received from the terminal device 20.

[0075] For example, in the observation device 10 according to the above-described embodiment, the control unit 15 has been described as acquiring information about the distance in real space from the observation device 10 to a predetermined point as information about the position in real space of the predetermined point in the observation area of ​​the first sensor 11 or the second sensor 12. However, the control unit 15 may also acquire position information in real space of the predetermined point as information about the position in real space of the predetermined point in the observation area of ​​the first sensor 11 or the second sensor 12. In this case, the control unit 15 may generate correspondence data by associating coordinates in the first coordinate system, the second coordinate system, and the third coordinate system with position information in real space of an object depicted in the pixel at those coordinates.

[0076] For example, the observation device 10 according to the above-described embodiment has been described as including two sensors, the first sensor 11 and the second sensor 12. However, the observation device 10 may include three or more sensors. When the observation device 10 includes three or more sensors, the control unit 15 may execute the same or similar processing as in the above-described embodiment. An example of processing when the observation device 10 includes three or more sensors will be described below. Hereinafter, a case in which a portion of the observation area of ​​one of the multiple sensors included in the observation device 10 overlaps with a portion of the observation areas of all of the other two or more sensors will be referred to as a "first case." Furthermore, a case in which a portion of the observation area of ​​one of the multiple sensors included in the observation device 10 overlaps with a portion of the observation areas of some of the other sensors will be referred to as a "second case." In the first and second cases, the control unit 15 may set a coordinate system that is the same as or similar to the first coordinate system described above in the overlapping areas of the observation areas of the multiple sensors included in the observation device 10. In the first and second cases, the control unit 15 may set different coordinate systems in the regions of the observation regions of each sensor included in the observation device 10 that do not overlap with other sensors. In the first and second cases, the control unit 15 may set the coordinate systems as described above and perform calibration processing in the same or similar manner as in the above-described embodiment. Furthermore, when the above-described first and second cases occur simultaneously in the observation device 10, the control unit 15 may perform processing for both the first case 1 and the second case.

[0077] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, configurations distinguished by descriptions such as "first" and "second" can have their numbers swapped. For example, a first sensor can exchange identifiers "first" and "second" with a second sensor. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

[0078] REFERENCE SIGNS LIST 1 Observation system 2 Road surface 3 Structure 4 Virtual surface 10 Observation device 11 First sensor 12 Second sensor 13 Communication unit 14 Memory unit 15 Control unit 20 Terminal device 21 Communication unit 22 Input unit 23 Display unit 24 Memory unit 25 Control unit 30 First image 31 Second image 32 Image 40 First area 41 Second area 42 Third area P1 to P3, P11 to P24 Feature points Q11 to Q24 Point d1 Distance p1 Coordinate p2 Position

Claims

1. a first region corresponding to an observation region of both the first sensor and the second sensor; a second region corresponding to an observation region of the first sensor and not overlapping with the first region; a third region corresponding to an observation region of the second sensor and not overlapping with the first region; an observation device comprising: a control unit that performs calibration based on feature points set for the first area and information relating to positions of the feature points in real space, and acquires information relating to positions in real space of specified points in the observation area of ​​the first sensor or the second sensor.

2. the information about the positions of the feature points in real space is position information about the feature points in real space, The control unit is forming a virtual surface based on position information of the feature points in real space; setting a first coordinate system based on the first region according to position information of the feature point in the first region; The observation device according to claim 1 , wherein information regarding the position of the predetermined point in real space is obtained using the virtual surface and the first coordinate system after it has been set.

3. A first coordinate system is set based on the first region; the information about the positions of the feature points in real space is position information about the feature points in real space, The control unit is forming a virtual surface based on position information of the feature points in real space; correcting the first coordinate system based on position information of the feature point in the first region; The observation apparatus according to claim 1 , wherein information regarding the position of the predetermined point in real space is obtained using the virtual surface and the first coordinate system after correction.

4. 3. The observation device according to claim 2, wherein the control unit acquires information regarding the position of the specified point in real space, further based on another feature point that is different from the feature point and is set for the second area or the third area, and information regarding the position of the other feature point in real space.

5. The observation device according to claim 4 , wherein at least a portion of the plurality of feature points is set at a boundary between the first region and the second region or a boundary between the first region and the third region.

6. the information about the positions of the other feature points in real space is position information about the other feature points in real space, The control unit is forming a virtual surface based on position information of the feature point and the other feature point in real space; setting a second coordinate system based on the second region or a third coordinate system based on the third region according to position information of the other feature points in the second region or the third region; 6. The observation device according to claim 4, wherein information relating to a position of the predetermined point in real space is obtained using the virtual plane, the first coordinate system after setting, and the second coordinate system or the third coordinate system after setting.

7. A second coordinate system is set based on the second region; A third coordinate system is set based on the third region; the information about the positions of the other feature points in real space is position information about the other feature points in real space, The control unit is forming a virtual surface based on position information of the feature point and the other feature point in real space; correcting the second coordinate system or the third coordinate system based on position information of the other feature points in the second region or the third region; 6. The observation device according to claim 4, wherein information relating to a position of the predetermined point in real space is obtained using the virtual plane, the corrected first coordinate system, and the corrected second coordinate system or the third coordinate system.

8. The first sensor observes an observation area of ​​the first sensor to generate a first image; The second sensor observes an observation area of ​​the second sensor to generate a second image; the first image includes the first region and the second region; The observation device according to claim 4 , wherein the second image includes the first region and the third region.

9. 2. The observation device according to claim 1, wherein one of the first sensor and the second sensor is a visible light camera, and the other is a Far Infrared Rays (FIR) camera.

10. The observation device according to claim 1 , wherein the first sensor and the second sensor are each a visible light camera or an FIR camera.

11. The observation device according to claim 1 , wherein one of the first sensor and the second sensor is a visible light camera or an FIR camera, and the other is a LiDAR (Light Detection And Ranging) camera.

12. The observation device according to claim 1 , wherein the first sensor and the second sensor are each a LiDAR.

13. a first region corresponding to an observation region of both the first sensor and the second sensor; a second region corresponding to an observation region of the first sensor and not overlapping with the first region; a third region corresponding to an observation region of the second sensor and not overlapping with the first region; performing a calibration based on feature points set for the first region and information relating to positions of the feature points in real space; and acquiring information regarding a position in real space of a predetermined point in an observation area of ​​the first sensor or the second sensor.

Images