Measurement system

Abstract

To easily grasp the three-dimensional position of an object measured by a radar.SOLUTION: A measurement system 1 includes a stereo camera 10 for imaging a measurement object area which is an area to be measured, a radar device 12 for transmitting transmission waves to the measurement object area, performing synthetic aperture processing on the basis of received reflection waves, and at least acquiring measurement intensity data indicating information indicating each position of the measurement object area by intensity, and a conversion data generation part 34 for generating conversion intensity data which is data obtained by converting the measurement intensity data into the image data in the imaging direction of the stereo camera 10.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a measurement system. [Background technology]

[0002] For example, Patent Document 1 discloses a so-called multiple input multiple output (MIMO) type synthetic aperture radar system having multiple transmitting and receiving antennas. In Patent Document 1, received signals acquired by multiple antennas are subjected to code modulation processing, and the signals after code modulation processing are multiplexed, thereby reducing the data capacity. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-201133 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, a MIMO-SAR with antennas arranged in a row can generate image data that correlates the direction of transmission of the transmitted wave, the distance of the reflected wave from the antenna, and the strength of the reflected wave. However, while this type of image data makes it possible to determine that an object is located at a distance where the strength of the reflected wave is high, it is difficult to intuitively determine the three-dimensional position of the object.

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a measurement system that can easily grasp the three-dimensional position of an object measured by radar. [Means for solving the problem]

[0006] In order to solve the above problem, a measurement system according to one embodiment of the present invention includes a stereo camera that captures an image of a measurement target area, which is the area to be measured, a radar device that transmits a transmission wave to the measurement target area and performs synthetic aperture processing based on the received reflected wave, and acquires measurement intensity data that indicates, by intensity, information indicating each position in the measurement target area, and a conversion data generation unit that generates converted intensity data, which is data obtained by converting the measurement intensity data into image data in the imaging direction of the stereo camera.

[0007] The measurement system may further include an image processing unit that derives camera coordinate data based on an image acquired by the stereo camera, in which each pixel of the image corresponds to a three-dimensional coordinate system in the image, and a coordinate data processing unit that derives radar coordinate data that indicates the coordinate system at the time of measurement by the radar device, and generates matching coordinate data that indicates coordinates in which the camera coordinate data and the radar coordinate data are integrated, and the conversion data generation unit may generate the conversion intensity data based on the matching coordinate data and the measurement intensity data.

[0008] In addition, the coordinate data processing unit may generate matching coordinate data by associating an index of the camera coordinate data with an index of the radar coordinate data so that the camera coordinate data corresponds to radar coordinate data indicating the three-dimensional position closest to the three-dimensional position of the camera coordinate data.

[0009] In addition, the conversion data generation unit may refer to the matching coordinate data to associate the measured intensity data with an index of the camera coordinate data, generate intensity pixel data in which the measured intensity data is associated with a pixel position of the captured image based on the camera coordinate data, and generate converted intensity data from the intensity pixel data.

[0010] The measurement system may further include an image processing unit that generates camera three-dimensional shape data indicating the three-dimensional shape of the measurement target area in the captured image based on the captured image acquired by the stereo camera, and a display processing unit that displays the camera three-dimensional shape data and the transformation intensity data in an overlaid manner.

[0011] In addition, the radar device may perform interferometry processing on the data obtained by the synthetic aperture processing to obtain measured displacement data indicating a displaced position in the measurement target area, and the converted data generation unit may generate converted displacement data, which is data obtained by converting the measured displacement data into image data in the imaging direction of the stereo camera.

[0012] The measurement system may further include an image processing unit that generates camera three-dimensional shape data indicating the three-dimensional shape of the measurement target area in the captured image based on the captured image acquired by the stereo camera, and a display processing unit that displays the camera three-dimensional shape data and the converted displacement data in an overlapping manner. Effect of the Invention

[0013] According to the present invention, it is possible to easily grasp the three-dimensional position of an object measured by a radar. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic block diagram showing the configuration of a measurement system according to this embodiment. [Diagram 2] FIG. 2 is a flowchart showing the flow of measurement by the measurement system. [Diagram 3] FIG. 3 is a flowchart illustrating the flow of captured image processing. [Figure 4] FIG. 4 is a flowchart illustrating the flow of radar measurement pre-processing. [Diagram 5] FIG. 5 is a flowchart illustrating the flow of the radar measurement process. [Figure 6] FIG. 6 is a flowchart illustrating the flow of the visualization process. [Figure 7] FIG. 7 is a diagram showing an example of an image in which an image of the camera three-dimensional shape data and an image of the 2D transformation intensity data are superimposed on the screen of a display device. [Figure 8] FIG. 8 is a diagram showing an example of an image of 3D transformation intensity data displayed on the screen of a display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The specific dimensions, materials, numerical values, etc. shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In this specification and drawings, elements having substantially the same functions and configurations are given the same reference numerals to avoid repeated explanations, and elements not directly related to the present invention are not shown.

[0016] 1 is a schematic block diagram showing the configuration of a measurement system 1 according to this embodiment. The measurement system 1 includes a stereo camera 10, a radar device 12, a user interface 14, and a control device 16.

[0017] The stereo camera 10 includes two cameras each including an imaging element such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The two cameras are arranged spaced apart in a substantially horizontal direction such that their optical axes are substantially parallel. The stereo camera 10 captures an image of a measurement target area, which is an area to be measured. For example, the stereo camera 10 captures an image of the measurement target area with the imaging direction being substantially horizontal.

[0018] The radar device 12 is, for example, a millimeter wave MIMO (Multi Input Multi Output) type synthetic aperture radar (SAR). The radar device 12 includes an array antenna 20 and a signal processing unit 22.

[0019] The array antenna 20 is configured by arranging a plurality of antennas in a row in the horizontal direction. The array antenna 20 is installed, for example, near the stereo camera 10. The aperture plane of the array antenna 20 is oriented in substantially the same direction as the imaging direction of the stereo camera 10. The array antenna 20 is capable of transmitting a transmission wave to a measurement target area. In addition, the array antenna 20 is capable of receiving a reflected wave from the measurement target area.

[0020] The signal processing unit 22 generates a transmission wave to be transmitted from the array antenna 20. The signal processing unit 22 also performs synthetic aperture processing based on the reflected wave received by the array antenna 20, and acquires measurement intensity data indicating at least information indicating each position in the measurement target area by intensity. The measurement intensity data associates, for example, the transmission direction of the transmission wave, the distance of the reflected wave from the array antenna 20, and the intensity of the reflected wave. The transmission direction of the transmission wave and the distance of the reflected wave from the array antenna 20 correspond to each position in the measurement target area.

[0021] The user interface 14 includes a display device 24 that displays predetermined information and images on a screen for the user, etc. The user interface 14 also has an input function that accepts input operations by the user, etc.

[0022] The control device 16 includes a processor and a memory that stores a program. The processor executes the program, whereby the control device 16 controls the entire measurement system 1. The processor executes the program, whereby the control device 16 also functions as a captured image processing unit 30, a coordinate data processing unit 32, a conversion data generating unit 34, and a display processing unit 36.

[0023] 2 is a flowchart showing the flow of measurement by the measurement system 1. The measurement system 1 has a plurality of display patterns, such as 2D intensity display, 2D displacement display, 3D intensity display, and 3D displacement display. The measurement system 1 can display the measurement results in a selected display pattern from among these plurality of display patterns. The control device 16 acquires, via the user interface 14, display pattern selection information, which is information indicating which display pattern to display is selected (S10).

[0024] The captured image processing unit 30 performs captured image processing to obtain captured images captured by the stereo camera 10 (S11). Next, the coordinate data processing unit 32 performs radar measurement pre-processing, which is processing performed in advance before performing radar measurement (S12). Next, the signal processing unit 22 of the radar device 12 performs radar measurement processing to execute radar measurement (S13). Next, the conversion data generating unit 34 and the display processing unit 36 ​​perform visualization processing to visualize the measurement results based on the results of the captured image processing and the results of the radar measurement processing (S14). In the visualization processing, visualization is performed using a display pattern selected from 2D intensity display, 2D displacement display, 3D intensity display, and 3D displacement display.

[0025] The control device 16 judges whether the termination condition is satisfied (S15). If the termination condition is not satisfied (NO in S15), the control device 16 judges whether the condition for updating the captured image is satisfied (S16). For example, the control device 16 may judge that the condition for updating the captured image is satisfied when a predetermined time has elapsed since the acquisition of the most recent captured image. In this case, the predetermined time from the acquisition of the most recent captured image is set to be equal to or greater than the total time required for the radar measurement process and the visualization process. If the condition for updating the captured image is not satisfied (NO in S16), the radar device 12 performs the radar measurement process (S13) again. If the condition for updating the captured image is satisfied (YES in S16), the captured image processing unit 30 performs the captured image process (S11) again. If the termination condition is satisfied (YES in S15), a series of measurements is terminated.

[0026] 3 is a flow chart for explaining the flow of the captured image processing (S11). The captured image processing unit 30 acquires a captured image captured by the stereo camera 10 (S30). The captured image data, which is image data of the captured image, is arranged pixel by pixel from the upper left to the right of the captured image and from the top to the bottom of the captured image. Therefore, the captured image data is associated with various indexes (indices) such as the position in the data string in the captured image, the position of the pixel, and color information in the pixel.

[0027] Next, the captured image processing unit 30 derives parallax information based on the two captured images obtained, using a so-called stereo method (S31).

[0028] Next, the captured image processing unit 30 derives camera coordinate data indicating a three-dimensional coordinate system in the captured image based on the parallax information (S32). For example, in the camera coordinate data, each pixel in the captured image is associated with a three-dimensional position in the captured image. That is, in the camera coordinate data, the position in the data string of the captured data and the position of the pixel of the captured image are associated with a three-dimensional position in the captured image. As a result, the position in the data string of the captured data, the position of the pixel of the captured image, and the three-dimensional position in the captured image corresponding to each camera coordinate data can be specified in the order in which the camera coordinate data are arranged.

[0029] Next, the captured image processing unit 30 generates camera three-dimensional shape data indicating the three-dimensional shape of the measurement target area in the captured image based on the parallax information (S33), and ends the captured image processing. The camera three-dimensional shape data is generated, for example, as point cloud data expressing the three-dimensional shape by a plurality of point clouds arranged at three-dimensional positions. As described above, since the camera coordinate data is derived based on the parallax information, the captured image processing unit 30 can derive the camera three-dimensional shape data, for example, by referring to the camera coordinate data and associating color information in the pixel with the three-dimensional position in the captured image to which the pixel of the captured image is associated.

[0030] 4 is a flowchart explaining the flow of the radar measurement pre-processing (S12). The coordinate data processing unit 32 acquires predetermined parameters related to the radar measurement (S40). The predetermined parameters may include, for example, the transmission direction of the transmission wave.

[0031] The coordinate data processing unit 32 derives radar coordinate data indicating the coordinate system at the time of measurement based on the acquired parameters (S41). In the radar coordinate data, various indexes (indicators) such as the transmission direction of the transmission wave, the distance until the reflected wave reaches the array antenna 20 (hereinafter, sometimes referred to as the distance of the reflected wave), and the position in the data string of the data acquired by the measurement are associated with each other. As a result, it is possible to specify the transmission direction of the transmission wave, the distance of the reflected wave, and the position in the data string of the data acquired by the measurement, which correspond to each radar coordinate data, in the order in which the radar coordinate data are arranged.

[0032] The coordinate data processing unit 32 performs a coordinate matching process (S42) based on the camera coordinate data and radar coordinate data derived in the captured image process (S11), and ends the radar measurement pre-processing. The coordinate matching process (S42) is a process for associating an index (index) related to the captured image with an index (index) related to the radar measurement. By performing the coordinate matching process (S42), the coordinate data processing unit 32 generates matching coordinate data indicating coordinates in which the camera coordinate data and the radar coordinate data are integrated.

[0033] More specifically, the coordinate data processing unit 32 extracts any one of the camera coordinate data, and searches the radar coordinate data for radar coordinate data corresponding to the extracted camera coordinate data. That is, the radar coordinate data indicating the three-dimensional position closest to the three-dimensional position of the extracted camera coordinate data is specified from the radar coordinate data. In the radar coordinate data, the three-dimensional position is indicated by the transmission direction of the transmission wave and the distance of the reflected wave, so the radar coordinate data is specified in which the three-dimensional position indicated by the transmission direction of the transmission wave and the distance of the reflected wave in the radar coordinate data is closest to the three-dimensional position of the extracted camera coordinate data. The coordinate data processing unit 32 associates the index of the extracted camera coordinate data with the index of the radar coordinate data identified by the search, and stores them in the memory as matching coordinate data. The coordinate data processing unit 32 repeatedly performs the extraction of the camera coordinate data, the search for the radar coordinate data, and the association until the extraction of all the camera coordinate data is completed.

[0034] Then, for example, matching coordinate data indicating a data string such as "camera: 1, radar: 30", "camera: 2, radar: 153", etc. is generated. In the example of "camera: 1, radar: 30", it means that the first camera coordinate data in the data string of the camera coordinate data is associated with the 30th radar coordinate data in the data string of the radar coordinate data. In other words, the position in the data string of the captured image, the position of the pixel of the captured image, and the three-dimensional position in the captured image, which correspond to the first camera coordinate data, are substantially associated with the transmission direction of the transmitted wave, the distance of the reflected wave, and the position in the data string of the data acquired by measurement, which correspond to the 30th radar coordinate data. In this way, in the coordinate matching process (S42), the radar coordinate data is integrated into the camera coordinate data through the matching coordinate data.

[0035] 5 is a flowchart illustrating the flow of the radar measurement process (S13). The signal processor 22 of the radar device 12 starts executing radar measurement, and acquires SAR data including information on reflected waves (S50).

[0036] The signal processing unit 22 performs calibration such as temperature correction on the acquired SAR data (S51). Note that the calibration may be omitted.

[0037] Next, the signal processing unit 22 performs synthetic aperture processing on the data after the calibration (S52), whereby the signal processing unit 22 acquires measurement intensity data that indicates information indicating each position in the measurement target area by intensity.

[0038] Here, the measured intensity data is numbered in the order in which the reflected wave intensity is acquired. As described above, the radar coordinate data includes an index indicating the position in the data string of the data acquired by measurement. Therefore, in the measured intensity data, each of the sequentially acquired intensities is associated with the transmission direction of the transmitted wave and the distance of the reflected wave through the radar coordinate data.

[0039] In radar measurement, a transmission wave is transmitted by the array antenna 20 arranged in the horizontal direction, and a reflected wave is received to obtain SAR data. The measurement intensity data based on the SAR data has horizontal position information (X-axis information and Y-axis information) based on the horizontal transmission direction of the transmission wave and the distance of the reflected wave, but the vertical position information (Z-axis information) is aggregated into horizontal position information according to the distance from the position of the array antenna 20. Therefore, the measurement intensity data based on the SAR data is obtained as a two-dimensional planar image in the horizontal direction. For example, the measurement intensity data is obtained as an image looking down on the ground from the sky.

[0040] Next, the signal processing unit 22 performs interferometry processing on the data (measurement intensity data) obtained by the synthetic aperture processing (S53), and ends the radar measurement processing at the current interrupt timing. As a result, the signal processing unit 22 obtains measurement displacement data indicating the displaced position in the measurement target area.

[0041] Here, the measured displacement data is numbered in the order in which the displacements are acquired. As described above, the radar coordinate data includes an index indicating the position in the data string of the data acquired by measurement. Therefore, in the measured displacement data, each of the sequentially acquired displacements is associated with the transmission direction of the transmitted wave and the distance of the reflected wave through the radar coordinate data.

[0042] 6 is a flow chart explaining the flow of the visualization process (S14). The control device converts the measured intensity data and measured displacement data acquired by the radar measurement process from numerical data to color data (S60). For example, for the measured intensity data, a numerical value indicating a relatively high intensity is converted to reddish color data, and a numerical value indicating a relatively low intensity is converted to bluish color data.

[0043] Hereinafter, the measured intensity data converted into color data may be referred to as measured intensity color data. Also, the measured displacement data converted into color data may be referred to as measured displacement color data. Note that the measured intensity color data is associated with the measured intensity data before being converted into color data. For example, the 30th measured intensity color data in the data string of measured intensity color data corresponds to the 30th measured intensity data in the data string of measured intensity data. Also, the measured displacement color data is associated with the measured displacement data before being converted into color data. For example, the 30th measured displacement color data in the data string of measured displacement color data corresponds to the 30th measured displacement data in the data string of measured displacement data.

[0044] Next, the display processing unit 36 ​​determines whether the 2D intensity display is selected as the display pattern (S61). If the 2D intensity display is selected (YES in S61), the conversion data generating unit 34 generates 2D converted intensity data, which is an example of converted intensity data, from the measured intensity color data (S62).

[0045] As described above, the measured intensity data is acquired as a two-dimensional planar image in the horizontal direction. In contrast, the converted intensity data is data obtained by converting the measured intensity data into image data in the imaging direction of the stereo camera 10. The image captured by the stereo camera 10 is acquired, for example, as a two-dimensional planar image in the approximately vertical direction. Therefore, the image data in the imaging direction of the stereo camera 10 means image data that is projected onto a screen in the approximately vertical direction. In other words, the measured intensity data as image data in the horizontal direction is converted into converted intensity data as image data in the approximately vertical direction.

[0046] The 2D converted intensity data is a type of converted intensity data, and the converted intensity data is generated as a two-dimensional planar image similar to that of the stereo camera 10. More specifically, in the 2D converted intensity data, measurement intensity color data based on the measured intensity data is represented by image data of a two-dimensional planar image in the imaging direction of the stereo camera 10.

[0047] More specifically, the conversion data generation unit 34 refers to the matching coordinate data, associates the measured intensity color data (or the measured intensity data) with an index of the camera coordinate data, and generates intensity pixel data in which the measured intensity color data (or the measured intensity data) is associated with the pixel position of the captured image based on the camera coordinate data.

[0048] For example, the 30th measurement intensity color data corresponds to the 30th measurement intensity data, and therefore corresponds to the 30th radar coordinate data. With reference to the matching coordinate data, it is identified that the 30th radar coordinate data corresponds to the first camera coordinate data. That is, the conversion data generating unit 34 associates the 30th measurement intensity color data with the first camera coordinate data. According to the camera coordinate data, the 30th measurement intensity color data is substantially associated with the position in the data string of the imaging data, the position of the pixel of the captured image, and the three-dimensional position in the captured image, which correspond to the first camera coordinate data. Then, it is identified which pixel of the captured image the 30th measurement intensity color data corresponds to. The conversion data generating unit 34 generates intensity pixel data by associating the 30th measurement intensity color data with the identified pixel. The conversion data generating unit 34 associates the index of the camera coordinate data with all of the measurement intensity color data, and generates intensity pixel data.

[0049] The conversion data generating unit 34 generates 2D converted intensity data as a two-dimensional planar image from the intensity pixel data generated for each measured intensity color data. The 2D converted intensity data generated in this manner corresponds to the measured intensity data converted into image data in the imaging direction of the stereo camera 10.

[0050] Next, the display processing unit 36 ​​displays the camera three-dimensional shape data on the screen of the display device 24 (S63). Next, the display processing unit 36 ​​displays the 2D conversion intensity data superimposed on the image of the camera three-dimensional shape data displayed on the screen (S64).

[0051] As a result, an image in which the image of the camera three-dimensional shape data and the image of the 2D conversion intensity data are superimposed, as exemplified in FIG.

[0052] In this example, the camera three-dimensional shape data and the 2D conversion intensity data are displayed in an overlapping manner, but the display of the camera three-dimensional shape data may be omitted and the 2D conversion intensity data may be displayed alone.

[0053] In step S61, if the 2D intensity display has not been selected (NO in S61), the display processing unit 36 ​​determines whether the 2D displacement display has been selected (S65).

[0054] If the 2D displacement display is selected (YES in S65), the converted data generating unit 34 generates 2D converted displacement data, which is an example of converted displacement data, from the measured displacement color data (S66).

[0055] The converted displacement data is data obtained by converting the measured displacement data into image data in the imaging direction of the stereo camera 10. As described above, the image data in the imaging direction of the stereo camera 10 means image data that is projected onto a screen in a substantially vertical direction. In other words, the measured displacement data as image data in the horizontal direction is converted into converted displacement data as image data in a substantially vertical direction.

[0056] The 2D converted displacement data is a type of converted displacement data, and the converted displacement data is generated as a two-dimensional planar image similar to that of the stereo camera 10. More specifically, in the 2D converted displacement data, measured displacement color data based on the measured displacement data is represented by image data of a two-dimensional planar image in the imaging direction of the stereo camera 10.

[0057] More specifically, the conversion data generation unit 34 refers to the matching coordinate data, associates the measured displacement color data (or the measured displacement data) with an index of the camera coordinate data, and generates displacement pixel data in which the measured displacement color data (or the measured displacement data) is associated with the pixel position of the captured image.

[0058] For example, the 30th measured displacement color data corresponds to the 30th measured displacement data, and therefore corresponds to the 30th radar coordinate data. With reference to the matching coordinate data, it is identified that the 30th radar coordinate data corresponds to the first camera coordinate data. That is, the conversion data generating unit 34 associates the 30th measured displacement color data with the first camera coordinate data. According to the camera coordinate data, the 30th measured displacement color data is substantially associated with the position in the data string of the imaging data, the position of the pixel of the captured image, and the three-dimensional position in the captured image, which correspond to the first camera coordinate data. Then, it is identified which pixel of the captured image the 30th measured displacement color data corresponds to. The conversion data generating unit 34 generates displacement pixel data by associating the 30th measured displacement color data with the identified pixel. The conversion data generating unit 34 associates the index of the camera coordinate data with all of the measured displacement color data, and generates displacement pixel data.

[0059] The conversion data generating unit 34 generates 2D converted displacement data as a two-dimensional planar image from the displacement pixel data generated for each measured displacement color data. The 2D converted displacement data generated in this manner corresponds to the measured displacement data converted into image data in the imaging direction of the stereo camera 10.

[0060] Next, the display processing unit 36 ​​displays the camera three-dimensional shape data on the screen of the display device 24 (S67). Next, the display processing unit 36 ​​displays the 2D converted displacement data superimposed on the image of the camera three-dimensional shape data displayed on the screen (S68).

[0061] As a result, an image in which the image of the camera three-dimensional shape data and the image of the 2D converted displacement data are superimposed is displayed on the screen of the display device 24.

[0062] In this example, the camera three-dimensional shape data and the 2D converted displacement data are displayed in an overlapping manner, but the display of the camera three-dimensional shape data may be omitted and the 2D converted displacement data may be displayed alone.

[0063] In step S65, if the 2D displacement display has not been selected (NO in S65), the display processing unit 36 ​​determines whether the 3D intensity display has been selected (S69).

[0064] If 3D intensity display is selected (YES in S69), the converted data generating unit 34 generates 3D converted intensity data, which is an example of converted intensity data, from the measured intensity color data (S70).

[0065] The 3D converted intensity data is a type of converted intensity data, and the converted intensity data is generated as a three-dimensional image. At this time, the 3D converted intensity data may be generated as point cloud data. More specifically, in the 3D converted intensity data, the measured intensity color data based on the measured intensity data is represented by a three-dimensional image in the imaging direction of the stereo camera 10, which is made up of point cloud data.

[0066] More specifically, the conversion data generator 34 first generates intensity pixel data in a manner similar to the process of deriving the 2D conversion intensity data, and then combines the camera coordinate data and the intensity pixel data to generate 3D conversion intensity data.

[0067] As described above, in the camera coordinate data, the pixel position of the captured image is associated with a three-dimensional position in the captured image. Based on this, the conversion data generating unit 34 refers to the camera coordinate data, identifies a three-dimensional position in the captured image from the pixel position indicated by the intensity pixel data, and associates the measured intensity color data indicated by the intensity pixel data with the identified three-dimensional position in the captured image. For all of the intensity pixel data, the conversion data generating unit 34 associates the measured intensity color data with the three-dimensional position in the captured image, and generates 3D converted intensity data as point cloud data.

[0068] Next, the display processing unit 36 ​​causes the 3D conversion intensity data to be displayed on the screen of the display device 24 (S71).

[0069] As a result, an image of the 3D converted intensity data, such as that shown in FIG.

[0070] If the 3D intensity display has not been selected in step S69, the display processing unit 36 ​​determines whether the 3D displacement display has been selected (S72).

[0071] If 3D displacement display is selected (YES in S72), the conversion data generating unit 34 generates 3D converted displacement data, which is an example of converted displacement data, from the measured displacement color data (S73).

[0072] The 3D converted displacement data is a type of converted displacement data, and the converted displacement data is generated as a three-dimensional image. At that time, the 3D converted displacement data may be generated as point cloud data. More specifically, in the 3D converted displacement data, the measured displacement color data based on the measured displacement data is represented by a three-dimensional image in the imaging direction of the stereo camera 10, which is made up of point cloud data.

[0073] More specifically, the transformation data generator 34 first generates displacement pixel data in a manner similar to the process of deriving the 2D transformation displacement data, and then combines the camera coordinate data and the displacement pixel data to generate 3D transformation intensity data.

[0074] The conversion data generating unit 34 refers to the camera coordinate data, identifies a three-dimensional position in the captured image from the pixel position indicated by the displacement pixel data, and associates the measured displacement color data indicated by the displacement pixel data with the identified three-dimensional position in the captured image. The conversion data generating unit 34 associates the measured displacement color data with the three-dimensional position in the captured image for all of the displacement pixel data, and generates 3D converted displacement data as point cloud data.

[0075] Next, the display processing unit 36 ​​causes the 3D converted displacement data to be displayed on the screen of the display device 24 (S74).

[0076] As a result, an image of the 3D converted displacement data is displayed on the screen of the display device 24.

[0077] If the 3D displacement display is not selected in step S72, it is presumed that the selection of the display pattern is inappropriate, and the display processing unit 36 ​​causes the display device 24 to display an error message (S75).

[0078] As described above, in the measurement system 1 of this embodiment, converted intensity data is generated, which is data obtained by converting the measurement intensity data by the radar device 12 into image data in the imaging direction of the stereo camera 10. In the measurement system 1 of this embodiment, by viewing the converted intensity data as an image, it is possible to grasp the measurement intensity data when the measurement target area is viewed from the same direction as the imaging direction of the stereo camera 10. Therefore, in the measurement system 1 of this embodiment, it becomes possible to grasp the three-dimensional position of an object in the measurement target area in the same way as when viewing an image captured by the stereo camera 10.

[0079] Therefore, according to the measurement system 1 of the present embodiment, it is possible to easily and intuitively grasp the three-dimensional position of an object in the measurement target area measured by the radar device 12.

[0080] Furthermore, in the measurement system 1 of this embodiment, the camera three-dimensional shape data showing the three-dimensional shape of the measurement target area in the captured image and the conversion intensity data are displayed in an overlapping manner. This makes it possible in the measurement system 1 of this embodiment to grasp the three-dimensional position of an object in the measurement target area, and to easily grasp, from the camera three-dimensional shape data, what the object grasped by the conversion intensity data is like.

[0081] Furthermore, in the measurement system 1 of this embodiment, converted displacement data is generated, which is data obtained by converting the measured displacement data by the radar device 12 into image data in the imaging direction of the stereo camera 10. In the measurement system 1 of this embodiment, if the converted displacement data is viewed as an image, it is possible to grasp the measured displacement data when the measurement target area is viewed from the same direction as the imaging direction of the stereo camera 10. Therefore, in the measurement system 1 of this embodiment, it becomes possible to grasp the displacement of an object in the measurement target area in the same way as when viewing an image captured by the stereo camera 10.

[0082] Therefore, according to the measurement system 1 of the present embodiment, the displacement of an object in the measurement target area measured by the radar device 12 can be easily and intuitively grasped.

[0083] Furthermore, in the measurement system 1 of this embodiment, the camera three-dimensional shape data showing the three-dimensional shape of the measurement target area in the captured image and the converted displacement data are displayed in an overlapping manner. This makes it possible in the measurement system 1 of this embodiment to grasp the displacement of an object in the measurement target area, and to easily grasp, from the camera three-dimensional shape data, what the object grasped by the converted displacement data is like.

[0084] Although the embodiment of the present invention has been described above with reference to the attached drawings, it goes without saying that the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can think of various modified or altered examples within the scope of the claims, and it is understood that such examples also naturally belong to the technical scope of the present invention. [Explanation of symbols]

[0085] 1. Measurement system 10 Stereo Camera 12 Radar equipment 24 Display device 30 Image capture processing section 32 Coordinate data processing section 34 Conversion data generation unit 36 Display processing section

Claims

1. A stereo camera that captures an image of a measurement target area, which is an area to be measured; a radar device that transmits a transmission wave to the measurement target area, performs synthetic aperture processing based on the received reflected wave, and acquires measurement intensity data that indicates at least information indicating each position in the measurement target area by intensity; a conversion data generation unit that generates converted intensity data by converting the measured intensity data into image data in the imaging direction of the stereo camera; A measurement system comprising:

2. a captured image processing unit that derives camera coordinate data in which each pixel of the captured image corresponds to a three-dimensional coordinate system in the captured image based on the captured image acquired by the stereo camera; a coordinate data processing unit that derives radar coordinate data indicating a coordinate system at the time of measurement by the radar device, and generates matching coordinate data indicating coordinates obtained by integrating the camera coordinate data and the radar coordinate data; Further equipped with The measurement system according to claim 1 , wherein the conversion data generating section generates the conversion intensity data based on the matching coordinate data and the measurement intensity data.

3. The measurement system according to claim 2, wherein the coordinate data processing unit generates the matching coordinate data by associating an index of the camera coordinate data with an index of the radar coordinate data so that the camera coordinate data corresponds to the radar coordinate data indicating a three-dimensional position closest to the three-dimensional position of the camera coordinate data.

4. The conversion data generation unit referring to the matching coordinate data, and associating the measured intensity data with an index of the camera coordinate data; generating intensity pixel data by associating the measured intensity data with pixel positions of the captured image based on the camera coordinate data; The metrology system of claim 3 , further comprising: a measuring system for measuring intensity data from said intensity pixel data;

5. an image processing unit that generates camera three-dimensional shape data indicating a three-dimensional shape of the measurement target area in the captured image based on the captured image acquired by the stereo camera; a display processing unit that displays the camera three-dimensional shape data and the transformation intensity data in an overlapping manner; The measurement system of claim 1 further comprising:

6. the radar device performs an interferometry process on the data obtained by the synthetic aperture process to obtain measured displacement data indicating a displaced position in the measurement target area; The measurement system according to claim 1 , wherein the converted data generating unit generates the converted displacement data, the converted displacement data being data obtained by converting the measured displacement data into image data in an imaging direction of the stereo camera.

7. an image processing unit that generates camera three-dimensional shape data indicating a three-dimensional shape of the measurement target area in the captured image based on the captured image acquired by the stereo camera; a display processing unit that displays the camera three-dimensional shape data and the converted displacement data in an overlapping manner; The measurement system of claim 6 further comprising:

Images