Computer system, method, and program

Abstract

Provided is a computer system that performs operations including fusing first frame data generated with a first interval and second frame data that include a plurality of items of sub-frame data generated with an interval shorter than the first interval and that, as a whole, are generated with a second interval which is longer than the first interval, selecting at least two of the items of sub-frame data, included in the second frame data, that are different from one another, on the basis of a reference time point of the first frame data, and generating the second frame data from the selected sub-frame data, wherein the fusing includes fusing the second frame data generated from the selected sub-frame data with the first frame data.

Description

Computer system, method and program

[0001] The present invention relates to a computer system, a method and a program.

[0002] Time-of-Flight (ToF) sensors, which measure distance based on the time of flight of light, are used, for example, to obtain three-dimensional information about a subject. They are broadly classified into two types: direct ToF (dToF), which measures the time difference until reflected light is detected, and indirect ToF (iToF), which measures distance by accumulating reflected light and detecting the phase difference with emitted light. Patent Document 1, for example, describes a technology that uses a ToF sensor in combination with a camera. Patent Document 1 describes a technology for more easily and automatically performing coordinate calibration in a distance measurement system that generates a distance image of a target object by installing a distance measurement sensor and an imaging camera.

[0003] Japanese Patent Application Laid-Open No. 2022-101310

[0004] By using a combination of a ToF sensor and a camera as in the above example, it is possible to, for example, color a point cloud obtained by ranging with the ToF sensor from an RGB image, upsample ranging points with low spatial resolution based on an RGB image, or fuse the RGB image and ranging points using SLAM, etc. However, because the frame intervals of the ToF sensor and the camera are different, there are cases where the image and ranging results cannot be properly associated.

[0005] Therefore, the present invention aims to provide a computer system, method, and program that can more appropriately match frames generated by sensors (including cameras) that sense space, even if the frame intervals of the sensors are different.

[0006] According to one aspect of the present invention, there is provided a computer system having a memory for storing program code and a processor for executing operations in accordance with the program code, wherein the operations include fusing first frame data generated at a first interval by a first sensor sensing a space with second frame data generated at a second interval that is longer than the first interval as a whole and that includes a plurality of subframe data generated at intervals shorter than the first interval by a second sensor sensing the space; selecting at least two of the subframe data, each of which is included in a different second frame data, based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, wherein the fusing includes fusing the second frame data generated from the selected subframe data with the first frame data.

[0007] According to another aspect of the present invention, there is provided a method, which includes, by operations executed by a processor according to program code stored in a memory, fusing first frame data generated at a first interval by sensing a space with a first sensor and second frame data generated at a second interval that is longer than the first interval as a whole, the second frame data including a plurality of subframe data generated at intervals shorter than the first interval by sensing the space with a second sensor; selecting at least two of the subframe data, each of which is included in a different second frame data, based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, wherein the fusing includes fusing the second frame data generated from the selected subframe data with the first frame data.

[0008] According to yet another aspect of the present invention, there is provided a program, the operations of which a processor executes according to the program include fusing first frame data generated at a first interval by sensing a space with a first sensor and second frame data generated at a second interval longer than the first interval as a whole, the second frame data including a plurality of subframe data generated at intervals shorter than the first interval by sensing the space with a second sensor; selecting at least two of the subframe data, each of which is included in a different second frame data, based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, wherein the fusing includes fusing the second frame data generated from the selected subframe data with the first frame data.

[0009] FIG. 1 is a diagram illustrating an example of a system according to an embodiment of the present invention. FIG. 2 is a timing chart illustrating an example of exposure timing of a camera and a sensor in the system illustrated in FIG. 1. FIG. 3 is a diagram for conceptually explaining a problem that may occur in the example of FIG. 2. FIG. 4 is a flowchart illustrating an example of processing executed in a first embodiment of the present invention. FIG. 5 is a diagram for conceptually explaining the processing of FIG. 4. FIG. 5 is a flowchart illustrating an example of processing executed in a second embodiment of the present invention. FIG. 7 is a diagram for conceptually explaining the processing of FIG. 7. FIG. 7 is a flowchart illustrating an example of processing executed in a third embodiment of the present invention. FIG. 10 is a diagram for conceptually explaining the processing of FIG. 10.

[0010] Hereinafter, several embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] 1 is a diagram illustrating an example of a system according to an embodiment of the present invention. In the illustrated example, the system 10 includes a computer 100, an RGB camera 210, a direct time of flight (dToF) sensor 220, and an event-based vision sensor (EVS) 230. The RGB camera 210 and the EVS 230 are arranged so that their positional relationship with the dToF sensor 220 is known, and images captured by the RGB camera 210 and event information acquired by the EVS 230 can be associated with distance measurements by the dToF sensor 220.

[0012] The computer 100 may be, for example, a game console, a personal computer (PC), or a network-connected server device. The computer 100 includes a processor 110 and a memory 120. The processor 110 is configured with processing circuits such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA). The memory 120 is configured with storage devices such as various read-only memories (ROMs), random access memories (RAMs), and / or hard disk drives (HDDs). The processor 110 operates according to program code stored in the memory 120. The computer 100 also includes a communication device 130 and a recording medium 140. For example, program code for the processor 110 to operate as described below may be received from an external device via the communication device 130 and stored in the memory 120. Alternatively, the program code may be read into the memory 120 from the recording medium 140. The recording medium 140 includes, for example, a removable recording medium such as a semiconductor memory, a magnetic disk, an optical disk, or a magneto-optical disk, and a driver for the same.

[0013] The RGB camera 210 includes an imaging element and acquires an RGB image. The dToF sensor 220 includes a laser light source and a light-receiving element arranged for each pixel, and outputs the measurement result of the time difference between the laser light emitted from the light source and the laser light received as reflected light for each pixel. The EVS 230, also known as an EDS (Event Driven Sensor), event camera, or DVS (Dynamic Vision Sensor), includes a sensor array composed of sensors including light-receiving elements. When the EVS 230 detects a change in the intensity of incident light, more specifically, a change in the luminance of the object surface, it generates an event signal including a timestamp, sensor identification information, and information on the polarity of the luminance change. In the embodiments of the present invention described below, the RGB camera 210 and the dToF sensor 220 are examples of sensors that generate frame data by sensing space. Frame data is data containing sensing results generated by the sensor at a predetermined interval and is simply referred to as a frame in the following description. Image data generated by the RGB camera 210 and distance data for each ranging point generated by the dToF sensor 220 are examples of frame data.

[0014] FIG. 2 is a timing chart showing an example of the exposure timing of the camera and sensor in the system shown in FIG. 1 . Note that the EVS 230 does not perform exposure on a frame-by-frame basis, but is included in the exposure timing diagram for convenience of explanation. The RGB camera 210 (RGB Exposure) generates an image by exposing at each vertical synchronization (V Sync). FIG. 2 shows two consecutive RGB frames (RGB Frame #0 and RGB Frame #1). On the other hand, the dToF sensor 220 (Depth Exposure) does not perform distance measurement for all distance measurement points simultaneously, but rather divides the distance measurement points into groups called banks and performs distance measurement by sequentially emitting and receiving laser light. In the illustrated example, the distance measurement points are divided into four banks (Bank0, Bank1, Bank2, and Bank3), and the time required to perform distance measurement for each of these banks is shown as a frame of the dToF sensor 220 (part of Depth Frame #-1, Depth Frame #0, and Depth Frame #1). In this example, distance data for each ranging point in each bank is a plurality of subframe data constituting frame data generated by the dToF sensor 220. The EVS 230 (EVS Exposure) generates event signals asynchronously in response to changes in light intensity, and has a higher time resolution than the RGB camera 210 and the dToF sensor 220. In other words, the interval at which the EVS 230 generates an event signal in response to changes in light intensity due to, for example, object movement is shorter than the exposure time of the RGB camera 210 and the dToF sensor 220.

[0015] In the above example, the exposure time for one frame of the dToF sensor 220 is longer than the exposure time for one frame of the RGB camera 210 and is close to the exposure time for two frames. In other words, the depth frames generated by the dToF sensor 220 are generated at intervals longer than the intervals at which the RGB frames are generated by the RGB camera 210. On the other hand, the individual banks (subframe data) included in the depth frames are generated at intervals shorter than the intervals at which the RGB frames are generated.

[0016] FIG. 3 is a diagram for conceptually explaining a problem that may occur in the example of FIG. 2 . As already explained, in Depth Frame #0, ranging is performed sequentially for the ranging points in Bank0, Bank1, Bank2, and Bank3. As shown in the timing chart of FIG. 2 , RGB Frame #0 and RGB Frame #1 are both included in Depth Frame #0. The center time of exposure for RGB Frame #0 is located between the exposure times of Bank0 and Bank1 in Depth Frame #0, and the center time of exposure for RGB Frame #1 is located between the exposure times of Bank2 and Bank3 in Depth Frame #0. Because the center time of the entire Depth Frame #0 is roughly between the exposure times of Bank#1 and Bank#2, the center times of both RGB frames are shifted from the center time of the depth frame.

[0017] When the distance measurement results of the Depth frame are fused with an RGB frame image with the above-described exposure center offset, the offset between the area where the subject existed in the RGB frame and the area where the depth position of the same subject is detected in the Depth frame becomes large, which may result in an inappropriate correspondence between the image and the distance measurement results. Specifically, in RGB Frame #0 Depth, to which the depth position of Depth Frame #0 is associated, the area where the depth of the subject is detected is significantly shifted to the right relative to the area where the subject existed during the exposure time (shown in a box; due to motion blur, this is larger than the area of ​​the subject similarly shown in each Bank of the Depth frame). Similarly, in RGB Frame #1 Depth, to which the depth position of Depth Frame #0 is associated, the area where the depth of the subject is detected is shifted to the left relative to the area where the subject existed during the exposure time.

[0018] First Embodiment FIG. 4 is a flowchart illustrating an example of processing executed in the first embodiment of the present invention. In the illustrated example, the processor 110 of the computer 100 receives RGB data captured by the RGB camera 210 (step S101), buffers EVS data including event signals acquired by the EVS 230 in the memory 120, and also buffers the depth data acquired by the dToF sensor 220 for each bank (step S102). Then, when fusing the RGB data and the depth data on a frame-by-frame basis, the processor 110 first calculates the EVS optical flow (OF) between the time of each bank of the depth data and the frame center time of the RGB data (step S103). As described above, the temporal resolution of the EVS 230 is higher than that of the RGB camera 210 and the dToF sensor 220, and therefore, the optical flow between the center times of each frame can be generated. The processor 110 corrects the depth detection position in each bank based on the optical flow generated from the EVS data, and regenerates the depth data (step S104). Furthermore, the processor 110 executes a fusion process between the RGB data and the depth data regenerated in step S104 (step S105).

[0019] 5 and 6 are diagrams for conceptually explaining the processing of FIG. 4 . FIG. 5 shows an example in which the depth detection positions associated with the RGB image of RGB Frame #0 in the example of FIG. 3 have been corrected. As described above in step S103, in this embodiment, the depth detection positions in Bank 2 and Bank 3 are corrected based on optical flows generated from EVS data, specifically, optical flow OF1 from the center time of Bank 2 of Depth Frame #0 to the center time of RGB Frame #0, and optical flow OF2 from the center time of Bank 3 to the center time of RGB Frame #0. This correction moves the depth detection positions in Bank 2 and Bank 3 in a direction that reduces the deviation from the area where the subject existed in RGB Frame #0. As a result, in RGB Frame #0 associated with the corrected (compensated) depth position, the deviation between the area where the subject existed during the exposure time and the area where the depth position was detected is smaller than in RGB Frame #0 Depth in the example of FIG. 3 .

[0020] On the other hand, Figure 6 shows an example in which the depth detection positions associated with the RGB image of RGB Frame #1 in the example of Figure 3 have been corrected. As in the example of Figure 5 above, the depth detection positions in Bank0 and Bank1 are corrected based on optical flow OF1 from the center time of Bank0 of Depth Frame #0 to the center time of RGB Frame #1, and optical flow OF2 from the center time of Bank1 to the center time of RGB Frame #1. This correction moves the depth detection positions in Bank0 and Bank1 in a direction that reduces the deviation from the area where the subject existed in RGB Frame #1. As a result, in RGB Frame #1, which is associated with the corrected (compensated) depth position, the deviation between the area where the subject existed during the exposure time and the area where the depth position was detected is smaller than in RGB Frame #1 Depth in the example of Figure 3.

[0021] In the examples of Figures 5 and 6, the depth detection position is corrected when the magnitude of the optical flow exceeds a threshold. For example, it is possible to generate optical flows between Bank0 and Bank1 in the example of Figure 5 and Bank2 and Bank3 in the example of Figure 6 and the center times of the RGB frames, but no correction is performed because the magnitude of the optical flows does not exceed the threshold. In such a case, for example, optical flows may be generated for all banks and then the magnitudes may be compared with the threshold, or optical flows may not be calculated for banks whose differences from the center times of the RGB frames are below the threshold.

[0022] Second Embodiment Next, a second embodiment of the present invention will be described. In the first embodiment, the deviation between the subject area in the RGB image and the depth detection position was reduced by correcting the depth detection position in each bank based on optical flow. In this embodiment, however, the deviation is reduced by selecting an appropriate bank for association with the RGB frame beyond the depth frame based on the center time of the RGB frame. Examples of this are Depth Frame #-1 (the frame immediately before the depth frame corresponding to the RGB frame) and Depth Frame #1 (the frame immediately after the depth frame corresponding to the RGB frame) shown in FIG. 2. Note that, because optical flow is not used in this embodiment, the EVS 230 does not need to be included in the system 10 shown in FIG. 1.

[0023] FIG. 7 is a flowchart showing an example of processing executed in the second embodiment of the present invention. Following steps S101 and S102 (similar to the example in FIG. 4 , except that EVS data may not be included), the processor 110 of the computer 100 selects a bank of depth data based on the center time of the RGB data frame (step S201). At this time, as in the example described below, the bank with the center time closest to the center time of the RGB frame is selected regardless of the relationship between the depth frame containing the bank and the RGB frame. Next, depth data is generated using the bank selected in step S201 (step S202). Here, in the example in FIG. 2 , the depth data includes four types of banks (subframe data), one for each bank. Therefore, in step S201, it is desirable to select one of the four types of banks (Bank0, Bank1, Bank2, and Bank3) even if the depth frames are different. Furthermore, the processor 110 performs a fusion process (step S105) between the RGB data and the depth data generated in step S202.

[0024] 8 and 9 are diagrams for conceptually explaining the processing of FIG. 7 . FIG. 8 shows an example in which a Bank associated with RGB Frame #0 in the example of FIG. 3 is selected across depth frames. As described above in step S201, in this embodiment, regardless of the relationship between the depth frame and the RGB frame, a Bank with a center time closest to the center time of the RGB frame is selected. In the example shown in FIG. 8 , Bank 3 of Depth Frame #−1 and Banks 0, 1, and 2 of Depth Frame #0 are selected as Banks closest to the center time (center) of RGB Frame #0. In other words, the subframe data of the Banks selected to generate the depth frame include at least two subframe data each included in a different frame (specifically, Bank 3 of Depth Frame #−1 and Bank 0 of Depth Frame #0). More specifically, because the center time of RGB Frame #0 is earlier than the center time of Depth Frame #0, Bank 3 of Depth Frame #−1, which is included in the previous Depth Frame #−1, is selected. With this selection, the depth data associated with RGB Frame #0 will include Bank3 of the previous Depth Frame #-1 instead of Bank3 of Depth Frame #0, which was the furthest from the center time of RGB Frame #0 in the example shown in Figure 3. Because the center time of Bank3 of Depth Frame #-1 is closer to the center time of RGB Frame #0, the discrepancy between the area in RGB Frame #0 where the subject existed during the exposure time and the area where the depth position was detected is reduced compared to the example in Figure 3.

[0025] On the other hand, FIG. 9 shows an example in which a Bank associated with RGB Frame #1 in the example of FIG. 3 is selected across depth frames. In the example of FIG. 9, Banks 1, 2, and 3 of Depth Frame #0 and Bank 0 of Depth Frame #1 are selected as Banks closest to the center time (center) of RGB Frame #1. In this example, the subframe data of the Bank selected for the depth frame includes at least two subframe data each contained in a different frame (specifically, Bank 3 of Depth Frame #0 and Bank 0 of Depth Frame #1). More specifically, because the center time of RGB Frame #1 is later than the center time of Depth Frame #0, Bank 0 of Depth Frame #1, which is contained in the subsequent Depth Frame #1, is selected. With this selection, Bank 0 of the next Depth Frame #1 is included in the depth data associated with RGB Frame #1, instead of Bank 0 of Depth Frame #0, which was furthest from the center time of RGB Frame #1 in the example shown in FIG. 3. Since the center time of Bank 0 of Depth Frame #1 is closer to the center time of RGB Frame #1, the difference between the area in RGB Frame #1 where the subject existed during the exposure time and the area where the depth position was detected is reduced compared to the example in Figure 3.

[0026] In this embodiment, different combinations of Banks may be selected for two consecutive RGB frame data (RGB Frame #0 and RGB Frame #1), as in the examples shown in Figures 8 and 9. Note that as a result of selection based on the center time of the RGB frames, Banks that are all included in the same Depth frame may be selected, or the same combination of Banks may be selected for two consecutive RGB frame data.

[0027] Third Embodiment FIG. 10 is a flowchart illustrating an example of processing executed in a third embodiment of the present invention. This embodiment is described as a combination of the first and second embodiments. The processor 110 of the computer 100 executes steps S101, S102, S201, and S202, as in the example of FIG. 7, to generate depth data from a bank whose center time is closest to the center time of the RGB frame. Furthermore, the processor 110 executes steps S103 and S104, as in the example of FIG. 4, to correct the depth detection position based on the EVS optical flow (OF) from the time of each bank of depth data to the center time of the RGB data frame, thereby regenerating the depth data. Then, the processor 110 executes a fusion process (step S105) between the RGB data and the depth data regenerated in step S104.

[0028] 11 and 12 are diagrams for conceptually explaining the processing of FIG. 10 . FIG. 11 shows an example in which banks associated with the RGB image of RGB Frame #0 in the example of FIG. 3 are selected across depth frames, and the depth detection position is corrected. In the example of FIG. 11 , similar to the example shown in FIG. 8 above, Bank 3 of Depth Frame #−1 and Banks 0, 1, and 2 of Depth Frame #0 are selected as banks closest to the center of RGB Frame #0. Furthermore, the depth detection position in each bank is corrected based on optical flows generated from EVS data, specifically, optical flow OF1 from the center of Bank 3 of Depth Frame #−1 to the center of RGB Frame #0, and optical flow OF2 from the center of Bank 2 of Depth Frame #0 to the center of RGB Frame #0. Through these processes, in RGB Frame #0 associated with the compensated depth position, the discrepancy between the area in which the subject existed during the exposure time and the area in which the depth position was detected is reduced.

[0029] On the other hand, Figure 12 shows an example in which banks associated with the RGB image of RGB Frame #1 in the example of Figure 3 are selected across depth frames, and the depth detection position is corrected. In the example of Figure 12, similar to the example shown in Figure 9 above, Banks 1, 2, and 3 of Depth Frame #0 and Bank 0 of Depth Frame #1 are selected as banks closest to the center of GB Frame #1. Furthermore, the depth detection position in each bank is corrected based on optical flows generated from the EVS data, specifically, optical flow OF1 from the center of Bank 1 of Depth Frame #0 to the center of RGB Frame #1, and optical flow OF2 from the center of Bank 0 of Depth Frame #1 to the center of RGB Frame #1. Through these processes, the discrepancy between the area where the subject existed during the exposure time and the area where the depth position was detected is reduced in RGB Frame #1, to which the corrected (compensated) depth position is associated.

[0030] As described above, according to an embodiment of the present invention, even if the sampling rates of the imaging camera and the ToF sensor are different, the deviation between the area in the RGB frame where the subject was present during the exposure time and the area in the Depth frame where the depth position is detected can be reduced, thereby making it possible to more appropriately associate the image with the distance measurement results obtained by the ToF sensor.

[0031] 3, 5, 6, 8, 9, 11, and 12 are schematic diagrams, and the ranging points of the ToF sensor do not necessarily correspond to pixels of the RGB image, and the deviation between the area where the subject existed and the area where the depth position was detected is not recognized on a pixel-by-pixel basis. Also, the ranging points of each bank of the ToF sensor do not necessarily have to be arranged in a grid pattern, and may be arranged, for example, in a vertical or horizontal line.

[0032] In addition, in the above embodiment, an RGB camera is used as the camera and a ToF sensor is used as the ranging sensor, but in other embodiments, other types of cameras such as infrared cameras or other types of ranging sensors such as LiDAR may be used. In the above embodiment, the sensing results of the ranging sensor are associated with the image of the camera, but the embodiments of the present invention are applicable to any combination of sensors (including cameras) as long as the relationship between the frame intervals of frames and subframes is similar. In other embodiments, images from multiple cameras or detection results from multiple sensors may be associated with each other.

[0033] In the above embodiment, the optical flow is generated from the EVS data using the EVS, but the optical flow can also be generated using another camera or sensor with a higher temporal resolution than the camera or sensor to which the images or detection results are associated. For example, the optical flow may be generated using a high frame rate camera instead of the EVS.

[0034] Furthermore, in the above embodiment, the center time of exposure is exemplified as the reference time for the RGB frame and the depth frame. However, in other embodiments, a time other than the center time of exposure may be used as the reference time. For example, if RGB frame data is converted to a high frame rate through post-capture processing, the reference time may be determined from the frame start and end times for each RGB frame data after the high frame rate conversion. Furthermore, for example, if an RGB camera acquires RGB images using a rolling shutter method, the center time of exposure may be used as the reference time for each line included in the RGB frame data to combine the ranging results of the depth frame. In such cases, the reference time when combining the RGB frame data and the ranging results of the depth frame may be different from the center time of exposure during image capture by the RGB camera.

[0035] 10: System, 100: Computer, 110: Processor, 120: Memory, 130: Communication device, 140: Recording medium, 210: RGB camera, 220: dToF sensor, 230: EVS.

Claims

1. A computer system having a memory for storing program code and a processor for executing operations in accordance with the program code, the operations including: fusing first frame data generated at a first interval by a first sensor sensing a space with second frame data generated at a second interval longer than the first interval as a whole, the second frame data including a plurality of subframe data generated at intervals shorter than the first interval by a second sensor sensing the space; selecting at least two of the subframe data each included in a different one of the second frame data based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, the fusing including fusing the second frame data generated from the selected subframe data into the first frame data.

2. The computer system of claim 1, wherein selecting the subframe data includes selecting different combinations of the subframe data for two consecutive first frame data.

3. The computer system of claim 1, wherein the second frame data includes a predetermined number of at least two types of subframe data, and selecting the subframe data includes selecting the subframe data such that the at least two types of subframe data are included in the predetermined number.

4. The computer system of claim 1, wherein selecting the subframe data includes selecting at least one of the subframe data included in the second frame data that is one frame before the first frame data when the reference time of the first frame data is earlier than the reference time of the second frame data, and selecting at least one of the subframe data included in the second frame data that is one frame after the first frame data when the reference time of the first frame data is later than the reference time of the second frame data.

5. The computer system of claim 1, wherein the first sensor is a camera and the second sensor is a distance sensor.

6. The computer system of claim 1, wherein the operation includes: generating an optical flow between a reference time of at least one subframe data included in the second frame data generated from the selected subframe data and a reference time of the first frame data based on a result of sensing the space by a third sensor having a higher temporal resolution than the first sensor and the second sensor; and correcting a position of a sensing result included in the second frame data generated from the selected subframe data according to the optical flow; and wherein the fusing includes fusing the second frame data with the position of the sensing result corrected into the first frame data.

7. A method comprising: fusing first frame data generated at a first interval by a first sensor sensing a space with second frame data generated at a second interval longer than the first interval as a whole, the second frame data including a plurality of subframe data generated at intervals shorter than the first interval by a second sensor sensing the space; selecting at least two of the subframe data each included in a different one of the second frame data based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, the fusing including fusing the second frame data generated from the selected subframe data into the first frame data.

8. A program, the operations of which a processor executes according to the program include: fusing first frame data generated at a first interval by a first sensor sensing a space with second frame data generated at a second interval longer than the first interval as a whole, the second frame data including a plurality of subframe data generated at intervals shorter than the first interval by a second sensor sensing the space; selecting at least two of the subframe data each included in a different one of the second frame data based on a reference time of the first frame data; and generating the second frame data from the selected subframe data, the fusing including fusing the second frame data generated from the selected subframe data into the first frame data.

Images