Information processing device, information processing method, and program
The information processing device addresses the dynamic range difference issue in EVS by extracting saturated regions and correcting event data, enhancing application performance by reducing artifacts and maintaining functionality in saturated areas.
Patent Information
- Application Number
- JP2023079037
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The dynamic range difference between gradation signals and event data in Event-based Vision Sensors (EVS) leads to performance deterioration in applications utilizing both signals, particularly causing artifacts in saturated regions.
An information processing device and method that includes a saturated region extraction unit to identify saturated pixels and an event data correction unit to perform deletion correction on event data within these regions, using techniques like probabilistic deletion based on distance or subject movement to minimize the impact of dynamic range differences.
This approach reduces the number of event data in saturated regions, thereby suppressing performance deterioration and artifacts in applications like Deblur, while maintaining optimal performance levels.
Smart Images

Figure 2024163442000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that are capable of further suppressing performance degradation. [Background technology]
[0002] In recent years, image sensors that detect luminance changes of each pixel as events in real time (hereinafter referred to as EVS (Event based Vision Sensor)) have been developed.
[0003] For example, Patent Document 1 discloses a solid-state imaging device that can obtain a grayscale image in conjunction with detecting an address event. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2020-127186 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, since the grayscale signal used to capture an image has a linear response, whereas the event data acquired by the EVS has a logarithmic response, a difference in dynamic range (hereinafter referred to as a DR (Dynamic Range) difference) occurs between the grayscale signal and the event data. As a result, the EVS may detect the occurrence of an event even in a saturated region where the grayscale signal is saturated.
[0006] Therefore, in an application that uses both grayscale signals and event data, if the DR difference is not taken into consideration, there is a concern that the performance of the application may deteriorate. For example, in a Deblur application that uses both grayscale signals and event data to remove blur, artifacts may occur in the saturated region. Therefore, there is a demand for suppressing such performance deterioration.
[0007] The present disclosure has been made in consideration of such circumstances, and is intended to make it possible to further suppress performance degradation. [Means for solving the problem]
[0008] An information processing device according to one embodiment of the present disclosure includes a saturated region extraction unit that extracts a saturated region occupied by pixels whose gradation signals are saturated, and an event data correction unit that performs deletion correction on event data present in the saturated region.
[0009] An information processing method or program according to one embodiment of the present disclosure includes extracting a saturated region occupied by pixels whose gradation signals are saturated, and performing deletion correction on event data present in the saturated region.
[0010] In one aspect of the present disclosure, a saturated region occupied by pixels whose grayscale signals are saturated is extracted, and deletion correction is performed on event data that exists within the saturated region. [Brief description of the drawings]
[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a DR difference correction processing unit to which the present technology is applied. [Diagram 2] FIG. 13 is a diagram illustrating a deletion correction for event data. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of a sensor. [Figure 4]FIG. 2 is a block diagram showing a configuration example of a hybrid EVS sensor and an application processor. [Diagram 5] 11 is a flowchart illustrating a first processing example of a saturated region extraction process. [Figure 6] 13 is a flowchart illustrating a second processing example of the saturated region extraction processing. [Figure 7] 13 is a flowchart illustrating a first example of an event data correction process. [Figure 8] 13 is a diagram showing an example of DR difference corrected event data in which deletion correction has been performed on all event data present in a saturated region of the gradation signal; FIG. [Figure 9] 13 is a flowchart illustrating a second example of the event data correction process. [Figure 10] 13 is a diagram showing an example of DR difference corrected event data in which probabilistic deletion correction has been performed on event data present in a saturated region of a gradation signal; FIG. [Figure 11] 13 is a flowchart illustrating a third example of the event data correction process. [Figure 12] 11 is a diagram illustrating a process of converting a saturation region into a distance and then converting the distance into an event deletion rate. FIG. [Figure 13] FIG. 13 is a diagram illustrating the relationship between distance and an event deletion rate. [Figure 14] FIG. 13 is a diagram showing an example of DR difference corrected event data subjected to deletion correction based on an event deletion rate according to the distance from the boundary of a saturated region. [Figure 15] 13 is a flowchart illustrating a fourth example of the event data correction process. [Figure 16] 11A and 11B are diagrams illustrating calculation of the movement amount of a subject in a saturated region. [Figure 17] FIG. 13 is a diagram illustrating the relationship between the amount of movement and an event deletion rate. [Figure 18] FIG. 11 is a diagram illustrating a method for checking an input / output response. [Figure 19] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. [Figure 20] It is a block diagram showing an example of a schematic configuration of a vehicle control system. [Figure 21] It is an explanatory diagram showing an example of the installation positions of an external information detection unit and an imaging unit.
Embodiments for Carrying Out the Invention
[0012] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.
[0013] <Configuration Example of DR Differential Correction Processing Unit> FIG. 1 is a block diagram showing a configuration example of an embodiment of a DR differential correction processing unit to which the present technology is applied.
[0014] As shown in FIG. 1, the DR differential correction processing unit 11 includes a saturation region extraction unit 21 and an event data correction unit 22, and a gradation signal and event data are supplied to the DR differential correction processing unit 11. For example, since the frame rate of the event data is significantly higher than that of the gradation signal, the event data of a plurality of frames are included in the gradation signal for one frame.
[0015] The gradation signal is a signal indicating the luminance value of the light received by each of the R pixel, G pixel, and B pixel for photographing an image. The event data is composed of positive events indicating that the luminance has changed in the direction of increasing luminance, negative events indicating that the luminance has changed in the direction of decreasing luminance, or data indicating no event indicating that the luminance value has not changed, for each EVS pixel for detecting the occurrence of an event. Note that the gradation signal and the event data may be input to the DR differential correction processing unit 11 as they are output from the sensor, or may be input to the DR differential correction processing unit 11 after being subjected to signal processing from the sensor.
[0016] As described above, the occurrence of an event may be detected even in the saturation region of the gradation signal due to the DR difference between the gradation signal and the event data. Therefore, the DR difference correction processing unit 11 performs DR difference correction processing on the event data existing in the saturation region of the gradation signal, and outputs the event data after the processing, which is the DR difference corrected event data.
[0017] The saturated region extraction unit 21 determines whether or not the gradation signal supplied to the DR difference correction processing unit 11 is saturated for each pixel, extracts a saturated region that is an area occupied by pixels determined to be saturated, and supplies saturated region information indicating the saturated region to the event data correction unit 22. For example, the saturated region extraction unit 21 can determine that the gradation signal is saturated when the luminance value of any of the R pixel, G pixel, and B pixel is equal to or greater than a threshold (see the flowchart in FIG. 5), or can determine that the gradation signal is saturated when the luminance values of all of the R pixel, G pixel, and B pixel are equal to or greater than a threshold (see the flowchart in FIG. 6).
[0018] The event data correction unit 22 refers to the saturated region information supplied from the saturated region extraction unit 21 and performs deletion correction on event data present in the saturated region of the gradation signal among the event data supplied to the DR difference correction processing unit 11. For example, the event data correction unit 22 can perform deletion correction on all event data present in the saturated region of the gradation signal (see the flowchart in FIG. 7), probabilistic deletion correction on event data present in the saturated region of the gradation signal (see the flowchart in FIG. 9), deletion correction based on an event deletion rate according to the distance from the boundary of the saturated region (see the flowchart in FIG. 11), deletion correction based on an event deletion rate according to the amount of movement of the subject in the saturated region (see the flowchart in FIG. 15), and the like.
[0019] Referring to FIG. 2, the deletion correction of event data by the event data corrector 22 will be described.
[0020] 2A shows an example of the data format of the event data supplied to the event data correction unit 22. As shown in the figure, a positive event (P) is represented by two bits, 01, a negative event (N) is represented by two bits, 10, and no event is represented by two bits, 00. Then, when the event data surrounded by a dashed line among the event data shown in FIG. 2A is subjected to deletion correction, the DR difference corrected event data output from the event data correction unit 22 is shown in FIG. 2B and FIG. 2C.
[0021] An example of the data format of DR difference corrected event data in which deletion correction has been performed on event data by replacing the positive and negative events that were subject to deletion correction with no events is shown in B of Figure 2. As shown in B of Figure 2, in the DR difference corrected event data, one positive event that was subject to deletion correction has been replaced with no events (01 → 00), and three negative events that were subject to deletion correction have been replaced with no events (10 → 00).
[0022] C in Fig. 2 shows an example of the data format of DR difference corrected event data in which deletion correction has been performed on the event data by adding 1-bit deletion information indicating whether or not it is subject to deletion correction. As shown in C in Fig. 2, in the DR difference corrected event data, one positive event that is subject to deletion correction is assigned the deletion information 1 (01→011), and three negative events that are subject to deletion correction are assigned the deletion information 1 (10→101). Event data that is not subject to deletion correction is assigned the deletion information 0, and no event is represented as 000 so that each event data has a fixed length of 3 bits.
[0023] For example, when one bit of deletion information is added as shown in C of FIG. 2, the deletion information is referenced in a downstream processing unit (e.g., application processor 41A shown in A of FIG. 4), and a process of deleting positive events or negative events is performed as necessary.
[0024] An example of the configuration of a sensor that supplies a grayscale signal and event data to the DR difference correction processing unit 11 will be described with reference to FIG.
[0025] As shown in A of Fig. 3, the hybrid EVS sensor 31 is configured such that R pixels that receive red light, G pixels that receive green light, and B pixels that receive blue light are combined with EVS pixels for detecting the occurrence of an event and arranged on the sensor surface, and the gradation signal and event data are output in parallel to the DR difference correction processing unit 11. In the configuration example of the hybrid EVS sensor 31 shown in A of Fig. 3, in the 4 x 4 pixel array, an EVS pixel is arranged at the bottom right of the upper left 2 x 2 pixel array, and an EVS pixel is arranged at the bottom right of the lower right 2 x 2 pixel array.
[0026] 3B, the image sensor 32 has R pixels for receiving red light, G pixels for receiving green light, and B pixels for receiving blue light arranged in a 2×2 pixel array on the sensor surface in a Bayer array, and the EVS sensor 33 has EVS pixels arranged over the entire sensor surface for detecting the occurrence of an event. In this manner, the image sensor 32 and the EVS sensor 33 can be configured to output grayscale signals and event data to the DR difference correction processing unit 11, respectively.
[0027] As shown in Fig. 3C, the switching-type hybrid EVS sensor 34 is configured to switch between R pixels that receive red light, G pixels that receive green light, and B pixels that receive blue light, and EVS pixels for detecting the occurrence of an event by switching, and to output grayscale signals and event data in a time-division manner to the DR difference correction processing unit 11. In the configuration example of the switching-type hybrid EVS sensor 34 shown in Fig. 3C, it is possible to switch between a configuration in which the R pixels, G pixels, and B pixels are arranged on the sensor surface in a 2 x 2 pixel array in a Bayer array, and a configuration in which the EVS pixels are arranged over the entire surface of the sensor surface by switching.
[0028] A configuration example of the hybrid EVS sensor 31 and the application processor 41 will be described with reference to Fig. 4. Note that, although Fig. 4 describes the hybrid EVS sensor 31 shown in Fig. 3A, the same configuration can also be applied to the combination of the image sensor 32 and the EVS sensor 33 shown in Fig. 3B and the switching type hybrid EVS sensor 34 shown in Fig. 3C.
[0029] FIG. 4A shows a block diagram of a hybrid EVS sensor 31A and an application processor 41A according to a first configuration example.
[0030] 4A, the hybrid EVS sensor 31A includes a grayscale signal acquisition unit 51 that acquires grayscale signals output from the R pixels, G pixels, and B pixels, an event data acquisition unit 52 that acquires event data output from the EVS pixels, and a DR difference correction processing unit 11. The grayscale signals and the event data after DR difference correction are supplied from the hybrid EVS sensor 31A to the application processor 41A.
[0031] In this manner, a configuration example in which the DR difference correction processing unit 11 is implemented inside the hybrid EVS sensor 31A can be adopted.
[0032] FIG. 4B shows a block diagram of a hybrid EVS sensor 31B and an application processor 41B according to a second configuration example.
[0033] 4B, the hybrid EVS sensor 31B includes a grayscale signal acquisition unit 51 that acquires grayscale signals output from the R pixels, G pixels, and B pixels, and an event data acquisition unit 52 that acquires event data output from the EVS pixels, and the application processor 41B includes a DR difference correction processing unit 11. The grayscale signals and the event data are supplied from the hybrid EVS sensor 31B to the application processor 41B.
[0034] In this manner, a configuration example can be adopted in which the DR difference correction processing unit 11 is implemented in the application processor 41B provided in the subsequent stage of the hybrid EVS sensor 31B.
[0035] The DR difference correction processing unit 11 configured as above can output DR difference corrected event data that has been subjected to deletion correction for event data that exists within the saturation region of the gradation signal. This reduces the number of event data that exists within the saturation region of the gradation signal, thereby reducing the influence of the DR difference, and as a result, it is possible to suppress performance degradation of applications that utilize both the gradation signal and the event data. For example, in a Deblur application that does not consider the DR difference, it is possible to suppress the occurrence of artifacts that exist within the saturation region of the gradation signal.
[0036] <Example of saturated region extraction processing> The saturated region extraction process executed in the saturated region extraction unit 21 will be described with reference to Figures 5 and 6. For example, the saturated region extraction process can be performed for each predetermined number of pixels to be processed, and in the hybrid EVS sensor 31 shown in Figure 3A, the process can be performed for a 4 x 4 pixel array.
[0037] FIG. 5 is a flowchart illustrating a first processing example of the saturated region extraction processing.
[0038] In step S11, the saturated region extraction unit 21 reads out, from among the gradation signals for one frame acquired by the gradation signal acquisition unit 51, a gradation signal to be processed.
[0039] In step S12, the saturated region extraction unit 21 sets 0 to data indicating the saturated state of the grayscale signal to be processed.
[0040] In step S13, the saturated region extraction unit 21 determines whether or not the luminance value of any of the R pixels, G pixels, and B pixels of the gradation signal to be processed is equal to or greater than a threshold value.
[0041] In step S13, if the saturated region extraction unit 21 determines that the luminance value of any of the R, G, and B pixels of the gradation signal to be processed is equal to or greater than the threshold value, the process proceeds to step S14. In step S14, the saturated region extraction unit 21 sets 1 to data indicating the saturation state of the gradation signal to be processed.
[0042] On the other hand, in step S13, if it is determined that the luminance value of any of the R pixels, G pixels, and B pixels for the gradation signal to be processed is not greater than or equal to the threshold (all luminance values are less than the threshold), or after processing in step S14, the process proceeds to step S15.
[0043] In step S15, the saturated region extraction unit 21 outputs data indicating the saturated state of the gradation signal to be processed, and the saturated region extraction process is terminated. After that, the saturated region extraction unit 21 performs the saturated region extraction process using the gradation signal to be processed next, and by performing the saturated region extraction process for all processing targets for one frame, it is possible to obtain saturated region information consisting of data indicating the saturated state for one frame.
[0044] FIG. 6 is a flowchart illustrating a second processing example of the saturated region extraction processing.
[0045] In steps S21 and S22, processing similar to steps S11 and S12 in FIG. 5 is performed, and in step S23, the saturated region extraction unit 21 determines whether or not all of the luminance values of the R pixels, G pixels, and B pixels of the gradation signal to be processed are equal to or greater than a threshold value.
[0046] If the saturated region extraction unit 21 determines in step S23 that all of the luminance values of the R, G, and B pixels of the gradation signal to be processed are equal to or greater than the threshold value, the process proceeds to step S24. On the other hand, if the saturated region extraction unit 21 determines in step S23 that all of the luminance values of the R, G, and B pixels of the gradation signal to be processed are not equal to or greater than the threshold value (any one of the luminance values is less than the threshold value), the process proceeds to step S25.
[0047] 5, the saturated region extraction process is terminated. After that, the saturated region extraction unit 21 performs the saturated region extraction process using the gradation signal of the next processing target, and performs the saturated region extraction process for all processing targets for one frame, thereby obtaining saturated region information consisting of data indicating the saturated state for one frame.
[0048] <Example of event data correction processing> 7 to 17, the event data correction process executed in the event data corrector 22 will be described. For example, the event data correction process can process the event data one by one.
[0049] FIG. 7 is a flowchart illustrating a first processing example of the event data correction processing.
[0050] In step S31, the event data corrector 22 reads out the saturated region information from the saturated region extractor 21.
[0051] In step S32, the event data corrector 22 reads out from the event data acquirer 52 the event data to be processed.
[0052] In step S33, the event data corrector 22 refers to the saturation region information read in step S31 and determines whether or not the event data read in step S32 is present within a saturation region.
[0053] In step S33, if the event data correcting unit 22 determines that the event data exists in the saturation region, the process proceeds to step S34. In step S34, the event data correcting unit 22 performs deletion correction on the event data to be processed.
[0054] On the other hand, if it is determined in step S33 that the event data does not exist within the saturation region, or after the process of step S34, the process proceeds to step S35.
[0055] In step S35, the event data correction unit 22 outputs the event data, and the event data correction process is terminated. At this time, if the process of step S34 has been performed, the event data in which the positive event or negative event has been replaced with no event as described above with reference to B of Fig. 2, or the event data to which 1 has been added as deletion information as described above with reference to C of Fig. 2, is output. On the other hand, if the process of step S34 has not been performed, the event data read out in step S32 is output as is.
[0056] Thereafter, the event data correction unit 22 performs an event data correction process on the next event data to be processed, and by performing the event data correction process on all event data for one frame, it is possible to obtain event data after DR difference correction for one frame.
[0057] In this manner, in the first processing example of the event data correction process, as shown in FIG. 8, DR difference corrected event data in which deletion correction has been performed on all event data existing in the saturation region is output.
[0058] FIG. 9 is a flowchart illustrating a second processing example of the event data correction processing.
[0059] In steps S41 to S43, the same processes as in steps S31 to S33 in FIG. 7 are performed. If the event data corrector 22 determines in step S43 that the event data exists within the saturation region, the process proceeds to step S44.
[0060] In step S44, the event data corrector 22 generates uniform random numbers between 0.0 and 1.0.
[0061] In step S45, the event data corrector 22 determines whether the value of the random number generated in step S44 is equal to or greater than a threshold value. Here, the threshold value is set according to the deletion rate. For example, when the deletion rate is 50%, the threshold value is set to 0.5.
[0062] In step S45, if the event data corrector 22 determines that the value of the random number is equal to or greater than the threshold value, the process proceeds to step S46. In step S46, the event data corrector 22 performs deletion correction on the event data being processed.
[0063] On the other hand, if it is determined in step S43 that the event data does not exist within the saturation region, if it is determined in step S45 that the value of the random number is not greater than or equal to the threshold value, or after processing in step S46, the process proceeds to step S47.
[0064] Thereafter, in step S47, the same process as in step S35 in FIG. 7 is carried out, and the event data correction process ends.
[0065] In this way, in the second processing example of the event data correction process, DR difference corrected event data in which the event data existing in the saturated region of the gradation signal has been subjected to probabilistic deletion correction is output. For example, Fig. 10 shows an example of DR difference corrected event data in which the event data existing in the saturated region of the gradation signal has been subjected to deletion correction at a deletion rate of 50%.
[0066] Here, in an application that utilizes both grayscale signals and event data, by reducing the number of event data present in the saturation region of the grayscale signal, it is possible to suppress performance degradation in some performances, but when deletion correction is performed on all event data present in the saturation region of the grayscale signal, performance degradation may occur in other performances. For example, in a Deblur application, by reducing the number of event data present in the saturation region of the grayscale signal, it is possible to suppress the occurrence of artifacts in the saturation region, but when deletion correction is performed on all event data present in the saturation region of the grayscale signal, Deblur performance in the saturation region may deteriorate.
[0067] Therefore, by performing probabilistic deletion correction on event data that exists within the saturated region of the gradation signal, as in the second processing example of the event data correction process, it is possible to adjust the trade-off in the performance of applications that utilize both the gradation signal and the event data.
[0068] For example, in a Deblur application, suppression of artifact occurrence and suppression of deterioration of Deblur performance can be adjusted by controlling the deletion rate for event data present in the saturation region of the gradation signal. In other words, when the deletion rate is 100%, no artifacts occur but the Deblur performance is reduced. When the deletion rate is 50%, artifacts occur at a medium level and the Deblur performance can also be set to a medium level. When the deletion rate is 0% (i.e., DR difference correction processing is not performed), many artifacts occur but the Deblur performance can be improved.
[0069] In this way, it is necessary to appropriately perform deletion correction on event data that exists within the saturation region of the gradation signal, depending on the performance required for an application that utilizes both the gradation signal and the event data.
[0070] FIG. 11 is a flowchart illustrating a third example of the event data correction process.
[0071] In steps S51 to S53, the same processes as in steps S31 to S33 in FIG. 7 are performed. If the event data corrector 22 determines in step S53 that the event data exists in the saturation region, the process proceeds to step S54.
[0072] In step S54, the event data corrector 22 converts the saturated region into a distance by referring to the saturated region information acquired in step S51, and in step S55, the event data corrector 22 converts the distance into an event deletion rate.
[0073] For example, as shown in Fig. 12, in the saturated region information, pixels that are saturated are set to 1, and pixels that are not saturated are set to 0. First, the event data correction unit 22 calculates the distance (the number of pixels from the boundary) from the boundary of the saturated region for each pixel in the saturated region occupied by pixels whose gradation signals are saturated, and converts the saturated region into a distance. Then, the event data correction unit 22 converts the distance into an event removal rate for each pixel in the saturated region such that the event removal rate is lower if the pixel is closer to the boundary of the saturated region, and higher if the pixel is farther from the boundary of the saturated region.
[0074] For example, the event data corrector 22 can convert the distance into an event deletion rate according to the relationship between the distance and the event deletion rate as shown in Fig. 13. For example, if the distance is less than the parameter d0, the distance is converted into an event deletion rate of 0.0, if the distance is equal to or greater than the parameter d0 and less than the parameter d1, the distance is linearly converted into an event deletion rate between 0.0 and 1.0, and if the distance is equal to or greater than the parameter d1, the distance is converted into an event deletion rate of 1.0. In this way, the event deletion rate can be controlled by the parameters d0 and d1.
[0075] Thereafter, in steps S56 to S59, the same processes as in steps S44 to S47 in FIG. 9 are performed, and the event data correction process ends.
[0076] In this way, in the third processing example of the event data correction process, as shown in Figure 14, DR difference corrected event data is output, which has been subjected to deletion correction based on an event deletion rate according to the distance from the boundary of the saturated region.
[0077] Therefore, for example, when used in a Deblur application, it is possible to leave positive or negative events near the saturation boundary, which is likely to affect Deblur performance, while increasing the rate at which positive or negative events are deleted as the image approaches the center of the saturation region, which is likely to affect artifacts. This allows the Deblur application to perform optimally.
[0078] FIG. 15 is a flowchart illustrating a fourth example of the event data correction process.
[0079] In steps S61 to S63, the same processes as in steps S31 to S33 in FIG. 7 are performed. If the event data corrector 22 determines in step S63 that the event data exists within the saturation region, the process proceeds to step S64.
[0080] In step S64, the event data correcting unit 22 calculates the amount of movement of the subject in the saturated region by referring to the saturated region information acquired in step S61, and in step S65, the event data correcting unit 22 converts the amount of movement into an event deletion rate.
[0081] For example, as shown in Fig. 16, the event data correction unit 22 extracts event data corresponding to the exposure start timing of the gradation signal and event data corresponding to the exposure end timing of the gradation signal. Then, the event data correction unit 22 calculates the center of gravity of the event existing in the saturation region for each, and regards the movement amount of the center of gravity from the start of exposure of the gradation signal to the end of exposure as the movement amount of the object. At this time, as shown in Fig. 16, if the object moves at a low speed, the movement amount of the center of gravity is small, and if the object moves at a high speed, the movement amount of the center of gravity is large. Then, for each pixel in the saturation region, the event data correction unit 22 converts the movement amount of the object into an event deletion rate so that if the movement amount of the object is small, the event deletion rate is high, and if the movement amount of the object is large, the event deletion rate is low.
[0082] For example, the event data correction unit 22 can convert the movement amount into an event deletion rate according to the relationship between the movement amount and the event deletion rate as shown in Fig. 17. For example, if the movement amount is less than the parameter d0, the movement amount is converted into an event deletion rate of 1.0, if the distance is equal to or greater than the parameter d0 and less than the parameter d1, the movement amount is linearly converted into an event deletion rate between 0.0 and 1.0, and if the distance is equal to or greater than the parameter d1, the movement amount is converted into an event deletion rate of 0.0. In this way, the event deletion rate can be controlled by the parameters d0 and d1.
[0083] Thereafter, in steps S66 to S69, the same processes as in steps S44 to S47 in FIG. 9 are performed, and the event data correction process ends.
[0084] In this manner, in the fourth processing example of the event data correction process, DR difference corrected event data that has been subjected to deletion correction based on the event deletion rate according to the amount of movement of the subject in the saturated region is output.
[0085] For example, when used in the Deblur application, artifacts tend to appear more easily when the subject is moving slowly compared to when the subject is moving quickly. Therefore, by increasing the rate at which positive or negative events are deleted when the subject is moving slowly, artifacts can be suppressed and the performance of the Deblur application can be properly demonstrated.
[0086] <How to check input / output response> A method for checking an input / output response will be described with reference to FIG.
[0087] First, a light source having an internal texture is prepared, and the hybrid EVS sensor 31 captures an image while moving the light source at a constant speed to obtain a grayscale signal and event data. At this time, the hybrid EVS sensor 31 captures an image multiple times while changing the luminance value of the grayscale signal, for example, by changing the illuminance of the light source. The luminance value of the grayscale signal acquired by the hybrid EVS sensor 31 may be changed by changing the exposure time of the pixel for acquiring the grayscale signal. The hybrid EVS sensor 31 is configured to capture an image even when the grayscale signal is saturated.
[0088] Then, when the gradation signal and event data obtained by photographing the light source in this manner are input, the number of events in the light source section (the part where the light source is photographed by the hybrid EVS sensor 31) is used as the output, and a difference can be confirmed between the output when the DR difference correction process is not performed and the output when the DR difference correction process is performed, as shown in Figure 18.
[0089] 18A shows the relationship between the illuminance of the light source and the number of events of the light source unit when the DR difference correction process is not performed. As shown in the figure, when the DR difference correction process is not performed, even if the illuminance of the light source is increased until the grayscale signal is saturated, it is confirmed that there is no change that reduces the number of events of the light source unit.
[0090] 18B shows the relationship between the illuminance of the light source and the number of events of the light source unit when the DR difference correction process is being performed. As shown in the figure, when the DR difference correction process is being performed, it is confirmed that when the illuminance of the light source is increased until the gradation signal is saturated, a change occurs such that the number of events of the light source unit is reduced.
[0091] In this way, it can be confirmed that the input / output response differs depending on whether or not the DR difference correction process is performed.
[0092] <Computer configuration example> Next, the above-mentioned series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.
[0093] FIG. 19 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.
[0094] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) 104 are interconnected by a bus 105. An input / output interface 106 is further connected to the bus 105, and the input / output interface 106 is connected to the outside.
[0095] In the computer configured as above, the series of processes described above is performed by the CPU 101, for example, loading the programs stored in the ROM 102 and EEPROM 104 into the RAM 103 via the bus 105 and executing them. Furthermore, the programs executed by the computer (CPU 101) can be written in advance into the ROM 102, or can be installed or updated in the EEPROM 104 from the outside via the input / output interface 106.
[0096] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).
[0097] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.
[0098] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.
[0099] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0100] Furthermore, for example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.
[0101] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.
[0102] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.
[0103] In addition, the processing of the steps of the program executed by the computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a necessary timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the processing of the steps of the program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0104] In addition, the present technologies described in this specification can be implemented independently and individually, as long as there is no contradiction. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.
[0105] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.
[0106] Fig. 20 is a block diagram showing a schematic configuration example of a vehicle control system 7000 which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in Fig. 20, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark).
[0107] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various control target devices. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle by wired communication or wireless communication. In FIG. 20, a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690 are illustrated as functional configurations of the integrated control unit 7600. Other control units also include a microcomputer, a communication I / F, a storage unit, and the like.
[0108] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle according to various programs. For example, the drivetrain control unit 7100 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle. The drivetrain control unit 7100 may also function as a control device such as an ABS (Antilock Brake System) or an ESC (Electronic Stability Control).
[0109] A vehicle state detection unit 7110 is connected to the drive system control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor for detecting the amount of operation of an accelerator pedal, the amount of operation of a brake pedal, the steering angle of a steering wheel, the engine rotation speed, or the rotation speed of wheels, for example. The drive system control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.
[0110] The body system control unit 7200 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 7200. The body system control unit 7200 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.
[0111] The battery control unit 7300 controls the secondary battery 7310, which is a power supply source for the drive motor, according to various programs. For example, information such as battery temperature, battery output voltage, or remaining capacity of the battery is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs calculation processing using these signals, and controls the temperature regulation of the secondary battery 7310 or controls a cooling device or the like equipped in the battery device.
[0112] The outside-vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside-vehicle information detection unit 7420 is connected to the outside-vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside-vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, and the like around the vehicle equipped with the vehicle control system 7000.
[0113] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may be provided as independent sensors or devices, or may be provided as a device in which multiple sensors or devices are integrated.
[0114] Here, FIG. 21 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield in the vehicle interior of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield in the vehicle interior mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield in the vehicle interior is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.
[0115] 21 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, image data captured by the imaging units 7910, 7912, 7914, and 7916 are superimposed to obtain an overhead image of the vehicle 7900.
[0116] The outside information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, and corners of the vehicle 7900 and on the upper part of the windshield inside the vehicle cabin may be, for example, ultrasonic sensors or radar devices. The outside information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and on the upper part of the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These outside information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, and the like.
[0117] Returning to FIG. 20, the description will be continued. The outside-vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle, and receives the captured image data. The outside-vehicle information detection unit 7400 also receives detection information from the outside-vehicle information detection unit 7420 connected thereto. When the outside-vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside-vehicle information detection unit 7400 transmits ultrasonic waves or electromagnetic waves, and receives information on the received reflected waves. The outside-vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, or characters on the road surface, based on the received information. The outside-vehicle information detection unit 7400 may perform environment recognition processing for recognizing rainfall, fog, road surface conditions, and the like, based on the received information. The outside-vehicle information detection unit 7400 may calculate the distance to an object outside the vehicle based on the received information.
[0118] Furthermore, the outside vehicle information detection unit 7400 may perform image recognition processing or distance detection processing to recognize people, cars, obstacles, signs, or characters on the road surface, based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or position adjustment on the received image data, and may generate an overhead image or a panoramic image by synthesizing image data captured by different imaging units 7410. The outside vehicle information detection unit 7400 may perform viewpoint conversion processing using image data captured by different imaging units 7410.
[0119] The in-vehicle information detection unit 7500 detects information inside the vehicle. For example, a driver state detection unit 7510 that detects the state of the driver is connected to the in-vehicle information detection unit 7500. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the bioinformation of the driver, or a microphone that collects sound in the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the bioinformation of a passenger sitting in the seat or a driver gripping the steering wheel. The in-vehicle information detection unit 7500 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing. The in-vehicle information detection unit 7500 may perform processing such as noise canceling processing on the collected sound signal.
[0120] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. The input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is realized by a device that can be operated by an occupant to input, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of a voice input by a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a PDA (Personal Digital Assistant) that supports the operation of the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the occupant can input information by gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the occupant using the above-mentioned input unit 7800 and outputs the input signal to the integrated control unit 7600. A passenger or the like operates the input unit 7800 to input various data to the vehicle control system 7000 and to instruct processing operations.
[0121] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may be realized by a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device, etc.
[0122] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication between various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (registered trademark), WiMAX (registered trademark), LTE (registered trademark) (Long Term Evolution) or LTE-Advanced (LTE-A), or other wireless communication protocols such as wireless LAN (also called Wi-Fi (registered trademark)) and Bluetooth (registered trademark). The general-purpose communication I / F 7620 may connect to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via, for example, a base station or an access point. The general-purpose communication I / F 7620 may also connect to a terminal (e.g., a driver's, pedestrian's, or store's terminal, or a Machine Type Communication (MTC) terminal) present in the vicinity of the vehicle using, for example, a Peer To Peer (P2P) technology.
[0123] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in a vehicle. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of a lower layer IEEE802.11p and a higher layer IEEE1609, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.
[0124] The positioning unit 7640 performs positioning by receiving, for example, a GNSS signal from a Global Navigation Satellite System (GNSS) satellite (for example, a GPS signal from a Global Positioning System (GPS) satellite) and generates position information including the latitude, longitude, and altitude of the vehicle. The positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone having a positioning function.
[0125] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closure, required time, etc. The function of the beacon receiving unit 7650 may be included in the dedicated communication I / F 7630 described above.
[0126] The in-vehicle device I / F 7660 is a communication interface that mediates a connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). In addition, the in-vehicle device I / F 7660 may establish a wired connection such as a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI (registered trademark), or a Mobile High-definition Link (MHL)) via a connection terminal (and a cable, if necessary) not shown. The in-vehicle device 7760 may include, for example, at least one of a mobile device or a wearable device owned by a passenger, or an information device carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.
[0127] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.
[0128] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired through at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate a control target value of a driving force generating device, a steering mechanism, or a braking device based on the acquired information inside and outside the vehicle, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 may perform cooperative control for the purpose of realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.
[0129] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including peripheral information of the current position of the vehicle, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. The microcomputer 7610 may also predict dangers such as vehicle collisions, the approach of pedestrians, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.
[0130] The audio / image output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of FIG. 20, an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as output devices. The display unit 7720 may include at least one of an on-board display and a head-up display, for example. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may be other devices such as headphones, a wearable device such as a glasses-type display worn by the passenger, a projector, or a lamp, other than these devices. When the output device is a display device, the display device visually displays the results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats such as text, image, table, graph, etc. When the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or acoustic data into an analog signal and audibly outputs it.
[0131] In the example shown in FIG. 20, at least two control units connected via the communication network 7010 may be integrated into one control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In the above description, some or all of the functions performed by any control unit may be provided by another control unit. In other words, as long as information is transmitted and received via the communication network 7010, a predetermined arithmetic processing may be performed by any control unit. Similarly, a sensor or device connected to any control unit may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.
[0132] The DR difference correction processing unit 11 according to this embodiment described using FIG. 1 can be applied to the application example 7410 and the outside-vehicle information detection unit 7420 shown in FIG. 20, and the outside-vehicle information detection unit 7400 can detect information outside the vehicle based on the DR difference corrected event data.
[0133] <Examples of configuration combinations> The present technology can also be configured as follows. (1) a saturated region extraction unit that extracts a saturated region occupied by pixels whose gradation signals are in a saturated state; an event data correction unit that performs deletion correction on event data present within the saturation region; An information processing device comprising: (2) The saturated region extraction unit determines that the gradation signal is in a saturated state when any one of the luminance values of the R pixel, the G pixel, and the B pixel indicated by the gradation signal is equal to or greater than a threshold value. The information processing device according to (1) above. (3) The saturated region extraction unit determines that the gradation signal is in a saturated state when all of the luminance values of the R pixels, the G pixels, and the B pixels indicated by the gradation signal are equal to or greater than a threshold value. The information processing device according to (1) above. (4) The event data correction unit performs deletion correction on all of the event data present within the saturation region. An information processing device according to any one of (1) to (3) above. (5) The event data correction unit performs a probabilistic deletion correction on the event data present in the saturation region. An information processing device according to any one of (1) to (3) above. (6) The event data correction unit performs deletion correction on the event data based on an event deletion rate according to a distance from a boundary of the saturated region. An information processing device according to any one of (1) to (3) above. (7) The event data correction unit performs deletion correction on the event data based on an event deletion rate according to an amount of movement of the subject in the saturated region. An information processing device according to any one of (1) to (3) above. (8) the event data is comprised of data indicative of a positive event, a negative event, or no event; The event data correction unit replaces the data indicating the positive event and the negative event that are the subject of the deletion correction with data indicating the absence of the event. An information processing device according to any one of (1) to (7) above. (9) the event data is comprised of data indicative of a positive event, a negative event, or no event; The event data correction unit adds deletion information indicating that the data indicating the positive event and the negative event that are the subject of deletion correction to the data indicating the positive event and the negative event that are the subject of deletion correction. An information processing device according to any one of (1) to (7) above. (10) The grayscale signal and the event data are output in parallel from a sensor arranged on a sensor surface in which pixels for acquiring the grayscale signal and pixels for acquiring the event data are combined. An information processing device according to any one of (1) to (9) above. (11) The grayscale signal and the event data are output from a first sensor, the sensor surface of which includes pixels for acquiring the grayscale signal, and a second sensor, the sensor surface of which includes pixels for acquiring the event data. An information processing device according to any one of (1) to (9) above. (12) The grayscale signal and the event data are output in a time-division manner from a sensor in which pixels for acquiring the grayscale signal and pixels for acquiring the event data are switched by switching. An information processing device according to any one of (1) to (9) above. (13) The grayscale signal and the event data are implemented inside a sensor that acquires the grayscale signal and the event data. An information processing device according to any one of (1) to (12) above. (14) The signal processing unit is installed in a downstream of a sensor that acquires the grayscale signal and the event data. An information processing device according to any one of (1) to (12) above. (15) An information processing device, Extracting a saturated region occupied by pixels whose grayscale signals are saturated; performing a deletion correction on event data present within the saturation region; An information processing method comprising: (16) The computer of the information processing device Extracting a saturated region occupied by pixels whose grayscale signals are saturated; performing a deletion correction on event data present within the saturation region; A program for executing information processing including the above.
[0134] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present disclosure. In addition, the effects described in this specification are merely examples and are not limiting, and other effects may be obtained. [Explanation of symbols]
[0135] 11 DR difference correction processing unit, 21 saturated region extraction unit, 22 event data correction unit, 31 hybrid EVS sensor, 32 image sensor, 33 EVS sensor, 34 switching type hybrid EVS sensor, 41 application processor, 51 grayscale signal acquisition unit, 52 event data acquisition unit
Claims
1. a saturated region extraction unit that extracts a saturated region occupied by pixels whose gradation signals are in a saturated state; an event data correction unit that performs deletion correction on event data present within the saturation region; An information processing device comprising:
2. The saturated region extraction unit determines that the gradation signal is in a saturated state when any one of the luminance values of the R pixel, the G pixel, and the B pixel indicated by the gradation signal is equal to or greater than a threshold value. The information processing device according to claim 1 .
3. The saturated region extraction unit determines that the gradation signal is in a saturated state when all of the luminance values of the R pixels, the G pixels, and the B pixels indicated by the gradation signal are equal to or greater than a threshold value. The information processing device according to claim 1 .
4. The event data correction unit performs deletion correction on all of the event data present within the saturation region. The information processing device according to claim 1 .
5. The event data correction unit performs a probabilistic deletion correction on the event data present in the saturation region. The information processing device according to claim 1 .
6. The event data correction unit performs deletion correction on the event data based on an event deletion rate according to a distance from a boundary of the saturated region. The information processing device according to claim 1 .
7. The event data correction unit performs deletion correction on the event data based on an event deletion rate according to an amount of movement of the subject in the saturated region. The information processing device according to claim 1 .
8. the event data is comprised of data indicative of a positive event, a negative event, or no event; The event data correction unit replaces the data indicating the positive event and the negative event that are the subject of the deletion correction with data indicating the absence of the event. The information processing device according to claim 1 .
9. the event data is comprised of data indicative of a positive event, a negative event, or no event; The event data correction unit adds deletion information indicating that the data indicating the positive event and the negative event that are the subject of deletion correction to the data indicating the positive event and the negative event that are the subject of deletion correction. The information processing device according to claim 1 .
10. The gray scale signal and the event data are output in parallel from a sensor arranged on a sensor surface in which pixels for acquiring the gray scale signal and pixels for acquiring the event data are combined. The information processing device according to claim 1 .
11. The grayscale signal and the event data are output from a first sensor having pixels arranged on a sensor surface for acquiring the grayscale signal, and a second sensor having pixels arranged on a sensor surface for acquiring the event data. The information processing device according to claim 1 .
12. The grayscale signal and the event data are output in a time-division manner from a sensor in which pixels for acquiring the grayscale signal and pixels for acquiring the event data are switched by switching. The information processing device according to claim 1 .
13. The grayscale signal and the event data are implemented inside a sensor that acquires the grayscale signal and the event data. The information processing device according to claim 1 .
14. The signal processing unit is installed in a downstream of a sensor that acquires the grayscale signal and the event data. The information processing device according to claim 1 .
15. An information processing device, Extracting a saturated region occupied by pixels whose grayscale signals are saturated; performing a deletion correction on event data present within the saturation region; An information processing method comprising:
16. The computer of the information processing device Extracting a saturated region occupied by pixels whose grayscale signals are saturated; performing a deletion correction on event data present within the saturation region; A program for executing information processing including the above.
Citation Information
Patent Citations
Solid-state imaging system and imaging apparatus
JP2020127186A
