Image processing device, image processing method, and program
Patent Information
- Application Number
- PCT/CN2023/095711
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-28
Smart Images

Figure CN2023095711_28112024_PF_FP_ABST
Abstract
Description
IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM[Technical Field]
[0001] The present disclosure relates to an image processing device, an image processing method, and a program.[Background Art]
[0002] There is a method of detecting an object by using image data and event data of a subject. An object may be detected by using event data even when the object cannot be easily detected in image data.[Disclosure of Invention]
[0003] [Problem to be Solved by the Invention]
[0004] However, even if the event data is used, there are a case where the use of event data has a good result and a case where the non-use of event data has a good result, it is difficult for a user to determine a shooting scene for which the event data should be used.
[0005] The present disclosure has been made in view of the above-described problem, and an aim of the present disclosure is to provide an image processing device, an image processing method, and a program, which can recommend a shooting scene to use event data.
[0006] [Means for Solving Problem]
[0007] To solve the problem described above and achieve an aim of the present disclosure, an image processing device according to one aspect of the present disclosure includes a first acquiring means configured to acquire image data; a second acquiring means configured to acquire event data; and a determining means configured to determine whether a process using the event data is effective based on the image data acquired by the first acquiring means.
[0008] [Effect of the Invention]
[0009] According to one aspect of the present disclosure, it can recommend a shooting scene to use event data.[Brief Description of Drawings]
[0010] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing device according to an embodiment.
[0011] FIG. 2 is a diagram illustrating an example of a functional block configuration of a processing unit in the image processing device according to the embodiment.
[0012] FIG. 3 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device according to the embodiment.
[0013] FIG. 4 is a diagram illustrating an example of an effectiveness determination process flow in the image processing device according to the embodiment.
[0014] FIG. 5 is a diagram illustrating an example of a configuration of the image processing device according to a first modification example of the embodiment.
[0015] FIG. 6 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device according to the first modification example of the embodiment.
[0016] FIG. 7 is a diagram illustrating an example of a configuration of the image processing device according to a second modification example of the embodiment.
[0017] FIG. 8 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device according to the second modification example of the embodiment.
[0018] FIG. 9 is a diagram illustrating an example of a configuration of the image processing device according to a third modification example of the embodiment.
[0019] FIG. 10 is a diagram illustrating an example of a effectiveness determination process flow in the image processing device according to the third modification example of the embodiment.
[0020] FIG. 11 is a diagram illustrating an example of a configuration of the image processing device according to a fourth modification example of the embodiment.
[0021] FIG. 12 is a diagram illustrating a display example of "recommended information" in the image processing device according to the fourth modification example of the embodiment.
[0022] [Embodiment (s) of Carrying Out the Invention]
[0023] Hereinafter, an image processing device, an image processing method, and a program according to an embodiment will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiment.
[0024] Embodiment
[0025] An image processing device according to an embodiment is an image processing device configured to acquire and process output data of a frame camera and an event camera. The image processing device according to the embodiment can be applied to an image processing device that can acquire output data output from a frame camera and an event camera. As an example, the image processing device according to the embodiment is applicable to a smartphone or a tablet type portable terminal. Moreover, the image processing device is also applicable to an on-vehicle device that detects an object.
[0026] Overall Configuration of Image Processing Device
[0027] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing device 1 according to the embodiment. The image processing device 1 includes an image acquiring unit 10, an event acquiring unit 20, a processing unit 30, a display module 40, a memory 50, an input module 60, a GNSS module 3, a wireless communication module 4, a speaker 5, a microphone 6, a CODEC 7, an IMU (inertial measurement unit) 8, a bus 9, an antenna 3-1, and an antenna 3-2. The processing unit 30 includes an ISP (image signal processor) 31 and a processor 32. As an example, the embodiment illustrates the hardware configuration of a smartphone, but the present invention is not limited to this.
[0028] The ISP 31, the processor 32, the display module 40, the memory 50, the input module 60, the GNSS module 3, the wireless communication module 4, the CODEC 7, and the IMU 8 are connected via the bus 9 to be able to communicate with one another. The speaker 5 and the microphone 6 are connected to the bus 9 via the CODEC 7. The antenna 3-1 is connected to the bus 9 via the GNSS module 3. The antenna 3-2 is connected to the bus 9 via the wireless communication module 4. The wireless communication module 4 communicates with a server 900 via a communication network 800.
[0029] The image acquiring unit 10 is a camera, and is also referred to as a frame camera. The image acquiring unit 10 outputs frame-based pixel signals to the processing unit 30 in synchronization with the clock of the processing unit 30.
[0030] As an example, the image acquiring unit 10 includes an optical system 11, an image sensor 12, an image sensor driver 13, a focus driver 14, and a focus actuator 15.
[0031] The optical system 11 includes a lens 111 and a lens 112. The lens 111 is arranged on the subject side. The lens 112 can move in the direction of an optical axis 100. The focus driver 14 moves the lens 112 in the direction of the optical axis 100 via the focus actuator 15 under the control of the processing unit 30 to focus the optical system 11. Note that the lens 111 and the lens 112 are illustrated as the optical system 11 but the optical system 11 may be configured of a plurality of lens groups.
[0032] The light is injected from a shooting range by the first lens of the lenses provided in the optical system 11, and the emitted light from the final lens forms an image on a pixel array surface 121 of the image sensor 12.
[0033] The image sensor 12 is driven by the image sensor driver 13 under the control of the processing unit 30. The image sensor 12 photo-electrically converts light at each pixel of the pixel array surface 121 having two-dimensional array pixels, and outputs pixel signals for the shooting range output from the pixels to the processing unit 30.
[0034] The image acquiring unit 10 may be a color image sensor configured to acquire a color image such as an RGB (Red, Green, Blue) image, or may be a monochromatic image sensor configured to acquire a monochromatic image.
[0035] The event acquiring unit 20 is a sensor configured to acquire an event per pixel, and is also referred to as an event camera. The event acquiring unit 20 detects pixels with the brightness change by an event detecting unit. The brightness change is detected by time resolution having the order of microseconds (μs) . The event acquiring unit 20 asynchronously outputs events of the pixels with the brightness change to the processing unit 30. The event indicates the coordinate of the pixels, the polarity of the brightness change, the time, and the like, as an example.
[0036] As an example, the event acquiring unit 20 includes an optical system 21, an image sensor 22, an image sensor driver 23, a focus driver 24, and a focus actuator 25.
[0037] The optical system 21 includes a lens 211 and a lens 212. The lens 211 is arranged on the subject side. The lens 212 can move in the direction of an optical axis 200. The focus driver 24 moves the lens 212 in the direction of the optical axis 200 via the focus actuator 25 under the control of the processing unit 30 to focus the optical system 21. Note that the lens 211 and the lens 212 are illustrated as the optical system 21 but the optical system 21 may be configured of a plurality of lens groups.
[0038] The light is injected from the shooting range by the first lens of the optical system 21, and the emitted light from the final lens forms an image on a pixel array surface 221 of the image sensor 22.
[0039] The image sensor 22 is driven by the image sensor driver 23 under the control of the processing unit 30.
[0040] The image sensor 22 includes the event detecting unit, and detects pixels for which a change in a brightness level of the received light exceeds a threshold and asynchronously outputs events of the pixels to the processing unit 30.
[0041] Note that the example of the hardware configuration according to the present embodiment illustrates the optical axis 100 and the optical axis 200 having the same direction. Moreover, because the focus of the event acquiring unit 20 is controlled in accordance with the movement to focus the image acquiring unit 10, the angle of view of the optical system 11 and the angle of view of the optical system 21 correspond to each other.
[0042] The processing unit 30 includes the ISP 31 and the processor 32. The ISP 31 mainly performs processing by hardware. The processor 32 mainly performs processing by software.
[0043] The ISP 31 performs signal processing on frame-based pixel signals output from the image acquiring unit 10 to generate frame-by-frame image data. Moreover, the ISP 31 performs signal processing on the events output from the event acquiring unit 20 to generate event data.
[0044] The processor 32 acquires the image data and the event data generated by the ISP 31 and performs predetermined processing on these data.
[0045] The display module 40 displays information on a display screen 41. The information includes an image for display of the image data, information to perform setting and an instruction, notification information, and the like. As an example, the display screen 41 is a micro LED, an LCD (liquid crystal display) , an organic light-emitting diode, or the like.
[0046] The input module 60 is a user interface to input an operation instruction, a set value, and the like. As an example, the input module 60 is an operating panel, hardware input keys, a touch panel, or the like.
[0047] The memory 50 is a memory such as ROM (read only memory) and RAM (random access memory) . The memory 50 stores therein a program, data, and the like.
[0048] Note that the image acquiring unit 10 and the event acquiring unit 20 may be unified as one camera. For example, one camera may acquire image data on a frame basis and simultaneously acquire events, and output frame-based pixel signals and event data indicating the generated events.
[0049] Functional Blocks
[0050] FIG. 2 is a diagram illustrating an example of a functional block configuration of the processing unit 30 in the image processing device 1 according to the embodiment. As illustrated in FIG. 2, the processing unit 30 includes an image data acquiring module 301, an event data acquiring module 302, an effectiveness determining module 303, and an output processing module 304.
[0051] The image data acquiring module 301 acquires frame-by-frame image data.
[0052] The event data acquiring module 302 acquires event data.
[0053] The effectiveness determining module 303 determines whether a process using the event data is effective.
[0054] The output processing module 304 performs an output process according to a determination result of the effectiveness determining module 303. Herein, the output process is an output process to be performed on the image data acquired by the image data acquiring module 301, and the output process means an image correction process, an object detection process, or the like, as an example.
[0055] As an example, the effectiveness determining module 303 determines whether the process using the event data is effective based on an amount of edges included in the image data.
[0056] The event data is data consisting of events of pixels with the brightness change. The event data has time resolution higher than the image data of the predetermined frame rate by the image acquiring unit 10, and catches a slight change in brightness that occurs in theμs order between these frames. If the amount of the edges is not less than a predetermined threshold, it is conceivable that the event data to be generated at the edges has a sufficient amount. By using this approach, when the amount of the edges is not less than the predetermined threshold, the effectiveness determining module 303 determines that the process using the event data is effective.
[0057] For example, the effectiveness determining module 303 performs an edge extraction process with reference to the image data acquired by the image data acquiring module 301, and determines whether the process using the event data is effective from the amount of the edges of the subject obtained by the edge extraction process.
[0058] Note that, when an amount of the event data to be generated at the edge is not sufficient, the event data acts as noise if the event data is used. For this reason, it is desirable that the threshold is previously adjusted to a value at which the output process becomes effective.
[0059] The edge extraction process extracts edges for the whole image from the image data acquired by the image data acquiring module 301. The extraction method of the edges includes a Sobel filter, Hough transform, or the like, as an example.
[0060] One of a ratio of an area occupied by the edges to the whole image, a total length obtained by extending the extracted edges, and the number of the edges may be used as the amount of the edges.
[0061] The ratio of the area occupied by the edges to the whole image is a ratio of total number of pixels corresponding to the extracted edges to total number of pixels of the whole image, for example.
[0062] The total length obtained by extending the extracted edges is a total length obtained by calculating the lengths for the extracted edges and adding the lengths of the edges.
[0063] The number of the edges is the number of edges extracted from the whole image.
[0064] FIG. 3 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device 1 according to the embodiment. For example, as illustrated in FIG. 3, it is assumed that the edge extraction process is performed on image data 1000 illustrated in (a) of FIG. 3. (b) of FIG. 3 is an image diagram after an edges is extracted. In the case of the image data 1000 illustrated in (a) of FIG. 3, edges 1001 as illustrated by a line drawing in (b) of FIG. 3 is extracted. Because sufficient-amount event data is highly likely generated in the case of such the subject, it is determined that the use of the event data is effective if an amount of the edges 1001 is not less than the threshold.
[0065] Effectiveness Determination Process Flow
[0066] FIG. 4 is a diagram illustrating an example of an effectiveness determination process flow in the image processing device 1 according to the embodiment. As illustrated in FIG. 4, the processing unit 30 acquires image data from the image acquiring unit 10 (Step S1) .
[0067] Moreover, the processing unit 30 acquires event data from the event acquiring unit 20 (Step S2) .
[0068] Next, based on the image data acquired in Step S1, the processing unit 30 determines whether the process using the event data is effective.
[0069] As an example, the processing unit 30 extracts edges from the image data acquired in Step S1, and determines whether an amount of the extracted edges is not less than a predetermined threshold (Step S3) .
[0070] When the amount of the edges is not less than the predetermined threshold (Step S3: Yes) , the processing unit 30 determines that the process using the event data is effective and sets the use of the event data to "recommendation" (Step S4) .
[0071] On the other hand, when the amount of the edges is less than the predetermined threshold (Step S3: No) , the processing unit 30 determines that the process using the event data is not effective and sets the use of the event data to "non-recommendation" (Step S5) .
[0072] After that, the output processing module 304 performs the output process depending on "recommendation" or "non-recommendation" that is the determination result of the effectiveness determination process. In other words, the output processing module 304 performs the output process using the event data when the process using the event data is "recommendation" , and performs the output process without using the event data when the process using the event data is "non-recommendation" .
[0073] Note that the effectiveness determination process flow of Step S1 to Step S5 may be repeated for each frame, may be performed every plural frames, may be performed in a predetermined shooting state, or may be performed at an arbitrary timing.
[0074] As described above, the image processing device 1 according to the present embodiment determines whether the process using the event data is effective from the amount of the edges of the subject, and recommends the event data and performs the output process when the amount of the edges is not less than the predetermined threshold. On the other hand, in a shooting scene for which the use of the event data is not suitable, the image processing device 1 can perform the output process without using the event data. As described above, the image processing device 1 can recommend a shooting scene to use the event data.
[0075] In the image processing device 1 according to the embodiment, the acquiring unit configured to acquire the image data is an example of a "first acquiring means" , and the acquiring unit configured to acquire the event data is an example of a "second acquiring means" . The image processing device 1 according to the embodiment includes the image acquiring unit 10 and the event acquiring unit 20. However, if the image data and the event data acquired by the image acquiring unit 10 and the event acquiring unit 20 can be acquired in pairs, the image acquiring unit 10 and the event acquiring unit 20 may be practiced separately from the image processing device 1. When they are separated from the image processing device, the image data acquiring module 301 corresponds to "the first acquiring means" , and the event data acquiring module 302 corresponds to "the second acquiring means" . The effectiveness determining module 303 is an example of a "determining means" . The output processing module 304 is an example of an "output processing means" .
[0076] First Modification Example of Embodiment
[0077] In the embodiment, it has been explained that the effectiveness determining module 303 determines effectiveness by using the amount of the edges as a criterion, but the criterion of determination of effectiveness is not limited to the amount of the edges. The other criterion for determining effectiveness may be provided.
[0078] As the first modification example, a case where the effectiveness determining module 303 determines effectiveness by using lightness information as a criterion will be described.
[0079] The lightness information is information on brightness obtained by aggregating brightness values of pixels of the image data, for example. When using brightness as the criterion of determination, the effectiveness determining module 303 determines that the process using the event data is effective when the aggregated value of brightness satisfies the condition of a threshold by using the threshold corresponding to the brightness. For example, if a shooting place is a bright place as the image is overexposed, the effectiveness determining module 303 is set to determine that the process using the event data is effective when the aggregated value of brightness is not less than a first threshold by using the first threshold of detecting overexposure.
[0080] The lightness information is not limited to the value obtained by aggregating the brightness values of pixels. The lightness information may be actual brightness of the subject obtained from a light measuring means.
[0081] FIG. 5 is a diagram illustrating an example of a configuration of the image processing device 1 according to the first modification example of the embodiment. In the configuration illustrated in the embodiment, the image processing device 1 further includes a light measuring unit 400 and a lightness information acquiring module 401 as an example of the light measuring means.
[0082] The light measuring unit 400 is a sensor configured to measure actual brightness of the subject.
[0083] The lightness information acquiring module 401 acquires information indicating the brightness from the light measuring unit 400.
[0084] When the brightness acquired from the light measuring unit 400 is not less than the first threshold, the effectiveness determining module 303 determines that the process using the event data is effective.
[0085] Note that the effectiveness determining module 303 may determine whether the process using the event data is effective by using only the lightness information, or may determine whether the process using the event data is effective by using the lightness information and the amount of the edges illustrated in the embodiment together.
[0086] Shooting Scene for which Determination by Brightness is suitable
[0087] FIG. 6 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device 1 according to the first modification example of the embodiment. If the image is an image of a normal shooting scene for which the extraction of the edges of the subject can be appropriately performed from the whole image, for example, like an image 2000 illustrated in (a) of FIG. 6, the event data is used for the subject suitable for the process using the event data by performing determination by using the amount of the edges of the whole image as a criterion.
[0088] However, as illustrated in (b) of FIG. 6, even if the image is an image to be processed as the subject suitable for the process using the event data in normal circumstances, edges are not appropriately extracted and the amount of the edges is hard to exceed the threshold when the image becomes an image 2001 such that the subject is totally overexposed by the shooting at a too bright place. When the amount of the edges does not exceed the threshold, the image is processed as the image 2001 itself illustrated in (b) of FIG. 6. For example, when the object detection is performed as the image 2001 itself, one object may be detected as other objects obtained by dividing the one object into two. Moreover, even when the image is saved as a recording image, the image looks unattractive and also it is hard to recognize the overexposed object.
[0089] In the first modification example, in consideration of the shooting scene that is likely overexposed, a brightness is set as the criterion of determination. When the shooting is likely overexposed, the image processing device 1 determines that the process using the event data is effective at the brightness having the first threshold or more.
[0090] An output image 2002 of (c) of FIG. 6 is an example of an output image after the output processing module 304 corrects the image 2001 that is overexposed illustrated in (b) of FIG. 6 by using the event data. As illustrated in the output image 2002 of (c) of FIG. 6, because portions that are overexposed in the image 2001 of (b) of FIG. 6 are corrected as edges by the event data, the actual subject is output.
[0091] Note that the threshold uses the first threshold indicating a high value to detect overexposure because a bright place such as "overexposure" is assumed herein, but the threshold uses the second threshold indicating a low value to detect underexposure in the case of a dark place such as "underexposure" . When using the second threshold, the image processing device 1 determines that the process using the event data is effective when a brightness is not more than the second threshold.
[0092] Moreover, when detecting "overexposure" and "underexposure" together, the first threshold of "overexposure" and the second threshold of "underexposure" are used together. In this case, the image processing device 1 determines that the process using the event data is effective when the brightness is not less than the first threshold, and also determines that the process using the event data is effective when the brightness is not more than the second threshold.
[0093] As described above, because an event camera has a wider dynamic range than a general frame camera, even in a shooting scene such as overexposure to overexpose and underexposure to underexpose in the frame camera, data of pixels can be accurately acquired by the event camera without vanishing data. For this reason, when the number of pixels saturated in the image sensor of the frame camera or pixels on which light is not incident, for which the brightness departs from a constant range, exceeds the threshold, the image processing device determines that the process using the event data is effective.
[0094] Therefore, the image processing device 1 according to the first modification example of the embodiment can recommend the shooting scene to use the event data.
[0095] Second Modification Example of Embodiment
[0096] As the second modification example, a case where the effectiveness determining module 303 determines effectiveness by using a value of acceleration as a criterion will be described.
[0097] FIG. 7 is a diagram illustrating an example of a configuration of the image processing device 1 according to the second modification example of the embodiment. In the configuration illustrated in the embodiment, the image processing device 1 further includes an acceleration measuring unit 500 and an acceleration information acquiring module 501 as an example of an acceleration measuring means.
[0098] The acceleration measuring unit 500 is an acceleration sensor configured to measure the acceleration of the image processing device 1.
[0099] The acceleration information acquiring module 501 acquires acceleration information from the acceleration measuring unit 500.
[0100] The effectiveness determining module 303 determines that the process using the event data is effective when the acceleration information acquired from the acceleration measuring module 500 is not less than the third threshold.
[0101] Note that the effectiveness determining module 303 may use only the value of acceleration but may determine whether the process using the event data is effective by using together the value of acceleration and the amount of the edges illustrated in the embodiment or the lightness information illustrated in the first modification example.
[0102] Shooting Scene for which Determination by Value of Acceleration is suitable
[0103] The subject blur in the image may occur by a camera shake etc. when shooting. The blur may be solved by a change of setting of a shutter speed, but the image is darkened and the subject etc. does not appear clearly in many cases. On the other hand, because data can be acquired in theμs order if it is the event data of the event acquiring unit 20, data of a position at which the subject exists can be accurately obtained even if there is an intense camera shake. Therefore, it is effective to use the event data in such a shooting scene.
[0104] Therefore, when a measured value of the acceleration measuring unit 500 mounted on the image processing device 1 is not less than the third threshold, it is determined that the intense subject blur occurs and the output process using the event data is performed.
[0105] The image data is deblurred by using the event data.
[0106] FIG. 8 is a diagram illustrating an example of a shooting scene suitable for a criterion of the image processing device 1 according to the second modification example of the embodiment. With respect to a normal shooting scene on which the extraction of edges of the subject is appropriately performed from the whole image, like an image 3000 illustrated in (a) of FIG. 8, an image with the subject blur like an image 3001 illustrated in (b) of FIG. 8 may be acquired.
[0107] When the intense camera shake occurs, the image 3001 with the subject blur as illustrated in (b) of FIG. 8 is acquired.
[0108] In the second modification example, in consideration of such a shooting scene, the value of acceleration is set as the criterion of determination, and the image processing device 1 illustrated in (c) of FIG. 8 determines that the process using the event data is effective when the blur with the value of acceleration greater than or equal to the third threshold is detected.
[0109] An output image 3002 of (d) of FIG. 8 is an example of an output image after the output processing module 304 corrects the image 3001 with the subject blur illustrated in (b) of FIG. 8 by using the event data. Like the output image 3002 illustrated in (d) of FIG. 8, the blurred image of the image 3001 in (b) of FIG. 8 is deblurred by the event data, and thus the subject can be accurately caught.
[0110] Thus, the image processing device 1 according to the second modification example of the embodiment can also recommend a shooting scene to use the event data.
[0111] Third Modification Example of Embodiment
[0112] In the embodiment, the first modification example, and the second modification example until now, plural methods to determine effectiveness of the process using the event data have been described, but, because these methods are independent methods, these methods may be used singly or may be used by combining two or more methods among these methods. When using these methods by combining two or more methods, determination results may be integrated at a predetermined ratio to output the final determination result. How to decide the ratio of the determination results of the methods is arbitrary, and the ratio may be adjusted to an optimum value as appropriate. Moreover, the ratio may be manually set, or may be automatically set by machine learning etc.
[0113] As the third modification example, a configuration that determination results determined by the effectiveness determining module 303 based on the plurality of criteria are integrated to output the final determination result will be described. As an example, there is described a case where the determination result by the amount of edges illustrated in the embodiment, the determination result by the lightness information illustrated in the first modification example, and the determination result by the value of acceleration illustrated in the second modification example are used and these determination results are integrated. Note that at least two of three criteria may be used. Moreover, another criterion may be provided without being limited to three criteria. Moreover, four or more criteria may be used.
[0114] FIG. 9 is a diagram illustrating an example of a configuration of the image processing device 1 according to the third modification example of the embodiment. The image processing device 1 further includes a first effectiveness determining module 303-1, a second effectiveness determining module 303-2, a third effectiveness determining module 303-3, and an integrated determination module 601. The integrated determination module 601 is an example of an "integrated determination means" .
[0115] The first effectiveness determining module 303-1 is an effectiveness determining module configured to determine effectiveness by using an amount of edges as a criterion. The second effectiveness determining module 303-2 is an effectiveness determining module configured to determine effectiveness by using lightness information as a criterion. The third effectiveness determining module 303-3 is an effectiveness determining module configured to determine effectiveness by using a value of acceleration as a criterion.
[0116] The integrated determination module 601 integrates determination results output from the first effectiveness determining module 303-1, the second effectiveness determining module 303-2, and the third effectiveness determining module 303-3 at a predetermined ratio, and outputs the final determination result based on the integration result.
[0117] As an example, assuming that the determination results ( "recommendation: 1" or "non-recommendation: 0" ) are A, B, and C, and the corresponding ratios are n1, n2, and n3, the integration result D is expressed by the following Equation 1. D=n1×A+n2×B+n3×C (1)
[0118] Effectiveness Determination Process Flow
[0119] FIG. 10 is a diagram illustrating an example of an effectiveness determination process flow in the image processing device 1 according to the third modification example of the embodiment. As illustrated in FIG. 10, the processing unit 30 acquires image data from the image acquiring unit 10 (Step S11) .
[0120] Moreover, the processing unit 30 acquires sensor information from various sensors (Step S12) . The sensor information acquired from the various sensors includes the event data acquired from the event acquiring unit 20, the information indicating brightness acquired from the light measuring unit 400, and the value of acceleration acquired from the acceleration measuring unit 500.
[0121] Next, the processing unit 30 determines whether the process using the event data is effective by using the following three methods. As the first method, the processing unit 30 extracts an amount of edges from the image data acquired in Step S11 to determine whether the process using the event data is effective (Step S13) . As the second method, the processing unit 30 determines whether the process using the event data is effective from the brightness acquired from the light measuring unit 400 (Step S14) . As the third method, the processing unit 30 determines whether the process using the event data is effective from the value of acceleration acquired from the acceleration measuring unit 500 (Step S15) .
[0122] Next, the processing unit 30 together integrates the value A that is the determination result of Step S13, the value B that is the determination result of Step S14, and the value C that is the determination result of Step S15 at a predetermined ratio (Step S16) . As an example, the processing unit 30 obtains the integration result D by using Equation (1) .
[0123] Next, the processing unit 30 performs comprehensive final determination by determining whether the integration result D is not less than a predetermined reference value (Step S17) .
[0124] When the integration result D is not less than the predetermined reference value (Step S17: Yes) , the processing unit 30 finally determines that the process using the event data is effective, and sets the use of the event data to "recommendation" (Step S18) .
[0125] On the other hand, when the integration result D is less than the predetermined reference value (Step S17: No) , the processing unit 30 finally determines that the process using the event data is not effective, and sets the use of the event data to "non-recommendation" (Step S19) .
[0126] Note that the effectiveness determination process flow of Step S11 to Step S19 may be repeated for each frame, may be performed every plural frames, may be performed in a predetermined shooting state, or may be performed at an arbitrary timing.
[0127] As described above, the image processing device 1 according to the third modification example of the embodiment can comprehensively determine whether a scene using the event data is a shooting scene of obtaining effective output by combining the plurality of methods to perform the output process. Thus, the image processing device 1 according to the third modification example of the embodiment can recommend the use of the event data in accordance with the determination results according to shooting scenes.
[0128] Fourth Modification Example of Embodiment
[0129] In the embodiment and the first to third modification examples, the methods of performing the determination of whether the process using the event data is effective have been described. In the embodiment and the first to third modification examples, in the case of the determination result of recommending the process using the event data, the output processing module 304 automatically performs the output process by using the event data with reference to the image data acquired by the image data acquiring module 301.
[0130] Herein, as the fourth modification example, a case where the execution of the process in the output processing module 304 is made manually will be described as an example.
[0131] FIG. 11 is a diagram illustrating an example of a configuration of the image processing device 1 according to the fourth modification example of the embodiment. The image processing device 1 further includes a notifying unit 700 as an example of a notifying means.
[0132] When the determination result of "recommendation" is obtained in the effectiveness determining module 303, the notifying unit 700 presents "recommended information" to a user to notify the user that the present shooting scene is a shooting scene for which the process using the event data is effective. The image processing device causes the user to decide whether the process using the event data is performed actually.
[0133] The output processing module 304 performs the output process according to the response from the user. For example, when accepting an execution instruction of the process using the event data from the user, the output processing module 304 performs the output process by using the event data with reference to the image data acquired by the image data acquiring module 301.
[0134] FIG. 12 is a diagram illustrating a display example of "recommended information" in the image processing device 1 according to the fourth modification example of the embodiment. (a) of FIG. 12 is an example of display when image data 4000 is previewed on the display screen 41. When the determination result of "recommendation" is obtained in the effectiveness determining module 303, the notifying unit 700 displays "recommended information" 4001 on the display screen 41 during preview display as illustrated in (b) of FIG. 12.
[0135] The user touches the "recommended information" 4001, for example, and instructs to execute the process using the event data from the input module 60.
[0136] Note that the "recommended information" 4001 illustrated in (b) of FIG. 12 is an example. The "recommended information" 4001 may have another form if the fact that the present shooting scene is a shooting scene for which the process using the event data is effective can be presented to the user.
[0137] In the embodiment and the fourth modification example, when the determination result of "recommendation" is obtained in the effectiveness determining module 303, the "recommended information" is presented to the user. Thus, the process using the event data can be actually executed by the final determination of the user.
[0138] It is also considered that the learned library is substituted for the effectiveness the determination methods illustrated in the embodiment and the first to third modification examples. The learned library means a model obtained by performing machine learning on the determination result indicating "recommendation" or "non-recommendation" by using information on the plurality of shooting scenes.
[0139] The learned library has a neural network that includes one or more intermediate layers between an input layer and an output layer. The learned library is previously learned by using a frame image of a predetermined shooting scene, information indicating brightness, and a value of acceleration, to adjust weights in nodes of the input layer, the one or more intermediate layers, and the output layer so that the determination result is appropriately output. The learned library whose weights are adjusted is stored in the memory 50.
[0140] For example, the image processing device 1 inputs the image data acquired by the image data acquiring module 301, the information indicating brightness acquired from the light measuring unit 400, and the value of acceleration acquired from the acceleration measuring unit 500 into the learned library of the memory 50. If there is the input of a scene similar to the learned scene, the learned library may output a determination result with high probability. Thus, it may be also considered that the learned library is substituted for the effectiveness determination methods illustrated in the embodiment and the first to third modification examples.
[0141] The functional blocks of the processing unit 30 illustrated in the embodiment and the first modification example to the fourth modification example may be realized by hardware or by software. Moreover, some of the blocks may be realized by software. When being realized by software, the processor 32 executes a program stored in the memory 50 to realize some or all of the functional blocks of the processing unit 30 by software.
[0142] The processor 32 executes the program to realize each functional block, and performs the effectiveness determination process in steps as illustrated in FIG. 4 or FIG. 10, for example.
[0143] Note that the program may be incorporated into ROM etc. to be provided, may be provided by a transportable recording medium, or may be downloaded in such a manner that the image processing device 1 communicates with a predetermined server by way of the Internet.
[0144] While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
[0145] [Explanations of Letters or Numerals]
[0146] 1: Image processing device 10: Image acquiring unit 20: Event acquiring unit 30: Processing unit 301: Image data acquiring module 302: Event data acquiring module 303: Effectiveness determining module 303-1: First effectiveness determining module 303-2: Second effectiveness determining module 303-3: Third effectiveness determining module 304: Output processing module 400: Light measuring unit 401: Lightness information acquiring module 500: Acceleration measuring unit 501: Acceleration information acquiring module 601: Integrated determination module 700: Notifying unit
Claims
1.An image processing device comprising:a first acquiring means configured to acquire image data;a second acquiring means configured to acquire event data; anda determining means configured to determine whether a process using the event data is effective based on the image data acquired by the first acquiring means.2.The image processing device according to claim 1, wherein the determining means is configured to determine whether the process using the event data is effective based on an amount of edges detected from the image data acquired by the first acquiring means.3.The image processing device according to claim 2, wherein the amount of the edges includes one of a ratio of an area occupied by the edges, a total length obtained by extending the edges, and a number of the edges.4.The image processing device according to claim 1, wherein the determining means is configured to determine whether the process using the event data is effective based on lightness information of the image data acquired by the first acquiring means.5.The image processing device according to claim 1, wherein the determining means is configured to determine whether the process using the event data is effective based on lightness information indicating brightness acquired from a light measuring means configured to measure brightness of a subject.6.The image processing device according to claim 1, wherein the determining means is configured to determine whether the process using the event data is effective based on a value of acceleration acquired from an acceleration measuring means configured to measure acceleration.7.The image processing device according to any one of claims 2 to 6, further comprising:an integrated determination means configured to integrate two or more determination results of the amount of edges, the lightness information, and the value of acceleration at a predetermined ratio and determine whether the process using the event data is effective from an integration result.8.The image processing device according to any one of claims 1 to 7, further comprising:an output processing means configured to perform an output process using the event data based on a determination result by the determining means or a determination result by the integrated determination means.9.The image processing device according to any one of claims 1 to 7, further comprising:a notifying means configured to notify a user that the process using the event data is effective based on a determination result by the determining means or a determination result by the integrated determination means.10.The image processing device according to claim 9, further comprising:an output processing means configured to perform an output process using the event data by a final determination by the user.11.The image processing device according to claim 8 or 10, wherein the output processing means is configured to correct the image data by using the event data when performing the output process by using the event data.12.An image processing method that is executed by an image processing device, the method comprising:acquiring image data;acquiring event data; anddetermining whether a process using the event data is effective based on the acquired image data.13.A program causing a computer to function as each means of the image processing device according to any one of claims 1 to 11.
Citation Information
Patent Citations
Method of correcting dynamic vision sensor (DVS) events and image sensor performing the same
US20210014435A1
Image processing apparatus, imaging apparatus, system, image processing method, and program
US20220284593A1
Information processing apparatus, information processing method, and storage medium
US20220400199A1
Information processing device and information processing method
WO2022215371A1
