Image processing device, image processing method, imaging device, and program

Abstract

To make it possible to avoid the occurrence of erroneous association in which another object or the like is associated with an object to be tracked.SOLUTION: An image processing device generates a frame image from an imaging signal of a frame-based sensor, and detects a size of a region in which a luminance change occurs based on an address event signal in which an event-based sensor that shares an imaging range of an actual space with the frame-based sensor detects a luminance change. Then, the image processing device outputs the frame image when the size of the region in which the luminance change occurs is larger than a predetermined size threshold.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for processing captured images. [Background technology]

[0002] Event-based sensors (event-driven image sensors, also known as event-based vision sensors) are known to output address event signals in real time, such as the location and time of a brightness change and the polarity of the brightness change. One example of how event-based sensors are expected to be used is in surveillance cameras. By installing an event-based sensor in a surveillance camera, it is possible to realize a surveillance camera that can detect the target of surveillance with greater accuracy. Furthermore, Patent Document 1 proposes a method for outputting an image using an address event signal, in which whether or not a predetermined address event has occurred is detected based on whether or not the amount of change in the amount of incident light exceeds a predetermined threshold, and a pixel signal is output when an address event has occurred. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-96347 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of Patent Document 1 outputs all pixel signals whose address event signals exceed a threshold value. Therefore, when applied to a surveillance camera, for example, it may output unnecessary images that capture the movement of small objects other than the target of surveillance.

[0005] Therefore, an object of the present invention is to make it possible to suppress the output of unnecessary images. [Means for solving the problem]

[0006] The image processing device of the present invention is characterized by having a generation means for generating a frame image from an imaging signal of a frame-based sensor, a detection means for detecting the size of an area in which a luminance change has occurred based on an address event signal in which an event-based sensor that shares a real-space imaging range with the frame-based sensor has detected the luminance change, and an output means for outputting the frame image when the size of the area in which the luminance change has occurred is greater than a predetermined size threshold. [Effects of the Invention]

[0007] According to the present invention, it is possible to suppress the output of unnecessary images. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an imaging device including an image processing device according to an embodiment. [Figure 2] 10 is a flowchart of image processing according to the embodiment. [Figure 3] FIG. 10 is a diagram used to explain a threshold value for the size of a detected object. [Figure 4] FIG. 10 is a diagram used to explain an example of changing a threshold value for size within an image. [Figure 5] FIG. 10 is a diagram used to explain a cutout range. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). In the following embodiments, the same or similar components are designated by the same reference symbols, and redundant explanations will be omitted.

[0010] 1 is a diagram showing an example of the configuration of an imaging device 100 including an image processing device according to this embodiment. The imaging device 100 of this embodiment is configured to include an imaging unit 101, an image processing unit 105, a control unit 106, a transmission unit 107, and a storage unit 108. Note that in this embodiment, a case will be described in which the imaging device 100 is used as a surveillance camera.

[0011] The imaging unit 101 includes a frame-based sensor 102 , an event-based sensor 103 , and a lens 104 . The frame-based sensor 102 is a sensor in which a plurality of light-receiving elements (pixels) are arranged on a light-receiving surface, and outputs an image signal on a frame-by-frame basis. Examples of the frame-based sensor 102 include a color sensor and a monochrome sensor. The frame-based sensor 102 made of a color sensor outputs values ​​related to brightness and color on a frame-by-frame basis. The frame-based sensor 102 made of a monochrome sensor outputs values ​​related to brightness on a frame-by-frame basis. Note that the image sensor of the color sensor or monochrome sensor may be, for example, a CMOS sensor or a CCD sensor. The image signal output on a frame-by-frame basis from the frame-based sensor 102 is input to an image processing unit 105.

[0012] The event-based sensor 103 is a sensor with an array of multiple light-receiving elements (pixels) on its light-receiving surface. It detects luminance changes (increases or decreases in luminance) for each pixel and outputs the pixel's position and time (time) at which the luminance change occurred as an address event signal. Each pixel in the event-based sensor 103 consists of a light-receiving section and a luminance change detection section. The light-receiving section includes a photodiode and a voltage conversion circuit. The photodiode converts incident light into a current, and the voltage conversion circuit converts the current into a voltage. The converted voltage is sent to the luminance change detection section. The luminance change detection section outputs an address event signal when the difference between the converted voltage and a reference voltage exceeds a predetermined difference threshold. The address event signal output from the event-based sensor 103 is input to the image processing section 105. As described above, the image capturing unit 101 of this embodiment has a configuration capable of outputting an image capturing signal in frame units from the frame-based sensor 102 and an address event signal corresponding to the occurrence of a luminance change from the event-based sensor 103.

[0013] Lens 104 is shared by frame-based sensor 102 and event-based sensor 103, and focuses an optical image of a subject or the like, formed by condensing incident light, at the same position on the light-receiving surfaces of both frame-based sensor 102 and event-based sensor 103. For example, if the light-receiving surfaces of frame-based sensor 102 and event-based sensor 103 are arranged on the same plane by stacking or the like, lens 104 focuses an optical image of a subject or the like at the same position on both light-receiving surfaces. Alternatively, if the light-receiving surfaces of frame-based sensor 102 and event-based sensor 103 are not on the same plane, lens 104 focuses an optical image of a subject or the like at the same position on both light-receiving surfaces via a half mirror or the like.

[0014] In this way, in imaging unit 101, lens 104 is shared by frame-based sensor 102 and event-based sensor 103. That is, the imaging range in real space captured by imaging unit 101 is shared by frame-based sensor 102 and event-based sensor 103 via lens 104. In other words, event-based sensor 103 detects a change in luminance for each pixel within the imaging range of real space captured by frame-based sensor 102 via lens 104, and acquires an address event signal that indicates the position and time of the pixel where the luminance change was detected.

[0015] The image processing unit 105 corresponds to the image processing device of this embodiment. The image processing unit 105 generates an image in a frame unit (hereinafter referred to as a frame image) from the image pickup signal acquired by the frame-based sensor 102 in a frame unit. Furthermore, image processing unit 105 generates an event image showing the location where a luminance change event occurred, based on the address event signal acquired by event-based sensor 103. Image processing unit 105 then detects the area and size of the event occurrence location from the event image. The area and size of the event occurrence location detected from the event image correspond to the area and size of an object detected as a luminance change due to movement, etc., within the shooting range of imaging unit 101 in real space. Since the area detected as the event occurrence location is the area of ​​the object detected due to movement, etc., in the following description, the area detected as the event occurrence location will be referred to as an "object." Based on the object detection result, image processing unit 105 then determines whether to output a frame image generated from the imaging signal of frame-based sensor 102. Details of the process of detecting an object and its size as an event from the event image, and the process of outputting a frame image based on the detected object and its size, will be described later.

[0016] The storage unit 108 stores frame images output from the image processing unit 105. In addition to storing frame images output from the image processing unit 105, the storage unit 108 may also store address event signals or event images. The storage unit 108 may include a ROM, a RAM, and a large-capacity memory. Furthermore, the storage unit 108 may store an image processing program for the CPU or GPU to operate as the image processing unit 105, and may similarly store a control program for the CPU or the like to operate as the control unit 106.

[0017] The transmission unit 107 transmits the frame images output from the image processing unit 105 or the frame images stored in the storage unit 108 to the outside of the imaging device 100. Furthermore, when control data for controlling the imaging device 100 is transmitted from, for example, an external imaging control device 120, the transmission unit 107 receives the control data and sends it to the control unit 106.

[0018] Control unit 106 controls each unit within imaging device 100. That is, control unit 106 controls the operations of frame-based sensor 102, event-based sensor 103, image processing unit 105, storage unit 108, and transmission unit 107. Furthermore, control unit 106 controls the driving of the focus lens and zoom lens included in lens 104, and, if imaging unit 101 also includes a pan / tilt or other head driving unit (not shown), also controls the driving of pan / tilt or the like by the head driving unit. Furthermore, when control data is received from external imaging control device 120 via transmission unit 107, control unit 106 controls the operation of each unit based on the control data.

[0019] The external imaging control device 120 is connected to the transmission unit 107 of the imaging device 100. The imaging control device 120 receives frame images from the imaging device 100 and transmits them to the display device 130. As a result, the frame images are displayed on the screen of the display device 130. The imaging control device 120 also generates control data for controlling the imaging device 100 as needed and transmits it to the imaging device 100. If the imaging device 100 of this embodiment is a surveillance camera, the imaging control device 120 may be, for example, a personal computer operated by a user (surveillance person, etc.) of the surveillance camera. The display device 130 may be a monitor that displays images sent from the surveillance camera and a GUI (graphical user interface) for operating the surveillance camera.

[0020] FIG. 2 is a flowchart showing the flow of image processing performed by the image processing unit 105 of the imaging device 100 of this embodiment. In this embodiment, the image processing unit 105 detects the area and size of the event occurrence location from the event image, thereby detecting the size and size of an object that has moved or otherwise moved within the shooting range of the imaging device 100, and further detects objects that are larger than a predetermined size among the detected objects. Then, based on the object detection result, the image processing unit 105 determines whether or not to output frame images acquired during the same period as the event image. Note that the image processing unit 105 performs processing each time an event image is generated until it has been determined whether to output frame images acquired during the same period as the event image, and once it has been determined whether to output frame images for the same period, it waits until the next frame period to perform processing.

[0021] First, in step S201, image processing unit 105 receives frame-by-frame imaging signals from frame-based sensor 102 and receives address event signals from event-based sensor 103. Image processing unit 105 then generates frame images from the frame-by-frame imaging signals. Image processing unit 105 also generates an event image from the address event signal and detects the area where the event occurred and its size from the event image. In other words, image processing unit 105 detects, as an event, the movement of an object or the like within the shooting range in real space captured by imaging unit 101, and detects the area and size of the object.

[0022] Next, in step S202, the image processing unit 105 determines whether the size of the object detected in step S201 is larger than a predetermined size threshold. The predetermined size threshold can be set to a desired value (any value). However, when the imaging device 100 is used as a surveillance camera as in this embodiment, the predetermined size threshold is set to a value that takes into account the size of monitored objects such as people and vehicles. In other words, in this embodiment, the predetermined size threshold is set to exclude small objects such as small animals, excluding people, vehicles, and other monitored objects. If it is determined that the size of the object is larger than the predetermined size threshold, the image processing unit 105 proceeds to outputting frame images from step S203 onwards. On the other hand, if the size of the object is equal to or smaller than the predetermined size threshold, the image processing unit 105 does not proceed to outputting frame images from step S203 onwards (i.e., does not output frame images), but waits until the next frame period. Note that if multiple objects are detected in step S201, the image processing unit 105 performs a determination using a predetermined size threshold for each of the objects in step S202. If there is one or more objects larger than the predetermined size threshold, the process of the image processing unit 105 proceeds to step S203.

[0023] In step S203, the image processing unit 105 calculates the number of objects that are determined to be larger than the predetermined size threshold in step S202. Next, in step S204, the image processing unit 105 determines whether the number of objects larger than the predetermined size threshold is equal to or greater than a predetermined number threshold. For example, when the image capture device 100 is used as a surveillance camera, the predetermined number threshold is set based on the assumption that when the number of surveillance objects, such as people, is greater than normal, the user of the surveillance camera (surveillant) will need to check these surveillance objects. In other words, in this embodiment, the predetermined number threshold is set to exclude cases where the number of surveillance objects is the same as normal. If the number of objects larger than the predetermined size threshold is equal to or greater than the predetermined number threshold, the processing of the image processing unit 105 proceeds to step S205 and subsequent steps. On the other hand, if the number of objects larger than the predetermined size threshold is less than the predetermined number threshold, the image processing unit 105 does not output a frame image and waits until the next frame period to perform processing.

[0024] In step S205, image processing unit 105 determines whether the frame image can be transmitted at the transmission speed based on the data amount of the frame image generated from the imaging signal of frame based sensor 102 and the transmission speed between transmission unit 107 and the device connected to it. If it is determined that the frame image can be transmitted, image processing unit 105 proceeds to step S206. On the other hand, if it is determined that the frame image cannot be transmitted, image processing unit 105 proceeds to step S207 and subsequent steps.

[0025] In step S206, image processing unit 105 outputs the frame image created from the imaging signal of frame-based sensor 102 to transmission unit 107. As a result, the frame image is sent to the outside via transmission unit 107. Note that image processing unit 105 may store the frame image in storage unit 108.

[0026] On the other hand, when the process proceeds to step S207, image processing unit 105 performs a cutout process on the frame image in order to reduce the size of the image to be output and reduce the amount of data. Details of the setting of the range to be cut out from the frame image by the cutout process (hereinafter referred to as the cutout range), the size of that range, etc. will be described later. Thereafter, the process of image processing unit 105 proceeds to step S208.

[0027] In step S208, the image processing unit 105 determines whether the cut-out process has been completed. For example, in step S208, the image processing unit 105 determines whether all of the objects detected in step S201 as being larger than a predetermined size threshold fit within the cut-out range set in the cut-out process. If all of the objects fit within the cut-out range, the image processing unit 105 determines that the cut-out process has been completed, and proceeds to step S209. On the other hand, if the cut-out process has not been completed, that is, if the objects do not fit within the cut-out range, the image processing unit 105 proceeds to step S211. Note that examples of cases in which an object does not fit within the cut-out range include when multiple objects are located far apart and not all of the objects fit within the cut-out range, or when an object is too large to fit within the cut-out range.

[0028] In step S209, image processing unit 105 determines, based on the size of the image in the cutout range (i.e., the amount of data), whether the image in the cutout range can be transmitted as a frame image from transmission unit 107. If it is determined that the image can be transmitted, image processing unit 105 proceeds to step S210. On the other hand, if it is determined that the image cannot be transmitted, image processing unit 105 proceeds to step S211.

[0029] In step S210, the image processing unit 105 outputs the image of the cutout range as a frame image, and outputs it to the outside via the transmission unit 107. Thereafter, the image processing unit 105 waits for the next frame period to perform processing. On the other hand, if the process proceeds to step S211, the image processing unit 105 sets the frame rate at which frame images are output to be lowered. Details of the frame rate lowering process will be described later. Thereafter, the process of the image processing unit 105 proceeds to step S212.

[0030] In step S212, image processing unit 105 determines whether transmission via transmission unit 107 is possible based on the size of the frame image (i.e., the amount of data) of the cutout range after the frame rate has been reduced. If it is determined that transmission is not possible, image processing unit 105 waits until the next frame period to perform processing. On the other hand, if it is determined that transmission is possible, the processing of image processing unit 105 proceeds to step S213. In step S213, the image processing unit 105 outputs the frame image of the cutout range after the frame rate has been reduced in step S211, and outputs it to the outside via the transmission unit 107. Thereafter, the image processing unit 105 waits for the next frame period to complete processing.

[0031] The image processing operation performed by the image processing unit 105 according to this embodiment will be described below in accordance with the flow chart shown in FIG. 2, with reference to FIGS. 3 to 5 showing more specific examples. FIG. 3 shows an example of an event image 300 generated by image processing unit 105 from an address event signal generated by event-based sensor 103. Image processing unit 105 creates an address image by plotting the positions of address event signals generated within a frame period. For example, if an object such as a subject is moving within the imaging range of imaging unit 101, event-based sensor 103 can capture the change in brightness occurring at the outline of the moving object as an event. Therefore, the event image generated by image processing unit 105 is an image capturing the outline of the object. Furthermore, event-based sensor 103 detects the movement of the object in real time and outputs it as an address event signal. Therefore, image processing unit 105 can generate outline data of the object from the address event signal regardless of the frame period in which the frame image is generated.

[0032] Fig. 3 shows an example of an event image 300 in which the contours of three objects, objects 320 to 340, are captured within the shooting range of the imaging unit 101. In Fig. 3, the contours of objects 320 to 340 are shown by solid-line squares, and a predetermined size threshold 310 for the sizes of objects 320 to 340 is shown by dotted-line squares.

[0033] In step S202, the image processing unit 105 determines whether the sizes of the three objects 320 to 340 detected in the address image generated from the address event signal are larger than a predetermined size threshold 310. The image processing unit 105 compares the sizes of the three objects 320 to 340 detected in the address image with the predetermined size threshold, and detects objects larger than the predetermined size threshold 310. In the example of FIG. 3, the sizes of objects 320 and 330 are larger than the predetermined size threshold 310, so in step S202, the image processing unit 105 determines that there are one or more objects larger than the predetermined size threshold. Note that whether the size of an object is larger than the predetermined size threshold may be determined, for example, by determining whether the number of pixels in the area of ​​the detected object is greater than the number of pixels set as the predetermined size threshold.

[0034] Furthermore, the size and shape of an object seen by imaging device 100 (how the object appears) vary greatly depending on the location where imaging device 100 is installed in real space and the orientation of the imaging direction of imaging device 100 relative to real space. For example, when an imaging device installed at a high location captures an image diagonally downward, the higher the position in the captured image, the farther away the object is. In this case, even if the object is the same, the higher the position in the image, the smaller the object will appear, and conversely, the lower the position in the image, the larger the object will appear. Therefore, the predetermined size threshold used to determine the size of an object detected from an event image may be changed depending on the imaging direction of imaging device 100 and the position in the image.

[0035] FIG. 4 is a diagram illustrating an example in which the predetermined size threshold is changed depending on the imaging direction of the imaging device 100 and the position within the image. The example in FIG. 4 shows an event image 400 in which the contours of two objects, object 430 and object 440, are captured. As in the example in FIG. 3, the contours of object 430 and object 440 are indicated by solid-line rectangles, and the predetermined size threshold is indicated by dotted-line rectangles. In FIG. 4, it is assumed that the imaging device is installed at a high location and captures an image obliquely downward, so the upper side of the image is farther away, and the object appears smaller the higher in the image, and conversely, the lower in the image, the larger it appears. Note that, although object 430 and object 440 in the event image 400 shown in FIG. 4 appear to be approximately the same size, because the imaging device 100 captures an image obliquely downward, object 430 is actually farther away than object 440 and is larger than object 440.

[0036] In this way, when the imaging device is installed at a high position and the imaging direction of the imaging device is directed diagonally downward, the predetermined size threshold is set to a threshold that decreases as the position in the image increases. That is, predetermined size threshold 410 for object 430 detected at an upper position in the image is set to a smaller value than predetermined size threshold 420 for object 440 detected at a lower position in the image. In the image processing unit 105 of this embodiment, the predetermined size threshold for the size of an object in the image can be changed depending on the imaging direction of the imaging device and the position in the image, thereby improving the accuracy of detecting the size of an object.

[0037] After detecting the size of the object as described above, the image processing unit 105 determines in step S204 whether the number of objects larger than the predetermined size threshold is equal to or greater than a predetermined number threshold. Then, when the number of objects larger than the predetermined size threshold is equal to or greater than the predetermined number threshold, the image processing unit 105 outputs a frame image. For example, when the predetermined number threshold is "2," the image processing unit 105 outputs a frame image when there are two or more objects that exceed the predetermined size threshold. Note that, although the predetermined number threshold is "2" in this embodiment, it is not limited to this and may be set to any number (desired number). In the event image 300 illustrated in FIG. 3, the objects that exceed the predetermined size threshold 310 are object 320 and object 330, which is equal to or greater than the predetermined number threshold of 2, and therefore the processing of the image processing unit 105 proceeds to step S205.

[0038] In step S205, image processing unit 105 calculates the data amount of the frame image generated from the imaging signal of frame-based sensor 102, and also calculates the amount of data that can be transmitted in one frame period from the transmission rate between transmission unit 107 and imaging control device 120. Image processing unit 105 then compares these data amounts to determine whether the frame image generated from the imaging signal of frame-based sensor 102 can be transmitted from transmission unit 107. If it is determined that the frame image can be transmitted, the process of image processing unit 105 proceeds to step S206, where it outputs the frame image. That is, image processing unit 105 outputs the frame image of the same period as when objects 320 and 330 were detected in event image 300 to transmission unit 107. This causes the frame image to be sent to imaging control device 120.

[0039] In this manner, image processing unit 105 of this embodiment determines whether to output a frame image depending on the size of an object detected in an event image based on the address event signal of event-based sensor 103. This prevents, for example, an image from being output from the surveillance camera in response to the detection of an object smaller than the object being monitored, i.e., prevents unnecessary images that do not show the object being monitored, when imaging device 100 is a surveillance camera. Image processing unit 105 also references the number of objects in the event image that are larger than a predetermined size threshold when determining whether to output a frame image. This allows a frame image to be output only when multiple objects are detected, a condition that should be confirmed by the surveillance camera monitor, further preventing unnecessary output of images from the surveillance camera.

[0040] Furthermore, when transmitting frame images from transmission unit 107 to an external device, if there is a large amount of communication traffic other than image data or if the communication environment during wireless communication is poor, the transmission speed may slow down and it may not be possible to transmit frame images for each frame period. In this embodiment, image processing unit 105 performs the above-mentioned clipping process to reduce the amount of data transmitted per frame, thereby enabling the transmission of frame images. That is, in step S207, image processing unit 105 generates a frame image consisting of an image of a clipped range that includes an object detected from the frame image, thereby reducing the amount of data for the frame image to be transmitted.

[0041] FIG. 5 is a diagram used to explain the cropping range that includes a detected object. FIG. 5 shows the same event image 300 as described in FIG. 3. The image processing unit 105 sets a range in the event image 300 that includes objects 320 and 330, each of which is equal to or larger than a predetermined size threshold 310, as the cropping range 500 for a frame image generated from the image capture signal of the frame-based sensor 102. The image processing unit 105 then crops an image within the cropping range 500 from the frame image based on the image capture signal of the frame-based sensor 102 and outputs it as a frame image. This reduces the data volume of the output frame image, i.e., the amount of data transmitted from the transmission unit 107 to an external device. For example, the image processing unit 105 sets the cropping range 500 to a predetermined ratio of the vertical and horizontal sizes of the original frame image generated from the image capture signal of the frame-based sensor 102. For example, if the ratio of the cutout range 500 to the original frame image is 70%, the data volume of the cutout image will be 49% of the data volume of the original frame image, making it possible to transmit a frame image with approximately half the data volume of the original frame image.

[0042] Alternatively, image processing unit 105 may determine the size of cropping range 500 according to, for example, an existing resolution format. For example, image processing unit 105 may crop a range equivalent to full HD (1920 x 1080) from a frame image with a so-called 4k (3840 x 2160) resolution as the cropping range. In this case, the data volume of the frame image is reduced to about 25%, and the transmission speed required to transmit the frame image is also about 25%, making it possible to transmit it from transmission unit 107 to an external device.

[0043] Furthermore, in step S208, image processing unit 105 determines whether all objects are contained within cut-out range 500. In the example of Fig. 5, objects 320 and 330, each of which is equal to or larger than predetermined size threshold 310, are contained within cut-out range 500. However, there may be cases where, for example, multiple objects are located far apart and not all of the objects are contained within the cut-out range, or where an object is too large to be contained within the cut-out range. In this way, if not all of the objects are contained within cut-out range 500, the processing of image processing unit 105 proceeds to step S211.

[0044] On the other hand, if the object falls within the cut-out range 500, the image processing unit 105 determines in step S209 whether transmission is possible based on the data volume of the frame image of the cut-out range 500 and the transmission speed of the transmission unit 107. For example, even if the entire object falls within the cut-out range 500, the frame image may not be able to be transmitted depending on the data volume of the image of the cut-out range, the communication environment, other communication volume, etc. For this reason, the image processing unit 105 determines in step S209 whether the frame image of the cut-out range 500 can be transmitted. Then, if it is determined that transmission is possible, the image processing unit 105 outputs the frame image of the cut-out range 500 to the transmission unit 107 in step S210. On the other hand, if it is determined that transmission is not possible, the processing of the image processing unit 105 proceeds to step S211.

[0045] In step S211, the image processing unit 105 performs processing to reduce the frame rate at which frame images of the cropping range are transmitted to a predetermined rate. For example, if the frame rate of the frame-based sensor 102 is 60 Hz, the image processing unit 105 reduces the frame rate to 30 Hz, thereby approximately halving the data rate. Processing to reduce the frame rate from 60 Hz to 30 Hz can be performed by thinning out every other frame of frame images generated from the image capture signal of the frame-based sensor 102. Note that the image processing unit 105 may control the operation of the frame-based sensor 102 via the control unit 106, and set the frame rate of the image capture signal output by the frame-based sensor 102 to be reduced to 30 Hz.

[0046] After performing the above processing, the processing of the image processing unit 105 proceeds to step S212. In step S212, the image processing unit 105 further determines whether the frame image after the frame rate has been lowered can be transmitted. That is, even after the frame rate has been lowered, the frame image may not be able to be transmitted depending on the data amount of the frame image, the communication environment, other communication traffic, etc., so in step S212, the image processing unit 105 further determines whether the frame image can be transmitted. If it is determined that the frame image can be transmitted, the processing of the image processing unit 105 proceeds to step S213, and the image processing unit 105 outputs the frame image of the cropped range after the frame rate has been lowered to the transmission unit 107. As a result, the transmission unit 107 outputs the frame image to the imaging control device 120. In this way, the image processing unit 105 can reduce the transmission speed required for transmission by reducing the frame rate of the frame images.

[0047] Furthermore, in this embodiment, as described above, image processing unit 105 can output frame images from transmission unit 107 to the outside and also store them in storage unit 108. Therefore, by reading out the stored frame images and displaying them on display device 130, a monitor can check the images from the surveillance camera, etc. Note that while this embodiment has been described as an example in which imaging device 100 is equipped with transmission unit 107, imaging device 100 may also be configured without transmission unit 107. In this case as well, image processing unit 105 stores frame images in storage unit 108. By later retrieving the frame images stored in storage unit 108 and displaying them on display device 130, etc., a monitor can check the images from the surveillance camera, etc.

[0048] In the above description, an example was given in which the operation of the image processing unit 105 is realized by a GPU or a CPU executing the image processing program according to this embodiment, but the image processing unit 105 may be configured by a circuit or the like. Similarly, the control unit 106 may also be configured by a circuit or the like. In the above description, the image processing unit 105 and the control unit 106 are configured separately, but for example, a CPU may execute the image processing program and a control program according to this embodiment to realize the functions of both the image processing unit 105 and the control unit 106.

[0049] Furthermore, for example, at least one of the above-mentioned clipping process and the process of lowering the frame rate may be performed as a predetermined process regardless of the transmission speed. Furthermore, for example, when image quality adjustment is performed to adjust the brightness, color, etc. of a frame image output from imaging device 100, the output frame image is displayed on display device 130 and the adjustment is performed while viewing the displayed image. In other words, image quality adjustment of the brightness and color of an image needs to be performed while viewing the displayed frame image, regardless of whether or not a monitored object, etc. is captured. For this reason, when image quality adjustment is performed, image processing unit 105 does not perform the processing described above, but simply outputs the frame image generated from the imaging signal of imaging unit 101.

[0050] In the above embodiment, an example has been given in which the image processing unit 105 performs all of the processes in the flowchart of Fig. 2, but some of the processes may be omitted. For example, the image processing unit 105 may be configured not to perform at least one of steps S203 and S204, steps S207 to S210, and steps S211 to S213.

[0051] 2, if the image processing unit 105 determines in step S202 that an object larger than a predetermined size threshold is present, the image processing unit 105 proceeds to step S205. If the image processing unit 105 determines in step S205 that transmission is not possible, the image processing unit 105 proceeds to step S207. Furthermore, for example, if the processes of steps S207 to S210 are not performed, the image processing unit 105 advances the process to step S211 if it is determined in step S205 that transmission is not possible. Furthermore, for example, if the processes of steps S211 to S213 are not performed, the image processing unit 105 waits until the next frame period when it determines "No" in step S208 or step S209. Furthermore, for example, if the processes of steps S203, S204, and S207 to S209 are not performed, the image processing unit 105 advances the process to step S205 if it determines in step S202 that an object larger than a predetermined size threshold is present. If it determines in step S205 that transmission is not possible, the process of the image processing unit 105 advances to step S211. Furthermore, for example, if the processes of steps S203, S204, and S211 to S213 are not performed, and if the image processing unit 105 determines in step S202 that an object larger than a predetermined size threshold is present, the process proceeds to step S205. If the image processing unit 105 determines in step S205 that transmission is not possible, the process proceeds to step S207. Furthermore, if the image processing unit 105 determines "No" in step S208 or step S209, the image processing unit 105 waits until the next frame period to perform the process. Furthermore, for example, if the processes of steps S207 to S210 and steps S211 to S213 are not performed, and if the image processing unit 105 determines in step S202 that an object larger than a predetermined size threshold is present, the process proceeds to step S205. If the image processing unit 105 determines in step S205 that transmission is not possible, the image processing unit 105 waits until the next frame period to perform the process.

[0052] In addition, although the above-described embodiment has been described as an example of application to a surveillance camera, the present invention is not limited to this and can also be applied to, for example, an in-vehicle camera for taking images while parked, an agricultural camera, and the like.

[0053] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of specific embodiments for implementing the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0054] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) generating means for generating a frame image from an imaging signal of the frame-based sensor; a detection means for detecting a size of an area where a luminance change has occurred based on an address event signal detected by an event-based sensor that shares a real-space imaging range with the frame-based sensor; an output means for outputting the frame image when the size of the area where the luminance change has occurred is greater than a predetermined size threshold; 1. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the detection means generates an event image according to the address event signal, and detects the size of the area in which the luminance change occurs from the event image. (Configuration 3) 3. The image processing device according to claim 1, wherein the output means further outputs the frame image when the number of the regions larger than the predetermined size threshold is equal to or greater than a predetermined number threshold. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the output means cuts out an image of a predetermined cutout range including the area larger than the predetermined size threshold from the frame image generated from the imaging signal, and sets the image of the cutout cutout range as the frame image to be output. (Configuration 5) The image processing device according to configuration 4, wherein the output means, when the amount of data of the frame image generated from the imaging signal is greater than the amount of data that can be transmitted in one frame period, cuts out an image of the cutout range from the frame image generated from the imaging signal. (Configuration 6) The image processing device according to configuration 4 or 5, wherein the output means reduces the frame rate at which frame images are transmitted to a predetermined rate when it is not possible to include all of the areas larger than the predetermined size threshold within the cutout range. (Configuration 7) 7. The image processing device according to any one of configurations 4 to 6, wherein the predetermined cropping range is a range that is a predetermined ratio to the size of a frame image generated from an imaging signal of the frame base sensor. (Configuration 8) The image processing device according to configuration 4 or 5, wherein the output means cuts out a cutout range corresponding to a second format having a lower resolution than the first format from a frame image of a first format generated from an imaging signal of the frame-based sensor. (Configuration 9) The image processing device according to any one of configurations 1 to 3, wherein the output means reduces a frame rate for transmitting the frame image to a predetermined rate when the data amount of the frame image generated from the imaging signal is larger than the data amount that can be transmitted in one frame period. (Configuration 10) 10. The image processing device according to any one of configurations 1 to 9, wherein the output means outputs the frame image generated from the imaging signal when the data amount of the frame image generated from the imaging signal is smaller than the data amount that can be transmitted in one frame period. (Configuration 11) 11. The image processing device according to any one of configurations 1 to 10, wherein the predetermined size threshold for the size of the area where the luminance change has occurred is set according to a direction in which an imaging device equipped with the frame-based sensor and the event-based sensor images an imaging range in real space and a position in the frame image generated from the imaging signal. (Configuration 12) 12. The image processing device according to any one of configurations 1 to 11, wherein the output means outputs the frame image generated from the imaging signal of the frame-based sensor when image quality adjustment is performed on the frame image generated from the imaging signal. (Configuration 13) an imaging means including a frame-based sensor that captures an image and an event-based sensor that shares a real-space imaging range with the frame-based sensor; an image processing means including a generating means for generating a frame image from an imaging signal of the frame-based sensor, a detecting means for detecting the size of an area where a luminance change has occurred based on an address event signal in which the frame-based sensor has detected a luminance change, and an output means for outputting the frame image when the size of the area where the luminance change has occurred is greater than a predetermined size threshold; a means for transmitting the frame image output from the image processing means to an external device and / or storing the frame image output from the image processing means; An imaging device comprising: (Method 1) a generating step of generating a frame image from an imaging signal of the frame-based sensor; a detection step of detecting a size of an area where a luminance change has occurred based on an address event signal detected by an event-based sensor that shares a real-space imaging range with the frame-based sensor; an output step of outputting the frame image when the size of the region where the luminance change has occurred is greater than a predetermined size threshold; An image processing method comprising: (Program 1) 13. A program that causes a computer to function as the image processing device according to any one of the first to twelfth configurations. [Explanation of symbols]

[0055] 100: imaging device, 101: imaging unit, 102: frame-based sensor, 103: event-based sensor, 105: image processing unit, 106: control unit, 107: transmission unit, 108: storage unit

Claims

1. generating means for generating a frame image from an imaging signal of the frame-based sensor; a detection means for detecting a size of an area where a luminance change has occurred based on an address event signal detected by an event-based sensor that shares a real-space imaging range with the frame-based sensor; an output means for outputting the frame image when the size of the area where the luminance change has occurred is greater than a predetermined size threshold; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the detecting means generates an event image according to the address event signal, and detects the size of the area in which the luminance change occurs from the event image.

3. 2. The image processing apparatus according to claim 1, wherein said output means further outputs said frame image when the number of said areas larger than said predetermined size threshold is equal to or greater than a predetermined number threshold.

4. The image processing device according to any one of claims 1 to 3, characterized in that the output means cuts out an image of a predetermined cut-out range from the frame image generated from the imaging signal, the image including the area larger than the predetermined size threshold, and uses the image of the cut-out cut-out range as the frame image to be output.

5. The image processing device according to claim 4, characterized in that the output means cuts out an image of the cut-out range from the frame image generated from the imaging signal when the data amount of the frame image generated from the imaging signal is greater than the data amount that can be transmitted in one frame period.

6. 5. The image processing device according to claim 4, wherein the output means reduces the frame rate at which frame images are transmitted to a predetermined rate if it is not possible to include all of the areas larger than the predetermined size threshold within the cutout range.

7. 5. The image processing device according to claim 4, wherein the predetermined cropping range is a range that is a predetermined ratio to the size of a frame image generated from an image pickup signal of the frame base sensor.

8. 5. The image processing device according to claim 4, wherein the output means cuts out a cutout range corresponding to a second format having a lower resolution than the first format from a frame image of a first format generated from an imaging signal of the frame-based sensor.

9. 4. The image processing device according to claim 1, wherein the output means reduces the frame rate at which the frame image is transmitted to a predetermined rate when the amount of data of the frame image generated from the imaging signal is greater than the amount of data that can be transmitted in one frame period.

10. 4. The image processing device according to claim 1, wherein the output means outputs the frame image generated from the imaging signal when the data amount of the frame image generated from the imaging signal is smaller than the data amount that can be transmitted in one frame period.

11. 2. The image processing device according to claim 1, wherein the predetermined size threshold for the size of the area in which the luminance change has occurred is set according to the direction in which an imaging device equipped with the frame-based sensor and the event-based sensor captures an imaging range in real space and the position in the frame image generated from the imaging signal.

12. 2. The image processing device according to claim 1, wherein the output means outputs the frame image generated from the imaging signal of the frame-based sensor when image quality adjustment is performed on the frame image generated from the imaging signal.

13. an imaging means including a frame-based sensor that captures an image and an event-based sensor that shares a real-space imaging range with the frame-based sensor; an image processing means including a generating means for generating a frame image from an imaging signal of the frame-based sensor, a detecting means for detecting the size of an area where a luminance change has occurred based on an address event signal in which the frame-based sensor has detected a luminance change, and an output means for outputting the frame image when the size of the area where the luminance change has occurred is greater than a predetermined size threshold; a means for transmitting the frame image output from the image processing means to an external device and / or storing the frame image output from the image processing means; An imaging device comprising:

14. a generating step of generating a frame image from an imaging signal of the frame-based sensor; a detection step of detecting a size of an area where a luminance change has occurred based on an address event signal detected by an event-based sensor that shares a real-space imaging range with the frame-based sensor; an output step of outputting the frame image when the size of the region where the luminance change has occurred is greater than a predetermined size threshold; An image processing method comprising:

15. Computer, generating means for generating a frame image from an imaging signal of the frame-based sensor; a detection means for detecting a size of an area where a luminance change has occurred based on an address event signal detected by an event-based sensor that shares a real-space imaging range with the frame-based sensor; an output means for outputting the frame image when the size of the area where the luminance change has occurred is greater than a predetermined size threshold; A program that causes the image processing device to function as an image processing device having the above.

Images