Apparatus, computer program and method

Abstract

To provide a device, computer program and method that makes a determination quickly when a neural network is in a system operating in real time and there is an obstacle moving relative to a vehicle.SOLUTION: A method includes the steps of: receiving an image from a dynamic vision sensor and an image from an image sensor; determining, from the image received from the dynamic vision sensor, an operation region in a visual field of the image from the image sensor; making a comparison with the operation region in the image from the image sensor and applying higher-level image compression to a non-operation region in the image from the image sensor to generate a processed image; and outputting the processed image to a neural network, the dynamic vision sensor and image sensor capturing visual fields which at least overlap with each other.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present technology relates to an apparatus, a computer program and a method. [Background technology]

[0002] The "Background" discussion provided herein is intended to generally provide background to the present disclosure. The work of the presently named inventors is not expressly or impliedly admitted as prior art to the present technology, as are aspects of the specification that do not qualify as prior art at the time of filing, to the extent that they are described in the Background section.

[0003] As the use of neural networks is increasingly used in a variety of technologies, there is an increasing need for the classification probabilities output by the neural network to be more accurate and faster, which is particularly important in imaging systems used in safety-critical systems such as fully and partially autonomous vehicles. In these types of systems, the classification probabilities output by the neural network must be very accurate to ensure that the chances of making an incorrect decision are reduced. Furthermore, if the neural network is in a system that operates in real time and there are obstacles that are moving relative to the vehicle, decisions need to be made quickly.

[0004] It is an aim of the present disclosure to address one or more of these problems. Summary of the Invention

[0005] According to an embodiment of the present disclosure, a method includes receiving an image from a dynamic vision sensor and an image from an image sensor; determining from the image received from the dynamic vision sensor motion areas within a field of view of the image from the image sensor; applying a higher level of image compression to non-motion areas in the image from the image sensor compared to motion areas in the image from the image sensor to generate a processed image; and outputting the processed image to a neural network, A method is provided in which the dynamic vision sensor and the image sensor capture at least overlapping fields of view.

[0006] The preceding paragraphs have been provided by way of general introduction and are not intended to limit the scope of the claims which follow. The embodiments described herein, together with further advantages, are best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0007] A more complete understanding of the present disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description, when considered in conjunction with the accompanying drawings. [Figure 1] 1 illustrates a system 100 according to one embodiment of the present disclosure. [Diagram 2] 1 shows an apparatus 110 according to one embodiment of the present disclosure forming part of the system of FIG. [Figure 3-1A] 1 shows a representative image captured by an image sensor in system 100, illustrating one embodiment of the present disclosure. [Figure 3-1B] 1 shows a representative image captured by an image sensor in system 100, illustrating one embodiment of the present disclosure. [Figure 3-1C] 1 shows a representative image captured by an image sensor in system 100, illustrating one embodiment of the present disclosure. [Figure 3-1D] 1 shows a representative image captured by an image sensor in system 100, illustrating one embodiment of the present disclosure. [Figure 3-2A] 1 shows a representative image captured by a dynamic vision sensor in system 100 illustrating one embodiment of the present disclosure. [Figure 3-2B] 1 shows a representative image captured by a dynamic vision sensor in system 100 illustrating one embodiment of the present disclosure. [Figure 3-2C] 1 shows a representative image captured by a dynamic vision sensor in system 100 illustrating one embodiment of the present disclosure. [Figure 3-2D] 1 shows a representative image captured by a dynamic vision sensor in system 100 illustrating one embodiment of the present disclosure. [Figure 4A] 1 shows a representative image output from device 110, according to one embodiment of the present disclosure. [Figure 4B] 1 shows a representative image output from device 110, according to one embodiment of the present disclosure. [Figure 4C] 1 shows a representative image output from device 110, according to one embodiment of the present disclosure. [Figure 4D] 1 shows a representative image output from device 110, according to one embodiment of the present disclosure. [Diagram 5] 1 illustrates a process according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Referring now to the drawings, like reference numerals indicate identical or corresponding parts throughout the several views.

[0009] 1 illustrates a system 100 according to an embodiment of the present disclosure. The system 100 includes a dynamic vision sensor (DVS) 115, an image sensor 120, a device 110, and a machine learning algorithm 105. In the system 100 of FIG. 1, the DVS 115 and the image sensor 120 are connected to the device 110 and provide the device 110 with one or more images of the same field of view. Device 110 processes the image as described below and outputs the compressed image to machine learning algorithm 105. Machine learning algorithm 105 uses the compressed image to classify elements in the compressed image more quickly and with improved accuracy. In embodiments, machine learning algorithm 105 is used to determine pedestrians in autonomous or partially autonomous vehicles, for example. Of course, the disclosure is not so limited and machine learning algorithm 105 may be any machine learning algorithm.

[0010] As mentioned above, system 100 includes a dynamic vision sensor (DVS) 115 and an image sensor 120. As understood herein, DVS 115 is a known sensor that determines areas of motion within a captured video stream. In other words, DVS 115 captures a video stream of its field of view and determines any areas of movement within the video stream. One example of DVS 115 is the SEES Dynamic Vision Sensor manufactured by Insightness®, a Sony Group Company, although the disclosure is not limited thereto and any image sensor having DVS capabilities is contemplated.

[0011] With respect to image sensor 120, in some embodiments, image sensor 120 is an active pixel image sensor. These types of image sensors are known, and each pixel sensor unit cell includes a photodetector and one or more active transistors. The output from the active pixel image sensor is an RGB image of its field of view. An example of an image sensor 120 is the IMX304LLR manufactured by Sony Corporation, although this disclosure is not so limited and any image sensor is contemplated.

[0012] It should be understood that in embodiments of the present disclosure, the fields of view of DVS 115 and image sensor 120 are the same in embodiments. In other words, the motion regions at pixel locations in the overlapping fields of view as determined by DVS 115 occur at the same (or similar) pixel locations (regions of interest) and at any time (times of interest) in image sensor 120. This combination of region of interest and time of interest has the added benefit of more accurate triggering. In this regard, although DVS 115 and image sensor 120 are shown in FIG. 1 as separate image sensors, in embodiments DVS 115 and image sensor 120 may be integrated on the same semiconductor die (i.e., in the same semiconductor package) or may be two separate image sensors located within the same housing. This is advantageous because any operation applied to one image sensor is applied to the other image sensor as well, which makes it easier to maintain at least an overlapping field of view between DVS 115 and image sensor 120.

[0013] Of course, although the above embodiments have been described as having the same field of view for DVS 115 and image sensor 120, the disclosure is not so limited. The fields of view for DVS 115 and image sensor 120 may at least overlap. In this case, there will be a mapping between pixel locations in DVS 115 and corresponding pixel locations in image sensor 120. In other words, a particular pixel in DVS 115 will have a corresponding pixel in image sensor 120.

[0014] The device 110 will now be described with reference to FIG.

[0015] 2, an apparatus 110 according to multiple embodiments is shown. The apparatus 110 includes a processing circuit 1102. The processing circuit 1102 may be any type of circuit that performs steps according to embodiments of the present disclosure. For example, the processing circuit 1102 may be an application specific integrated circuit (ASIC) or a microprocessor operating under the control of a computer program (or computer software) to perform steps according to embodiments of the present disclosure.

[0016] The processing circuit 1102 is connected to the DVS 115 and the image sensor 120. This connection may be a wired or wireless connection. The purpose of the connection is to receive images from both the DVS 115 and the image sensor 120. As mentioned above, both the DVS 115 and the image sensor 120 have the same field of view in the embodiment of the present disclosure.

[0017] Additionally connected to the processing circuit 1102 is a memory device 1101. The memory device is any type of storage capable of storing computer programs or computer software executed on the processing circuit 1102 to control the processing circuit 1102 to perform methods according to embodiments of the present disclosure. Storage device 1101 can store images from one or both of DVS 115 and image sensor 120. In embodiments, storage device 1101 can be a solid-state storage device, such as a semiconductor storage device, or a magnetically or optically readable storage device. Although storage 1101 is shown as being included in device 110, the disclosure is not so limited and storage 1101 may be located remotely from device 110, such as across a network such as a local area network or the Internet.

[0018] Additionally connected to the processing circuitry 1102 are one or more position sensors 1103. The purpose of the position sensors is to determine the position, attitude, velocity and / or orientation of the image sensor 120 and / or the DVS 115. This means that the position sensor 1103 communicates with both the DVS sensor 115 and the image sensor 120. Of course, it will be understood that if the DVS sensor 115 and the image sensor 120 are located on the same semiconductor die, the position sensor 1103 need only communicate with one of the DVS sensor 115 and the image sensor 120.

[0019] As described below, device 110 is configured to use the motion information provided by the images captured by DVS 115 to determine the regions within the image sequence (video stream) in which the motion captured by image sensor 120 occurs. This is possible because DVS 115 and image sensor 120 have correspondence between pixels within each of DVS 115 and image sensor 120. In other words, processing circuitry 1102 knows the mapping between each pixel in DVS 115 and the corresponding pixel in image sensor 120. In some embodiments, this mapping is performed so that DVS 115 and image sensor 120 capture the same or at least overlapping fields of view.

[0020] Processing circuitry 1102 then applies a compression algorithm to the non-motion areas in the images received from image sensor 120. This compression algorithm applies a higher level of compression than that applied to the motion areas in the images received from image sensor 120. In some embodiments, no compression is applied to the motion areas in the images received from image sensor 120. In some embodiments, this difference in compression is achieved by having higher dynamic binning applied to pixels located in motion areas compared to those applied to non-motion areas. For example, dynamic binning may be applied to the RAW images captured by image sensor 120. In some embodiments, for non-dynamic regions, a higher or maximum level of compression can be applied in a lossy compression format such as JPEG, or a lossless compression format such as PNG, or any highly efficient image file format, whereas for regions of the image that are in motion, a lesser amount of compression (or no compression at all) is applied to the regions.

[0021] In some embodiments, the amount of motion in a region can be determined (i.e., not just whether motion occurs), in which case if the amount of motion in that region is below a threshold, no increased amount of compression is applied to that region, in other embodiments, the level of compression is increased in proportion to the amount of decrease in the level of motion in that region.

[0022] By selectively applying compression in this manner, a higher level of compression is applied to non-motion regions compared to motion regions, which means that data loss in the motion regions is much less than data loss in the non-motion regions, which is useful since the motion regions are typically most relevant for determining the classification probabilities output by a neural network. However, by selectively compressing the image as described above, the size of the compressed image is reduced, in other words, the amount of data passed between the device 110 and the machine learning algorithm 105. The compressed image is provided to the machine learning algorithm 105. Importantly, however, a higher signal-to-noise ratio is achieved for moving regions by applying less compression to them compared to non-moving regions, which lowers the maximum pixel deviation providing a better contrast ratio in the compressed image fed to the machine learning algorithm 105. Thus, compressed images result in greater accuracy in the classification performed by the machine learning algorithm 105. Furthermore, by reducing the size of the images passed to the machine learning algorithm 105, the time it takes for the classification to occur is reduced, in other words, the speed at which the classification occurs is increased.

[0023] 3-1A-3-1D show representative images captured by an image sensor in a system 100 illustrating an embodiment of the present disclosure. Figures 3-2A to 3-2D show representative images captured by a dynamic vision sensor in a system 100 illustrating an embodiment of the present disclosure, and Figures 4A to 4D show representative images output from an apparatus 110 according to an embodiment of the present disclosure.

[0024] An image of the person is captured by both DVS 115 and image sensor 120 because DVS 115 and image sensor 120 both have the same field of view, and therefore the image captured by DVS 115 is the same as the image captured by image sensor 120. This image is captured at time t = 0. The image captured by image sensor 120 is shown in Figure 3-1A, and the image captured by DVS 115 is shown in Figure 3-2A.

[0025] As will be appreciated by those skilled in the art, the output from DVS 115 is a binary image that indicates motion within the image. In particular, the output image from DVS 115 is black when no motion is detected and white when motion is detected. Since there is no motion in this image compared to the previous image in the image sequence (video stream), no areas of motion have been identified by the DVS 115. This means that the output from the DVS 115 is completely black. This is indicated by the diagonal lines in Figure 3-2 A. Of course, the present disclosure is not so limited, and the color representations shown above are merely exemplary, not limiting.

[0026] As mentioned above, the level of image compression applied to the image from image sensor 120 is at a higher level for non-motion areas as compared to motion areas within the image from the image sensor. Thus, the entire image from image sensor 120 at time t=0 has the same level of compression applied to the entire image since no motion is detected by DVS 115. Thus, referring to FIG 4A, an output image 310 having the same level of compression across the entire image is output to machine learning algorithm 105.

[0027] At time t=2, another image following time t=1 is captured. For example, this subsequent image may be captured as the next frame of a video stream. Referring now to FIG. 3-1C, an image of a person is shown captured by image sensor 120. When the person moves their arm between time t=1 and time t=2, the DVS 115 image identifies region 315 as a motion region. Device 110 determines the areas of motion from the image provided by DVS 115. As previously mentioned, in some embodiments, DVS 115 may transmit the entire binary image (i.e., an image having only black and white regions) so that processing circuitry 1102 in device 110 can determine the areas of motion. In some embodiments, DVS 115 may send only the pixel locations of the motion area, rather than the entire image, so that processing circuitry 1102 in device 110 can determine the motion area. For example, pixel locations of the four corners of area 305 may be sent to device 110. This reduces the amount of information provided between DVS 115 and device 110.

[0028] Since there is a mapping between the pixel locations of the motion areas in the image provided by the DVS 115 and the image provided by the image sensor, the unit 110 determines the motion areas in the image provided by the image sensor. As previously mentioned, although this disclosure is not limited thereto, this mapping is provided because in some embodiments the fields of view of DVS 115 and image sensor 120 are identical.

[0029] The device 110 then applies compression to non-motion areas in the image provided by the image sensor 120. Again, the compression applied to the non-motion areas in the image is higher than the compression applied to the motion areas in the image. As noted above, in some embodiments, no compression is applied to areas of motion within the image. In other embodiments, compression is applied to areas of motion within the image. However, the level of compression applied to non-motion areas within the image is greater than the compression applied to areas of motion within the image.

[0030] 4C, which shows the output of apparatus 110. Specifically, the amount of compression applied to non-motion areas in the image from image sensor 120 is higher than the amount of compression applied to motion areas in the image from image sensor 120. In particular, less compression is applied to the image in region 305 ″. The compressed image 310 ″ produced by device 110 is then output to machine learning algorithm 105 .

[0031] At time t=3, another image is captured after time t=2. For example, this subsequent image may be captured as the next frame of a video stream. Referring now to FIG. 3-1D, an image of a person is shown captured by image sensor 120. When the person moves their arm between time t=2 and time t=3, the DVS 115 image identifies region 320 as a motion region. The device 110 determines the motion area from the image provided by the DVS 115. As mentioned above, in some embodiments, the DVS 115 can transmit the entire binary image to enable the device 110 to determine the motion area. In some embodiments, the DVS 115 can send only the pixel locations of the motion area, rather than the entire image, to enable the device 110 to determine the motion area. For example, the pixel locations of the four corners of the area 305 can be sent to the device 110. This reduces the amount of information provided between DVS 115 and device 110.

[0032] Since there is a mapping between the pixel locations of the motion areas in the image provided by the DVS 115 and the image provided by the image sensor, the unit 110 determines the motion areas in the image provided by the image sensor. As previously mentioned, although this disclosure is not limited in this respect, in some embodiments, this mapping is provided because the fields of view of DVS 115 and image sensor 120 are identical.

[0033] The device 110 then applies compression to non-motion areas in the image provided by the image sensor 120. Again, the compression applied to the non-motion areas in the image is higher than the compression applied to the motion areas in the image. As noted above, in some embodiments, no compression is applied to areas of motion within the image. In other embodiments, compression is applied to areas of motion within the image. However, the level of compression applied to non-motion areas within the image is greater than the compression applied to areas of motion within the image.

[0034] 4D, where the output of apparatus 110 is shown. Specifically, a higher amount of compression is applied to non-motion regions in the image from image sensor 120 than to motion regions in the image from image sensor 120. In particular, less compression is applied in region 305''', and compressed image 310''' generated by apparatus 110 is output to machine learning algorithm 105.

[0035] As mentioned above, in an embodiment of the present disclosure, the system 100 is incorporated into a vehicle. For example, the system 100 may be included in a car, truck, motorcycle, etc. In this case, the system 100 is subject to the motion of the vehicle. Of course, the system 100 is subject to motion in other scenarios, such as when the system 100 is attached to a human, etc.

[0036] When system 100 is moving relative to its surroundings, the motion that affects the images captured by DVS 115 and image sensor 120 is due to the motion of system 100, not the motion of the subject in the captured image. For example, if the subject is a person walking down a street, if the system 100 is stationary, the person will move relative to the system 100 as he walks down the street. However, if the system 100 moves, the system 100 will detect the relative motion between the person and the system 100, since although the person may be stationary, there is relative motion between the system 100 and the person. This results in the DVS 115 inaccurately indicating the person's motion.

[0037] To reduce the likelihood of this situation occurring, according to an embodiment, position sensor 1103 is used as described below. As mentioned above, position sensor 1103 is connected to DVS 115 and image sensor 120. The purpose of position sensor 1103 is to determine the position (i.e., geographic position, attitude, velocity, and / or orientation) of DVS 115 and image sensor 120 when capturing an image. In other words, position sensor 1103 determines the position of DVS 115 and image sensor 120 between successively captured images. The positions of DVS 115 and image sensor 120 for each captured image are provided to processing circuitry 1102. Thus, processing circuitry 1102 determines the change in position of DVS 115 and image sensor 120 between successively captured images. This allows the movement of the DVS 115 and image sensor 120 to be determined by the processing circuitry 1102 .

[0038] Once the images captured by DVS 115 and image sensor 120 exhibit the same motion experienced by DVS 115 and image sensor 120, processing circuit 1102 compares the motion of DVS 115 and image sensor 120 to any motion of one or more objects in the images captured by DVS 115 and image sensor 120, respectively.

[0039] If the movement of one or more objects in the images captured by DVS 115 and image sensor 120 is greater than the amount of movement of DVS 115 and image sensor 120 detected by position sensor 1103, processing circuit 1102 determines that the objects in the captured images are moving.

[0040] In other examples, one or more objects in an image may be moving, but the amount of movement of the one or more objects in the image captured by DVS 115 and image sensor 120 may be less than the amount of movement of DVS 115 and image sensor 120 detected by position sensor 1103. Thus, in several embodiments, if the determined motion of the image sensor 120 and the dynamic vision sensor 115 differs from the motion determined from the image received from the dynamic vision sensor, the processing circuit 1102 determines that the subject in the captured image appears to be moving due to movement of the DVS 115 and the image sensor 120. If the movement of DVS 115 and image sensor 120 causes apparent motion in the captured image, the image is not output to the neural network. This reduces the likelihood of an erroneous decision being made by the neural network. Of course, this disclosure is not limited in this respect, and in some embodiments, the movement of DVS 115 and image sensor 120 causes apparent motion in the captured image, but the same level of compression is applied to the entire image.

[0041] Thus, in several embodiments, processing circuit 1102 determines that an object in an image captured by image sensor 120 is moving if the amount of object movement in successive images differs from that determined by processing circuit 1102. This allows the system 100 (and in particular the DVS 115 and image sensor 120 of the system 100) to be mounted on a moving object, such as a vehicle.

[0042] Although the foregoing description describes using position sensor 1103 to determine that detected motion is due to movement of system 100, this disclosure is not limited in this respect. In embodiments, motion of the entire image captured by DVS 115 and image sensor 120 also indicates that the detected motion is due to movement of DVS 115 and image sensor 120. In other words, if the entire image (as opposed to just a portion of the image) moves, this is most likely due to movement of DVS 115 and image sensor 120, rather than a subject in the image.

[0043]

[0046] Referring to Figure 5, embodiments of the present disclosure are illustrated. Specifically, Figure 5 depicts a flowchart 500 illustrating a method performed by the processing circuit 1102 according to embodiments of the present disclosure. The process begins at step 505. The process then moves to step 510 where the movement of the DVS 115 and image sensor 120 is determined by the processing circuit 1102. This is determined from position information provided by the position sensor 1103.

[0044] The process then moves to step 515, where processing circuit 1102 receives images captured by DVS 115 and image sensor 120. The process then proceeds to step 520, where processing circuit 1102 compares the motion of the object or objects in the images captured by image sensor 120 to the motion (movement) of DVS 115 and / or image sensor 120 determined in step 510. If the actions are not different then the "no" path is followed to step 535 where the process ends, whereas if the actions are different then the "yes" path is followed to step 525.

[0045] In step 525, processing circuitry 1102 determines motion areas within the field of view of the image captured by image sensor 120. The process then moves to step 530, where processing circuitry 1102 applies a higher level of image compression to non-motion areas compared to motion areas in the image from the image sensor.

[0046] It should be noted that steps 510 and 520 are optional and may be used in multiple embodiments of the present disclosure.

[0047] Numerous modifications and variations of the present disclosure are possible in light of the above teachings, and it is therefore to be understood that, within the scope of the appended claims, the present disclosure may be practiced otherwise than as specifically described herein.

[0048] To the extent that embodiments of the present disclosure are described as being implemented, at least in part, by a software-controlled data processing apparatus, it will be understood that non-transitory machine-readable media carrying such software, such as optical disks, magnetic disks, semiconductor memories, and the like, are also considered to represent an embodiment of the present disclosure.

[0049] It will be appreciated that for clarity of the invention, the above description has described embodiments with reference to different functional units, circuits and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuits and / or processors can be used without deviating from embodiments of the invention.

[0050] The embodiments described herein may be implemented in any suitable form including hardware, software, firmware or any combination of these. The embodiments described herein may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally, and logically implemented in any suitable way. Indeed, functionality may be implemented in a single unit, in multiple units, or as part of other functional units. Thus, embodiments of the present disclosure may be implemented in a single unit, or may be physically and functionally distributed between different units, circuits, and / or processors.

[0051] Although the present disclosure has been described in connection with several embodiments, it is not intended to be limited to the specific form set forth herein. Moreover, while features of the present disclosure appear to be described in connection with particular embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined in any manner suitable for implementing the present techniques.

[0052] Embodiments of the present technique can generally be described by the following numbered paragraphs: (1) receiving an image from a dynamic vision sensor and an image from an image sensor; determining, from images received from the dynamic vision sensor, an area of ​​motion within a field of view of an image from the image sensor; applying a higher level of image compression to non-motion areas in an image from the image sensor compared to motion areas in an image from the image sensor to generate a processed image; outputting the processed image to a neural network; Including, The dynamic vision sensor and the image sensor capture at least overlapping fields of view. method. (2) The method according to (1), receiving a plurality of positions from the dynamic vision sensor that define the motion area; method. (3) The method according to (2), The plurality of locations are pixel locations. method. (4) The method according to (2) or (3), The plurality of positions define a plurality of corners of the working area. method. (5) The method according to any one of (1) to (4), The amount of image compression is determined according to the amount of motion within the motion region. method. (6) The method according to any one of (1) to (5), Prior to applying said image compression, determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; Do not output images to the neural network. Also includes method. (7) The method according to any one of (1) to (5), Prior to applying said image compression, determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; applying the same image compression to the entire image; outputting the compressed image to the neural network; Also includes method. (8) The method according to (6) or (7), The motion of the dynamic vision sensor and the image sensor is determined using a position sensor. method. (9) receiving an image from the dynamic vision sensor and an image from the image sensor; determining, from the images received from the dynamic vision sensor, an area of ​​motion within a field of view of an image from the image sensor; applying a higher level of image compression to non-motion areas in an image from the image sensor compared to motion areas in an image from the image sensor to generate a processed image; outputting the processed image to a neural network; The dynamic vision sensor and the image sensor capture at least overlapping fields of view. The circuit is configured as follows: Device. (10) The device according to (9), The circuit comprises: receiving a plurality of positions from the dynamic vision sensor that define the motion area; It is configured as follows: Device. (11) The device according to (10), The plurality of locations are pixel locations. Device. (12) The device according to (10) or (11), The plurality of positions define a plurality of corners of the working area. Device. (13) The device according to any one of (9) to (12), The amount of image compression is determined according to the amount of motion within the motion region. Device. (14) The device according to any one of (9) to (13), Prior to applying the image compression, the circuit determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; Do not output images to the neural network. It is configured as follows: Device. (15) The device according to any one of (9) to (13), Prior to applying the image compression, the circuit determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; applying the same image compression to the entire image; Output the compressed image to the neural network. It is configured as follows: Device. (16) The device according to (14) or (15), The motion of the dynamic vision sensor and the image sensor is determined using a position sensor. Device. (17) A method comprising the steps of: (a) providing a method for performing a method for a computer comprising: Computer program. (18) A computer program product configured to store therein or thereon a computer program according to (17).

Claims

1. receiving an image from a dynamic vision sensor and an image from an image sensor; determining, from images received from the dynamic vision sensor, an area of ​​motion within a field of view of an image from the image sensor; applying a higher level of image compression to non-motion areas in an image from the image sensor compared to motion areas in an image from the image sensor to generate a processed image; outputting the processed image to a neural network; Including, Prior to applying said image compression, determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; Do not output images to the neural network. Further comprising: The dynamic vision sensor and the image sensor capture at least overlapping fields of view. method.

2. 2. The method of claim 1 , receiving a plurality of positions from the dynamic vision sensor that define the motion area; method.

3. 3. The method of claim 2, The plurality of locations are pixel locations. method.

4. 4. The method according to claim 2 or 3, The plurality of positions define a plurality of corners of the working area. method.

5. 2. The method of claim 1 , The amount of image compression is determined according to the amount of motion within the motion region. method.

6. 2. The method of claim 1 , Prior to applying said image compression, determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; applying the same image compression to the entire image; outputting the compressed image to the neural network; Also includes method.

7. 10. The method of claim 1 or 6, The motion of the dynamic vision sensor and the image sensor is determined using a position sensor. method.

8. Receives an image from the dynamic vision sensor and an image from the image sensor; determining, from the images received from the dynamic vision sensor, an area of ​​motion within a field of view of an image from the image sensor; applying a higher level of image compression to non-motion areas in an image from the image sensor compared to motion areas in an image from the image sensor to generate a processed image; outputting the processed image to a neural network; The dynamic vision sensor and the image sensor capture at least overlapping fields of view. The present invention provides a circuit configured as follows: Prior to applying the image compression, the circuit determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; Do not output images to the neural network. It is configured as follows: Device.

9. 9. The apparatus of claim 8, The circuit comprises: receiving a plurality of positions from the dynamic vision sensor that define the motion area; It is configured as follows: Device.

10. 10. The apparatus of claim 9, The plurality of locations are pixel locations. Device.

11. 11. An apparatus according to claim 9 or 10, The plurality of positions define a plurality of corners of the working area. Device.

12. 9. The apparatus of claim 8, The amount of image compression is determined according to the amount of motion within the motion region. Device.

13. 9. The apparatus of claim 8, Prior to applying the image compression, the circuit determining operation of the dynamic vision sensor and the image sensor; if the determined motion of the image sensor and the dynamic vision sensor differs from the motion determined from the image received from the dynamic vision sensor; applying the same image compression to the entire image; Output the compressed image to the neural network. It is configured as follows: Device.

14. 14. An apparatus according to claim 8 or 13, comprising: The motion of the dynamic vision sensor and the image sensor is determined using a position sensor. Device.

15. comprising computer readable instructions which, when loaded into a computer, configure the computer to carry out the method of claim 1. Computer program.

16. A computer program product adapted to store in or on it a computer program according to claim 15.

Images