Method, apparatus and storage medium for processing image

Abstract

The present application relates to a method, apparatus and storage medium for processing an image. The method comprises the steps of: receiving an image from a dynamic vision sensor and an image from an image sensor, wherein the dynamic vision sensor and the image sensor capture at least overlapping fields of view; determining a moving region in the field of view of the image from the image sensor from the image received from the dynamic vision sensor; applying a higher level of image compression to a non-moving region than to the moving region in the image from the image sensor to produce a processed image; and outputting the processed image to a neural network.

Description

Technical Field

[0001] This technology relates to an apparatus, computer program, and method. Background Technology

[0002] The “background” description provided herein is for the purpose of generally presenting the context of this disclosure. To the extent described in the background section, the work of the currently attributed inventors, and aspects of the description that may not conform to the prior art at the time of submission, are neither expressly nor implicitly acknowledged as prior art opposing this technology.

[0003] As neural networks are increasingly used in various technologies, the demand for more accurate classification probabilities and faster output from neural network outputs is growing. This is particularly important in image systems used in safety-critical systems such as fully autonomous and partially autonomous vehicles. In these types of systems, the classification probabilities output by the neural network must be highly accurate to ensure a low probability of making incorrect decisions. Furthermore, given that neural networks operate in real-time systems with obstacles moving relative to the vehicle, decisions must be made rapidly.

[0004] The purpose of this disclosure is to address one or more of these problems. Summary of the Invention

[0005] According to embodiments of the present disclosure, a method is provided, comprising the steps of: receiving an image from a dynamic vision sensor and an image from an image sensor, wherein the dynamic vision sensor and the image sensor capture at least overlapping fields of view; determining a moving region in the field of view of the image from the image sensor based on the image received from the dynamic vision sensor; applying a higher level of image compression to the non-moving region compared to the moving region in the image from the image sensor to produce a processed image; and outputting the processed image to a neural network.

[0006] The preceding paragraphs are provided by way of overview and are not intended to limit the scope of the technical solutions. The described embodiments and other advantages will be best understood by referring to the following detailed description taken in conjunction with the accompanying drawings. Attached Figure Description

[0007] A more complete understanding of this disclosure and its many accompanying advantages will be readily obtained by referring to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0008] Figure 1 A system 100 according to an embodiment of the present disclosure is shown;

[0009] Figure 2 A device 110 according to an embodiment of the present disclosure is shown, the device 110 being formed Figure 1 Part of the system;

[0010] Figures 3-1A to 3-1D Representative images captured by the image sensor in system 100 are shown to illustrate embodiments of this disclosure;

[0011] Figures 3-2A to 3-2D Representative images captured by a dynamic vision sensor in system 100 are shown to illustrate embodiments of this disclosure;

[0012] Figures 4A to 4D A representative image output from slave device 110 according to an embodiment of the present disclosure is shown; and

[0013] Figure 5 A process according to an embodiment of this disclosure is illustrated. Detailed Implementation

[0014] Referring now to the accompanying drawings, which run through several views, the same reference numerals denote the same or corresponding parts.

[0015] Figure 1 A system 100 according to an embodiment of the present disclosure is illustrated. System 100 includes a dynamic vision sensor (DVS) 115, an image sensor 120, a device 110, and a machine learning algorithm 105. Figure 1 In system 100, a DVS 115 and an image sensor 120 are connected to device 110 and provide device 110 with one or more images of the same field of view. Device 110 processes the images as will be described and outputs a compressed image to machine learning algorithm 105. Machine learning algorithm 105 uses the compressed image to improve accuracy and classify elements within the compressed image more quickly. In an embodiment, machine learning algorithm 105 is used to identify pedestrians, for example, in an autonomous or partially autonomous vehicle. Of course, this disclosure is not limited thereto, and machine learning algorithm 105 can be any machine learning algorithm.

[0016] As described above, system 100 includes a dynamic vision sensor (DVS) 115 and an image sensor 120. As will be understood, the DVS 115 is a known sensor that determines moving regions within a captured video stream. In other words, the DVS 115 captures the video stream of its field of view and determines any moving regions within the video stream. An example of the DVS 115 is the SEES dynamic vision sensor manufactured by Insightness®, Sonygroup company, although this disclosure is not limited thereto, and any image sensor with DVS capability is contemplated.

[0017] Regarding image sensor 120, in this embodiment, image sensor 120 is an active pixel image sensor. These types of image sensors are known, and each pixel sensor unit has a photodetector and one or more active transistors. The output of an active pixel image sensor is an RGB image of its field of view. An example of image sensor 120 is the IMX304LLR manufactured by Sony Corporation®, although this disclosure is not limited thereto, and any image sensor is contemplated.

[0018] It should be understood that in the embodiments of this disclosure, the fields of view of DVS 115 and image sensor 120 are the same. In other words, the region of movement at the pixel location within the overlapping field of view determined by DVS 115 will occur at the same (or similar) pixel location (region of interest) and optionally at a time (time of interest) within image sensor 120. This combination of region of interest and time of interest has the additional advantage of more precise triggering. In this regard, although DVS 115 and image sensor 120 are in Figure 1 While shown as separate image sensors, in embodiments, DVS 115 and image sensor 120 may be integrated onto the same semiconductor die (i.e., within the same semiconductor package), or they may be two separate image sensors located within the same housing. This is advantageous because any movement applied to one image sensor will be equally applied to the other. This makes it easier to maintain at least some overlap in the field of view between DVS 115 and image sensor 120.

[0019] Of course, although the above embodiments discussed that the fields of view of DVS 115 and image sensor 120 are the same, this disclosure is not limited thereto. The fields of view of DVS 115 and image sensor 120 may at least overlap. In this case, there will be a mapping between pixel positions in DVS 115 and corresponding pixel positions in image sensor 120. In other words, a particular pixel in DVS 115 will have a corresponding pixel in image sensor 120.

[0020] Now refer to Figure 2 Describe device 110.

[0021] Reference Figure 2 The diagram illustrates a device 110 according to an embodiment. Device 110 includes processing circuitry 1102. Processing circuitry 1102 can be any type of circuitry that performs the steps according to embodiments of this disclosure. For example, processing circuitry 1102 can be an application-specific integrated circuit (ASIC) or microprocessor that operates under the control of a computer program (or computer software) to perform the steps according to embodiments of this disclosure.

[0022] Processing circuitry 1102 is connected to DVS 115 and image sensor 120. This connection can be wired or wireless. The purpose of the connection is to receive images from both DVS 115 and image sensor 120. As described above, in embodiments of this disclosure, both DVS 115 and image sensor 120 have the same field of view.

[0023] Additionally connected to the processing circuitry 1102 is a memory 1101. The memory can be any kind of memory capable of storing a computer program or computer software that runs on the processing circuitry 1102 to control the processing circuitry 1102 to execute methods according to embodiments of this disclosure. The memory 1101 can store images from one or both of the DVS 115 and the image sensor 120. In embodiments, the memory 1101 can be a solid-state memory, such as a semiconductor memory, a magnetically readable memory, or an optically readable memory. Although the memory 1101 is shown as included in the device 110, this disclosure is not limited thereto, and the memory 1101 can be located at a remote location, such as via a network connected to the device 110, such as a local area network or the Internet.

[0024] Additionally connected to the processing circuitry 1102 are one or more position sensors 1103. The purpose of the position sensors is to determine the position, attitude, velocity, and / or orientation of the image sensor 120 and / or the DVS 115. This means that the position sensor 1103 communicates with both the DVS sensor 115 and the image sensor 120. Of course, it will be understood that if the DVS sensor 115 and the image sensor 120 are located on the same semiconductor die, the position sensor 1103 will need to communicate with only one of the DVS sensor 115 and the image sensor 120.

[0025] As will be explained later, device 110 is configured to use motion information provided by images captured by DVS 115 to determine the region of motion occurring within an image sequence (video stream) captured by image sensor 120. This is possible because DVS 115 and image sensor 120 have a correspondence between pixels within each of DVS 115 and image sensor 120. In other words, processing circuitry 1102 knows the mapping between each pixel in DVS 115 and its corresponding pixel in image sensor 120. In this embodiment, this mapping occurs because DVS 115 and image sensor 120 capture the same or at least overlapping fields of view.

[0026] Then, processing circuitry 1102 applies a compression algorithm to non-moving regions of the image received from image sensor 120. This compression algorithm applies a higher level of compression compared to that applied to moving regions of the image received from image sensor 120. In some embodiments, compression is not applied to moving regions of the image received from image sensor 120. In embodiments, this compression difference is achieved by applying a higher level of dynamic merging to pixels located in moving regions compared to dynamic merging applied to non-moving regions. For example, dynamic merging may be applied to RAW (raw) images captured by image sensor 120. In embodiments, for non-moving regions, a higher or maximum level of compression may be applied to lossy compression formats such as JPEG, lossless compression formats such as PNG, or any efficient image file format. In moving image regions, less compression (or no compression at all) is applied to that region.

[0027] In one embodiment, the amount of movement within a region can be determined (i.e., not just whether movement occurs). In this case, if the amount of movement in the region is below a threshold, the amount of compression applied to that region does not increase. In other embodiments, the compression level increases based on the amount of decrease in the level of movement within the region.

[0028] By selectively applying compression in this way, a higher level of compression is applied to non-moving regions compared to moving regions. This means that much less data is lost in moving regions than in non-moving regions. This is useful because moving regions are generally most relevant to determining the classification probability of the neural network output. However, by selectively compressing the image as described, the size of the compressed image is reduced. In other words, the amount of data transferred between device 110 and machine learning algorithm 105 is reduced. The compressed image is provided to machine learning algorithm 105. Importantly, however, a higher signal-to-noise ratio is achieved in moving regions by applying less compression to them compared to non-moving regions. This reduces the maximum pixel deviation, which provides better contrast in the compressed image fed into machine learning algorithm 105. Therefore, image compression leads to higher accuracy in the classification performed by machine learning algorithm 105. Furthermore, by reducing the size of the image passed to machine learning algorithm 105, the time spent on classification is reduced. In other words, the speed of classification is increased.

[0029] Figures 3-1A to 3-1D Representative images captured by the image sensor in system 100 are shown, illustrating embodiments of this disclosure. Figures 3-2A to 3-2D Representative images captured by the dynamic vision sensor in system 100 are shown, illustrating embodiments of the invention, and Figures 4A to 4D A representative image output from device 110 according to an embodiment of the present invention is shown.

[0030] The image of the person was captured by both DVS115 and image sensor 120. This is because both DVS115 and image sensor 120 have the same field of view, therefore the image captured by DVS115 is the same as the image captured by image sensor 120. This image was captured at time t=0. The image captured by image sensor 120 at... Figure 3-1A The image shown, and captured by the DVS 115, is in Figure 3-2A As shown in the image.

[0031] As those skilled in the art will understand, the output from the DVS 115 is a binary image indicating movement within an image. Specifically, the output image from the DVS 115 will be black where no movement was detected and white where movement was detected. Since there is no movement in this image compared to previous images in the image sequence (video stream), the DVS 115 has not identified any moving areas. This means the output from the DVS 115 is completely black. This is in Figure 3-2A The colors are shown as shaded lines. Of course, this disclosure is not limited thereto, and the color representations shown above are merely illustrative and therefore not restrictive.

[0032] As described above, the image compression level applied from image sensor 120 to the image is higher for areas without movement compared to moving areas in the image from the image sensor. Therefore, at time t=0, the entire image from image sensor 120 has the same level of compression applied to the entire image because DVS 115 did not detect any movement. Therefore, referring to... Figure 4A The output image 310, which has the same compression level across the entire image, is output to the machine learning algorithm 105.

[0033] At time t=2, capture another image that occurs after time t=1. For example, this subsequent image can be captured as the next frame in the video stream. Now refer to... Figure 3-1C The image shown is an image of a person captured by image sensor 120. Because the person moved their arm between time t=1 and time t=2, the DVS 115 image identifies region 315 as a moving region. Device 110 determines the moving region based on the image provided by DVS 115. As previously described, in some embodiments, DVS 115 may send the entire binary image (i.e., an image with only black and white areas) to allow processing circuitry 1102 within device 110 to determine the moving region. In embodiments, DVS 115 may send only the pixel positions of the moving region instead of the entire image to allow processing circuitry 1102 within device 110 to determine the moving region. For example, the pixel positions of the four corners of region 305 may be sent to device 110. This reduces the amount of information provided between DVS 115 and device 110.

[0034] Because there is a mapping between the pixel positions of the moving area in the image provided by DVS 115 and the image provided by image sensor 120, device 110 determines the moving area in the image provided by image sensor 120. As previously described, in embodiments, this mapping is provided because DVS 115 and image sensor 120 have the same field of view, although this disclosure is not limited thereto.

[0035] Then, device 110 applies compression to the non-moving regions within the image provided by image sensor 120. Again, the compression applied to the non-moving regions is higher than any compression applied to the moving regions within the image. As described above, in this embodiment, no compression is applied to the moving regions within the image. In other embodiments, compression is applied to the moving regions within the image. However, the level of compression applied to the non-moving regions within the image is higher than the level of compression applied to the moving regions within the image.

[0036] This is Figure 4C The output of device 110 is illustrated in the figure. Specifically, the compression applied to the motionless regions of the image from image sensor 120 is higher than the compression applied to the moving regions of the image from image sensor 120. In particular, less compression is applied to region 305''. The compressed image 310'' generated by device 110 is then output to machine learning algorithm 105.

[0037] At time t=3, another image is captured after time t=2. For example, this subsequent image can be captured as the next frame in the video stream. Now refer to... Figure 3-1D The image shown is an image of a person captured by image sensor 120. Because the person moved their arm between time t=2 and time t=3, the DVS 115 image identifies region 320 as a moving region. Device 110 determines the moving region based on the image provided by DVS 115. As previously described, in some embodiments, DVS 115 may send the entire binary image to allow the device to determine the moving region. In embodiments, DVS 115 may send only the pixel positions of the moving region instead of the entire image to allow the device to determine the moving region. For example, the pixel positions of the four corners of region 305 may be sent to device 110. This reduces the amount of information provided between DVS 115 and device 110.

[0038] Because there is a mapping between the pixel positions of the moving area in the image provided by DVS 115 and the image provided by image sensor 120, device 110 determines the moving area in the image provided by image sensor 120. As previously described, in embodiments, this mapping is provided because DVS 115 and image sensor 120 have the same field of view, although this disclosure is not limited thereto.

[0039] Then, device 110 applies compression to the non-moving regions within the image provided by image sensor 120. Again, the compression applied to the non-moving regions is higher than any compression applied to the moving regions within the image. As described above, in this embodiment, no compression is applied to the moving regions within the image. In other embodiments, compression is applied to the moving regions within the image. However, the level of compression applied to the non-moving regions within the image is higher than the level of compression applied to the moving regions within the image.

[0040] This is Figure 4D The output of device 110 is illustrated in the figure. Specifically, the compression applied to the motionless regions of the image from image sensor 120 is higher than the compression applied to the moving regions of the image from image sensor 120. In particular, less compression is applied to region 305'''. The compressed image 310''' generated by device 110 is then output to machine learning algorithm 105.

[0041] As described above, in embodiments of this disclosure, system 100 is integrated into a vehicle. For example, system 100 may be included in a car, truck, motorcycle, etc. In this case, system 100 will be subject to vehicle movement. Of course, system 100 will be subject to movement in other situations, such as if system 100 is on a person.

[0042] When system 100 moves relative to its surroundings, the movement affecting the image captured by DVS 115 and image sensor 120 may be due to the movement of system 100 rather than the movement of objects within the captured image. For example, if the object is a person walking along a street, when system 100 is stationary, the person will move relative to system 100 as they walk along the street. However, when system 100 is moving, the person may be stationary, but due to the relative movement between system 100 and the person, system 100 will detect the relative movement between the person and system 100. This will cause DVS 115 to incorrectly indicate the person's movement.

[0043] To reduce the likelihood of this occurring, according to an embodiment, a position sensor 1103 is used as will be explained. As described above, the position sensor 1103 is connected to the DVS 115 and the image sensor 120. The purpose of the position sensor 1103 is to determine the position (i.e., geographic location, attitude, velocity, and / or orientation) of the DVS 115 and the image sensor 120 during image capture. In other words, the position sensor 1103 determines the position of the DVS 115 and the image sensor 120 between consecutive image captures. The position of the DVS 115 and the image sensor 120 for each captured image is provided to the processing circuitry 1102. Therefore, the processing circuitry 1102 determines the positional changes of the DVS 115 and the image sensor 120 between consecutive image captures. This allows the processing circuitry 1102 to determine the movement of the DVS 115 and the image sensor 120.

[0044] Since the images captured by DVS 115 and image sensor 120 will undergo the same movement as DVS 115 and image sensor 120, the processing circuit 1102 compares the movement of DVS 115 and image sensor 120 with any movement of one or more objects within the images captured by DVS 115 and image sensor 120, respectively.

[0045] If the movement of one or more objects in an image captured by DVS 115 and image sensor 120 is greater than the amount of movement of DVS 115 and image sensor 120 detected by position sensor 1103, then processing circuit 1102 determines that the objects in the captured image are moving.

[0046] In other instances, although one or more objects in the image may move, the amount of movement of one or more objects within the image captured by DVS 115 and image sensor 120 may be less than the amount of movement of DVS 115 and image sensor 120 detected by position sensor 1103. Therefore, in an embodiment, when the movement determined by image sensor 120 and dynamic vision sensor 115 is the same as the movement determined based on the image received from the dynamic vision sensor, processing circuitry 1102 determines that the movement of DVS 115 and image sensor 120 makes the object in the captured image appear to be moving. In cases where the movement of DVS 115 and image sensor 120 results in significant movement in the captured image, no image is output to the neural network. This reduces the likelihood of the neural network making incorrect decisions. Of course, this disclosure is not limited to this, and in embodiments, for instances where the movement of DVS 115 and image sensor 120 results in significant movement in the captured image, the same level of compression is applied to the entire image.

[0047] Therefore, in the embodiment, the processing circuit 1102 determines that an object is moving within an image captured by the image sensor 120 when the amount of movement of an object within a continuous image differs from the amount of movement determined by the processing circuit 1102. This allows the system 100 (especially the DVS 115 of the system 100 and the image sensor 120) to be mounted on a moving object such as a vehicle.

[0048] While the foregoing describes the use of position sensor 1103 to determine that detected movement is caused by movement of system 100, this disclosure is not limited thereto. In embodiments, movement of the entire image captured by DVS 115 and image sensor 120 will also indicate that the detected movement is caused by movement of DVS 115 and image sensor 120. In other words, if the entire image moves (rather than a portion of the image), it is likely due to movement of DVS 115 and image sensor 120 rather than movement of objects in the image.

[0049] Reference Figure 5 The following illustrates embodiments of this disclosure. Specifically, Figure 5 A flowchart 500 describes a method performed by the processing circuit 1102 according to embodiments of the present disclosure.

[0050] The process begins at step 505. Then, the process moves to step 510, where processing circuitry 1102 determines the movement of DVS 115 and image sensor 120. This is determined based on position information provided by position sensor 1103.

[0051] The process then moves to step 515, where processing circuitry 1102 receives images captured by DVS 115 and image sensor 120. The process then moves to step 520, where processing circuitry 1102 compares the movement of one or more objects within the image captured by image sensor 120 with the movement of DVS 115 and / or image sensor 120 determined in step 510. If the movements are not different, the "No" path proceeds to step 535, and the process ends. However, if the movements are different, the "Yes" path proceeds to step 525.

[0052] In step 525, processing circuit 1102 determines the moving region in the field of view of the image captured by image sensor 120. The process then moves to step 530, where processing circuit 1102 applies a higher level of image compression to the non-moving region compared to the moving region in the image from the image sensor.

[0053] It should be noted that steps 510 and 520 are optional and used in the embodiments of this disclosure.

[0054] Based on the foregoing teachings, many modifications and variations of this disclosure are possible. Therefore, it should be understood that, within the scope of the appended technical solutions, this disclosure can be implemented in ways other than those specifically described herein.

[0055] In the context of embodiments of this disclosure being described as being implemented at least in part by a data processing apparatus controlled by software, it will be understood that non-transitory machine-readable media carrying such software, such as optical discs, magnetic disks, semiconductor memories, etc., are also considered to represent embodiments of this disclosure.

[0056] It should be understood that, for clarity, the above description refers to different functional units, circuits, and / or processors in the embodiments. However, it will be apparent that any suitable functional distribution among the different functional units, circuits, and / or processors can be used without departing from the embodiments.

[0057] The described embodiments can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The described embodiments can optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. Elements and components of any embodiment can be implemented physically, functionally, and logically in any suitable manner. In practice, the functionality can be implemented in a single unit, in multiple units, or as part of other functional units. Thus, the disclosed embodiments can be implemented in a single unit or can be physically and functionally distributed among different units, circuits, and / or processors.

[0058] Although this disclosure has been described in conjunction with some embodiments, it is not intended to limit it to the specific forms set forth herein. Furthermore, while features may appear to be described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments can be combined in any manner suitable for implementing the technology.

[0059] Embodiments of this technology can generally be described by the following numbered clauses:

[0060] 1. A method comprising the following steps:

[0061] Receives images from a dynamic vision sensor and an image from an image sensor, wherein the dynamic vision sensor and the image sensor capture at least overlapping fields of view;

[0062] The moving region in the field of view of the image from the image sensor is determined based on the image received from the dynamic vision sensor;

[0063] A higher level of image compression is applied to non-moving regions compared to moving regions in images from an image sensor to produce a processed image; and

[0064] The processed image is output to the neural network.

[0065] 2. The method described in Clause 1, comprising:

[0066] Receive multiple locations within a defined movement area from a dynamic vision sensor.

[0067] 3. The method according to Clause 2, wherein the plurality of locations are pixel locations.

[0068] 4. The method according to any one of Clauses 2 or 3, wherein the plurality of positions define the angles of the movement area.

[0069] 5. The method according to any of the preceding clauses, wherein the image compression amount is determined based on the amount of movement within the region.

[0070] 6. The method according to any of the preceding clauses, wherein, before applying image compression, the method comprises:

[0071] Determining the movement of a dynamic vision sensor and an image sensor; and where the determined movement of the image sensor and the dynamic vision sensor is the same as the movement determined based on an image received from the dynamic vision sensor, the method includes:

[0072] Do not output images to the neural network.

[0073] 7. The method according to any one of clauses 1 to 5, wherein, before applying image compression, the method comprises:

[0074] Determining the movement of a dynamic vision sensor and an image sensor; and where the determined movement of the image sensor and the dynamic vision sensor is the same as the movement determined based on an image received from the dynamic vision sensor, the method includes:

[0075] Apply the same compression to the entire image, and;

[0076] The compressed image is output to the neural network.

[0077] 8. The method according to Clause 6 or 7, wherein the movement of the dynamic vision sensor and the image sensor is performed using a position sensor.

[0078] 9. An apparatus comprising a circuit configured to:

[0079] Receives images from a dynamic vision sensor and an image from an image sensor, wherein the dynamic vision sensor and the image sensor capture at least overlapping fields of view;

[0080] The moving region in the field of view of the image from the image sensor is determined based on the image received from the dynamic vision sensor;

[0081] A higher level of image compression is applied to non-moving regions compared to moving regions in images from an image sensor to produce a processed image; and

[0082] The processed image is output to the neural network.

[0083] 10. The apparatus according to clause 9, wherein the circuitry is configured to:

[0084] Receive multiple locations within a defined movement area from a dynamic vision sensor.

[0085] 11. The apparatus according to Clause 10, wherein the plurality of locations are pixel locations.

[0086] 12. The apparatus according to any one of clauses 10 or 11, wherein a plurality of positions define an angle of the movement area.

[0087] 13. The apparatus according to any one of clauses 9 to 12, wherein the amount of image compression is determined based on the amount of movement within the region.

[0088] 14. The apparatus according to any one of clauses 9 to 13, wherein, prior to applying image compression, the circuitry is configured to:

[0089] The movement of the dynamic vision sensor and the image sensor is determined; and if the determined movement of the image sensor and the dynamic vision sensor is the same as the movement determined based on the image received from the dynamic vision sensor, the circuit is configured to:

[0090] Do not output images to the neural network.

[0091] 15. The apparatus according to any one of clauses 9 to 13, wherein, prior to applying image compression, the circuitry is configured to:

[0092] The movement of the dynamic vision sensor and the image sensor is determined; and if the determined movement of the image sensor and the dynamic vision sensor is the same as the movement determined based on the image received from the dynamic vision sensor, the circuit is configured to:

[0093] Apply the same compression to the entire image, and;

[0094] The compressed image is output to the neural network.

[0095] 16. The apparatus according to clause 16 or 15, wherein a position sensor is used to perform the movement of the dynamic vision sensor and the image sensor.

[0096] 17. A computer program comprising computer-readable instructions that, when loaded onto a computer, configure the computer to perform the method pursuant to any one of clauses 1 to 8.

[0097] 18. A computer program product configured to store the computer program described in Article 17 thereon or thereon.

Claims

1. A method for processing images, comprising the steps of: determining movement of a dynamic vision sensor and an image sensor; receiving images from the dynamic vision sensor and images from the image sensor, wherein the dynamic vision sensor and the image sensor capture at least overlapping fields of view; comparing the determined movement of the dynamic vision sensor and the image sensor and movement determined from images received from the dynamic vision sensor; based on a result of the comparison, determining a region of movement in the field of view from the image sensor from images received from the dynamic vision sensor; applying a higher level of image compression to regions of no movement compared to the region of movement in images from the image sensor to produce a processed image; and outputting the processed image to a neural network.

2. The method of claim 1, comprising: receiving a plurality of locations from the dynamic vision sensor defining the region of movement.

3. The method of claim 2, wherein, The plurality of locations are pixel locations.

4. The method of any one of claims 2 or 3, wherein, The plurality of locations define corners of the region of movement.

5. The method of claim 1, wherein, An amount of image compression is determined from an amount of movement within a region.

6. The method of claim 1, wherein, Prior to applying the image compression, the method comprises: in the event that the determined movement of the image sensor and the dynamic vision sensor is the same as movement determined from images received from the dynamic vision sensor, the method comprises: not outputting an image to the neural network.

7. The method of claim 1, wherein, Prior to applying the image compression, the method comprises: in the event that the determined movement of the image sensor and the dynamic vision sensor is the same as movement determined from images received from the dynamic vision sensor, the method comprises: applying the same compression to the entire image, and; outputting the compressed image to the neural network.

8. The method of claim 6 or 7, wherein, detecting movement of the dynamic vision sensor and the image sensor using a location sensor.

9. An apparatus for processing images, the apparatus comprising circuitry configured to: determine movement of a dynamic vision sensor and an image sensor; receiving images from the dynamic vision sensor and images from the image sensor, wherein, the dynamic vision sensor and the image sensor capture at least overlapping fields of view; compare the determined movement of the dynamic vision sensor and the image sensor and movement determined from images received from the dynamic vision sensor; based on a result of the comparison, determine a region of movement in the field of view from the image sensor from images received from the dynamic vision sensor; apply a higher level of image compression to regions of no movement compared to the region of movement in images from the image sensor to produce a processed image; and output the processed image to a neural network. The circuitry is configured to:

10. The apparatus of claim 9, wherein, receive a plurality of locations from the dynamic vision sensor defining the region of movement. The plurality of locations are pixel locations.

11. The apparatus of claim 10, wherein, The plurality of locations define corners of the region of movement.

12. The apparatus of any one of claims 10 or 11, wherein, An amount of image compression is determined from an amount of movement within a region.

13. The apparatus of claim 9, wherein, Prior to applying the image compression, the circuitry is configured to:

14. The apparatus of claim 9, wherein, in the event that the determined movement of the image sensor and the dynamic vision sensor is the same as movement determined from images received from the dynamic vision sensor, the circuitry is configured to: not output an image to the neural network. in case the determined movement of the image sensor and the dynamic vision sensor is the same as determined from images received from the dynamic vision sensor, the circuit is configured to: not output images to the neural network.

15. The apparatus of claim 9, wherein, before applying the image compression, the circuit is configured to: in case the determined movement of the image sensor and the dynamic vision sensor is the same as determined from images received from the dynamic vision sensor, the circuit is configured to: apply the same compression to the entire image; output the compressed image to the neural network.

16. The apparatus of claim 14 or 15, wherein, The movement of the dynamic vision sensor and the image sensor is detected using a position sensor.

17. A storage medium having stored therein a computer program comprising computer readable instructions which, when loaded onto a computer comprising the storage medium, configure the computer to perform the method according to claim 1.

Images