Image processing device, image processing method, computer program, and computer readable recording medium

Abstract

To provide an image recognition technique having a low computational load.SOLUTION: An image processing device is configured to: calculate a difference between an acquired current frame image and a previous frame image stored in a memory and identify a first region in which pixels have changed and a second region in which pixels have not changed on the current frame image; subject the first region to convolution processing to generate first region convolution data; extract second region convolution data corresponding to the second region from previous frame convolution data stored in the memory; generate current frame convolution data on the basis of the generated first region convolution data and the extracted second region convolution data; and store the current frame image in the memory as the previous frame image and store the current frame convolution data in the memory as the previous frame convolution data. Acquisition processing, identification processing, generation processing or extraction processing, and synthesis processing are repeatedly executed in this order.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology disclosed in this specification relates to an image processing device that uses a neural network.

[0002] Image recognition technology using convolutional neural networks is known. This technology requires a large number of convolutional layers, resulting in a very large amount of computation. Patent Document 1 discloses a method for reducing the amount of computation by extracting objects using a recognition algorithm different from convolutional neural networks and performing computations on only the extracted objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-98250 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 requires an extraction process for extracting objects in addition to the image recognition process using a convolutional neural network, which increases the computational load required for the extraction process. [Means for solving the problem]

[0005] The image processing device disclosed in this specification is an image processing device (10) that includes a memory (20) and an arithmetic circuit (30) and is configured to be capable of processing video data including a plurality of frame images. The arithmetic circuit is configured to be capable of executing an acquisition process (S10) that acquires a current frame image (CF) at the current time. The arithmetic circuit is configured to be capable of executing an identification process (S40) that calculates the difference between the acquired current frame image and a previous frame image (PF) that is stored in memory and that is acquired at a time earlier than the current time, and identifies, on the current frame image, a first region (A1) where pixels have changed within the frame and a second region (A2) where pixels have not changed. The arithmetic circuit is configured to be capable of executing a generation process (S60) that causes a convolution layer (41) of a neural network to perform convolution processing on the first region of the current frame image, and generates first region convolution data (A1C) that is convolution data corresponding to the first region. The arithmetic circuit is configured to be able to execute an extraction process (S70) that extracts second-region convolution data (A2C), which is convolution data corresponding to a second region, from previous-frame convolution data (PC), which is convolution data for a previous frame image and is stored in memory. The arithmetic circuit is configured to be able to execute a synthesis process (S80) that generates current-frame convolution data (CC), which is convolution data for a current frame image, based on the generated first-region convolution data and the extracted second-region convolution data. The arithmetic circuit is configured to be able to execute a storage process (S100) that stores the current-frame image acquired in the acquisition process as a previous-frame image in memory, and stores the current-frame convolution data synthesized in the synthesis process as previous-frame convolution data in memory. The acquisition process, identification process, generation process or extraction process, and synthesis process are repeatedly executed in this order.

[0006] With the above configuration, the previous frame convolution data stored in memory can be reused for the second region of the current frame image, where the pixels have not changed from the previous frame image. Therefore, it is sufficient to perform convolution processing only for the first region of the current frame image, where the pixels have changed. Since the region for which convolution processing is performed can be reduced, it is possible to reduce the amount of calculation. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of a vehicle 1. [Figure 2] 10 is a flowchart illustrating the operation of the image processing device 10. [Figure 3] 10 is an example of a current frame image CF. [Figure 4] 10 is an example of a previous frame image PF. [Figure 5] 10 is an example of a difference frame image DF. [Figure 6] 10 is an example of an image obtained by performing threshold processing on the difference frame image DF. [Figure 7] This is an example of an image in which the difference frame image DF is divided into a first area A1 and a second area A2. [Figure 8] 1 is an example of a histogram. [Figure 9] 1 is an example of a histogram. DETAILED DESCRIPTION OF THE INVENTION

[0008] In the image processing device described above, the current frame image, current frame convolution data, previous frame image, and previous frame convolution data may comprise pixel groups having the same matrix size. The identification process may include the steps of generating a difference frame image (DF) that is the difference between the current frame image and the previous frame image, arranging a filter corresponding to n pixels at a predetermined position in the difference frame image, and, if at least one of the n pixels in the filter has changed, identifying the region in which the filter is located as a first region, and, if the n pixels in the filter have not changed, identifying the region in which the filter is located as a second region. The generation process may, if the first region is identified by the identification process, generate first-region convolution data corresponding to the n pixels by performing a convolution process on the n pixels in the filter arranged at a predetermined position in the current frame image. The extraction process, if the second region is identified by the identification process, may extract second-region convolution data corresponding to the n pixels by extracting n pixels in the filter arranged at a predetermined position in the previous frame convolution data. The synthesis process may generate convolution data for the current frame at a predetermined position using either the first-region convolution data or the second-region convolution data. The identification process, generation process or extraction process, and synthesis process may be repeatedly performed while moving the predetermined position of the filter.

[0009] According to this configuration, it is possible to reduce the area where convolution processing is performed using a filter.

[0010] In the image processing device described above, the identification process may determine that a pixel has changed if, among a plurality of pixels included in the difference frame image, the amount of change in luminance exceeds a predetermined threshold value (TP, TM).

[0011] This configuration makes it possible to appropriately determine which pixels have changed.

[0012] The image processing device described above may be further configured to set a predetermined threshold value, which may be set so that the proportion of pixels determined to be unchanged relative to the total number of pixels in the frame image is a predetermined proportion.

[0013] According to this configuration, it is possible to omit calculation processing for a predetermined percentage of the total number of pixels.

[0014] In the image processing device described above, the setting process may be such that the smaller the number of pixels with zero luminance change amount is, the smaller the predetermined ratio is set.

[0015] This configuration makes it possible to set an appropriate predetermined threshold value according to the driving scene.

[0016] Furthermore, the processing executed by the image processing device described above can also be realized as an image processing method, a computer program, or a computer-readable recording medium on which a program is recorded. [Example]

[0017] (Vehicle 1 configuration) FIG. 1 shows a block diagram of a vehicle 1 according to this embodiment. The vehicle 1 may be, for example, an automobile, a motorcycle, or the like. The vehicle 1 mainly includes a camera 2, an image processing device 10, and an output device 90. The camera 2 captures an image of the area ahead of the vehicle. The camera 2 is capable of generating video data including a plurality of frame images. The image processing device 10 detects objects (e.g., pedestrians, other vehicles, obstacles, etc.) from images captured by the camera 2. The image processing device 10 transmits the detection results to an output device 90 mounted on the vehicle. For example, the output device 90 may be a display device. In this case, the output device 90 can display the position and type of the object identified by the image processing device 10. In addition, in a vehicle with a driving assistance function or an autonomous vehicle, the output device 90 may be a device that controls the traveling of the vehicle (e.g., a device that controls acceleration / deceleration or a device that controls a steering angle). In this case, the output device 90 can control the speed and steering angle of the vehicle according to the detection results of the image processing device 10.

[0018] The image processing device 10 mainly comprises a memory 20 and an arithmetic circuit 30. The memory 20 may comprise a non-volatile memory or a volatile memory. The memory 20 mainly stores a previous frame image PF, previous frame convolution data PC, and an object detection program 23. Note that the current frame image CF, current frame convolution data CC, previous frame image PF, and previous frame convolution data PC comprise pixel groups having the same matrix size. This makes it possible to easily perform subtraction and synthesis processes, as will be described later.

[0019] The arithmetic circuit 30 is configured by a microprocessor equipped with an NPU, a GPU, etc. By executing the object detection program 23 stored in the memory 20, the arithmetic circuit 30 functions as an acquisition unit 31, an identification unit 32, a generation unit 33, an extraction unit 34, a synthesis unit 35, storage units 36 and 37, and a neural network unit 40. Details of the acquisition unit 31 to the storage unit 37 will be described later.

[0020] The neural network unit 40 is a convolutional neural network that has undergone machine learning. The neural network unit 40 includes a first convolutional layer 41, a first pooling layer 42, a second convolutional layer 43, a second pooling layer 44, a fully connected layer 45, and an output layer 46. In the first half of the neural network unit 40, the first convolutional layer 41, the first pooling layer 42, the second convolutional layer 43, and the second pooling layer 44 are alternately arranged to extract features from an image. In the second half of the neural network unit 40, a fully connected layer 45 is arranged to identify objects in the image. The output layer 46 outputs the type and position of the object identified in the image (i.e., its position in the image) as a detection result. Note that the details of convolutional neural networks are well known, so a detailed description thereof will be omitted here.

[0021] (Operation of image processing device 10) When the vehicle 1 starts up, the arithmetic circuit 30 executes the object detection program 23. Specifically, the arithmetic circuit 30 captures an image of the area ahead of the vehicle using the camera 2, and executes neural network arithmetic processing (hereinafter referred to as NN arithmetic processing) on ​​the captured image data. In the NN arithmetic processing, the arithmetic circuit 30 inputs the image data to the neural network unit 40, performs calculations according to each intermediate layer, and outputs a detection result from the output layer 46. The arithmetic circuit 30 repeatedly executes the NN arithmetic processing while the vehicle 1 is traveling, and detects objects ahead of the vehicle 1 in real time.

[0022] The specific operation of the image processing device 10 will be described using the flowchart in Fig. 2 and example images in Fig. 3 to Fig. 7. In step S10, the acquisition unit 31 acquires a current frame image CF from the camera 2. Fig. 3 shows an example of the current frame image CF.

[0023] In step S20, the determination unit 32 reads out the previous frame image PF stored in the memory 20. The previous frame image PF is a frame image captured by the camera 2 at a time (t-1) prior to the current time (t). FIG. 4 shows an example of the previous frame image PF. The determination unit 32 then calculates the difference (i.e., the amount of change in luminance) between the current frame image CF (see FIG. 3) captured by the capture unit 31 and the read previous frame image PF (see FIG. 4). This results in a difference frame image DF that indicates the change in pixel luminance. FIG. 5 shows an example of the difference frame image DF. In FIG. 5, the darker the color, the greater the amount of change in pixel luminance.

[0024] In step S30, the specifying unit 32 places filters corresponding to n pixels at predetermined positions in the difference frame image DF. In this embodiment, a 3×3 filter corresponding to 9 pixels was used.

[0025] In step S40, the identification unit 32 identifies a first region A1, which is a region where pixels have changed, and a second region A2, which is a region where pixels have not changed, on the current frame image CF. Specifically, if at least one of the n pixels in the filter has changed, the region where the filter is located is identified as the first region A1. On the other hand, if the n pixels in the filter have not changed, the region where the filter is located is identified as the second region A2.

[0026] In this embodiment, if at least one of the nine pixels has changed, the 3x3 pixel area in which the filter is located is identified as the first area A1, and if none of the nine pixels has changed, the 3x3 pixel area in which the filter is located is identified as the second area A2.

[0027] Whether or not a pixel in the filter has changed is determined by whether or not the amount of change in luminance exceeds a predetermined threshold. The specific details of how to determine the predetermined threshold and how to determine whether or not a pixel has changed will be described later.

[0028] The threshold processing in step S40 can be performed on the entire difference frame image DF by loop processing, which will be described later. Fig. 6 shows an example of an image in which threshold processing has been performed on the entire difference frame image DF. The areas shown in black are areas where the amount of change in luminance exceeds the threshold, and are areas where it has been determined that pixels have changed. It can be seen that the area of ​​the black areas has decreased in the image after threshold processing (Fig. 6) compared to the difference frame image DF before threshold processing (Fig. 5).

[0029] Furthermore, the process of identifying the first region A1 and the process of identifying the second region A2 in step S40 can be performed on the entire difference frame image DF by loop processing. FIG. 7 shows an example of an image in which the entire difference frame image DF is divided into the first region A1 and the second region A2. The region shown in gray is the first region A1, and the region shown in white is the second region A2. The first region A1 is a region that requires calculation in the first convolutional layer 41. Furthermore, the second region A2 is a region that does not require calculation in the first convolutional layer 41 because the previous frame convolutional data PC can be reused. It can be seen that the process of step S40 reduces the size of the region that requires calculation in the first convolutional layer 41 from the entire frame to the first region A1.

[0030] In step S50, it is determined whether the n pixel region in the filter has been identified as the first region A1 or the second region A2. If the n pixel region has been identified as the first region A1 (S50: first region A1), the process proceeds to step S60. In step S60, the generation unit 33 performs a process of generating first-region convolution data A1C. Specifically, the first convolution layer 41 executes a convolution process on the n pixels in the first region A1 of the current frame image CF. As a result, first-region convolution data A1C corresponding to the n pixels is generated.

[0031] On the other hand, if the n pixel region is identified as the second region A2 (S50: second region A2), the process proceeds to step S70. In step S70, the extraction unit 34 performs a process of extracting second region convolution data A2C. Specifically, the extraction unit 34 extracts n pixels corresponding to the second region A2 from the previous frame convolution data PC stored in the memory 20.

[0032] In step S80, the synthesis unit 35 generates current frame convolution data CC. The current frame convolution data CC is convolution data for the current frame image CF. Specifically, if the n pixel region is identified as the first region A1, the first region convolution data A1C generated by the first convolution layer 41 is incorporated into the corresponding n pixel region of the current frame convolution data CC. On the other hand, if the n pixel region is identified as the second region A2, the second region convolution data A2C extracted by the extraction unit 34 is incorporated into the corresponding n pixel region of the current frame convolution data CC.

[0033] In step S90, it is determined whether the current frame convolution data CC is complete. That is, it is determined whether the filter has scanned the entire area of ​​the current frame image CF. If scanning of the current frame image CF is not complete (S90: NO), the process returns to step S30, and the filter is slid to the next predetermined position. Steps S40 to S90 are then looped. On the other hand, if scanning of the current frame image CF is complete (S90: YES), the process proceeds to step S100.

[0034] In step S100, the synthesis unit 35 transmits the synthesized current frame convolution data CC to the neural network unit 40. As a result, NN calculation processing from the first pooling layer 42 to the output layer 46 is executed.

[0035] Furthermore, storage unit 36 ​​overwrites memory 20 with the current frame image CF acquired by acquisition unit 31 as the previous frame image PF. Storage unit 37 also overwrites memory 20 with the current frame convolution data CC combined by combination unit 35 as the previous frame convolution data PC. This updates the previous frame image PF and the previous frame convolution data PC, making it possible to receive the next frame image.

[0036] In step S110, the acquisition unit 31 determines whether a new next frame image has been captured by the camera 2. If the next frame image has not yet been captured (S110: NO), the acquisition unit 31 waits. If the next frame image has been captured by the camera 2 (S110: YES), the acquisition unit 31 returns to step S10. The acquisition unit 31 then acquires the captured next frame image as a new current frame image CF.

[0037] As a result, processing by the acquisition unit 31 (S10), processing by the identification unit 32 (S40), processing by the generation unit 33 (S60) or processing by the extraction unit 34 (S70), and processing by the synthesis unit 35 (S80) are repeatedly executed in this order.

[0038] (Predetermined threshold setting process) The image processing device 10 is further configured to be able to execute a setting process for setting a predetermined threshold value. As described above, the predetermined threshold value is a value used in determining whether or not a pixel has changed in step S40. The specific contents of the setting process for the predetermined threshold value will be described below.

[0039] First, test image data is prepared. The test image data is images of the area ahead of the vehicle captured while the vehicle is traveling on various types of roads (e.g., urban areas, expressways) under various conditions (e.g., daytime, nighttime, rainy weather). The test image data may be video data of a predetermined duration (e.g., several tens of seconds). Then, the difference between frame images from different times is calculated for the entire image frame (i.e., all pixels). This allows the amount of change in luminance to be calculated for all pixels.

[0040] Second, the obtained brightness change amounts for all pixels are created into a histogram. FIG. 8 shows an example of a histogram. The vertical axis represents frequency (number of pixels) and the horizontal axis represents class (brightness change amount). Pixels with a brightness change amount of 0 are pixels whose brightness did not change between the previous and next image frames. Pixels with a positive brightness change amount are pixels whose brightness increased between the previous and next image frames. Pixels with a negative brightness change amount are pixels whose brightness decreased between the previous and next image frames. Note that while FIG. 8 illustrates a brightness change amount range of ±10, this is not a limitation and various change ranges can be used. Also, while FIG. 8 illustrates a case where the histogram resembles a normal distribution, this is not a limitation. The histogram will have various distributions depending on the driving scene.

[0041] Third, a predetermined threshold is set using a histogram. Specifically, the predetermined threshold is set so that the proportion of pixels determined to be unchanged in the obtained histogram is a predetermined proportion. In the example of FIG. 8, among the pixels having a positive luminance change, pixels corresponding to a predetermined proportion of 10% are determined to be "unchanged." In this case, the number of pixels falling within the range of luminance change amounts from "0" to "+2" corresponds to 10% of all pixels. Therefore, the positive-side predetermined threshold TP is "+2." Similarly, among the pixels having a negative luminance change, pixels corresponding to a predetermined proportion of 10% are determined to be "unchanged." In this case, the number of pixels falling within the range of luminance change amounts from "0" to "-2" corresponds to 10% of all pixels. Therefore, the negative-side predetermined threshold TM is "-2."

[0042] As a result, pixels whose luminance change amount is in the range of "-2" to "+2" (20% of all pixels) can be determined to be "no change." In other words, it is possible to omit the calculation process of the first convolution layer 41 for 20% of all pixels.

[0043] The predetermined ratio may be determined in various ways. As described above, the higher the predetermined ratio, the smaller the amount of computational processing required by the first convolutional layer 41. Therefore, the predetermined ratio can be determined appropriately depending on the processing speed of the arithmetic circuit 30, the number of pixels in the frame image, etc.

[0044] The predetermined threshold value may be determined in various ways. For example, one of the predetermined threshold values ​​TP and TM may be calculated, and the calculated value may be used to set the other. As a specific example, if the predetermined threshold value TP is calculated as "+3.5," the negative value "-3.5" may be set as the predetermined threshold value TM.

[0045] Then, the values ​​of the predetermined thresholds TP and TM are calculated for each of the various driving scenes. Based on the obtained multiple predetermined thresholds TP and TM, one representative predetermined threshold TP and TM is obtained. For example, the median value or average value of the multiple predetermined thresholds TP and TM may be used as the representative value.

[0046] (Method of determining pixel changes using a predetermined threshold) The method of determining pixel changes performed by the specification unit 32 will now be described in detail. In step S40, it is determined whether or not the amount of change in luminance of each of the n pixels in the filter exceeds a predetermined threshold. Three examples of this determination method will be described below.

[0047] The first example is a method of integrating RGB. In this method, the amount of change in R, G, and B for the pixel being evaluated is added together to calculate the total amount of change. If the total amount of change exceeds a predetermined threshold, the pixel being evaluated is determined to have changed.

[0048] The second example is a method of determining RGB individually. In this method, for a pixel to be determined, the amount of change in R, G, and B is individually compared with a predetermined threshold. If at least one of the amount of change in R, G, and B exceeds the predetermined threshold, it is determined that the pixel to be determined has changed. In the second example, the process of generating first-region convolved data A1C (step S60) and the process of generating second-region convolved data A2C (step S70) may also be performed for RGB individually. Some networks calculate RGB individually. Therefore, the second example is effective for such types.

[0049] A third example is a method of making a judgment by converting RGB to YUV. In this method, if the amount of change in the luminance signal Y exceeds a predetermined threshold, it is determined that the pixel being judged has changed.

[0050] (effect) In the technology of this embodiment, for the second region A2 where pixels have not changed from the previous frame image PF, the previous frame convolution data PC stored in memory 20 can be reused (steps S70, S80). Therefore, it is sufficient to perform convolution processing only for the first region A1, which is the region where pixels have changed in the current frame image CF (step S60). Since the region where convolution processing is performed can be reduced, it is possible to reduce the amount of calculations.

[0051] To reduce the amount of computation, conventional methods exist that extract objects using a recognition algorithm other than a convolutional neural network and then perform computations on only the extracted objects. However, this increases the computational load required for object extraction processing. Furthermore, if the object extraction processing fails, image recognition becomes impossible. On the other hand, the technology of this embodiment obtains the difference between the current frame image CF and the previous frame image PF (step S20), and only the first region A1 where pixels have changed is subject to computation (step S60). Because the object extraction processing is unnecessary, it is possible to reduce the computational load associated with object extraction and prevent problems caused by failed object extraction processing. [Example]

[0052] In the second embodiment, a first modified example of the process of setting a predetermined threshold value will be described. Description of the content common to the first embodiment will be omitted. In the first embodiment, the predetermined threshold value is set in advance. On the other hand, in the second embodiment, the predetermined threshold value is set in real time. This will be described below.

[0053] The image processing device 10 stores video data captured by the camera 2 for a predetermined time in the memory 20. Then, the amount of change in luminance between frame images is calculated. A histogram is generated using the calculated amount of change in luminance. A predetermined threshold is set using the generated histogram. Since the video data for the predetermined time stored in the memory 20 is updated sequentially, the amount of change in luminance is also updated sequentially. Therefore, the predetermined threshold can be updated sequentially in real time.

[0054] The effect of this method is as follows. The histogram of brightness variation changes significantly depending on the driving scene. Therefore, by sequentially updating the predetermined threshold in real time, it becomes possible to use the optimum predetermined threshold according to the change in the histogram. This makes it possible to further enhance the effect of reducing the amount of calculation. [Example]

[0055] In the third embodiment, a second modified example of the process of setting the predetermined threshold value will be described. Description of the content common to the first embodiment will be omitted. In the first embodiment, the proportion of unchanged pixels in the histogram (predetermined proportion) is set to a fixed value. On the other hand, in the third embodiment, the predetermined proportion is set to be variable. This will be described below.

[0056] The histogram of brightness change varies greatly depending on the driving scene. For example, in monotonous driving scenes with little image change (e.g., driving on a highway, driving at night), the brightness change is small. Therefore, as shown in the example histogram in FIG. 8, the proportion of pixels with brightness change = 0 is high (see area R1). On the other hand, in complex driving scenes with large image change (e.g., driving in an urban area, driving during the day), the brightness change is large. Therefore, as shown in the example histogram in FIG. 9, the proportion of pixels with brightness change = 0 is low (see area R2).

[0057] In the technology of the third embodiment, the smaller the number of pixels with a luminance change amount of 0 (i.e., the more complex the driving scene), the smaller the predetermined percentage is set. Note that various methods for setting the predetermined percentage may be used. For example, a table defining the relationship between the number of pixels with a luminance change amount of 0 and the predetermined percentage may be prepared in advance. In the example of FIG. 8, the percentage of the number of pixels with a luminance change amount of 0 is high, so the predetermined percentage is set to "10%." On the other hand, in the example of FIG. 9, the percentage of the number of pixels with a luminance change amount of 0 is low, so the predetermined percentage is set to "3%."

[0058] The effect of the third embodiment will be described. When the same predetermined percentage is used, the predetermined threshold value will be larger when the percentage of pixels with a luminance change of 0 is low ( FIG. 9 : complex driving scene) than when the percentage of pixels with a luminance change of 0 is high ( FIG. 8 : monotonous driving scene). For example, when the same predetermined percentage of 10% is used, the predetermined threshold values ​​TP and TM are ±2 in the example of FIG. 8 (monotonous driving scene), but the predetermined threshold values ​​TPa and TMa are larger at ±5 in the example of FIG. 9 (complex driving scene). If the predetermined threshold value is unnecessarily large, pixels with large luminance changes will be excluded from the calculation, which may reduce the accuracy of image recognition. Therefore, in the technology of the third embodiment, the smaller the number of pixels with a luminance change of 0 (i.e., the more complex the driving scene), the smaller the predetermined percentage, thereby preventing the predetermined threshold value from becoming unnecessarily large. In the example of FIG. 9 (complex driving scene), the predetermined percentage is set low at 3%, which allows the predetermined threshold values ​​TP and TM to be kept at ±2. This makes it possible to set an appropriate predetermined threshold value according to the driving scene, thereby maintaining the accuracy of image recognition at a certain level or higher.

[0059] Although the embodiments have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. The technical elements described in this specification or drawings exhibit technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technology exemplified in this specification or drawings simultaneously achieves multiple objectives, and achieving one of these objectives itself has technical utility.

[0060] (Variation) The process of calculating the difference between the current frame image CF and the previous frame image PF may be performed outside the image processing device 10, for example, by the camera 2.

[0061] The configuration of the neural network unit 40 is an example. It may also include other types of layers, such as ReLU layers. The number of stages may also vary. [Explanation of symbols]

[0062] 1: Vehicle 10: Image processing device 20: Memory 30: Arithmetic circuit 41: Convolution layer CF: Current frame image PF: Previous frame image A1: First region A2: Second region A1C: First region convolution data A2C: Second region convolution data PC: Previous frame convolution data CC: Current frame convolution data

Claims

1. An image processing device (10) comprising a memory (20) and an arithmetic circuit (30) and configured to be able to process video data comprising a plurality of frame images, An acquisition process (S10) for acquiring a current frame image (CF) at the current time; an identification process (S40) for calculating a difference between the acquired current frame image and a previous frame image (PF) taken at a time earlier than the current time and stored in the memory, and identifying, on the current frame image, a first region (A1) in which pixels have changed within the frame and a second region (A2) in which pixels have not changed; A generation process (S60) of causing a convolution layer (41) of a neural network to perform a convolution process on the first region of the current frame image, thereby generating first region convolution data (A1C) that is convolution data corresponding to the first region; an extraction process (S70) for extracting second region convolution data (A2C) that is convolution data corresponding to the second region from previous frame convolution data (PC) that is convolution data for the previous frame image and is stored in the memory; a synthesis process (S80) for generating current frame convolution data (CC), which is convolution data for the current frame image, based on the generated first region convolution data and the extracted second region convolution data; a storage process (S100) in which the current frame image acquired in the acquisition process is stored in the memory as the previous frame image, and the current frame convolution data synthesized in the synthesis process is stored in the memory as the previous frame convolution data; The arithmetic circuit is configured to be able to execute the acquisition process, the identification process, the generation process or the extraction process, and the synthesis process are repeatedly executed in this order; Image processing device.

2. the current frame image, the current frame convolution data, the previous frame image, and the previous frame convolution data comprise pixel groups having the same matrix size; The specific processing is A process of generating a difference frame image (DF) which is a difference between the current frame image and the previous frame image; A process of placing filters corresponding to n pixels at predetermined positions of the difference frame image; identifying a region in which the filter is located as the first region if at least one of the n pixels in the filter has changed; identifying the region in which the filter is located as the second region if n pixels in the filter have not changed; Run the generation process, when the first region is identified by the identification process, performs the convolution process on n pixels in the filter that are arranged at the predetermined position of the current frame image, thereby generating the first region convolution data corresponding to the n pixels; the extraction process, when the second region is identified by the identification process, extracts n pixels in the filter that are arranged at the predetermined positions on the previous frame convolution data, thereby extracting the second region convolution data corresponding to the n pixels; the synthesis process uses either the first region convolution data or the second region convolution data to generate the current frame convolution data at the predetermined position; the identification process, the generation process or the extraction process, and the synthesis process are repeatedly executed while moving the predetermined position of the filter; The image processing device according to claim 1 .

3. The image processing device according to claim 2 , wherein the specifying process determines that a pixel has changed when, among a plurality of pixels included in the difference frame image, a pixel has a luminance change amount exceeding a predetermined threshold value (TP, TM).

4. the image processing device is further configured to be able to execute a setting process for setting the predetermined threshold value; The image processing device according to claim 3 , wherein the predetermined threshold is set so that the ratio of the number of pixels determined to be unchanged to the total number of pixels in the frame image is a predetermined ratio.

5. The image processing device according to claim 4 , wherein the setting process sets the predetermined ratio to be smaller as the number of pixels for which the luminance change amount is zero decreases.

6. An image processing method executed by an image processing device that includes a memory and an arithmetic circuit and is configured to be able to process moving image data that includes a plurality of frame images, an acquisition process for acquiring a current frame image at a current time; an identification process for calculating a difference between the acquired current frame image and a previous frame image taken at a time earlier than the current time and stored in the memory, and identifying, on the current frame image, a first region in which pixels have changed within the frame and a second region in which pixels have not changed; a generation process of causing a convolution layer of a neural network to perform a convolution process on the first region of the current frame image, and generating first region convolution data that is convolution data corresponding to the first region; an extraction process of extracting second region convolution data, which is convolution data corresponding to the second region, from previous frame convolution data, which is convolution data for the previous frame image and is stored in the memory; a synthesis process for generating current frame convolution data, which is convolution data for the current frame image, based on the generated first region convolution data and the extracted second region convolution data; a storage process for storing the current frame image acquired in the acquisition process in the memory as the previous frame image, and storing the current frame convolution data synthesized in the synthesis process in the memory as the previous frame convolution data; Equipped with the acquisition process, the identification process, the generation process or the extraction process, and the synthesis process are repeatedly executed in this order; Image processing methods.

7. A computer program (23) for an image processing device that includes a memory and an arithmetic circuit and is configured to be able to process video data that includes a plurality of frame images, The arithmetic circuit of the image processing device is divided into the following parts: an acquisition unit that acquires a current frame image at a current time; an identification unit that calculates a difference between the acquired current frame image and a previous frame image that is taken at a time earlier than the current time and that is stored in the memory, and identifies, on the current frame image, a first region that is a region in which pixels have changed within the frame and a second region that is a region in which pixels have not changed; a generation unit that causes a convolution layer of a neural network to perform a convolution process on the first region of the current frame image, and generates first region convolution data that is convolution data corresponding to the first region; an extracting unit that extracts second region convolution data, which is convolution data corresponding to the second region, from previous frame convolution data, which is convolution data for the previous frame image and is stored in the memory; a synthesis unit that generates current frame convolution data, which is convolution data for the current frame image, based on the generated first region convolution data and the extracted second region convolution data; a storage unit that stores the current frame image acquired by the acquisition unit in the memory as the previous frame image, and stores the current frame convolution data synthesized by the synthesis unit in the memory as the previous frame convolution data; It functions as a process by the acquisition unit, a process by the identification unit, a process by the generation unit or a process by the extraction unit, and a process by the synthesis unit are repeatedly executed in this order; Computer program.

8. A computer-readable recording medium on which the computer program according to claim 7 is recorded.

Images