Image Processing Device

Abstract

To enable appropriate processing when an information processing unit is active while history information is unavailable.SOLUTION: When an information processing unit is active and history information held in a history information holding unit is available, the information processing unit executes information processing using an input image and the history information held in the history information holding unit to generate an information-processed image, and updates the history information held in the history information holding unit. An output unit outputs the information-processed image. When the information processing unit is active and if the history information held in the history information holding unit is unavailable, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information to generate an information-processed image, updates the history information held in the history information holding unit. The output unit outputs the information-processed image. When the information processing unit is inactive, the output unit outputs the input image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a processing method for an image processing device, and a program. [Background technology]

[0002] In recent years, devices have been proposed that use neural networks to improve the image quality of images captured by cameras, etc. By using neural networks, it becomes possible to perform image quality improvement processes such as noise reduction and super-resolution with higher performance and speed than conventional methods.

[0003] In particular, for video, the ability to perform high-quality image processing in real time is required. Recurrent neural networks (RNNs) are widely used as a method to achieve real-time video performance.

[0004] RNNs are characterized by their structure in which the output results from the RNN, i.e., history information, are recursively input into the RNN, making it possible to process time series data of any length using a single neural network.

[0005] In addition, because RNN has a recursive structure, it is not necessary to prepare a neural network for each time series, which makes it possible to reduce the size of the neural network. This allows for faster processing, i.e., real-time processing.

[0006] Patent Document 1 discloses an image processing device that uses an RNN to achieve low latency and low cost.

[0007] Patent document 2 discloses a device that, when a process is stopped and then restarted, extracts the history obtained in the history generation process, determines whether the input information in the extracted history is still valid or invalid, and does not use the input information in the history if it is invalid. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2021-149333 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-3743 Summary of the Invention [Problem to be solved by the invention]

[0009] While neural networks such as RNNs can perform complex processing such as improving image quality with high performance and speed, they tend to require a large amount of calculation, which in turn increases the power consumption of the device.

[0010] In recent years, edge devices for processing images captured by cameras and other devices have been installed in moving objects such as vehicles, ships, aircraft, and drones. These edge devices are required to be energy-efficient in order to conserve fuel for batteries and generators.

[0011] To achieve power saving, it is necessary to be able to temporarily turn off the functions of a device when they are not needed and to be able to instantly turn on the functions of the device when they are needed.

[0012] When using an RNN for processing, the RNN has a structure in which the previous output results, i.e., history information, are recursively input to the RNN. Therefore, to fully utilize the RNN's performance, it is necessary to continuously input images, etc.

[0013] However, if the operation of the RNN is stopped for reasons such as power saving, the history information remains old, which creates the problem that the RNN's performance cannot be fully utilized when it is resumed.

[0014] In particular, with low frame rate videos, which are often seen in surveillance cameras, the processing interval becomes longer, resulting in a longer period of time when the RNN cannot demonstrate its performance when it resumes execution. This can result in, for example, screen flickering or the failure to detect fast-moving targets (such as people or vehicles).

[0015] On the other hand, if the RNN processing is kept running constantly to avoid screen flickering and overlooking the monitored object, power consumption will be constantly high, making it impossible to use for long periods of time in an environment using batteries or a generator.

[0016] The method of Patent Document 2 has a problem in that, although it is possible to erase history information, there will be no valid history information when the process is restarted, and therefore performance cannot be immediately demonstrated after the process is restarted.

[0017] An object of the present disclosure is to enable appropriate processing to be performed when an information processing unit is in an operating state and history information is invalid. [Means for solving the problem]

[0018] The image processing device has an information processing unit and an output unit, and when the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit performs information processing using an input image and the history information stored in the history information storage unit, generates a processed image, and updates the history information stored in the history information storage unit, and the output unit outputs the processed image, and when the information processing unit is in an operating state and the history information stored in the history information storage unit is invalid, the information processing unit regenerates the history information based on the input image, performs information processing using the input image and the regenerated history information, generates a processed image, and updates the history information stored in the history information storage unit, and the output unit outputs the processed image, and when the information processing unit is in a stopped state, the output unit outputs the input image. [Effects of the Invention]

[0019] According to the present disclosure, when the information processing unit is in an operating state and the history information is invalid, appropriate processing can be performed. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an example of a hardware configuration of an image processing apparatus. [Figure 2] FIG. 2 illustrates an example of a functional configuration of an image processing apparatus. [Figure 3] FIG. 10 is a diagram illustrating an example of a processing flow of the image processing device. [Figure 4] FIG. 10 is a diagram illustrating an example of processing by an information processing unit. [Figure 5] FIG. 10 is a diagram illustrating an example in which a neural network is used. [Figure 6] FIG. 10 is a diagram illustrating another example of the processing flow of the image processing device. [Figure 7] FIG. 10 is a diagram illustrating an example in which a neural network is used. [Figure 8] FIG. 10 illustrates an example of a user interface. DETAILED DESCRIPTION OF THE INVENTION

[0021] Preferred embodiments will be described in detail below with reference to the drawings. Note that the configurations in the following embodiments are merely examples and are not intended to be limiting.

[0022] 1 is a diagram showing an example of the hardware configuration of an image processing device 10 according to this embodiment. The image processing device 10 has, as its hardware configuration, a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F device 16.

[0023] The control device 11 controls the entire image processing device 10. The storage device 12 stores programs and data required for the operation of the control device 11. The arithmetic device 13 executes necessary arithmetic processing based on the control of the control device 11.

[0024] The input device 14 is a human interface device or the like, and inputs user operations to the image processing device 10. The output device 15 is a display or the like, and presents the processing results of the image processing device 10 to the user.

[0025] The I / F device 16 is a wired interface such as a universal serial bus, Ethernet, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth. The I / F device 16 also has functions such as connecting a camera or the like to input captured images to the image processing device 10, transmitting processing results obtained by the image processing device 10 to the outside, and inputting programs, data, etc. required for the operation of the image processing device 10 to the image processing device 10. The camera is a video camera, a security camera, etc.

[0026] 2 is a diagram showing an example of the functional configuration of the image processing device 10 according to this embodiment. The image processing device 10 has, as its functional configuration, an image acquisition unit 201, an input image storage unit 202, a history information storage unit 203, an information processing unit 204, and an output unit 205.

[0027] The image acquisition unit 201 acquires an image to be subjected to image processing by the image processing device 10 as an input image.

[0028] The input image holding unit 202 holds the input image acquired by the image acquisition unit 201. The history information holding unit 203 holds history information.

[0029] The information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, and generates a processed image and updated history information.

[0030] If it is determined that the history information held by the history information holding unit 203 is invalid before the information processing unit 204 is executed, the history information holding unit 203 regenerates the history information from the input image held by the input image holding unit 202 and holds it as new history information.

[0031] Here, regeneration refers to discarding the invalid history information held by the history information holding unit 203 at the point in time when it is determined that the history information held by the history information holding unit 203 is invalid, and regenerating only the history information from the input image held by the input image holding unit 202.

[0032] The history information holding unit 203 receives the updated history information generated by the information processing unit 204 and holds it as new history information.

[0033] When the information processing unit 204 is in an operating state, the output unit 205 outputs the processed image generated by the information processing unit 204, and when the information processing unit 204 is in a stopped state, the output unit 205 outputs the input image stored in the input image storage unit 202.

[0034] FIG. 3(a) is a flowchart showing an example of a processing method of the image processing device 10 according to this embodiment.

[0035] In step S101, the image acquisition unit 201 acquires, as an input image, an image to be subjected to image processing by the image processing device 10. Here, the input image is, for example, streaming, a video file, a series of image files saved frame by frame, a video or image saved in a medium, or the like.

[0036] The image acquisition unit 201 may acquire an input image from a solid-state imaging element such as a CMOS sensor, a CCD sensor, or a SPAD sensor, or a camera equipped with such a solid-state imaging element, or may acquire an input image from a storage device or recording medium such as a hard disk or an SSD. There may be only one or more elements, cameras, storage devices, recording media, etc. that acquire the input image.

[0037] In step S102, the input image storage unit 202 stores the input images acquired by the image acquisition unit 201. The input image storage unit 202 may store only one latest image, or may store multiple images in chronological order.

[0038] Furthermore, the input image storage unit 202 may set an upper limit on the number of images that can be stored, and if the upper limit is exceeded, the oldest images may be discarded. If the information processing unit 204 requires multiple images when performing information processing, the upper limit may be set as the number of images required by the information processing unit 204.

[0039] In addition, the above upper limit may be set to a number of images with some leeway so that there is no conflict between the writing of the input image acquired by the image acquisition unit 201 to the input image storage unit 202 and the reading of the input image from the input image storage unit 202 by the information processing unit 204.

[0040] For example, if the image acquisition unit 201 writes one input image to the input image storage unit 202, and the information processing unit 204 requires three images when performing information processing, the input image storage unit 202 should store at least four images.

[0041] Furthermore, the upper limit of the number of images that can be stored may be increased to prevent processing delays when subsequent processing is delayed. For example, if processing delays of up to 20 frames of a moving image are allowed, the input image storage unit 202 may store an additional 20 frames.

[0042] The input image storage unit 202 may store additional images as many as are necessary when regenerating the history information in step S107. For example, if three consecutive frames of images are necessary when regenerating the history information in step S107, the input image storage unit 202 may store three additional frames. In this case, the input image storage unit 202 stores at least the number of input images necessary for regenerating the history information of the information processing unit 204.

[0043] If there is latency in the processing from steps S105 to S108, the input image storage unit 202 may store additional images for the latency. For example, if the latency is three frames, the input image storage unit 202 may store only one image from three frames ago, or may store three additional frames. In this case, the input image storage unit 202 stores at least as many input images as the latency of the information processing unit 204.

[0044] It is preferable that the input image storage unit 202 simultaneously stores an identifier for uniquely identifying the input image in addition to the input image acquired by the image acquisition unit 201. Here, the identifier may be, for example, the time of shooting.

[0045] It is more preferable that the identifier represents the chronological order of the input images. Examples of such identifiers include frame numbers and serial numbers. Here, the frame numbers and serial numbers may be obtained from, for example, a camera equipped with a solid-state image sensor, a storage device such as a hard disk or SSD, or a recording medium, or they may be assigned a time stamp or unique numbers generated within the input image storage unit 202.

[0046] In step S103, the image processing device 10 determines the state of the information processing unit 204. Here, the state of the information processing unit 204 refers to whether the information processing unit 204 is in a stopped state or in an operating state.

[0047] The information processing unit 204 being in a stopped state means that the information processing unit 204 cannot start new processing, and the information processing unit 204 being in an operating state means that the information processing unit 204 can start new processing.

[0048] When the information processing unit 204 is in a waiting state for processing, for example, waiting for initialization, waiting for reconfiguration due to an option change, or waiting for information processing to be completed, the information processing unit 204 cannot start new processing and is therefore considered to be in a stopped state.

[0049] Even when the processing of the information processing unit 204 is skipped, i.e., when the input image is output as is without performing any information processing on it, the information processing unit 204 is considered to be in a stopped state because it cannot start new processing since no input image is supplied.

[0050] If the result of the above determination is that the information processing unit 204 is in a stopped state, the process proceeds to step S104, and if the information processing unit 204 is in an operating state, the process proceeds to step S105.

[0051] In step S104, the output unit 205 outputs the input image held in the input image holding unit 202.

[0052] Here, if the input image storage unit 202 stores one input image, the output unit 205 outputs that input image. If the input image storage unit 202 stores two or more input images, the output unit 205 outputs any one of the input images stored in the input image storage unit 202.

[0053] As for any one of the images, for example, a method of outputting the oldest image in chronological order among the input images held by the input image holding unit 202 can be mentioned.

[0054] Furthermore, if there is latency in the processing from steps S105 to S108, the output unit 205 may acquire an image from the input image storage unit 202 as far back as the latency, and output the image in accordance with the latency.

[0055] By matching the latency of the processing in step S104 with the latency of the processing in step S105 and subsequent steps, it is possible to prevent inconveniences such as missing or skipping frames when the processing switches in step S103 during moving image processing.

[0056] If an identifier is assigned to the output image, the output unit 205 may output the identifier together with the image.

[0057] In step S105, the history information storage unit 203 determines whether the history information it stores is valid or invalid, and if the history information is valid, proceeds to step S106, and if the history information is invalid, proceeds to step S107.

[0058] In step S106, which follows step S105, the information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, and generates a processed image and updated history information.

[0059] If, at step S106, the history information held by the history information holding unit 203 is chronologically older than that expected by the information processing unit 204, the processing by the information processing unit 204 will not be performed as expected, which may result in a decrease in processing accuracy, output abnormalities, output deficiencies, etc.

[0060] Therefore, in step S105, it is determined whether the history information held by the history information holding unit 203 is chronologically older than that assumed by the information processing unit 204, and the history information is determined to be valid or invalid, and subsequent processing is branched based on the determination result.

[0061] In the above determination, if the history information is chronologically older than assumed by the information processing unit 204, the held history information is determined to be invalid, and otherwise it is determined to be valid.

[0062] For example, in the example of FIG. 4(a), it is assumed that the information processing unit 204 is designed to use an image 401 of the current frame N as an input image and history information 402 of the immediately preceding frame N-1.

[0063] In FIG. 4(a), an image 401 of frame N and history information 402 of frame N-1 are input to the information processing unit 204, and a processing result 403 of the image of the current frame N and history information 404 of frame N are generated.

[0064] Here, it is assumed that the frame number is assigned as an identifier to the input image held in the input image holding unit 202.

[0065] FIG. 4B shows an example in which the information processing unit 204 is operated up to frame N-10, immediately thereafter put into a stopped state, and then put into an operating state again when processing frame N.

[0066] In FIG. 4(b), an image 401 of frame N is input to the information processing unit 204 as the input image, but history information 405 of frame N-10 at the time of the previous execution is input as the history information.

[0067] As shown in FIG. 4(a), when an image 401 of frame N is used as an input image, the information processing unit 204 is designed to use history information 402 of the immediately preceding frame N-1.

[0068] 4(b), however, the information processing unit 204 stopped processing from frame N-10 until frame N-1, so there is no history information for the immediately preceding frame N-1. Therefore, if history information 405 for frame N-10, which should not have been used, is used for input image 401 of frame N, the information processing unit 204 will not be able to demonstrate its original performance and will output a degraded processed image 406 and degraded history information 407 for frame N.

[0069] In the case of Figure 4(b), the history information 405 of frame N-10 stored in the history information storage unit 203 is a frame that is chronologically older than the frame N-1 that is actually required, so the history information 405 of frame N-10 is determined to be invalid.

[0070] On the other hand, if the information processing unit 204 is not stopped, as shown in FIG. 4(a), when the image 401 of frame N is used as the input image, the history information holding unit 203 holds the history information 402 of the immediately preceding frame N-1, and therefore determines that the history information 402 is valid.

[0071] In the example of Figure 4(b), if the history information is delayed by one frame or more, it is judged to be invalid. However, if processing is performed at a high frame rate or the subject is moving slowly, there is little change in the image between adjacent frames, so processing may be possible even if the history information is delayed by several frames.

[0072] In such a case, M is a constant greater than 1, and if the history information is delayed by M frames or more, the history information is considered to be old and judged to be invalid, and if the history information is delayed by less than M frames, the history information is not considered to be old and is judged to be valid.

[0073] That is, the history information held in the history information holding unit 203 is valid when the history information held in the history information holding unit 203 is history information for a frame less than the threshold number of frames ago, and the history information held in the history information holding unit 203 is invalid when the history information held in the history information holding unit 203 is history information for a frame more than the threshold number of frames ago.

[0074] If there is no history information stored in the history information storage unit 203 at the time of step S106, the processing by the information processing unit 204 will not be performed as expected, which may result in a decrease in processing accuracy, output abnormalities, output loss, etc.

[0075] Therefore, in step S105, if the history information storage unit 203 does not store the history information, that is, if the history information does not exist, it may be determined that the history information is invalid.

[0076] On the other hand, if the history information storage unit 203 stores history information, i.e., if history information exists, for example, as described above, it is possible to determine whether the history information is chronologically old, determine whether the history information is valid or invalid, and branch subsequent processing based on the determination result.

[0077] If the processing of the information processing unit 204 is skipped, i.e., if the input image is output as is without performing information processing on it, the history information held by the history information holding unit 203 is not updated, and the history information becomes outdated in chronological order.

[0078] In such a case, the history information holding unit 203 may delete the chronologically older history information that it holds.

[0079] If the history information has been deleted, in step S105, the history information holding unit 203 does not hold the history information, that is, the history information does not exist, and therefore it is determined that the history information is invalid.

[0080] The history information holding unit 203 may hold an update flag that indicates whether the history information needs to be updated or does not need to be updated.

[0081] A case where updating of history information is necessary occurs when, for example, the processing of the information processing unit 204 is skipped and the history information held by the history information holding unit 203 is not updated, causing the history information to become chronologically outdated.

[0082] The update flag can be expressed as a true / false value, for example, which is true if an update is required and false if an update is not required.

[0083] For example, if the history information held by the history information holding unit 203 is updated, the update flag is set to false, and if the history information held by the history information holding unit 203 is not updated, the update flag is set to true.

[0084] In this case, in step S105, it is determined whether the history information is valid or invalid based on the update flag, and the subsequent processing is branched based on the determination result.

[0085] For example, if the update flag is true, the history information is determined to be invalid, and if the update flag is false, the history information is determined to be valid.

[0086] In step S106, the information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, generates a processed image and updated history information, and proceeds to step S108.

[0087] The information processing is various known image processing, such as noise removal, super-resolution, sharpening, dehazing, or dark area emphasis.

[0088] If an identifier has been assigned to the input image, the information processing unit 204 assigns the identifier that was assigned to the input image to the processed image and the updated history information.

[0089] The number of input images and information-processed images may be one or more. Furthermore, the number of input images and information-processed images may be the same or different.

[0090] The number of pieces of history information and updated history information may be one or more. Furthermore, the number of pieces of history information and updated history information may be the same or different.

[0091] Furthermore, the processed image may be treated as updated history information, in which case the information processing unit 204 does not need to output the updated history information.

[0092] The history information holding unit 203 holds the updated history information generated by the information processing unit 204 as new history information.

[0093] If an identifier has been assigned to the updated history information, the history information storage unit 203 stores the identifier together with the updated history information as new history information.

[0094] Methods for executing information processing include, for example, machine learning and neural networks.

[0095] Examples of machine learning methods include decision trees, random forests, and LightGBM.

[0096] Examples of neural networks include convolutional neural networks, deconvolutional neural networks, and autoencoders that combine both of them.

[0097] FIG. 5 is a diagram showing an example of a case where a neural network 501 is used as the information processing method of the information processing unit 204. In FIG.

[0098] In the neural network 501 shown in the example of FIG. 5, Conv represents a convolution layer, Pooling represents a pooling layer, Upsample represents an upsampling layer, Concat represents a concatenation layer, and Output represents an output layer.

[0099] In the example of FIG. 5, an input image 502 held in the input image holding unit 202 and history information 503 held in the history information holding unit 203 are input to a neural network 501.

[0100] When an input image 502 is input to Conv101 of a neural network 501 and history information 503 is input to Conv103, calculation processing is performed according to the flow of FIG.

[0101] When the processing of the neural network 501 is completed, the processed image 504 is output from the OutPut 101 of the neural network 501, and the updated history information 505 is output from the OutPut 102 of the neural network 501.

[0102] The information processing unit 204 may perform information processing by using the neural network 501 shown in the example of FIG. 5 in multiple stages.

[0103] For example, when the neural network 501 is used in two stages, in the first stage, the information processing unit 204 generates updated history information for frame N-1 using the input image of frame N-1 and history information for frame N-2.

[0104] In the subsequent second stage, the information processing unit 204 generates an information-processed image of frame N and updated history information of frame N using the input image of frame N and history information of frame N-1.

[0105] The information processing unit 204 can execute information processing using a recursive neural network (RNN). A neural network 501 is an example of an RNN. For example, the RNN recursively inputs history information 505 of a previous output result as history information 503 of a current input to the RNN. The history information 505 is, for example, an image that has been processed by the information processing unit 204.

[0106] The information processing unit 204 may treat intermediate outputs of machine learning or neural networks as updated historical information.

[0107] For example, in the example of neural network 501 shown in FIG. 5, the output from any layer other than the output layers Output1 and Output2 contained within neural network 501 may be extracted as the intermediate output and treated as updated history information.

[0108] When machine learning or neural networks are used in multiple stages, the information processing unit 204 may extract outputs from other than the final stage of machine learning or neural networks as the intermediate outputs and treat them as updated history information.

[0109] For example, when the neural network 501 in FIG. 5 is used in two stages, in the first stage, the output obtained from the output layer Output1 or Output2 using the input image of frame N-1 and the history information of frame N-2 may be extracted as the intermediate output and treated as updated history information.

[0110] In step S107, the information processing unit 204 regenerates history information based on the input image stored in the input image storage unit 202, and the process proceeds to step S106. Here, if an identifier has been assigned to the input image, the information processing unit 204 assigns the identifier that was assigned to the input image to the regenerated history information.

[0111] The number of input images may be one or more, and the number of updated history information may be one or more.

[0112] A method for regenerating the history information includes, for example, inputting the input image stored in the input image storage unit 202 and provisional history information into the information processing unit 204, performing information processing, discarding the processed image, and obtaining only the updated history information.

[0113] Alternatively, the processed image may be treated as updated history information instead of being discarded, in which case the information processing unit 204 does not need to output the updated history information.

[0114] As the provisional history information, for example, the latest history information stored in the history information storage unit 203 at the time of executing step S107 may be used, or the input image stored in the input image storage unit 202 may be used.

[0115] Other examples of provisional history information include a solid image, a pattern image, a noise image, and the like.

[0116] The history information storage unit 203 stores the regenerated history information as new history information. If an identifier is assigned to the regenerated history information, the history information storage unit 203 stores the identifier together with the regenerated history information as new history information.

[0117] It may take time to regenerate the history information in step S107. For this reason, for example, if the process shown in Fig. 3(a) is performed at regular time intervals, when the process in step S107 starts, it may appear that the entire process has temporarily stopped.

[0118] 3(b), the image processing device 10 may perform the process of step S104 in parallel with the process of step S107. That is, the image processing device 10 may perform the process of step S104 in parallel and output the input image as is until the process of step S107 is completed and the process of step S106 is resumed.

[0119] Execution of step S107 increases the amount of processing that must be performed by the image processing device 10. Execution of step S107 increases the processing time, which may cause a delay in processing.

[0120] In such a case, the image processing device 10 may eliminate the delay in processing due to the execution of step S107. For example, the image processing device 10 may perform the processing of step S107 in a separate thread or temporarily increase the processing frame rate.

[0121] If the history information holding unit 203 determines in step S105 that the history information it holds is invalid, and if step S107 is not performed, then in step S106 the information processing unit 204 will perform information processing using the invalid history information.

[0122] If the history information is invalid, the information processing unit 204 will not be able to fully utilize its capabilities, which will result in flickering and a decrease in accuracy during video image processing, leading to inconveniences such as overlooking a monitored object due to the flickering and a decrease in accuracy.

[0123] Therefore, to fully utilize the performance of the information processing unit 204, the information processing unit 204 must be kept in operation at all times, and the history information must always be kept up to date.

[0124] For this reason, the information processing unit 204 cannot be put into a stopped state when not needed, and the power consumption of the image processing device 10 cannot be reduced, so it cannot be used for a long time in an environment such as a battery or generator.

[0125] On the other hand, if the history information holding unit 203 determines in step S105 that the held history information is invalid and S106 is executed after the processing of S107, the information processing unit 204 can perform information processing using the regenerated latest history information, i.e., the valid history information. In step S106, the information processing unit 204 performs information processing using the input image and the regenerated history information, generates a processed image, and updates the history information held in the history information holding unit 203.

[0126] Because the history information is valid, the performance of the information processing unit 204 can be fully utilized. As a result, flickering and a decrease in accuracy do not occur when processing moving images, and it is possible to prevent inconveniences such as overlooking a monitored object due to flickering and a decrease in accuracy.

[0127] Furthermore, if the history information is invalid, the latest regenerated history information is used, so there is no need to keep the information processing unit 204 in an operating state at all times.

[0128] Therefore, the information processing unit 204 can be put into a stopped state when not needed, reducing the power consumption of the image processing device 10, allowing it to be used for a long period of time even in an environment where it is powered by a battery or a generator.

[0129] If it is determined in step S103 that the information processing unit 204 is in a stopped state, the image processing device 10 may execute the process of step S109 in parallel with step S104, as shown in FIG.

[0130] In step S109, the history information holding unit 203 executes the same process as in step S107. That is, the history information holding unit 203 regenerates history information based on the input image held in the input image holding unit 202, and updates the history information held in the history information holding unit 203. Here, if an identifier has been assigned to the input image, the history information holding unit 203 assigns the identifier that was assigned to the input image to the regenerated history information.

[0131] One of the reasons for wanting to stop the information processing unit 204 is to stop the information processing unit 204 when it is not needed, thereby reducing power consumption of the image processing device 10. Therefore, when executing step S109, it is preferable that the image processing device 10 perform processing while reducing power consumption.

[0132] One way to reduce power consumption in step S109 is to perform the regeneration process in step S109 using a lighter method than the method for regenerating history information used in the process in step S107, as shown in the example of Fig. 5. In this case, the regeneration of history information in step S109 is performed using a lighter method than the regeneration of history information in step S107.

[0133] Lightweight methods include, for example, machine learning, neural networks, etc. Machine learning methods include, for example, decision trees, random forests, and LightGBM.

[0134] Examples of neural networks include convolutional neural networks, deconvolutional neural networks, and autoencoders that combine both of them.

[0135] Fig. 7 is a diagram showing an example of a lightweight method using a neural network 701. In the neural network 701 shown in the example of Fig. 7, Conv represents a convolution layer, Pooling represents a pooling layer, Upsample represents an upsampling layer, Concat represents a concatenation layer, and Output represents an output layer.

[0136] In the example of FIG. 7, an input image 702 stored in the input image storage unit 202 is input to a neural network 701 .

[0137] When an input image 702 is input to Conv101 of a neural network 701, calculation processing is performed according to the flow of FIG.

[0138] When the processing of the neural network 701 is completed, the updated history information 703 is output from the OutPut 101 of the neural network 501 .

[0139] In the example of Figure 7, neural network 701 is a version of neural network 501 in Figure 5 with some parts omitted, but this is not limited to this and it may be made lighter by distillation or quantization, or a neural network with a completely different structure may be used.

[0140] In step S109, the history information storage unit 203 can execute regeneration of the history information by the neural network 701.

[0141] Another method for reducing power consumption in step S109 is to use a device that consumes less power than usual for the arithmetic unit 13 of the image processing device 10. In this case, the regeneration of history information in step S109 is performed by a device that consumes less power than the regeneration of history information in step S107. To enable execution on a power-saving device, a weight reduction method such as distillation or quantization may be used in combination.

[0142] Alternatively, a plurality of methods with different power consumption may be selected and used depending on the remaining amount of fuel in the battery or generator.

[0143] For example, when the remaining amount is large, history information may be regenerated with high accuracy using a method that requires a lot of calculations, and as the remaining amount decreases, the method may be gradually switched to a lighter method, the use of a power-saving device, etc. If the remaining amount becomes even smaller, the process of step S109 itself may be stopped.

[0144] If the information processing unit 204 is in a stopped state and the process of S109 is not executed in parallel with step S104, the history information becomes older by the time the information processing unit 204 is stopped.

[0145] In this state, if the information processing unit 204 is put into operation and it is determined in step S105 that the history information is invalid, processing is executed in the order of steps S107 and S106, and valid history information is regenerated in step S107.

[0146] As shown in FIG. 6, another method for determining whether to execute the process of S109 in parallel with step S104 is to use the operation of the user interface.

[0147] For example, it is assumed that the information processing unit 204 is switched between a stopped state and an operating state by a user's operation via the input device 14 or the I / F device 16 of the image processing device 10.

[0148] As an example, consider the case where a user interface such as that shown in Fig. 8(a) is used. In the example of Fig. 8(a), a screen 801 displays a processed image 802 output by the information processing unit 204 and a user interface 803 for changing the information processing unit 204 from a stopped state to an operating state.

[0149] In FIG. 8(a), it is assumed that the information processing unit 204 is in a stopped state, and that the information processing unit 204 can be put into an operating state by operating the user interface 803 by clicking or the like.

[0150] FIG. 8(b) shows a state in which the user operates a cursor 804 to place it on the user interface 803 and focus on the user interface 803 in order to operate the user interface 803 by clicking or the like.

[0151] In FIG. 8(b), the color of the button is changed from that in FIG. 8(a) to indicate that the user interface 803 is in focus.

[0152] When the user operates the user interface 803 in FIG. 8(b) to change the information processing unit 204 from a stopped state to an operating state, there is a high possibility that the user will then perform an operation such as clicking on the user interface 803, causing it to change to an operating state.

[0153] Therefore, when the information processing unit 204 is in a stopped state, as shown in FIG. 8B, the process of S109 may be started in parallel with step S104 in FIG. 6 when the user interface 803 is focused.

[0154] In the state of FIG. 8B, assume that the user interface 803 for changing from a stopped state to an operating state is focused on, and then an operation such as a click is performed, causing the information processing unit 204 to change from a stopped state to an operating state.

[0155] Here, it is assumed that the processing of step S109 is completed between the time when the user interface 803 is focused and the time when the information processing unit 204 is put into an operating state.

[0156] When the processing of step S109 is completed, the history information held by the history information holding unit 203 is updated to the latest information, so that when the information processing unit 204 enters an operating state, the history information is determined to be valid in S105, and S106 can be executed immediately without any additional overhead.

[0157] In step S108, the output unit 205 outputs the information-processed image generated in step S106 by the information processing unit 204. If an identifier is assigned to the output image, the output unit 205 may output the identifier simultaneously with the image.

[0158] The output unit 205 may present the processing results to the user via the output device 15 of the image processing device 10, or may present the processing results to another terminal or the like via the I / F device 16.

[0159] In this embodiment, image processing is taken as an example of information processing, but the information processing is not limited to image processing. For example, behavior recognition, posture estimation, fall detection, tracking, flow rate estimation, etc. may be mentioned.

[0160] As described above, according to this embodiment, the image processing device 10 temporarily stops the RNN processing of the information processing unit 204, and when restarting the processing, if the history information is invalid, regenerates appropriate history information. This feature makes it possible to immediately demonstrate the performance of the RNN when the RNN processing is restarted, so that flickering, accuracy degradation, etc. do not occur when processing moving images, and it is possible to prevent overlooking of a monitoring target due to flickering, accuracy degradation, etc.

[0161] Furthermore, since the image processing device 10 can suspend and resume processing without degrading the processing performance of the RNN of the information processing unit 204, it is not necessary to operate the RNN constantly to maintain the processing performance of the information processing unit 204, thereby reducing the power consumption of the image processing device 10. This allows the image processing device 10 to be used for a long period of time even in an environment where it is powered by a battery or a generator.

[0162] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0163] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0164] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an information processing unit; an output unit; When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, and updates the history information stored in the history information storage unit, and the output unit outputs the information-processed image. When the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and the output unit outputs the information-processed image. The image processing device, wherein when the information processing section is in a stopped state, the output section outputs an input image. (Configuration 2) 2. The image processing device according to configuration 1, wherein if the history information is not held in the history information holding section, the history information is determined to be invalid. (Configuration 3) 2. The image processing device according to configuration 1, wherein if the history information held in the history information holding unit is not updated, the history information is deleted. (Configuration 4) The image processing device according to configuration 1, wherein the history information storage unit stores an update flag indicating whether the history information needs to be updated or not, and refers to the update flag to determine that the history information is invalid if the history information needs to be updated, and determines that the history information is valid if the history information does not need to be updated. (Configuration 5) The image processing device according to any one of configurations 1 to 4, characterized in that when the information processing unit is in an operating state and the history information stored in the history information storage unit is invalid, the output unit outputs the input image in parallel with regenerating the history information of the information processing unit. (Configuration 6) The image processing device according to any one of configurations 1 to 5, characterized in that when the information processing unit is in a stopped state, in parallel with the output of the input image by the output unit, the history information holding unit regenerates history information based on the input image and updates the history information held in the history information holding unit. (Configuration 7) The history information held in the history information holding unit is valid when the history information held in the history information holding unit is history information of a frame less than a threshold number of frames ago, The image processing device according to any one of configurations 1 to 6, characterized in that the history information stored in the history information storage unit is invalid when the history information stored in the history information storage unit is history information for a frame that is more than a threshold number of frames ago. (Configuration 8) 8. The image processing device according to any one of configurations 1 to 7, wherein the history information is an image that has already been processed by the information processing unit. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the input image is stored in an input image storage unit. (Configuration 10) 10. The image processing device according to configuration 9, wherein the input image storage unit stores at least as many input images as are necessary to regenerate the history information of the information processing unit. (Configuration 11) 11. The image processing device according to configuration 9 or 10, wherein the input image storage unit stores at least as many input images as the latency of the information processing unit. (Configuration 12) The image processing device according to configuration 6, wherein when the information processing unit is in a stopped state and a user interface that changes the information processing unit from a stopped state to an operating state is focused on, the history information holding unit starts a process of regenerating history information based on the input image in parallel with the output of the input image by the output unit. (Configuration 13) 13. The image processing device according to configuration 6 or 12, wherein the regeneration of the history information by the history information holding unit is performed by a method that is lighter than the regeneration of the history information by the information processing unit. (Configuration 14) 13. The image processing device according to configuration 6 or 12, wherein the regeneration of the history information by the history information holding unit is performed by a device that consumes less power than the regeneration of the history information by the information processing unit. (Configuration 15) 15. The image processing device according to any one of configurations 1 to 14, wherein the information processing is noise removal, super-resolution, sharpening, dehazing, or dark area emphasis processing. (Configuration 16) 16. The image processing device according to any one of configurations 1 to 15, wherein the information processing section executes the information processing by a recursive neural network. (Configuration 17) 7. The image processing device according to configuration 6, wherein the history information storage unit executes the regeneration of the history information by a neural network. (Method 1) When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, updates the history information stored in the history information storage unit, and an output unit outputs the information-processed image; when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates a processed image, updates the history information held in the history information holding unit, and an output unit outputs the processed image; When the information processing unit is in a stopped state, the output unit outputs the input image. 10. A processing method for an image processing apparatus, comprising: (Program 1) When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, updates the history information stored in the history information storage unit, and an output unit outputs the information-processed image; when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates a processed image, updates the history information held in the history information holding unit, and an output unit outputs the processed image; When the information processing unit is in a stopped state, the output unit outputs the input image. A program that causes a computer to execute the following. [Explanation of symbols]

[0165] 10 Image processing device, 201 Image acquisition unit, 202 Input image storage unit, 203 History information storage unit, 204 Information processing unit, 205 Output unit

Claims

1. an information processing unit; an output unit; When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, and updates the history information stored in the history information storage unit, and the output unit outputs the information-processed image. When the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and the output unit outputs the information-processed image. The image processing device, wherein when the information processing section is in a stopped state, the output section outputs an input image.

2. 2. The image processing apparatus according to claim 1, wherein if the history information is not stored in the history information storage unit, the history information is determined to be invalid.

3. 2. The image processing apparatus according to claim 1, wherein the history information stored in the history information storage unit is deleted when the history information is not updated.

4. 2. The image processing device according to claim 1, wherein the history information storage unit stores an update flag indicating whether the history information needs to be updated or not, and by referring to the update flag, determines that the history information is invalid if the history information needs to be updated, and determines that the history information is valid if the history information does not need to be updated.

5. The image processing device according to claim 1, characterized in that, when the information processing unit is in an operating state and the history information stored in the history information storage unit is invalid, the output unit outputs the input image in parallel with regenerating the history information of the information processing unit.

6. The image processing device according to claim 1, characterized in that when the information processing unit is in a stopped state, in parallel with the output of the input image by the output unit, the history information holding unit regenerates history information based on the input image and updates the history information held in the history information holding unit.

7. The history information held in the history information holding unit is valid when the history information held in the history information holding unit is history information of a frame less than a threshold number of frames ago, The image processing device according to claim 1, characterized in that the history information stored in the history information storage unit is invalid when the history information stored in the history information storage unit is history information for a frame that is more than a threshold number of frames ago.

8. 2. The image processing apparatus according to claim 1, wherein the history information is an image that has already been processed by the information processing unit.

9. 2. The image processing apparatus according to claim 1, wherein the input image is stored in an input image storage unit.

10. 10. The image processing apparatus according to claim 9, wherein the input image storage unit stores at least as many input images as are necessary to regenerate the history information of the information processing unit.

11. 10. The image processing device according to claim 9, wherein the input image holding unit holds at least as many input images as the latency of the information processing unit.

12. The image processing device according to claim 6, characterized in that when the information processing unit is in a stopped state and focus is placed on a user interface that changes the information processing unit from a stopped state to an operating state, the history information holding unit starts a process of regenerating history information based on the input image in parallel with the output of the input image by the output unit.

13. 7. The image processing apparatus according to claim 6, wherein the regeneration of the history information by the history information holding unit is performed by a method that is lighter than the regeneration of the history information by the information processing unit.

14. 7. The image processing apparatus according to claim 6, wherein the regeneration of the history information by the history information holding unit is performed by a device that consumes less power than the regeneration of the history information by the information processing unit.

15. 2. The image processing device according to claim 1, wherein the information processing is noise reduction, super-resolution, sharpening, dehazing, or dark area emphasis processing.

16. 2. The image processing apparatus according to claim 1, wherein the information processing section executes the information processing using a recursive neural network.

17. 7. The image processing apparatus according to claim 6, wherein the history information holding unit regenerates the history information using a neural network.

18. When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, updates the history information stored in the history information storage unit, and an output unit outputs the information-processed image; when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates a processed image, updates the history information held in the history information holding unit, and an output unit outputs the processed image; When the information processing unit is in a stopped state, the output unit outputs the input image.

10. A processing method for an image processing apparatus, comprising:

19. When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit, generates an information-processed image, updates the history information stored in the history information storage unit, and an output unit outputs the information-processed image; when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information, generates a processed image, updates the history information held in the history information holding unit, and an output unit outputs the processed image; When the information processing unit is in a stopped state, the output unit outputs the input image. A program that causes a computer to execute the following.

Images