Image processing apparatus

Abstract

To enable appropriate processing when an information processing unit is active while history information is unavailable.SOLUTION: When an information processing unit is active and history information held in a history information holding unit is available, the information processing unit executes information processing using an input image and the history information held in the history information holding unit to generate an information-processed image, and updates the history information held in the history information holding unit. An output unit outputs the information-processed image. When the information processing unit is active and if the history information held in the history information holding unit is unavailable, the information processing unit regenerates the history information based on the input image, executes information processing using the input image and the regenerated history information to generate an information-processed image, updates the history information held in the history information holding unit. The output unit outputs the information-processed image. When the information processing unit is inactive, the output unit outputs the input image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a processing method for an image processing device, and a program. [Background technology]

[0002] In recent years, devices have been proposed that use neural networks to improve the image quality of images captured by cameras, etc. By using neural networks, it becomes possible to perform image quality improvement processes such as noise reduction and super-resolution with higher performance and speed than conventional methods.

[0003] In particular, in the case of video, the ability to perform high-quality processing in real time is required. Recurrent neural networks (RNNs) are widely used as a method to achieve real-time video performance.

[0004] RNNs are characterized by their structure in which the output results from the RNN, i.e., history information, are recursively input to the RNN. This feature makes it possible to process time series data of any length with a single neural network.

[0005] In addition, because RNN has a recursive structure, it is not necessary to prepare a neural network for each time series, and the size of the neural network can be reduced. This enables faster processing, i.e., real-time processing.

[0006] Patent Document 1 discloses an image processing device that uses an RNN to achieve low latency and low cost.

[0007] Patent document 2 discloses a device that, when processing is stopped and then restarted, extracts the history obtained in the history generation process, determines whether the input information in the extracted history is still valid or invalid, and does not use the input information in the history if it is invalid. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2021-149333 A [Patent Document 2] JP 2006-3743 A Summary of the Invention [Problem to be solved by the invention]

[0009] While neural networks such as RNNs can perform complex processing such as improving image quality with high performance and speed, they tend to require a large amount of calculations, which in turn increases the power consumption of the device.

[0010] In recent years, edge devices for processing images captured by cameras and other devices have been installed in moving objects such as vehicles, ships, aircraft, and drones. There is a demand for such edge devices to be power-efficient in order to conserve fuel for batteries and generators.

[0011] To achieve power saving, it is necessary to be able to temporarily turn off the functions of a device when they are not needed and to instantly turn on the functions when they are needed.

[0012] When using an RNN for processing, the RNN has a structure in which the previous output result, i.e., history information, is recursively input to the RNN. Therefore, to fully utilize the performance of the RNN, it is necessary to continuously input images, etc. to the RNN.

[0013] However, when the operation of an RNN is stopped for reasons such as power saving, the history information remains outdated, which creates the problem that the RNN's performance cannot be fully utilized when execution is resumed.

[0014] In particular, in low frame rate videos, which are often seen in surveillance cameras, the processing interval becomes longer, and the time during which the RNN cannot demonstrate its performance when it resumes execution becomes longer. This can result in, for example, screen flickering or overlooking fast-moving surveillance targets (such as people or vehicles).

[0015] On the other hand, if the RNN processing is run constantly to avoid screen flickering or missing monitored targets, power consumption will always be high, making it impossible to use the system for long periods of time on batteries or generators.

[0016] The method of Patent Document 2 has a problem in that, although it is possible to erase history information, there is no valid history information when the processing is restarted, and therefore performance cannot be demonstrated immediately after the processing is restarted.

[0017] An object of the present disclosure is to enable appropriate processing to be performed when an information processing unit is in an operating state and history information is invalid. [Means for solving the problem]

[0018] The image processing device has an information processing unit and an output unit, and when the information processing unit is in an operating state and the history information held in the history information holding unit is valid, the information processing unit performs information processing using an input image and the history information held in the history information holding unit, generates an information-processed image, and updates the history information held in the history information holding unit, and the output unit outputs the information-processed image, and when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on the input image, performs information processing using the input image and the regenerated history information, generates an information-processed image, and updates the history information held in the history information holding unit, and the output unit outputs the information-processed image, and when the information processing unit is in a stopped state, the output unit outputs the input image. Effect of the Invention

[0019] According to the present disclosure, when the information processing unit is in an operating state and the history information is invalid, appropriate processing can be performed. [Brief description of the drawings]

[0020] [Figure 1] FIG. 2 illustrates an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 illustrates an example of a functional configuration of an image processing apparatus. [Diagram 3] FIG. 2 is a diagram illustrating an example of a processing flow of the image processing device. [Figure 4] FIG. 4 illustrates an example of processing by an information processing unit. [Diagram 5] FIG. 13 is a diagram showing an example in which a neural network is used. [Figure 6] FIG. 11 is a diagram illustrating another example of the processing flow of the image processing device. [Figure 7] FIG. 13 is a diagram showing an example in which a neural network is used. [Figure 8] FIG. 13 illustrates an example of a user interface. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, preferred embodiments will be described in detail with reference to the drawings. Note that the configurations in the following embodiments are merely examples, and the present invention is not limited to these configurations.

[0022] 1 is a diagram showing an example of the hardware configuration of an image processing device 10 according to this embodiment. The image processing device 10 has, as its hardware configuration, a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F device 16.

[0023] The control device 11 controls the entire image processing device 10. The storage device 12 holds programs and data necessary for the operation of the control device 11. The arithmetic device 13 executes necessary arithmetic processing based on the control of the control device 11.

[0024] The input device 14 is a human interface device or the like, and inputs user operations to the image processing device 10. The output device 15 is a display or the like, and presents the processing results of the image processing device 10 to the user.

[0025] The I / F device 16 is a wired interface such as a universal serial bus, Ethernet, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth. The I / F device 16 also has functions such as connecting a camera or the like to input captured images to the image processing device 10, transmitting processing results obtained by the image processing device 10 to the outside, and inputting programs, data, etc. required for the operation of the image processing device 10 to the image processing device 10. The camera is a video camera, a security camera, etc.

[0026] 2 is a diagram showing an example of the functional configuration of the image processing device 10 according to this embodiment. The image processing device 10 has, as its functional configuration, an image acquisition unit 201, an input image storage unit 202, a history information storage unit 203, an information processing unit 204, and an output unit 205.

[0027] The image acquisition unit 201 acquires an image to be subjected to image processing by the image processing device 10 as an input image.

[0028] The input image storage unit 202 stores the input image acquired by the image acquisition unit 201. The history information storage unit 203 stores history information.

[0029] The information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, and generates a processed image and updated history information.

[0030] If it is determined before execution of the information processing unit 204 that the history information held by the history information holding unit 203 is invalid, the history information holding unit 203 regenerates the history information from the input image held by the input image holding unit 202 and holds it as new history information.

[0031] Here, regeneration refers to discarding the invalid history information held by history information holding unit 203 when it is determined that the history information held by history information holding unit 203 is invalid, and regenerating only the history information from the input image held by input image holding unit 202.

[0032] The history information holding unit 203 receives the updated history information generated by the information processing unit 204, and holds it as new history information.

[0033] When the information processing unit 204 is in an operating state, the output unit 205 outputs the information-processed image generated by the information processing unit 204, and when the information processing unit 204 is in a stopped state, the output unit 205 outputs the input image held by the input image holding unit 202.

[0034] FIG. 3A is a flowchart showing an example of a processing method of the image processing device 10 according to this embodiment.

[0035] In step S101, the image acquisition unit 201 acquires, as an input image, an image to be subjected to image processing by the image processing device 10. Here, the input image is, for example, streaming, a video file, or a series of image files saved for each frame, a video or image saved in a medium, or the like.

[0036] The image acquisition unit 201 may acquire an input image from a solid-state imaging element such as a CMOS sensor, a CCD sensor, or a SPAD sensor, or a camera equipped with such a solid-state imaging element, or may acquire an input image from a storage device and a recording medium such as a hard disk or an SSD. There may be only one element, camera, storage device, recording medium, etc. that acquires the input image, or there may be multiple elements.

[0037] In step S102, the input image storage unit 202 stores the input images acquired by the image acquisition unit 201. The input image storage unit 202 may store only one latest image, or may store multiple images in chronological order.

[0038] Furthermore, the input image storage unit 202 may set an upper limit on the number of images that can be stored, and when the upper limit is exceeded, images may be discarded beginning with the oldest. When the information processing unit 204 requires multiple images when executing information processing, the above upper limit may be set as the number of images required by the information processing unit 204.

[0039] In addition, the above upper limit may be set to a number of images with some leeway so that there is no conflict between the writing of the input images acquired by the image acquisition unit 201 to the input image storage unit 202 and the reading of the input images from the input image storage unit 202 by the information processing unit 204.

[0040] For example, if the input image acquired by the image acquisition unit 201 is written to the input image storage unit 202 as one image, and the information processing unit 204 requires three images when executing information processing, the input image storage unit 202 may be configured to store at least four images.

[0041] Furthermore, in order to prevent processing slowdown when subsequent processing is delayed, the upper limit of the number of images that can be stored may be increased. For example, if processing slowdown of up to 20 frames in a video is permitted, the input image storage unit 202 may store an additional 20 frames.

[0042] The input image storage unit 202 may store additional images as many as are necessary when regenerating the history information in step S107. For example, if three consecutive frames of images are necessary when regenerating the history information in step S107, the input image storage unit 202 may store an additional three frames. In this case, the input image storage unit 202 stores at least the number of input images necessary for regenerating the history information of the information processing unit 204.

[0043] If there is latency in the processing from step S105 to S108, the input image storage unit 202 may store additional images for the latency. For example, if the latency is three frames, the input image storage unit 202 may store only one image from three frames ago, or may store an additional three frames. In this case, the input image storage unit 202 stores at least as many input images as the latency of the information processing unit 204.

[0044] It is preferable that the input image storage unit 202 simultaneously stores an identifier for uniquely identifying the input image in addition to the input image acquired by the image acquisition unit 201. Here, the identifier may be, for example, the time of shooting.

[0045] It is more preferable that the identifier indicates the chronological order of the input images. Examples of such identifiers include frame numbers and serial numbers. Here, the frame numbers and serial numbers may be obtained from, for example, a camera equipped with a solid-state imaging device, a storage device such as a hard disk or SSD, or a recording medium, or may be assigned a time or a unique number may be generated inside the input image storage unit 202.

[0046] In step S103, the image processing device 10 determines the state of the information processing unit 204. Here, the state of the information processing unit 204 is whether the information processing unit 204 is in a stopped state or in an operating state.

[0047] The information processing unit 204 being in a stopped state means that the information processing unit 204 cannot start a new process, and the information processing unit 204 being in an operating state means that the information processing unit 204 can start a new process.

[0048] If the information processing unit 204 is waiting for processing, for example, waiting for initialization, waiting for reconfiguration due to an option change, or waiting for information processing to be completed, the information processing unit 204 is considered to be in a stopped state because it cannot start new processing.

[0049] Even when the processing of the information processing unit 204 is skipped, i.e., when no information processing is performed on the input image and the input image is output as is, the information processing unit 204 is considered to be in a stopped state because it cannot start new processing since no input image is supplied.

[0050] If it is determined that the information processing unit 204 is in a stopped state, the process proceeds to step S104, and if it is determined that the information processing unit 204 is in an operating state, the process proceeds to step S105.

[0051] In step S 104 , the output unit 205 outputs the input image held in the input image holding unit 202 .

[0052] Here, when the input image storage unit 202 stores one input image, the output unit 205 outputs the input image. When the input image storage unit 202 stores two or more input images, the output unit 205 outputs any one of the input images stored in the input image storage unit 202.

[0053] As for any one of the above images, for example, a method of outputting the oldest image in chronological order among the input images held in the input image holding unit 202 can be mentioned.

[0054] Furthermore, if there is latency in the processing from steps S105 to S108, the output unit 205 may obtain an image from the input image storage unit 202 as old as the latency, and output the image in accordance with the latency.

[0055] By matching the latency of the process in step S104 with the latency of the processes in steps S105 and after, it is possible to prevent inconveniences such as missing or skipping frames when the process switches in step S103 in moving image processing.

[0056] When an identifier is assigned to the output image, the output unit 205 may output the identifier simultaneously with the image.

[0057] In step S105, the history information holding unit 203 determines whether the history information it holds is valid or invalid, and if the history information is valid, it proceeds to step S106, and if the history information is invalid, it proceeds to step S107.

[0058] In step S106 following step S105, the information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, and generates a processed image and updated history information.

[0059] If, at step S106, the history information held by the history information holding unit 203 is chronologically older than that expected by the information processing unit 204, the processing by the information processing unit 204 will not be performed as expected, and this may result in a decrease in processing accuracy, output abnormalities, output deficiencies, etc.

[0060] Therefore, in step S105, it is determined whether the history information held by the history information holding unit 203 is chronologically older than what was assumed by the information processing unit 204, and the history information is determined to be valid or invalid, and subsequent processing is branched based on the determination result.

[0061] In the above determination, if the history information is chronologically older than what was assumed by the information processing unit 204, the held history information is determined to be invalid, and otherwise it is determined to be valid.

[0062] For example, in the example of FIG. 4(a), it is assumed that the information processing unit 204 is designed to use an image 401 of the current frame N as an input image, and history information 402 of the immediately preceding frame N-1.

[0063] In FIG. 4(a), an image 401 of frame N and history information 402 of frame N-1 are input to the information processing unit 204, so that a processing result 403 of the current image of frame N and history information 404 of frame N are generated.

[0064] Here, it is assumed that the frame number is given as an identifier to the input image held in the input image holding unit 202 .

[0065] FIG. 4B shows an example in which the information processing unit 204 is operated up to frame N-10, stopped immediately thereafter, and then operated again when processing frame N.

[0066] In FIG. 4B, an image 401 of frame N is input to the information processing unit 204 as the input image, but history information 405 of frame N-10 at the time of the previous execution is input as the history information.

[0067] As shown in FIG. 4(a), when an image 401 of frame N is used as an input image, the information processing unit 204 is designed to use history information 402 of the immediately preceding frame N-1.

[0068] 4B, however, the information processing unit 204 has been stopped from processing frame N-10 until frame N-1, so there is no history information for the immediately preceding frame N-1. Therefore, if history information 405 of frame N-10, which should not have been used, is used for input image 401 of frame N, the information processing unit 204 cannot demonstrate its original performance and outputs a degraded information-processed image 406 and degraded history information 407 of frame N.

[0069] In the case of FIG. 4(b), the history information 405 of frame N-10 stored in the history information storage unit 203 is a chronologically older frame than the frame N-1 which is actually required, so the history information 405 of frame N-10 is determined to be invalid.

[0070] On the other hand, if the information processing unit 204 is not stopped, as shown in FIG. 4(a), at the time when the image 401 of frame N is used as the input image, the history information holding unit 203 holds the history information 402 of the immediately preceding frame N-1, and therefore it is determined that the history information 402 is valid.

[0071] In the example of Figure 4(b), the history information is judged to be invalid if it is delayed by one frame or more. However, when processing at a high frame rate or when the subject is moving slowly, there is little change in the image between adjacent frames, so it may be possible to process the history information even if it is delayed by several frames.

[0072] In such a case, M is a constant greater than 1, and if the history information is delayed by M frames or more, it is deemed to be old and judged to be invalid, and if the history information is delayed by less than M frames, it is not deemed to be old and is judged to be valid.

[0073] That is, the history information held in history information holding unit 203 is valid when the history information held in history information holding unit 203 is history information for a frame less than the threshold number of frames ago, and the history information held in history information holding unit 203 is invalid when the history information held in history information holding unit 203 is history information for a frame more than the threshold number of frames ago.

[0074] If there is no history information stored in the history information storage unit 203 at the time of step S106, the processing by the information processing unit 204 will not be performed as expected, which may result in reduced processing accuracy, output abnormalities, output deficiencies, and the like.

[0075] Therefore, in step S105, if the history information holding unit 203 does not hold the history information, that is, if the history information does not exist, it may be determined that the history information is invalid.

[0076] On the other hand, when the history information storage unit 203 stores history information, i.e., when history information exists, for example, as described above, it is possible to determine whether the history information is chronologically old, determine whether the history information is valid or invalid, and branch subsequent processing based on the determination result.

[0077] If the processing of the information processing unit 204 is skipped, that is, if the input image is output as is without performing information processing on it, the history information held by the history information holding unit 203 is not updated, and the history information becomes chronologically outdated.

[0078] In such a case, the history information holding unit 203 may delete the history information that is held and that is chronologically older.

[0079] If the history information has been deleted, in step S105, the history information holding unit 203 does not hold the history information, that is, the history information does not exist, and therefore it is determined that the history information is invalid.

[0080] The history information holding unit 203 may hold an update flag indicating whether the history information needs to be updated or not.

[0081] The history information needs to be updated when, for example, the history information held by the history information holding unit 203 is not updated due to skipping the processing of the information processing unit 204, and the history information becomes chronologically outdated.

[0082] The update flag can be expressed, for example, as a true / false value that is true if an update is required and false if an update is not required.

[0083] For example, if the history information held by the history information holding unit 203 is updated, the update flag is set to false, and if the history information held by the history information holding unit 203 is not updated, the update flag is set to true.

[0084] In this case, in step S105, it is determined whether the history information is valid or invalid based on the update flag, and the subsequent processing is branched based on the determination result.

[0085] For example, if the update flag is true, the history information is determined to be invalid, and if the update flag is false, the history information is determined to be valid.

[0086] In step S106, the information processing unit 204 performs information processing using the input image stored in the input image storage unit 202 and the history information stored in the history information storage unit 203, generates a processed image and updated history information, and proceeds to step S108.

[0087] The above information processing is various known image processing, such as noise removal, super-resolution, sharpening, dehaze, or dark area emphasis.

[0088] If an identifier has been assigned to the input image, the information processing unit 204 assigns the identifier that was assigned to the input image to the processed image and the updated history information.

[0089] The number of input images and the number of information-processed images may be one or more. Furthermore, the number of input images and the number of information-processed images may be the same or different.

[0090] The number of pieces of history information and the number of pieces of updated history information may be one or more. Furthermore, the number of pieces of history information and the number of pieces of updated history information may be the same or different.

[0091] The processed image may be treated as updated history information, in which case the information processing unit 204 does not need to output the updated history information.

[0092] The history information holding unit 203 holds the updated history information generated by the information processing unit 204 as new history information.

[0093] If an identifier has been assigned to the updated history information, the history information storage unit 203 stores the identifier together with the updated history information as new history information.

[0094] Methods for executing information processing include, for example, machine learning and neural networks.

[0095] Machine learning methods include, for example, decision trees, random forests, and LightGBM.

[0096] Examples of neural networks include a convolutional neural network, a deconvolutional neural network, and an autoencoder that combines both of them.

[0097] FIG. 5 is a diagram showing an example of a case where a neural network 501 is used as the information processing method of the information processing unit 204. In FIG.

[0098] In the neural network 501 shown in the example of FIG. 5, Conv represents a convolution layer, Pooling represents a pooling layer, Upsample represents an upsampling layer, Concat represents a concatenated layer, and Output represents an output layer.

[0099] In the example of FIG. 5, an input image 502 held in the input image holding unit 202 and history information 503 held in the history information holding unit 203 are input to a neural network 501 .

[0100] When an input image 502 is input to Conv 101 of a neural network 501 and history information 503 is input to Conv 103, calculation processing is performed according to the flow of FIG.

[0101] When the processing of the neural network 501 is completed, a processed image 504 is output from the OutPut 101 of the neural network 501 , and updated history information 505 is output from the OutPut 102 of the neural network 501 .

[0102] The information processing unit 204 may perform information processing by using the neural network 501 shown in the example of FIG. 5 in multiple stages.

[0103] For example, when the neural network 501 is used in two stages, in the first stage, the information processing unit 204 generates updated history information of frame N-1 using the input image of frame N-1 and history information of frame N-2.

[0104] In the subsequent second stage, the information processing unit 204 generates a processed image of frame N and updated history information of frame N using the input image of frame N and the history information of frame N-1.

[0105] The information processing unit 204 can execute information processing using a recursive neural network (RNN). A neural network 501 is an example of an RNN. For example, the RNN recursively inputs history information 505 of a previous output result as history information 503 of a current input to the RNN. The history information 505 is, for example, an image that has been processed by the information processing unit 204.

[0106] The information processing unit 204 may treat intermediate outputs of machine learning or neural networks as updated historical information.

[0107] For example, in the example of the neural network 501 shown in FIG. 5, the output from any layer other than the output layers Output1 and Output2 contained within the neural network 501 may be extracted as the intermediate output and treated as updated history information.

[0108] When machine learning or neural networks are used in multiple stages, the information processing unit 204 may extract output from other than the final stage of machine learning or neural network as the intermediate output and treat it as updated history information.

[0109] For example, when the neural network 501 in FIG. 5 is used in two stages, in the first stage, the output obtained from the output layer Output1 or Output2 using the input image of frame N-1 and the history information of frame N-2 may be extracted as the intermediate output and treated as updated history information.

[0110] In step S107, the information processing unit 204 regenerates history information based on the input image stored in the input image storage unit 202, and proceeds to step S106. Here, if an identifier has been assigned to the input image, the information processing unit 204 assigns the identifier that was assigned to the input image to the regenerated history information.

[0111] The number of input images may be one or more. The number of updated history information may be one or more.

[0112] A method for regenerating the history information may include, for example, inputting the input image stored in the input image storage unit 202 and provisional history information into the information processing unit 204, executing information processing, discarding the processed image, and obtaining only the updated history information.

[0113] Alternatively, the processed image may be treated as updated history information instead of being discarded, in which case the information processing unit 204 does not need to output the updated history information.

[0114] As the provisional history information, for example, the latest history information stored in the history information storage unit 203 at the time of executing step S107 may be used, or the input image stored in the input image storage unit 202 may be used.

[0115] Other examples of provisional history information include a solid image, a pattern image, a noise image, and the like.

[0116] The history information storage unit 203 stores the regenerated history information as new history information. If an identifier is assigned to the regenerated history information, the history information storage unit 203 stores the identifier together with the regenerated history information as new history information.

[0117] It may take time to regenerate the history information in step S107. For this reason, for example, when the process shown in Fig. 3(a) is performed at regular time intervals, when the process in step S107 is started, it may appear that the entire process has temporarily stopped.

[0118] In such a case, as shown in the example of Fig. 3(b), the image processing device 10 may perform the process of step S104 in parallel with the process of step S107. That is, the image processing device 10 may perform the process of step S104 in parallel and output the input image as it is until the process of step S107 ends and the process of step S106 is resumed.

[0119] Execution of step S107 increases the amount of processing that must be performed by the image processing device 10. Execution of step S107 increases the processing time, which may cause a delay in processing.

[0120] In such a case, the image processing device 10 may eliminate the delay in processing due to the execution of step S107. For example, the image processing device 10 may perform the processing of step S107 in a separate thread, or may temporarily increase the processing frame rate.

[0121] If, in step S105, the history information holding unit 203 determines that the history information it holds is invalid, and if step S107 is not performed, in step S106, the information processing unit 204 will perform information processing using the invalid history information.

[0122] If the history information is invalid, the performance of the information processing unit 204 cannot be fully demonstrated. This causes flickering and a decrease in accuracy during image processing of moving images, and the flickering and the decrease in accuracy cause inconvenience such as overlooking a monitored object.

[0123] Therefore, to fully utilize the performance of the information processing unit 204, the information processing unit 204 must be kept in operation at all times, and the history information must always be kept up to date.

[0124] For this reason, the information processing unit 204 cannot be turned off when not needed, and the power consumption of the image processing device 10 cannot be reduced, so that the image processing device 10 cannot be used for a long time in an environment using a battery or a generator.

[0125] On the other hand, in step S105, if the history information holding unit 203 determines that the held history information is invalid and S106 is executed after the processing of S107, the information processing unit 204 can execute information processing using the regenerated latest history information, i.e., the valid history information. In step S106, the information processing unit 204 executes information processing using the input image and the regenerated history information, generates a processed image, and updates the history information held in the history information holding unit 203.

[0126] Since the history information is valid, the performance of the information processing unit 204 can be fully demonstrated. As a result, flickering, loss of accuracy, and the like does not occur during image processing of moving images, and inconveniences such as overlooking a monitored object due to flickering, loss of accuracy, and the like can be prevented.

[0127] Furthermore, when the history information is invalid, the latest regenerated history information is used, so that the information processing unit 204 does not need to be kept in an operating state at all times.

[0128] Therefore, the information processing unit 204 can be put into a stopped state when not needed, thereby reducing the power consumption of the image processing device 10, and thus enabling it to be used for a long period of time even in an environment where it is powered by a battery or a generator.

[0129] When it is determined in step S103 that the information processing unit 204 is in a stopped state, the image processing device 10 may execute the process of step S109 in parallel with step S104, as shown in FIG.

[0130] In step S109, the history information storage unit 203 executes the same process as in step S107. That is, the history information storage unit 203 regenerates history information based on the input image stored in the input image storage unit 202, and updates the history information stored in the history information storage unit 203. Here, if an identifier has been assigned to the input image, the history information storage unit 203 assigns the identifier that was assigned to the input image to the regenerated history information.

[0131] One of the reasons for wanting to stop the information processing unit 204 is to stop the information processing unit 204 when it is not needed, thereby reducing power consumption of the image processing device 10. Therefore, when executing step S109, it is preferable that the image processing device 10 performs processing while reducing power consumption.

[0132] A method of reducing power consumption in step S109 may be to perform the regeneration process in S109 using a method that is lighter than the method of regenerating the history information used in the process in step S107, as shown in the example of Fig. 5. In this case, the regeneration of the history information in step S109 is performed using a method that is lighter than the method of regenerating the history information in step S107.

[0133] Examples of lightweight methods include machine learning, neural networks, etc. Examples of machine learning methods include decision trees, random forests, LightGBM, etc.

[0134] Examples of neural networks include a convolutional neural network, a deconvolutional neural network, and an autoencoder that combines both of them.

[0135] Fig. 7 is a diagram showing an example of a case where a neural network 701 is used as a lightweight method. In the neural network 701 shown in the example of Fig. 7, Conv represents a convolution layer, Pooling represents a pooling layer, Upsample represents an upsampling layer, Concat represents a concatenated layer, and Output represents an output layer.

[0136] In the example of FIG. 7, an input image 702 stored in the input image storage unit 202 is input to a neural network 701 .

[0137] When an input image 702 is input to Conv 101 of a neural network 701, calculation processing is performed according to the flow of FIG.

[0138] When the processing of the neural network 701 is completed, the updated history information 703 is output from the OutPut 101 of the neural network 501 .

[0139] In the example of FIG. 7, neural network 701 is a neural network 501 of FIG. 5 with some parts omitted, but the present invention is not limited to this. The neural network may be made lighter by distillation or quantization, or a neural network with a completely different structure may be used.

[0140] In step S109, the history information storage unit 203 can execute regeneration of the history information by the neural network 701.

[0141] Another method for reducing power consumption in step S109 is, for example, to use a device that consumes less power than usual for the arithmetic unit 13 of the image processing device 10. In this case, the regeneration of history information in step S109 is performed by a device that consumes less power than the regeneration of history information in step S107. To enable execution by a power-saving device, a weight reduction method such as distillation or quantization may be used in combination.

[0142] Alternatively, a plurality of methods with different power consumption may be selected and used depending on the remaining amount of fuel in the battery or generator.

[0143] For example, when the remaining amount is large, the history information may be regenerated with high accuracy using a method with a large amount of calculation, and as the remaining amount decreases, the method may be gradually switched to a weight-saving method, the use of a power-saving device, etc. When the remaining amount becomes even smaller, the process of step S109 may be stopped.

[0144] If the information processing unit 204 is in a stopped state and the process of S109 is not executed in parallel with step S104, the history information becomes older by the time the process is stopped.

[0145] In this state, when the information processing unit 204 is put into operation, if it is determined in step S105 that the history information is invalid, processing is executed in the order of steps S107 and S106, and valid history information is regenerated in step S107.

[0146] As shown in FIG. 6, another method for determining whether to execute the process of S109 in parallel with step S104 is to use an operation of a user interface.

[0147] For example, it is assumed that the information processing unit 204 is switched between a stopped state and an operating state by a user's operation via the input device 14 or the I / F device 16 of the image processing device 10 .

[0148] As an example, consider the case where a user interface as shown in Fig. 8(a) is used. In the example of Fig. 8(a), a processed image 802 output by the information processing unit 204 and a user interface 803 for changing the information processing unit 204 from a stopped state to an operating state are displayed on a screen 801.

[0149] In FIG. 8(a), it is assumed that the information processing unit 204 is in a stopped state, and that the information processing unit 204 can be put into an operating state by operating the user interface 803 by clicking or the like.

[0150] FIG. 8B shows a state in which the user has operated a cursor 804 to place it on the user interface 803 and focus on the user interface 803 in order to operate the user interface 803 by clicking or the like.

[0151] In FIG. 8(b), the color of the buttons is changed from that in FIG. 8(a) to indicate that the user interface 803 is in focus.

[0152] When the user operates the user interface 803 in FIG. 8(b) to change the information processing unit 204 from a stopped state to an operating state, there is a high possibility that the user will then perform an operation such as clicking on the user interface 803, causing it to change to an operating state.

[0153] Therefore, when the information processing unit 204 is in a stopped state, as shown in FIG. 8B, the process of S109 may be started in parallel with step S104 in FIG. 6 at the point in time when the focus is placed on the user interface 803.

[0154] In the state of FIG. 8B, it is assumed that the user interface 803 for changing from a stopped state to an operating state is focused on, and then an operation such as a click is performed, causing the information processing unit 204 to change from a stopped state to an operating state.

[0155] Here, it is assumed that the process of step S109 is completed between the time when the user interface 803 is focused and the time when the information processing unit 204 is put into an operating state.

[0156] When the processing of step S109 is completed, the history information held in the history information holding unit 203 is updated to the latest information, so that when the information processing unit 204 enters an operational state, the history information is determined to be valid in S105, and S106 can be executed immediately without any additional overhead.

[0157] In step S108, the output unit 205 outputs the information-processed image generated in step S106 by the information processing unit 204. If an identifier is assigned to the output image, the output unit 205 may output the identifier simultaneously with the image.

[0158] The output unit 205 may present the processing results to the user via the output device 15 of the image processing device 10, or may present the processing results to another terminal or the like via the I / F device 16.

[0159] In this embodiment, image processing is taken as an example of information processing, but the information processing is not limited to image processing. For example, the information processing may include behavior recognition, posture estimation, fall detection, tracking, flow rate estimation, and the like.

[0160] As described above, according to this embodiment, the image processing device 10 temporarily stops the RNN processing of the information processing unit 204, and when it is restarted, if the history information is invalid, the image processing device 10 regenerates appropriate history information. This feature makes it possible to immediately demonstrate the performance of the RNN when the RNN processing is restarted, so that flickering, loss of accuracy, etc., does not occur during image processing of moving images, and it is possible to prevent overlooking of a monitoring target due to flickering, loss of accuracy, etc.

[0161] In addition, since the image processing device 10 can stop and resume processing without degrading the processing performance of the RNN of the information processing unit 204, it is not necessary to operate the information processing unit 204 all the time in order to maintain the processing performance of the RNN, and it is possible to reduce the power consumption of the image processing device 10. This allows the image processing device 10 to be used for a long time even in an environment where it is powered by a battery or a generator.

[0162] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0163] It should be noted that the above-described embodiments are merely illustrative of specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0164] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing unit; An output unit, When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using an input image and the history information stored in the history information storage unit to generate an information-processed image and update the history information stored in the history information storage unit, and the output unit outputs the information-processed image. When the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and the output unit outputs the information-processed image. 2. An image processing device comprising: an information processing unit that outputs an input image when the information processing unit is in a stopped state; (Configuration 2) 2. The image processing device according to configuration 1, wherein if no history information is held in the history information holding section, it is determined that the history information is invalid. (Configuration 3) 2. The image processing device according to configuration 1, wherein if the history information held in the history information holding unit is not updated, the history information is deleted. (Configuration 4) The image processing device according to configuration 1, wherein the history information storage unit stores an update flag indicating whether the history information needs to be updated or not, and refers to the update flag to determine that the history information is invalid if the history information needs to be updated, and determines that the history information is valid if the history information does not need to be updated. (Configuration 5) The image processing device according to any one of configurations 1 to 4, characterized in that when the information processing unit is in an operating state and the history information stored in the history information storage unit is invalid, the output unit outputs an input image in parallel with regenerating the history information of the information processing unit. (Configuration 6) The image processing device according to any one of configurations 1 to 5, characterized in that when the information processing unit is in a stopped state, in parallel with the output of the input image by the output unit, the history information holding unit regenerates history information based on the input image and updates the history information held in the history information holding unit. (Configuration 7) The history information held in the history information holding unit is valid when the history information held in the history information holding unit is history information of a frame less than a threshold number of frames back; The image processing device according to any one of configurations 1 to 6, characterized in that when the history information stored in the history information storage unit is invalid, the history information stored in the history information storage unit is history information for a frame that is more than a threshold number of frames ago. (Configuration 8) 8. The image processing device according to any one of configurations 1 to 7, wherein the history information is an image that has already been processed by the information processing section. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the input image is stored in an input image storage unit. (Configuration 10) 10. The image processing device according to configuration 9, wherein the input image holding section holds at least a number of input images necessary for regenerating history information of the information processing section. (Configuration 11) 11. The image processing device according to configuration 9 or 10, wherein the input image holding unit holds input images at least the number of which corresponds to the latency of the information processing unit. (Configuration 12) The image processing device according to configuration 6, characterized in that when the information processing unit is in a stopped state and a user interface that changes the information processing unit from a stopped state to an operating state is focused on, the history information holding unit starts a process of regenerating history information based on the input image in parallel with the output of the input image by the output unit. (Configuration 13) 13. The image processing device according to configuration 6 or 12, wherein the regeneration of the history information by the history information holding unit is performed by a method that is lighter than the regeneration of the history information by the information processing unit. (Configuration 14) 13. The image processing device according to configuration 6 or 12, wherein the regeneration of the history information by the history information holding unit is performed by a device consuming less power than the regeneration of the history information by the information processing unit. (Configuration 15) 15. The image processing device according to any one of configurations 1 to 14, wherein the information processing is noise removal, super-resolution, sharpening, mist removal, or dark area emphasis processing. (Configuration 16) 16. The image processing device according to any one of configurations 1 to 15, wherein the information processing section executes the information processing by a recursive neural network. (Configuration 17) 7. The image processing device according to configuration 6, wherein the history information storage unit executes regeneration of the history information by a neural network. (Method 1) when the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit to generate an information-processed image, and updates the history information stored in the history information storage unit; and an output unit outputs the information-processed image. when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and an output unit outputs the information-processed image; When the information processing unit is in a stopped state, the output unit outputs the input image. 13. A processing method for an image processing apparatus comprising: (Program 1) when the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit to generate an information-processed image, and updates the history information stored in the history information storage unit; and an output unit outputs the information-processed image. when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and an output unit outputs the information-processed image; When the information processing unit is in a stopped state, the output unit outputs the input image. A program for causing a computer to execute the following. [Explanation of symbols]

[0165] 10 image processing device, 201 image acquisition unit, 202 input image storage unit, 203 history information storage unit, 204 information processing unit, 205 output unit

Claims

1. An information processing unit; An output unit, When the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using an input image and the history information stored in the history information storage unit to generate an information-processed image and update the history information stored in the history information storage unit, and the output unit outputs the information-processed image. When the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and the output unit outputs the information-processed image.

2. An image processing device comprising: an information processing unit that outputs an input image when the information processing unit is in a stopped state;

2. 2. The image processing apparatus according to claim 1, wherein, when the history information is not held in the history information holding section, the history information is determined to be invalid.

3. 2. The image processing apparatus according to claim 1, wherein when the history information held in said history information holding section is not updated, the history information is deleted.

4. 2. The image processing device according to claim 1, wherein the history information storage unit stores an update flag indicating whether the history information needs to be updated or not, and by referring to the update flag, determines that the history information is invalid if the history information needs to be updated, and determines that the history information is valid if the history information does not need to be updated.

5. The image processing device according to claim 1, characterized in that when the information processing unit is in an operating state and the history information stored in the history information storage unit is invalid, the output unit outputs the input image in parallel with regenerating the history information of the information processing unit.

6. The image processing device according to claim 1, characterized in that when the information processing unit is in a stopped state, in parallel with the output of the input image by the output unit, the history information holding unit regenerates history information based on the input image and updates the history information held in the history information holding unit.

7. The history information held in the history information holding unit is valid when the history information held in the history information holding unit is history information of a frame less than a threshold number of frames back; 2. The image processing device according to claim 1, characterized in that the history information stored in the history information storage unit is invalid when the history information stored in the history information storage unit is history information for a frame that is more than a threshold number of frames ago.

8. 2. The image processing apparatus according to claim 1, wherein the history information is an image that has already been processed by the information processing unit.

9. 2. The image processing apparatus according to claim 1, wherein the input image is stored in an input image storage unit.

10. 10. The image processing apparatus according to claim 9, wherein the input image holding section holds at least a number of input images necessary for regenerating the history information of the information processing section.

11. 10. The image processing apparatus according to claim 9, wherein the input image holding section holds input images at least the number of which corresponds to the latency of the information processing section.

12. The image processing device according to claim 6, characterized in that when the information processing unit is in a stopped state and focus is placed on a user interface that changes the information processing unit from a stopped state to an operating state, the history information holding unit starts a process of regenerating history information based on the input image in parallel with the output of the input image by the output unit.

13. 7. The image processing apparatus according to claim 6, wherein the regeneration of the history information by said history information holding unit is performed by a method that is lighter than the regeneration of the history information by said information processing unit.

14. 7. The image processing apparatus according to claim 6, wherein the regeneration of the history information by said history information holding unit is performed by a device that consumes less power than the regeneration of the history information by said information processing unit.

15. 2 . The image processing device according to claim 1 , wherein the information processing is noise reduction, super-resolution, sharpening, mist elimination, or dark area emphasis processing.

16. 2. The image processing apparatus according to claim 1, wherein the information processing section executes the information processing by a recursive neural network.

17. 7. The image processing apparatus according to claim 6, wherein the history information holding unit regenerates the history information using a neural network.

18. when the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit to generate an information-processed image, and updates the history information stored in the history information storage unit; and an output unit outputs the information-processed image. when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and an output unit outputs the information-processed image; When the information processing unit is in a stopped state, the output unit outputs an input image.

13. A processing method for an image processing apparatus comprising:

19. when the information processing unit is in an operating state and the history information stored in the history information storage unit is valid, the information processing unit executes information processing using the input image and the history information stored in the history information storage unit to generate an information-processed image, and updates the history information stored in the history information storage unit; and an output unit outputs the information-processed image. when the information processing unit is in an operating state and the history information held in the history information holding unit is invalid, the information processing unit regenerates the history information based on an input image, executes information processing using the input image and the regenerated history information, generates an information-processed image, updates the history information held in the history information holding unit, and an output unit outputs the information-processed image; When the information processing unit is in a stopped state, the output unit outputs an input image. A program for causing a computer to execute the following.

Images