Image processing apparatus, image processing method, storage medium, and computer program product

Abstract

The invention provides an image processing apparatus, a processing method thereof, a storage medium, and a computer program product. The image processing apparatus includes an information processing unit that processes an input image using history information to generate an information-processed image, updates the history information, and outputs the history information when the history information held in a history information holding unit is valid while the information processing unit is operated and the history information held in the history information holding unit is valid. The output unit outputs the information-processed image, and in a case where the history information is invalid, the information processing unit regenerates the history information based on the input image, processes the input image and the regenerated history information to generate the information-processed image, updates the history information, and the output unit outputs the information-processed image, and updates the history information based on the information-processed image. The output unit outputs the input image when the information processing unit is stopped.

Description

Technical Field

[0001] The present invention relates to an image processing device, a processing method of the image processing device, a storage medium and a computer program product. Background Art

[0002] Recently, a device has been proposed that uses a neural network to improve the quality of images captured by cameras and other devices. The use of a neural network can provide image quality improvement processing such as noise reduction and super-resolution with high performance and speed compared to the prior art.

[0003] In particular, real-time image quality improvement processing performance is required in moving images. Recurrent Neural Networks (RNNs) are widely used as a method for providing real-time performance for moving images.

[0004] As a characteristic, RNN has a structure in which the output result or historical information from RNN is recursively input into RNN. This structure enables processing of time series data of arbitrary length using a single neural network.

[0005] RNN has a recursive structure that does not require the preparation of a neural network corresponding to time series data, thereby reducing the size of the neural network. This allows high-speed processing or real-time processing.

[0006] Japanese Patent Application Publication No. 2021-149333 discusses an image processing device that uses RNN to provide neural network operation processing with small latency and low cost.

[0007] Japanese Patent Application Publication No. 2006-3743 discusses a device that retrieves a history generated by a history generation process when the process is stopped and then restarted, and determines whether input information in the retrieved history is still valid or invalid. If the input information is invalid, the device does not use the input information in the history. Summary of the invention

[0008] According to aspects of the present invention, an image processing device includes an information processing unit and an output unit. When the information processing unit is running and the historical information retained in the historical information retaining unit is valid, the information processing unit is configured to perform information processing using an input image and the historical information retained in the historical information retaining unit to generate an information-processed image and update the historical information retained in the historical information retaining unit, and the output unit is configured to output the information-processed image. When the information processing unit is running and the historical information retained in the historical information retaining unit is invalid, the information processing unit is configured to regenerate the historical information based on the input image, perform information processing using the input image and the regenerated historical information to generate an information-processed image and update the historical information retained in the historical information retaining unit, and the output unit is configured to output the information-processed image. When the information processing unit is stopped, the output unit is configured to output the input image.

[0009] Further features of the present invention will become apparent from the following detailed description of embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram showing a hardware configuration example of an image processing apparatus.

[0011] Figure 2 is a block diagram showing a functional configuration example of an image processing apparatus.

[0012] Figure 3A and Figure 3B 1 and 2 are flowcharts each showing an example of a process flow of the image processing apparatus.

[0013] Figure 4A and Figure 4B Examples of processes performed by the information processing units are respectively shown.

[0014] Figure 5 An example using a neural network is shown.

[0015] Figure 6 Another example of the processing flow of the image processing apparatus is shown.

[0016] Figure 7 An example using a neural network is shown.

[0017] Fig. 8A and Figure 8B Examples of user interfaces are respectively shown. DETAILED DESCRIPTION

[0018] Neural networks, such as recurrent neural networks (RNNs), provide high performance and high speed for complex processing, such as image quality improvement processing. On the other hand, neural networks tend to increase arithmetic operations. The more arithmetic operations there are, the more power the device consumes.

[0019] Some recent mobile objects (such as vehicles, ships, aircraft, and drones) include edge devices for processing images captured by cameras or other devices. The use of such edge devices brings with it the need to reduce power consumption in order to conserve battery and generator fuel.

[0020] In order to reduce power consumption, some functions of the device are temporarily turned off when they become unnecessary, and turned on immediately when they become necessary.

[0021] In the case of using RNN for processing, RNN has a structure in which previous output results or historical information are recursively input into RNN. Therefore, in order to fully utilize the performance of RNN, it is necessary to continuously input images or other data into RNN.

[0022] However, if the RNN operation is stopped due to power consumption or other reasons, the historical information will become old or will no longer be updated, resulting in the RNN being unable to exert its full performance when the RNN operation is restarted.

[0023] In particular, low frame rate motion images commonly found in surveillance cameras require longer processing intervals, which results in a longer period of time when the RNN operation is restarted. This may cause, for example, screen flickering or miss high-speed moving surveillance targets (running people, running vehicles, etc.).

[0024] On the other hand, in order to avoid screen flickering or missing the monitoring target, it is conceivable that the RNN processing is constantly running. This will cause the power consumption to be always large, making it difficult to use the device for a long time in an environment using batteries or generators.

[0025] In the method discussed in Japanese Patent Application Laid-Open No. 2006-3743, history information may be deleted, which means that there is no valid history information when the process is restarted, so that the process cannot exert its full performance immediately after the process is restarted.

[0026] An object of the present disclosure is to perform appropriate processing when an information processing unit is operating and history information is invalid.

[0027] Embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The configurations described in the following embodiments are merely examples, and the present invention is not limited to these configurations.

[0028] Figure 11 is a block diagram showing a hardware configuration example of an image processing apparatus 10 according to an embodiment of the present invention. As a hardware configuration, the image processing apparatus 10 includes a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an interface (I / F) device 16.

[0029] The control device 11 generally controls the operation of the image processing apparatus 10. The storage device 12 holds programs and data for the operation of the control device 11. The calculation device 13 performs arithmetic processing under the control of the control device 11.

[0030] The input device 14 (eg, a human interface device) is used to input user operations to the image processing apparatus 10. The output device 15 (eg, a display) presents processing results and other information from the image processing apparatus 10 to the user.

[0031] The I / F device 16 is a wired interface such as a universal serial bus (USB), Ethernet, or an optical cable, or a wireless fidelity interface such as a wireless or The I / F device 16 has, for example, the following functions: connecting with a camera or other device to input a captured image to the image processing apparatus 10, sending a processing result obtained at the image processing apparatus 10 to an external device, and inputting a program, data, and other information used for the operation of the image processing apparatus 10 to the image processing apparatus 10. The camera is, for example, a video camera or a security camera.

[0032] Figure 2 1 is a block diagram showing a functional configuration example of the image processing device 10 according to the present embodiment. As a functional configuration, the image processing device 10 includes an image acquisition unit 201 (or image acquisition unit), an input image holding unit 202, a history information holding unit 203, an information processing unit 204, and an output unit 205.

[0033] The image acquisition unit 201 acquires an image subjected to image processing by the image processing device 10 as an input image.

[0034] The input image holding unit 202 holds the input image acquired by the image acquisition unit 201. The history information holding unit 203 holds history information.

[0035] The information processing unit 204 performs information processing using the input image held in the input image holding unit 202 and the history information held in the history information holding unit 203 , thereby generating an information-processed image and updated history information.

[0036] When determining that the history information held in the history information holding unit 203 before the operation of the information processing unit 204 is invalid, the history information holding unit 203 regenerates history information based on the input image held in the input image holding unit 202 and holds the regenerated history information as new history information.

[0037] The term “regeneration” used here refers to a process of discarding history information held in the history information holding unit 203 when it is determined that the history information held in the history information holding unit 203 is invalid, and regenerating history information based only on the input image held in the input image holding unit 202 .

[0038] The history information holding unit 203 receives the updated history information generated by the information processing unit 204 and holds the updated history information as new history information.

[0039] The output unit 205 outputs an information processed image generated by the information processing unit 204 when the information processing unit 204 is operating, and outputs an input image held in the input image holding unit 202 when the information processing unit 204 is not operating.

[0040] Figure 3A : is a flowchart showing an example of a processing method of the image processing device 10 according to the present embodiment.

[0041] In step S101, the image acquisition unit 201 acquires, as an input image, an image that is an image processing target of the image processing apparatus 10. Examples of the input image include a streaming image, a moving image file, a series of image files stored for each frame, and a moving image or image stored in a medium.

[0042] The image acquisition unit 201 may acquire an input image from a solid-state image sensor (e.g., a complementary metal oxide semiconductor (CMOS) sensor, a charge coupled device (CCD) sensor, or a single photon avalanche diode (SPAD) sensor), a camera, or other devices including the solid-state image sensor, or may acquire an input image from a storage device (e.g., a hard disk or a solid-state drive (SSD)) or a recording medium. One or more components, cameras, storage devices, or recording media may be used to acquire an input image.

[0043] In step S102, the input image holding unit 202 holds the input image acquired by the image acquiring unit 201. The input image holding unit 202 may hold a single latest image or a plurality of images in chronological order.

[0044] The input image holding unit 202 may set an upper limit on the number of images to be held, thereby discarding images in chronological order when the number of images to be held exceeds the upper limit. If the information processing unit 204 uses multiple images to perform information processing, the number of images used by the information processing unit 204 may be set to an upper limit.

[0045] The upper limit can be set to a value greater than the number of images to be used by the information processing unit 204, thereby preventing the writing of the input image acquired by the image acquisition unit 201 to the input image holding unit 202 from conflicting with the reading of the input image from the input image holding unit 202 by the information processing unit 204.

[0046] For example, if one input image acquired by the image acquisition unit 201 is written to the input image holding unit 202 and the information processing unit 204 performs information processing using three images, the input image holding unit 202 may be set to hold at least four images.

[0047] In addition, the upper limit of the number of images to be retained may be additionally increased to prevent delays in subsequent processing and skip processing. For example, if skip processing of up to 20 frames is allowed in a moving image, the input image retention unit 202 may be set to store 20 frames additionally.

[0048] The input image holding unit 202 may hold additional images corresponding to the number of images used in regenerating the history information in step S107. For example, if images corresponding to three consecutive frames are used in regenerating the history information in step S107, the input image holding unit 202 may additionally store images corresponding to three frames. In this case, the input image holding unit 202 holds at least the number of input images used by the information processing unit 204 to regenerate the history information.

[0049] In the case where there is a delay in the processing from step S105 to step S108, the input image holding unit 202 may additionally hold an image corresponding to the delay. For example, if the delay corresponds to three frames, the input image holding unit 202 may hold only one image three frames ago, or may additionally hold three frames. In this case, the input image holding unit 202 holds at least the number of input images corresponding to the number of delays in the information processing unit 204.

[0050] Preferably, the input image holding unit 202 stores both the input image acquired by the image acquisition unit 201 and an identifier for uniquely identifying the input image. Examples of the identifier include image capturing time.

[0051] Preferably, the identifier indicates the time sequence of the input images. Examples of the identifier include a frame number and a sequence number. For example, the frame number and the sequence number can be obtained from a camera including a solid-state image sensor, a storage device (such as a hard disk or SSD), or a recording medium. In the input image holding unit 202, a time can be assigned, or a unique number can be generated.

[0052] In step S103, the image processing apparatus 10 determines the state of the information processing unit 204. The state of the information processing unit 204 is a state in which the information processing unit 204 is stopped or a state in which the information processing unit 204 is running.

[0053] The information processing unit 204 being in a stopped state indicates a state in which the information processing unit 204 cannot start a new process, and the information processing unit 204 being in an operating state indicates a state in which the information processing unit 204 can start a new process.

[0054] In addition, while the information processing unit 204 is waiting for processing (eg, initialization) and resetting due to option change, or waiting for completion of information processing, the information processing unit 204 cannot start new processing, and therefore, it is determined that the information processing unit 204 is stopped.

[0055] In addition, when the information processing unit 204 skips processing (ie, outputs an input image without information processing), the input image is not provided to the information processing unit 204. Therefore, the information processing unit 204 cannot start new processing, and it is determined that the information processing unit 204 stops.

[0056] As a result of the above determination, if the information processing unit 204 is stopped, the process proceeds to step S104, and if the information processing unit 204 is operating, the process proceeds to step S105.

[0057] In step S104 , the output unit 205 outputs the input image held in the input image holding unit 202 .

[0058] If the input image holding unit 202 holds one input image, the output unit 205 outputs the input image. If the input image holding unit 202 holds two or more input images, the output unit 205 outputs one of the input images held in the input image holding unit 202.

[0059] For example, one of the images to be output is an older or oldest image in chronological order of the input images held in the input image holding unit 202 .

[0060] In the case where there is a delay in the processing from step S105 to step S108 , the output unit 205 may acquire the number of previous images corresponding to the delay from the input image holding unit 202 , thereby outputting the acquired image.

[0061] Matching the delay in the process of step S104 with the delay in the process of step S105 and subsequent steps eliminates frame drop or frame skipping when switching the process in step S103 in the moving image processing.

[0062] In the case where an identifier is assigned to the output image, the output unit 205 can output the identifier and the image simultaneously.

[0063] In step S105, the history information holding unit 203 determines whether the held history information is valid or invalid. If the history information is valid, the process proceeds to step S106. If the history information is invalid, the process proceeds to step S107.

[0064] In step S106 following step S105 , the information processing unit 204 performs information processing using the input image held in the input image holding unit 202 and the history information held in the history information holding unit 203 , thereby generating an information processed image and updated history information.

[0065] In step S106, if the history information held in the history information holding unit 203 is chronologically older than expected by the information processing unit 204, the processing of the information processing unit 204 cannot be performed as expected, which may result in low processing accuracy, abnormal output, or lost output.

[0066] Therefore, in step S105, it is determined whether the history information held in the history information holding unit 203 is chronologically older than expected by the information processing unit 204, thereby determining whether the history information is valid or invalid. Then, the subsequent processing is branched based on the determination result.

[0067] In the above determination, if the history information is chronologically older than expected by the information processing unit 204, the held history information is determined to be invalid. If the history information is not older than expected by the information processing unit 204, the held history information is determined to be valid.

[0068] For example, in Figure 4A In the example shown, the information processing unit 204 is designed to use the image 401 of the current frame N as an input image, and to use the history information about the frame N-1 immediately before the current frame N.

[0069] like Figure 4AAs shown, an image 401 of frame N and historical information 402 related to frame N-1 are input to the information processing unit 204, thereby generating a processing result 403 of the image of the current frame N and historical information 404 related to frame N.

[0070] Here, it is assumed that a frame number is assigned as an identifier to the input image held in the input image holding unit 202 .

[0071] Figure 4B An example is shown in which the information processing unit 204 is operated until frame N-10, and is stopped immediately after frame N-10, and then is operated again during processing of frame N.

[0072] like Figure 4B As shown, an image 401 of frame N is input as an input image to the information processing unit 204, and history information 405 regarding frame N-10 during the previous processing is input as history information.

[0073] like Figure 4A As shown, when an image 401 of frame N is input as an input image, the information processing unit 204 is designed to use history information 402 about frame N-1 immediately before frame N.

[0074] However, in Figure 4B In the example shown, the information processing unit 204 stops after processing frame N-10 until frame N-1, and therefore, there is no history information about frame N-1 immediately before frame N. If history information 405 about frame N-10 that should not be used is used for the input image 401 of frame N, the information processing unit 204 cannot operate with its original performance, and outputs a degraded information processed image 406 and degraded history information 407 about frame N.

[0075] exist Figure 4B In the illustrated example, the history information 405 regarding the frame N-10 held in the history information holding unit 203 indicates a frame that is chronologically older than the frame N-1 that should have been used, and therefore the history information 405 regarding the frame N-10 is determined to be invalid.

[0076] On the other hand, if the information processing unit 204 is not stopped, Figure 4A As shown, at the timing when the image 401 of frame N is input as an input image, the history information holding unit 203 holds the history information 402 about the frame N-1 immediately before the frame N, and therefore determines that the history information 402 is valid.

[0077] exist Figure 4BIn the example shown, if the historical information is delayed by one or more frames, the historical information is determined to be invalid. However, if the processing is performed at a high frame rate, or if the subject moves at a low speed, the image change between adjacent frames is small, so there is a possibility that the processing can be performed even if the historical information is delayed by several frames.

[0078] In this case, when M is set to a constant greater than 1, if the historical information is delayed by M frames, it can be determined that the historical information is old and therefore invalid, and if the historical information is delayed by less than M frames, it can be determined that the historical information is not old and therefore valid.

[0079] Specifically, when the history information held in the history information holding unit 203 is valid, the history information is history information about a frame that is a frame less than a predetermined number of frames before the current frame. When the history information held in the history information holding unit 203 is invalid, the history information is history information about a frame that is a predetermined number of frames or more before the current frame.

[0080] In step S106 , if the history information holding unit 203 does not include history information, the information processing unit 204 does not perform as expected, which may result in low processing accuracy, abnormal output, or lost output.

[0081] Therefore, in step S105 , if the history information holding unit 203 does not hold the history information, or if the history information does not exist, it can be determined that the history information is invalid.

[0082] On the other hand, if the history information holding unit 203 holds the history information, or if the history information exists, for example, it can be determined whether the history information is old in chronological order as described above, thereby determining whether the history information is valid or invalid. Then, the subsequent processing can be branched based on the determination result.

[0083] For example, if an input image is directly output by skipping the processing of the information processing unit 204 or without performing information processing, the history information held in the history information holding unit 203 is not updated, and thus the history information becomes older in chronological order.

[0084] In this case, the history information holding unit 203 can delete the held history information that becomes older in chronological order.

[0085] If the history information is deleted, then in step S105, the history information holding unit 203 does not hold any history information, or the history information does not exist. Therefore, it is determined that the history information is invalid.

[0086] The history information holding unit 203 may hold an update flag indicating whether the history information is updated.

[0087] Examples of the case where the history information is updated include a case where the history information held in the history information holding unit 203 is not updated and the history information becomes older in chronological order, for example, by skipping the processing of the information processing unit 204 .

[0088] The update flag may be represented by a true value, for example, "true" when the history information is to be updated, and "false" when the history information is not to be updated, for example.

[0089] For example, when the history information held in the history information holding unit 203 is updated, the update flag is set to “false”, and when the history information held in the history information holding unit 203 is not updated, the update flag is set to “true”.

[0090] In this case, in step S105 , it is determined whether the history information is valid or invalid based on the update flag, and the subsequent processing is branched based on the determination result.

[0091] For example, if the update flag indicates "true", it is determined that the history information is invalid, and if the update flag indicates "false", it is determined that the history information is valid.

[0092] In step S106, the information processing unit 204 performs information processing using the input image held in the input image holding unit 202 and the history information held in the history information holding unit 203, thereby generating an information processed image and updated history information. The process then proceeds to step S108.

[0093] The above-mentioned information processing includes various types of known image processing. Examples of various types of information processing include noise removal processing, super-resolution processing, sharpening processing, fog / haze removal processing, and dark portion emphasis processing.

[0094] If an identifier is assigned to the input image, the information processing unit 204 assigns the identifier assigned to the input image to the information-processed image and the updated history information.

[0095] One or more input images and one or more information processed images may be used.

[0096] The number of input images may be the same as or different from the number of processed images.

[0097] One or more pieces of history information and one or more pieces of updated history information may be used.

[0098] The amount of history information may be the same as or different from the amount of updated history information.

[0099] The image after information processing can be regarded as updated history information. In this case, the information processing unit 204 does not necessarily have to output the updated history information.

[0100] The history information holding unit 203 holds the updated history information generated by the information processing unit 204 as new history information.

[0101] If an identifier is assigned to the updated history information, the history information holding unit 203 holds the identifier and the updated history information as new history information.

[0102] Examples of methods for performing information processing include machine learning and neural networks.

[0103] Examples of machine learning methods include decision trees, random forests, and Lightweight Gradient Boosting Machines (LightGBM).

[0104] Examples of neural networks include convolutional neural networks, deconvolutional neural networks, and auto-decoders that connect convolutional neural networks and deconvolutional neural networks.

[0105] Figure 5 An example in which the neural network 501 is used in the information processing method of the information processing unit 204 is shown.

[0106] exist Figure 5 In the neural network 501 shown in the example of , “Conv” represents a convolutional layer, “Pooling” represents a pooling layer, “Upsample” represents an upsampling layer, “Concat” represents a connection layer, and “Output” represents an output layer.

[0107] exist Figure 5 In the illustrated example, an input image 502 held in the input image holding unit 202 and history information 503 held in the history information holding unit 203 are input to the neural network 501 .

[0108] When the input image 502 is input to Conv 101 of the neural network 501 and the history information 503 is input to Conv 103, based on Figure 5 The calculation process is performed according to the processing procedure shown.

[0109] After the processing of the neural network 501 is completed, the information-processed image 504 is output from the Output 101 of the neural network 501 , and the updated history information 505 is output from the Output 102 of the neural network 501 .

[0110] The information processing unit 204 can use Figure 5 The neural network 501 shown in the example performs information processing.

[0111] For example, when the neural network is used in two stages, in the first stage, the information processing unit 204 generates updated history information about frame N-1 using the input image of frame N-1 and history information about frame N-2.

[0112] In the following second stage, the information processing unit 204 generates an information-processed image of frame N and updated history information about frame N using the input image of frame N and the history information about frame N-1.

[0113] The information processing unit 204 can perform information processing using an RNN. The neural network 501 is an example of an RNN. For example, the RNN recursively inputs historical information 505 related to a previous output result as the current input historical information 503 of the RNN. For example, the historical information 505 is an image after information processing generated by the information processing unit 204.

[0114] The information processing unit 204 may process the intermediate output from the machine learning or neural network as the historical information as updated historical information.

[0115] For example, in Figure 5 In the example of the neural network 501 shown, the output of any layer included in the neural network 501 except the output layer of Output 101 and Output 102 can be obtained as an intermediate output, and the intermediate output can be used as updated history information.

[0116] If the information processing unit 204 uses machine learning or a neural network in multiple stages, the output of any stage except the final stage of machine learning or a neural network can be obtained as an intermediate output, and the intermediate output can be used as updated historical information.

[0117] For example, if you use Figure 5 The neural network 501 shown, in the first stage, uses the input image of frame N-1 and the historical information related to frame N-2 to obtain the output obtained from the output layer (Output 101 or Output 102) as an intermediate output, and the intermediate output can be used as the updated historical information.

[0118] In step S107, the information processing unit 204 regenerates the history information based on the input image held in the input image holding unit 202, and then the process proceeds to step S106. In this case, if an identifier is assigned to the input image, the information processing unit 204 assigns the identifier assigned to the input image to the regenerated history information.

[0119] One or more input images may be used. One or more updated historical information may be used.

[0120] Examples of a method for regenerating history information include a method of inputting an input image and temporary history information held in the input image holding unit 202 to the information processing unit 204, performing information processing, discarding the information processed image, and acquiring only updated history information.

[0121] In other embodiments, the processed image is used as updated historical information instead of discarding the processed image. In this case, the information processing unit 204 does not have to output the updated historical information.

[0122] As temporary history information, for example, during execution of the process of step S107 , the latest history information held in the history information holding unit 203 may be used, or the input image held in the input image holding unit 202 may be used.

[0123] Other examples of temporary history information include stereo images, pattern images, and noise images.

[0124] The history information holding unit 203 holds the regenerated history information as new history information. If an identifier is assigned to the regenerated history information, the history information holding unit 203 holds the identifier together with the regenerated history information as new history information.

[0125] In some cases, it takes a long time to regenerate the history information in step S107. Figure 3A In the case of the processing shown, when the processing of step S107 is started, the entire processing appears to be temporarily stopped.

[0126] In such a case, Figure 3B As shown in the example of , the image processing apparatus 10 may perform the process of step S104 in parallel with the process of step S107. Specifically, the image processing apparatus 10 may perform the process of step S104 in parallel and directly output the input image before the process of step S107 is completed and the process of step S106 is restarted.

[0127] Executing the process of step S107 will result in an increase in the processing performed by the image processing apparatus 10. Since the processing of step S107 is executed, the processing time will be increased, which may cause a processing delay.

[0128] In such a case, the image processing apparatus 10 can resolve the processing delay caused by executing the process of step S107. Examples of a method for resolving the delay include a method of executing the process of step S107 in another thread, and a method of temporarily increasing the processing frame rate.

[0129] In step S105 , if it is determined that the history information held in the history information holding unit 203 is invalid, assuming that the process of step S107 is skipped, in step S106 the information processing unit 204 will perform information processing using the invalid history information.

[0130] Invalid history information prevents the information processing unit 204 from exerting its full performance. This may cause screen flickering and reduced accuracy during image processing of moving images, which may cause the monitoring target to be missed due to the screen flickering and reduced accuracy.

[0131] Therefore, in order to enable the information processing unit 204 to exert its full performance, it is necessary to enable the information processing unit 204 to always operate using the latest historical information.

[0132] Therefore, this will prevent the information processing unit 204 from being stopped at an unnecessary time, preventing the reduction of power consumption from being suppressed, thereby making it impossible to use the information processing unit 204 for a long time in an environment using a battery or a generator.

[0133] On the other hand, in step S105, if it is determined that the history information held in the history information holding unit 203 is invalid and the process of step S106 is performed after the process of step S107, the information processing unit 204 can perform information processing using the latest history information or valid history information that has been regenerated. In step S106, the information processing unit 204 performs information processing using the input image and the regenerated history information, thereby generating an information-processed image, and updates the history information held in the history information holding unit 203.

[0134] The effective history information enables the information processing unit 204 to exert its full performance. This prevents screen flickering and reduced accuracy during image processing of moving images, and does not cause the monitoring target to be missed due to screen flickering and reduced accuracy.

[0135] When the history information is invalid, the latest history information that has been regenerated is used, so that it is not necessary to keep the information processing unit 204 running all the time.

[0136] This allows the information processing unit 204 to stop at unnecessary times, reduces power consumption, and enables the information processing unit 204 to be used for a long time even in an environment using a battery or a generator.

[0137] In step S103, if it is determined that the information processing unit 204 is stopped, Figure 6 As shown, the information processing device 10 may execute the process of step S109 in parallel with the process of step S104.

[0138] In step S109, the history information holding unit 203 performs processing similar to that of step S107. Specifically, the history information holding unit 203 regenerates history information based on the input image held in the input image holding unit 202, and updates the history information held in the history information holding unit 203. If an identifier is assigned to the input image, the history information holding unit 203 assigns the identifier assigned to the input image to the regenerated history information.

[0139] One reason for stopping the information processing unit 204 is to stop the information processing device at an unnecessary time to reduce power consumption of the image processing device 10. Therefore, in the process of step S109, it is preferable that the image processing device 10 performs the process with reduced power consumption.

[0140] As a method of reducing the power consumption in step S109, for example, by comparing Figure 5 The regeneration process in step S109 is performed by a more lightweight method (i.e., a less computationally intensive method) than the method used in the process of step S107 shown in the figure to regenerate the history information. In this case, the regeneration of the history information in step S109 is performed by a more lightweight method than the regeneration of the history information in step S107.

[0141] Examples of lighter-weight methods include machine learning and neural networks. Examples of machine learning methods include decision trees, random forests, and LightGBM.

[0142] Examples of neural networks include convolutional neural networks, deconvolutional neural networks, and auto-decoders that connect convolutional neural networks and deconvolutional neural networks.

[0143] As a lightweight method, Figure 7 An example of using a neural network 701 is shown. Figure 7 In the neural network 701 shown in the example of FIG. 1 , “Conv” represents a convolutional layer, “Pooling” represents a pooling layer, “Upsample” represents an upsampling layer, “Concat” represents a connection layer, and “Output” represents an output layer.

[0144] exist Figure 7 In the illustrated example, an input image 702 held in the input image holding unit 202 is input to the neural network 701 .

[0145] When the input image 702 is input to the Conv 101 of the neural network 701, based on Figure 7 The calculation process is performed according to the processing procedure shown.

[0146] After the processing of the neural network 701 is completed, the updated historical information 703 is output from the Output 101 of the neural network 701 .

[0147] exist Figure 7 In the example shown, neural network 701 is Figure 5 A portion of neural network 501 is shown. However, the configuration of neural network 701 is not limited to this example. The method may be lightweight by distillation and quantization, or a neural network with a completely different structure may be used.

[0148] In step S109 , the history information holding unit 203 may regenerate the history information using the neural network 701 .

[0149] Another example of a method for reducing power consumption in step S109 is a method of using a device with lower power than normal operation as the computing device 13 of the image processing apparatus 10. In this case, the regeneration of the history information in step S109 is performed by a device with lower power consumption than the regeneration of the history information in step S107. In order to enable a power-saving device to perform processing, a lightweight method such as distillation or quantization may also be used.

[0150] In some embodiments, one of a plurality of methods having different power consumptions may be selected according to the remaining amount of the battery or the remaining amount of the generator fuel.

[0151] For example, when the remaining power of the battery or the remaining fuel of the generator is large, the history information can be regenerated with high accuracy by a method with a large amount of calculation. When the amount is small, the method can be gradually changed to a lightweight method, such as the use of power-saving equipment. When the amount is further reduced, the processing of step S109 can be stopped.

[0152] In a case where the information processing unit 204 is not operating and the process of step S109 is not executed in parallel with the process of step S104 , the history information does not become older as the time of stopping the process increases.

[0153] When the information processing unit 204 is operated in this state and it is determined that the history information in step S105 is invalid, the process of step S107 and the process of step S106 are sequentially performed, and valid history information is regenerated in step S107.

[0154] It is used to determine whether the process of step S109 is executed in parallel with the process of step S104 (such as Figure 6 Another example of the method shown in FIG. 1 is a method using a user interface operation.

[0155] For example, a case is assumed in which the information processing unit 204 is switched to be stopped or operated by a user operation via the input device 14 or the I / F device 16 of the information processing apparatus 10 .

[0156] For example, suppose you use Fig. 8A The user interface shown in Fig. 8A In the illustrated example, an information-processed image 802 output from the information processing unit 204 and a user interface 803 for causing the information processing unit 204 to start from stop are displayed on a screen 801 .

[0157] exist Fig. 8A In the illustrated example, when the information processing unit 204 is stopped, the information processing unit 204 can be operated by clicking the operation user interface 803 .

[0158] Figure 8B A state is shown in which the user operates the cursor 804 to place it on the user interface 803 and focuses the cursor 804 on the user interface 803, thereby operating the user interface 803 by clicking or other operations.

[0159] exist Figure 8B In the example shown, Figure 8B The color of the button "Start image processing" is shown as Fig. 8A The color shown is different, indicating that the user interface 803 is focused.

[0160] In the user interface 803 for switching the information processing unit 204 from stop to operation, Figure 8B When the user operation shown is focused, it is likely that the user interface 803 is clicked or otherwise operated after the user operation, thereby causing the information processing unit 204 to operate.

[0161] Therefore, in the case where the information processing unit 204 stops, Figure 8B As shown, in response to the user interface 803 being focused, the processing of step S109 may be started in parallel with the processing of step S104 (eg, Figure 6 shown).

[0162] exist Figure 8B In the illustrated state, after the user interface 803 for switching the information processing unit 204 from stop to operation is focused, a click operation or other operations are performed, so that the information processing unit 204 switches from stop to operation.

[0163] Here, the process of step S109 is completed after the user interface 803 is focused and before the information processing unit 204 is operated.

[0164] After the processing of step S109 is completed, the history information held in the history information holding unit 203 is updated with the latest history information. Therefore, when the information processing unit 204 is operated, the history information in step S105 is determined to be valid, and the processing of step S106 can be immediately executed without overhead.

[0165] In step S108, the output unit 205 outputs the information processed image generated in step S106 by the information processing unit 204. If an identifier is assigned to the output image, the output unit 205 may output the identifier and the image simultaneously.

[0166] The output unit 205 can present the processing result to the user via the output device 15 of the image processing apparatus 10 , or present to another terminal via the I / F device 16 .

[0167] Although the present embodiment shows image processing as an example of information processing, information processing is not limited to image processing. Examples of information processing include behavior recognition, direction estimation, landing detection, tracking, and flow estimation.

[0168] As described above, according to the present embodiment, when the information processing device 10 temporarily stops the RNN processing of the information processing unit 204 and makes the information processing unit 204 run again, appropriate history information is regenerated in the case where the history information is invalid. This feature can fully use the performance of the RNN immediately when the RNN processing is restarted, without causing screen flickering or reduction in image processing accuracy, and prevents the monitoring target from being missed during the image processing of the moving image due to screen flickering or reduction in accuracy.

[0169] Furthermore, the information processing device 10 can stop or restart processing without reducing the RNN processing performance of the information processing unit 204, so that it is not necessary to keep the information processing unit 204 running to maintain the RNN processing performance of the information processing unit 204, thereby reducing the power consumption of the image processing device 10. Therefore, even in an environment using a battery or a generator, the image processing device 10 can be used for a long time.

[0170] Other embodiments

[0171] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0172] While the present disclosure has been described with reference to embodiments, it will be understood that the disclosure is not limited to the disclosed embodiments but is instead defined by the scope of the following claims.

Claims

1. An image processing device, comprising: Information processing unit; as well as Output unit, wherein, when the information processing unit is running and the history information held in the history information holding unit is valid, the information processing unit is configured to perform information processing using the input image and the history information held in the history information holding unit to generate an information-processed image and update the history information held in the history information holding unit, and the output unit is configured to output the information-processed image, wherein, when the information processing unit is running and the history information held in the history information holding unit is invalid, the information processing unit is configured to regenerate the history information based on the input image, perform information processing using the input image and the regenerated history information to generate an information-processed image, and update the history information held in the history information holding unit, and the output unit is configured to output the information-processed image, and Wherein, when the information processing unit stops, the output unit is configured to output the input image.

2. The image processing device according to claim 1, wherein: In a case where the history information holding unit does not hold the history information, it is determined that the history information is invalid.

3. The image processing device according to claim 1, wherein: In a case where the history information held in the history information holding unit is not updated, the history information holding unit is configured to delete the history information.

4. The image processing device according to claim 1, in, The history information holding unit is configured to hold an update flag indicating whether the history information is to be updated or not to be updated, and Here, referring to the update flag, in the case where the history information is to be updated, it is determined that the history information is invalid, and in the case where the history information is not to be updated, it is determined that the history information is valid.

5. The image processing device according to claim 1, wherein: In a case where the information processing unit operates and the history information held in the history information holding unit is invalid, the output unit is configured to output the input image in parallel with the regeneration of the history information by the information processing unit.

6. The image processing device according to claim 1, wherein: When the information processing unit stops, the history information holding unit is configured to regenerate the history information based on the input image and update the history information held in the history information holding unit in parallel with the output of the input image by the output unit.

7. The image processing device according to claim 1, in, The case where the history information held in the history information holding unit is valid corresponds to the case where the history information held in the history information holding unit is history information related to a frame less than a predetermined number of frames before the current frame, and Here, the case where the history information held in the history information holding unit is invalid corresponds to the case where the history information held in the history information holding unit is history information related to frames equal to or greater than a predetermined number of frames before the current frame.

8. The image processing device according to claim 1, wherein: The history information indicates images subjected to information processing performed by the information processing unit.

9. The image processing apparatus according to claim 1, wherein: The input image holding unit is configured to hold the input image.

10. The image processing device according to claim 9, wherein: The input image holding unit is configured to hold at least a plurality of input images for the information processing unit to regenerate the history information.

11. The image processing device according to claim 9, wherein: The input image holding unit is configured to hold at least a plurality of input images corresponding to the number of delays in the information processing unit.

12. The image processing apparatus according to claim 6, wherein: When the information processing unit is stopped and a user interface for switching the information processing unit from stop to operation is focused, the history information holding unit is configured to start processing for regenerating history information based on the input image in parallel with the output of the input image by the output unit.

13. The image processing device according to claim 6 or 12, wherein: The history information holding unit is configured to perform regeneration of the history information by a more lightweight method than regeneration of the history information by the information processing unit.

14. The image processing device according to claim 6 or 12, wherein: The regeneration of the history information by the history information holding unit is configured to be performed by a device having lower power consumption than the regeneration of the history information by the information processing unit.

15. The image processing apparatus according to claim 1, wherein: The information processing is noise removal processing, super-resolution processing, sharpening processing, fog / haze removal processing, and dark portion emphasis processing.

16. The image processing apparatus according to claim 1, wherein: The information processing unit is configured to perform information processing using a recurrent neural network.

17. The image processing apparatus according to claim 6, wherein: The history information holding unit is configured to perform regeneration of the history information using a neural network.

18. A processing method of an image processing device, the processing method comprising: In a case where the information processing unit is running and the history information held in the history information holding unit is valid, performing information processing by the information processing unit using the input image and the history information held in the history information holding unit to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, In a case where the information processing unit is running and the history information held in the history information holding unit is invalid, regenerating the history information based on the input image by the information processing unit, performing information processing by the information processing unit using the input image and the regenerated history information to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, and When the information processing unit is stopped, the input image is outputted through the output unit.

19. A non-transitory computer-readable storage medium storing a program for causing a computer to execute a method, the method comprising: In a case where the information processing unit is running and the history information held in the history information holding unit is valid, performing information processing by the information processing unit using the input image and the history information held in the history information holding unit to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, In a case where the information processing unit is running and the history information held in the history information holding unit is invalid, regenerating the history information based on the input image by the information processing unit, performing information processing by the information processing unit using the input image and the regenerated history information to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, and When the information processing unit is stopped, the input image is outputted through the output unit.

20. A computer program product comprising a program for causing a computer to execute a control method, the control method comprising: In a case where the information processing unit is running and the history information held in the history information holding unit is valid, performing information processing by the information processing unit using the input image and the history information held in the history information holding unit to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, In a case where the information processing unit is running and the history information held in the history information holding unit is invalid, regenerating the history information based on the input image by the information processing unit, performing information processing by the information processing unit using the input image and the regenerated history information to generate an information-processed image, updating the history information held in the history information holding unit by the information processing unit, and outputting the information-processed image by the output unit, and When the information processing unit is stopped, the input image is outputted through the output unit.

Images