Information processing device, information processing method, and program

Abstract

To enable an image in which the visibility of a specific subject is high to be obtained.SOLUTION: An information processing device generates a second image by subjecting a first image which is obtained by photographing to a prescribed image quality increasing process, and acquires an evaluation value which indicates a visibility evaluation result of a specific subject in each of the first image and the second image. Then, the information processing device selects either the first image or the second image on the basis of the evaluation values.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for processing captured images. [Background technology]

[0002] In recent years, machine learning has been applied to various image quality improvement application programs. Examples of image quality improvement processes that use machine learning include noise reduction, fog reduction, and super-resolution processing. Such image processing techniques that use machine learning are also applied to surveillance systems, for example. Specifically, a known technology is one that generates high-quality images from images captured by a surveillance system in low-visibility environments, such as at night or during foggy conditions, to make monitoring and analysis easier for observers. Furthermore, Patent Document 1 discloses a technology that, when characters, numbers, etc. are recognized in an identification medium area within an image, performs super-resolution processing to improve the visibility of the characters, numbers, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-21787 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a captured image is subjected to high-quality image processing such as noise reduction processing, the image may become one in which the visibility of specific objects, such as characters, contained in the image is reduced.

[0005] Therefore, an object of the present invention is to make it possible to acquire an image in which a specific target has high visibility. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized by having an image quality improvement means that applies a predetermined image quality improvement process to a first image obtained by photographing to generate a second image, an evaluation means that obtains an evaluation value for the first image and the second image that indicates an evaluation result of the visibility of a specific object in each image, and a selection means that selects either the first image or the second image based on the evaluation value. [Effects of the Invention]

[0007] According to the present invention, it is possible to acquire an image in which a specific target has high visibility. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating an application example of an information processing device. [Figure 2] FIG. 1 is a diagram illustrating an example of a system configuration including an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] 4 is a flowchart of information processing according to the first embodiment. [Figure 5] FIG. 10 is a diagram used to explain the evaluation process. [Figure 6] 10 is a flowchart of information processing according to a first modified example. [Figure 7] 10 is a flowchart of information processing according to a second modified example. [Figure 8] 10 is a flowchart of information processing according to the second embodiment. [Figure 9] 10 is a flowchart of information processing according to the third embodiment. [Figure 10] 10 is a flowchart of information processing according to a third modified example. [Figure 11] FIG. 10 is a diagram illustrating an application example of an information processing device according to a fourth embodiment. [Figure 12] 10 is a flowchart of information processing according to the fourth embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a system configuration including an information processing apparatus according to a fifth embodiment. [Figure 14] 13 is a flowchart of information processing according to the fifth embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of a display UI. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in each embodiment are necessarily essential to the solution of the present invention. The configuration of each embodiment may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, in the following embodiments, the same or similar configurations and processing steps are designated by the same reference numerals, and redundant explanations will be omitted.

[0010] First Embodiment First, an example of an environment in which the information processing apparatus according to the first embodiment is used will be described. In this embodiment, as an application example of an information processing device, a monitoring system will be described in which a monitored object such as a ship operating at sea at night is monitored, and images of the monitored object such as a ship are visually inspected by a monitor or the like to monitor and analyze the monitored object. When an observer visually checks an image of a monitored object such as a ship, the observer may focus on characters (strings of characters) such as an identification number or ship name painted on the port or starboard side or stern of the ship. The character strings such as the identification number or ship name that the observer focuses on are information that can be used to identify the ship's affiliation and the items being transported, and are referred to as "specific objects" in this embodiment. In this way, in a monitoring system, the observer visually checks specific objects consisting of strings of characters such as an identification number or ship name, so it is desirable that the images of the monitored object, such as a ship, have high visibility of the specific objects.

[0011] For example, images of a distant vessel or other monitored object captured at night or during hazy conditions often have poor visibility, such as high noise levels, low contrast, and low resolution. In such cases, methods are often adopted to improve the visibility of images by performing predetermined image quality enhancement processes, such as noise reduction, defogger, and super-resolution processing, on the images with poor visibility. In this embodiment, an image with high noise levels resulting from high-sensitivity photography at night will be described as an example of an image with low visibility. Furthermore, in this embodiment, a process for generating a noise-reduced image from a noisy image using a neural network model (hereinafter referred to as an NN model) trained for noise reduction will be described as an example of image quality enhancement processing. However, after image quality enhancement processing, the visibility of specific character strings, such as an identification number or ship name, may be reduced. This reduced visibility may occur not only in images subjected to noise reduction processing but also in images subjected to image quality enhancement processes such as defogger and super-resolution processing.

[0012] 1(a) is a diagram showing an overview of a surveillance system 100 as an example to which an information processing device 101 according to this embodiment is applied, and shows how a surveillance target is monitored using images captured of a target area 103. As shown in FIG. 1(a), the surveillance system 100 is configured to include an information processing device 101 and an imaging device 102.

[0013] The imaging device 102 is a surveillance camera that captures an area to be photographed 103, and outputs images of each frame in a time series of images of the area to be photographed 103. Fig. 1(a) shows an example in which a ship 104, which is an object to be monitored, is sailing on the sea at night, and the image captured by the imaging device 102 shows the ship 104 sailing on the sea at night.

[0014] FIG. 1(b) is a diagram showing a captured image (hereinafter referred to as input image 110) input to the information processing device 101 from the camera device 102 capturing an image of the target area 103, and an enlarged image 111 of the area around the ship 104 captured in the input image 110. As shown in the enlarged image 111, the ship 104 is depicted with character strings such as the identification number and ship name described above, and these character strings of the identification number and ship name are shown as examples of the specific object 105. The input image 110 is an image captured by the camera device 102 of the target area 103 at night, and therefore contains a lot of noise caused by capturing an image at high sensitivity in a dark environment. Note that although the input image 110 is an image containing a lot of noise, as can be seen from the enlarged image 111, the character strings of the specific object 105 are visible to a certain extent.

[0015] FIG. 1(c) shows a high-resolution image 112 obtained after the information processing device 101 has performed noise-reducing image enhancement processing on the input image 110 shown in FIG. 1(b), and an enlarged image 113 of the area around the ship 104 in the high-resolution image 112. The high-resolution image 112 shown in FIG. 1(c) is an image that has been subjected to noise-reducing image enhancement processing, and therefore has reduced noise compared to the input image 110 shown in FIG. 1(b). However, comparing the enlarged image 111 in FIG. 1(b) with the enlarged image 113 in FIG. 1(c), the high-resolution image 112 shows that the sharpness of the character string of the specific object 105 has been reduced by the image enhancement processing, reducing the visibility of each character. Therefore, this high-resolution image 112 may not be appropriate as an image for surveillance or analysis by an observer.

[0016] Therefore, the information processing device 101 of the first embodiment evaluates the visibility of the specific target for each of an input image 110, which is a first image obtained by capturing an image of a monitored object, and a high-quality image 112, which is a second image obtained by performing image quality improvement processing on the input image 110. Then, based on the evaluation results of both the input image 110 and the high-quality image 112, the information processing device 101 selects either the input image 110 or the high-quality image 112, thereby acquiring an image with high visibility of the specific target.

[0017] 2 is a diagram showing an example of the internal configuration of the information processing device 101 of the monitoring system 100. As shown in FIG. 2, the information processing device 101 according to this embodiment has an image acquisition unit 203, an image quality improvement unit 204, an evaluation unit 205, a selection unit 206, and an output unit 207.

[0018] The image acquisition unit 203 acquires an input image 110 of a frame at the time when the photographing device 102 photographs the photographing target area 103 or a plurality of frames in time series. The image quality improving unit 204 performs image quality improvement processing on the input image 110 acquired by the image acquisition unit 203 from the image capturing device 102. In this embodiment, the image quality improving unit 204 performs image quality improvement processing to reduce noise in the image using an NN model trained for the purpose of noise reduction (i.e., a noise reduction model: NR model). The image quality improving unit 204 then sends the high-quality image 112 generated by the image quality improvement processing to the evaluation unit 205 together with the input image 110.

[0019] The evaluation unit 205 performs an evaluation process for the input image 110 and the high-quality image 112 to evaluate the visibility of specific objects in the images, and obtains an evaluation value indicating the evaluation result. In this embodiment, characters are used as an example of specific objects, so the evaluation unit 205 performs an evaluation process from the perspective of visibility for the characters of the specific object 105 in the input image 110 and the characters of the specific object 105 in the high-quality image 112. Details of the evaluation process performed by the evaluation unit 205 will be described later. The evaluation unit 205 then outputs the input image 110, the high-quality image 112, and evaluation values ​​indicating the evaluation results for the specific object 105 in the input image 110 and the specific object 105 in the high-quality image 112, respectively, to the selection unit 206.

[0020] The selection unit 206 compares an evaluation value indicating the evaluation result for the specific target 105 in the input image 110 with an evaluation value indicating the evaluation result for the specific target 105 in the high-quality image 112. Then, based on the comparison result of these evaluation values, the selection unit 206 selects either the input image 110 or the high-quality image 112, and sends the selected image together with information related to the image, such as the evaluation value (hereinafter referred to as related information), to the output unit 207. Note that the related information may include, in addition to the evaluation value, images of frames before and after the frame of the selected image, as well as images that were not selected, as will be described in a sixth embodiment described later. Details of the selection process by the selection unit 206 will be described later.

[0021] The output unit 207 outputs the image and related information selected by the selection unit 206 to the outside. The image and related information output from the output unit 207 are recorded in a recording device (not shown). The image and related information output from the output unit 207 may be recorded in a recording area provided on a network (not shown). Furthermore, if the information processing device 101 is equipped with a recording device (not shown), the information processing device 101 may record the image and related information selected by the selection unit 206 in an internal recording device.

[0022] FIG. 3 is a diagram showing an example of the hardware configuration of the information processing device 101. A CPU (Central Processing Unit) 301 controls various devices connected to a bus 308 and executes information processing related to each functional unit of the information processing device 101 shown in FIG. A ROM (Read Only Memory) 302 stores a BIOS program and a boot program. A RAM (Random Access Memory) 303 is used as a main storage device for the CPU 301 . The external memory 304 stores an information processing program according to this embodiment. The information processing program stored in the external memory 304 is expanded into the RAM 303 and executed by the CPU 301. This realizes the various functional units of the information processing device 101 shown in Fig. 2. The external memory 304 can also record images output from the output unit 207.

[0023] The input unit 305 is a keyboard, mouse, touch panel, or the like, and performs processing related to the input of information from the user. The display unit 306 outputs the calculation results of the information processing device 101 to a display device in accordance with instructions from the CPU 301. The display device may be any type, such as a liquid crystal display device, a projector, or an LED (Light Emitting Diode) indicator. A display UI (user interface image) described in a sixth embodiment below is displayed on the screen of the display device by the display unit 306. The I / O (Input / Output) 307 is connected to the image capturing device 102, an external recording device and network (not shown), an external display device, and the like, and communicates with them. The bus 308 connects the CPU 301, the ROM 302, the RAM 303, the external memory 304, the input unit 305, the display unit 306, and the I / O 307 in a manner that allows them to communicate with one another.

[0024] The processing procedure and detailed processing method of the information processing device 101 according to this embodiment will be described below with reference to Figures 1, 2, 4, and 5. Figure 4 is a flowchart showing the flow of information processing performed by each functional unit of the information processing device 101 according to this embodiment shown in Figure 2.

[0025] First, in the process of step S401, the image acquisition unit 203 acquires an image captured by the image capture device 102 as the input image 110. As described in FIG. 1, the input image 110 is an image of a ship 104 (monitoring target) and the like that is captured by the image capture device 102 capturing an image of the capture target area 103, and specific objects such as letters are depicted on the ship 104. In the case of this embodiment, the input image 110 is an image that contains a lot of noise because it is an image captured at night with high sensitivity, as described above.

[0026] Next, in the process of step S402, the image quality improving unit 204 performs image quality improvement processing to reduce noise on the input image 110 from the image acquisition unit 203, thereby generating a high-image-quality image 112. In this embodiment, the image quality improving unit 204 performs image quality improvement processing using an NN model that has been trained with the aim of reducing noise around the specific target 105 in the input image 110 and improving the visibility of the specific target 105. Alternatively, the image quality improving unit 204 may perform image quality improvement processing using an NN model (i.e., an NR model) that has been trained to perform noise reduction separately between a region of the specific target and a region other than the specific target in the input image 110.

[0027] Next, in step S403, the evaluation unit 205 performs evaluation processing on the specific target 105 in the input image 110 and the specific target 105 in the high-quality image 112 from the perspective of character visibility, and acquires evaluation values ​​as the evaluation results. Details of the evaluation processing by the evaluation unit 205 will be described later using FIG. 5.

[0028] Next, in the process of step S404, the selection unit 206 compares the evaluation values ​​obtained by the evaluation unit 205 from both the specific objects in the input image 110 and the high-quality image 112, and selects the image with the higher evaluation value. For example, if the specific object 105 in the input image 110 has a higher evaluation value, the selection unit 206 selects the input image 110. On the other hand, if the specific object 105 in the high-quality image 112 has a higher evaluation value, the selection unit 206 selects the high-quality image 112. Details of image selection based on evaluation values ​​by the selection unit 206 will be explained later using FIG. 5.

[0029] Next, in the process of step S405, the output unit 207 outputs the image and related information selected by the selection unit 206. In the case of this embodiment, the image and related information output from the output unit 207 is output to a recording device (not shown).

[0030] Figure 5(a) is a diagram used to explain an example of the evaluation process related to the visibility of characters of a specific target, which is performed by the evaluation unit 205 in step S403. Image 510 in Figure 5(a) is an image in which the image portions of the ship 104 and the specific target 105 are extracted from the input image 110 shown in Figure 1(b). Similarly, image 511 in Figure 5(a) is an image in which the image portions of the ship 104 and the specific target 105 are extracted from the high-quality image 112 shown in Figure 1(c).

[0031] The evaluation unit 205 has a character recognizer 501, which performs character recognition processing on the input image 110 and the high-resolution image 112. Character recognition result 502 in FIG. 5(a) shows the result of recognition by the character recognizer 501 from the specific object 105 of the ship 104 in the input image 110. Similarly, character recognition result 503 in FIG. 5(a) shows the result of recognition by the character recognizer 501 from the specific object 105 of the ship 104 in the high-resolution image 112. Note that the ship 104 illustrated in FIG. 1 has the word "TRANSPORT" written on it, and in the character recognition processing on the input image 110, the character string "TRANSPORT" is recognized from the specific object 105 of the ship 104 as character recognition result 502. On the other hand, in the character recognition processing on the high-resolution image 112, the character string "18AN5PC81" is recognized from the specific object 105 of the ship 104 as character recognition result 503. In other words, in the high-resolution image 112, the sharpness of the characters of the specific object 105 on the ship 104 has been reduced due to the high-resolution processing, so the character string "18AN5PC81" is recognized as the character recognition result 503 instead of "TRANSPORT".

[0032] The character recognizer 501 performs character recognition processing using, for example, an OCR (Optical Character Recognition) model trained to read handwritten or printed characters from an image. In character recognition processing using the OCR model, characters are obtained as character recognition results, and likelihoods for the characters in the character recognition results are calculated. That is, in character recognition processing using the OCR model, for example, if the character in the character recognition results is "T," a likelihood indicating the likelihood that the character in the character recognition results is "T" is also calculated. The evaluation unit 205 of this embodiment acquires the likelihood calculated in character recognition processing using the OCR model as an evaluation value in terms of character visibility.

[0033] The evaluation unit 205 may also perform rule-based evaluation of the character string acquired by the character recognition process using the OCR model. For example, when recognizing the identification number of a small vessel by character recognition, the evaluation unit 205 may also evaluate whether the character string acquired by the character recognition process contains a "country code," "serial number," "month of manufacture," "year of manufacture," etc. The evaluation unit 205 may then perform processing such that if the character string acquired by the character recognition process is close to the identification number of the vessel 104, which is the monitored object, the evaluation unit 205 calculates a higher evaluation value. Note that while the present embodiment has been described as an example of recognizing the identification number of a small vessel, if the monitored object is a vehicle in Japan and the specific object is the characters on its license plate, the evaluation unit 205 may also evaluate whether the character string contains a "place name," "number string," "hiragana," etc.

[0034] Fig. 5(b) is a diagram used to explain another example of the evaluation process for the visibility of the specified target character, which is performed in step S403 by the evaluation unit 205. In Fig. 5(b), character recognition results 502 and 503 are assumed to be character recognition results obtained by the character recognizer 501 in Fig. 5(a).

[0035] In the case of FIG. 5(b), the evaluation unit 205 inputs both the character strings of the character recognition result 502 and the character recognition result 503 into an LLM (Large Language Models) recognizer 504 to obtain evaluation values ​​for these character strings. The LLM is a model trained using a large amount of text data, and in this embodiment, it is trained as a model that performs two-class classification based on whether an input character string is a ship name or not. The LLM recognizer 504 obtains, as evaluation values, the classification results of whether the character recognition result 502 and the character recognition result 503 are a ship name or not, or the likelihood that the result is a ship name, for each of the character recognition results 502 and 503. That is, the LLM recognizer 504 obtains evaluation values ​​that become higher the closer the character strings of the character recognition result 502 and the character recognition result 503 are to the character string (in this example, the ship name) associated with the ship 104, which is the monitored object.

[0036] 5(b) shows an example in which the LLM recognizer 504 acquires the likelihood that the character recognition result 502 is the name of a ship as evaluation value 505, and acquires the likelihood that the character recognition result 503 is the name of a ship as evaluation value 506. The evaluation value 505 shown in FIG. 5(b) shows an example in which "82" is acquired as the likelihood that the character string of the character recognition result 502 recognized by the character recognizer 501 from the specific object 105 of the ship 104 in the input image 110 is the name of a ship. On the other hand, the evaluation value 506 in FIG. 5(b) shows an example in which "20" is acquired as the likelihood that the character string of the character recognition result 503 recognized by the character recognizer 501 from the specific object 105 of the ship 104 in the high-quality image 112 is the name of a ship.

[0037] Fig. 5(c) is a diagram used to explain yet another example of the evaluation process for the visibility of the specified target character, which is performed in step S403 by the evaluation unit 205. In Fig. 5(c), character recognition results 502 and 503 are assumed to be character recognition results obtained by the character recognizer 501 in Fig. 5(a).

[0038] In the case of Figure 5(c), the evaluation unit 205 inputs both the character strings of the character recognition result 502 and the character recognition result 503 into the classifier 507, thereby obtaining an evaluation value for these character strings. The classifier 507 is a classifier that has trained to output object category names such as ships and vehicles that are closely related from character strings such as the identification numbers of multiple ships or vehicle license plates, and obtains the likelihood immediately before multiplying by the softmac function as an evaluation value. In other words, the classifier 507 obtains an evaluation value that becomes higher the closer the character strings of the character recognition result 502 and the character recognition result 503 are to the character string (identification number) attached to the ship 104, which is the monitored object.

[0039] 5(c) shows an example in which classifier 507 acquires the likelihood that character recognition result 502 is the name of a ship as evaluation value 508, and acquires the likelihood that character recognition result 503 is the name of a ship as evaluation value 509. Evaluation value 508 shown in FIG. 5(c) shows an example in which "70" is acquired as the likelihood that the character string of character recognition result 502 recognized by character recognizer 501 from specific object 105 of ship 104 in input image 110 is the name of a ship. On the other hand, evaluation value 509 in FIG. 5(c) shows an example in which "6" is acquired as the likelihood that the character string of character recognition result 503 recognized by character recognizer 501 from specific object 105 of ship 104 in high-quality image 112 is the name of a ship.

[0040] As described above, the information processing device 101 of the first embodiment selects and outputs the image with higher visibility for a specific target in an area of ​​the monitored object that the monitor focuses on, from the input image 110 and the high-quality image 112. As a result, according to this embodiment, it is possible to obtain an image of the specific target with good visibility.

[0041] <First Modification> In the first embodiment, an example was given in which a high-quality image is generated from an input image, and the visibility of a specific target is evaluated for both the input image and the high-quality image, and an image is selected based on the evaluation result. In the first modified example below, an example will be described in which the visibility of a specific target is first evaluated for the input image, and an NN model to be used in the image quality improvement process is selected based on the evaluation result, thereby performing image quality improvement processing specialized in improving the visibility accuracy of the specific target.

[0042] In the first modified example, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 1 to 3, and therefore illustrations and descriptions thereof will be omitted. The first modified example differs from the first embodiment described above in that the NN model to be used in the image quality improvement process is selected based on the evaluation value obtained by the evaluation process performed on the input image.

[0043] Fig. 6 is a flowchart showing the flow of information processing according to the first modified example. In the flowchart of Fig. 6, the processes of steps S401, S404, and S405 are the same as the corresponding steps in the flowchart of Fig. 4, and therefore their description will be omitted.

[0044] In the first modified example, after step S401 described above, the processing of the information processing device 101 proceeds to step S601. In step S601, the evaluation unit 205 of the information processing device 101 acquires an evaluation value related to character visibility for a specific object in the input image sent via the image quality improvement unit 204. The evaluation processing at this time is the same as that described with reference to FIG. 5 above, except that it is performed on the input image, and therefore a detailed description thereof will be omitted. Then, the evaluation unit 205 sends the evaluation value resulting from the evaluation processing performed on the input image to the image quality improvement unit 204.

[0045] Next, in step S602, the image quality improvement unit 204 selects an NN model based on the evaluation value acquired from the input image in step S601, and uses the selected NN model to perform image quality improvement processing on the input image to generate a high-quality image. In the first modified example, the image quality improvement unit 204 has multiple NN models (NR models) trained to perform noise reduction processing for each character type, such as Japanese and alphanumeric characters, and selects one of the multiple NN models based on the evaluation value of the input image. For example, if the character recognition result 502 in FIG. 5, which contains many alphanumeric characters, is obtained in step S601, the image quality improvement unit 204 selects an NN model specialized for noise reduction for alphanumeric characters from among multiple NN models trained to perform noise reduction for each character type, such as Japanese and alphanumeric characters. Note that the NN model selected by the image quality improvement unit 204 may be, for example, an NN model trained to perform noise reduction separately for specific target areas and other areas, or an NN model with adjusted image quality improvement processing strength (i.e., noise reduction strength).

[0046] Next, in the process of step S603, the evaluation unit 205 obtains an evaluation value for the high-quality image generated by the image quality improving unit 204 in step S602. The evaluation process here is the same as the process described above with reference to FIG. 5, so a detailed description thereof will be omitted. The evaluation unit 205 then outputs the evaluation value obtained from the input image in step S601 and the evaluation value obtained from the high-quality image in step S603 to the selection unit 206. After step S603, the process of the information processing device 101 proceeds to step S404 and subsequent steps as described above.

[0047] As described above, the information processing device 101 of the first modified example performs image quality improvement processing specialized for improving the visibility accuracy of a specific target by selecting an NN model for image quality improvement processing based on the evaluation result of the visibility of a specific target in an input image. Therefore, according to the first modified example, it is possible to provide the observer with an image with even higher character visibility accuracy than the example of the first embodiment.

[0048] <Second Modification> In the first embodiment, an example was given in which evaluation processing was performed on an input image and a high-quality image of a frame at a certain time point. In the second modified example below, an example will be described in which evaluation processing is performed on input images and high-quality images of multiple frames, and image selection is performed based on multiple evaluation results obtained from the images of the multiple frames.

[0049] In the second modified example, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 1 to 3, and therefore illustrations and descriptions thereof will be omitted. However, in the case of the second modified example, the information processing device 101 performs evaluation processing on images of multiple frames, and selects images based on the multiple evaluation results, which is a difference from the first embodiment described above.

[0050] FIG. 7 is a flowchart showing the flow of information processing according to the second modified example. First, in the process of step S701, the image acquisition unit 203 acquires input images of multiple frames from the image capturing device 102. Note that the image acquisition unit 203 may acquire images of multiple frames that are consecutive in time series, or may acquire multiple frames extracted every predetermined number of frames from the frames in time series, or multiple images of specific frames from the frames in time series. In the case of the second modified example, the image acquisition unit 203 acquires images of N consecutive frames at time t (t=1, 2,...N (N is a natural number)), for example.

[0051] Next, in the process of step S702, the image quality improving unit 204 performs image quality improvement processing using an NN model on the input image made up of N frames acquired by the image acquisition unit 203 in step S701, in the same manner as described in the first embodiment. As a result, the image quality improving unit 204 generates a high-quality image made up of N frames. The image quality improving unit 204 then sends the high-quality image made up of these N frames and the input image also made up of N frames to the evaluation unit 205.

[0052] Next, in step S703, the evaluation unit 205 performs a process of evaluating character visibility for each frame on the input image consisting of N frames acquired in step S701 and the high-quality image consisting of N frames generated in step S702. This results in N evaluation values ​​being obtained for each of the input image and the high-quality image, each consisting of N frames. In the second modified example, the evaluation process is the same as that described above with reference to FIG. 5, so a detailed description thereof will be omitted.

[0053] Next, in step S704, the selection unit 206 determines whether to select the input image or the high-quality image based on the evaluation value acquired for each frame in step S703. In the second modified example, the selection unit 206 determines whether to select the input image or the high-quality image based on, for example, statistics of N evaluation values ​​acquired from each image of N frames. As the statistics of the N evaluation values, at least one of the average, median, maximum, etc. of the N evaluation values ​​is used. Then, the selection unit 206 selects the image with the larger statistics of the evaluation values ​​as the image with the higher evaluation. Next, in the process of step S705, the output unit 207 outputs the image and its related information selected by the selection unit 206. The output destination of the output unit 207 is the same as in the above-described embodiment, and therefore a description thereof will be omitted.

[0054] As described above, the information processing device 101 of the second modified example selects and outputs either the input image or the high-quality image based on the statistics of multiple evaluation values ​​obtained from the input image and the high-quality image of multiple frames. That is, in the case of the second modified example, it becomes possible to evaluate the character visibility using images of multiple frames, and it is possible to reduce the occurrence of a situation where an appropriate image cannot be provided to the observer, for example, because an image of a frame in which the character visibility is accidentally low is selected and output.

[0055] <Second embodiment> In the first embodiment, an example was given in which a high-quality image is generated by performing image quality improvement processing on an input image containing noise, and the image with the highest evaluation value obtained from the high-quality image and the input image is selected. In the second embodiment below, an example is described in which multiple image quality improvement processes are performed on an input image to generate multiple high-quality images, and then evaluation values ​​for the multiple high-quality images and the input image are obtained and the image with the highest evaluation value is selected. In the second embodiment, multiple high-quality images are generated by image quality improvement processing using multiple NN models with noise reduction intensities adjusted to multiple patterns. That is, in the second embodiment, noise reduction intensity parameters for the input image are automatically adjusted, and the image with the highest visibility of a specific target is provided to the observer from among the multiple high-quality images and the input image that have been image quality improved using the adjusted noise reduction intensity parameters.

[0056] In the second embodiment, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 1 to 3, and therefore illustrations and descriptions thereof will be omitted. In the second embodiment, the image quality improvement unit 204 performs multiple image quality improvement processes on an input image to generate multiple high-quality images, and the evaluation unit 205 acquires evaluation values ​​for the multiple high-quality images and the input image, which is different from the above-mentioned embodiments.

[0057] Fig. 8 is a flowchart showing the flow of information processing according to the second embodiment. In the flowchart of Fig. 8, the processes of steps S401, S404, and S405 are the same as the corresponding steps in Fig. 4, and therefore their description will be omitted.

[0058] In the second embodiment, after step S401 described above, the process of the information processing device 101 proceeds to step S801. In step S801, the image quality improvement unit 204 of the information processing device 101 performs image quality improvement processing on the input image using multiple patterns of NN models, each with a different noise reduction strength, to generate multiple high-quality images. Note that the number of patterns of the NN model is not particularly limited, and the user may specify any number of patterns.

[0059] Next, in step S802, the evaluation unit 205 performs an evaluation process on the input image acquired in step S401 and the multiple high-quality images generated in step S802 to acquire an evaluation value related to character visibility. The evaluation process is the same as that described above with reference to FIG. 5, and therefore a detailed description thereof will be omitted. After step S802, the processing of the information processing device 101 proceeds to step S404 and subsequent steps. In the second embodiment, in step S404, the selection unit 206 selects the image with the highest evaluation value from among the input image and the multiple high-quality images based on a comparison of the evaluation values ​​acquired in step S802.

[0060] As described above, the information processing device 101 of the second embodiment selects an image to be output based on the evaluation values ​​of the input image and multiple high-quality images obtained by image quality improvement processes with different noise reduction intensities. That is, according to the second embodiment, it is possible to select and output an image that has high character visibility accuracy and is optimal for the monitor to perform monitoring and analysis.

[0061] <Third embodiment> In the first and second embodiments described above, an example was described in which an image with the highest evaluation value obtained from a noise-containing input image and one or more high-quality images was selected. In the third embodiment described below, an example will be described in which a threshold value for the evaluation value is set in advance, and an image with an evaluation value equal to or greater than the threshold value is selected. In the third embodiment, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 1 to 3, and therefore illustrations and descriptions thereof will be omitted. In the third embodiment, the processing in the image quality improvement unit 204 differs from that in the above-described embodiments.

[0062] Fig. 9 is a flowchart showing the flow of information processing according to the third embodiment. In the flowchart of Fig. 9, the processes of steps S401 and S405 are the same as the corresponding steps in Fig. 4, and therefore their description will be omitted. Also, steps S601, S602, and S603 in Fig. 9 are generally the same as the corresponding steps in Fig. 6.

[0063] In the third embodiment, the process of the information processing device 101 proceeds to step S601 after the above-described step S401. In step S601, the evaluation unit 205 acquires an evaluation value related to character visibility for a specific object in the input image. After step S601, the process of the information processing device 101 proceeds to step S901.

[0064] Next, when the process proceeds to step S901, the image quality improving unit 204 compares the evaluation value acquired for the input image by the evaluation unit 205 in step S601 with a predetermined threshold. Note that a value prepared before the information processing of this embodiment is performed is used as the threshold. The threshold may be, for example, an arbitrary numerical value designated by the user. In step S901, the image quality improving unit 204 determines whether the evaluation value of the input image is less than the threshold. If the evaluation value of the input image is less than the threshold, the process of the information processing device 101 proceeds to step S902; on the other hand, if the evaluation value is equal to or greater than the threshold, the process of the information processing device 101 proceeds to step S903.

[0065] In step S902, the image quality improving unit 204 determines whether the number of times the determination process using the threshold value in step S901 has been performed is less than a predetermined specified number of times. The specified number of times is a number that has been specified in advance as the maximum number of times (i.e., the upper limit number of times) the determination process using the threshold value in step S901 will be performed. If the number of times the determination process has been performed reaches the specified number of times, the information processing device 101 proceeds to step S903. On the other hand, if the number of times is less than the specified number of times, the information processing device 101 proceeds to step S602.

[0066] When the process proceeds to step S602, the image quality improving unit 204 selects an NN model based on the evaluation value acquired from the input image in step S601, and generates a high-quality image by performing image quality improvement processing on the input image using the selected NN model. As described above, in the information processing device 101 of the third embodiment, after the above-described steps S602 and S603, the process returns to step S901, and the processing from step S901 onwards is repeated again. However, if the repeated processing from step S901 onwards is the second or subsequent time, in step S602, the image quality improving unit 204 selects an NN model different from the previously selected NN model as the NN model to be selected based on the evaluation value of the input image. In the case of the third embodiment, if the repeated processing is the second or subsequent time, the image quality improving unit 204 selects an NN model with a different strength of image quality improvement processing from the initial time. In other words, while the number of judgment processes is less than the specified number, the information processing device 101 repeats different image quality improvement processes on the input image by going through steps S602 and S603 and returning to step S901 until the evaluation value of the high-quality image becomes higher than the threshold value.

[0067] Thereafter, when the process proceeds from step S901 or step S902 to step S903, the selection unit 206 selects either the input image or the high-quality image based on the determination results of steps S901 and S902. For example, if the process proceeds to step S903 because the evaluation value of the input image at the time of proceeding from step S601 to step S901 is equal to or greater than a threshold, the selection unit 206 selects the input image as the image to be output. Also, for example, when the processes of steps S901 to S603 are repeated, if it is determined in step S901 that the evaluation value of the high-quality image is equal to or greater than the threshold and the process proceeds to step S903, the selection unit 206 selects the high-quality image as the image to be output. Also, for example, if the evaluation value does not become equal to or greater than the threshold even after the processes of steps S901 to S603 are repeated and the number of determination processes reaches a specified number, the selection unit 206 selects the image with the highest evaluation value among the evaluation values ​​obtained so far as the image to be output. After step S903, in the next step S405, the output unit 207 outputs the image selected by the selection unit 206 and its related information.

[0068] As described above, the information processing device 101 of the third embodiment selects and outputs images having an evaluation value equal to or greater than a preset threshold, thereby enabling the monitor to select an image that achieves a desired character visibility accuracy preset by the monitor. Furthermore, the information processing device 101 of the third embodiment sets an upper limit on the number of times for the determination process of comparing the evaluation value with the threshold, thereby preventing the determination process from continuing indefinitely and enabling the monitor to be provided with an image having the highest visibility evaluation value within the upper limit of the number of times.

[0069] <Third Modification> In the third embodiment, an example was described in which a threshold value for the evaluation value was set in advance. In the following third modified example, an example will be described in which an evaluation value acquired by evaluation processing on an input image is used as a predetermined threshold. In the third modified example, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 1 to 3 above, so illustrations and descriptions thereof will be omitted.

[0070] Fig. 10 is a flowchart showing the flow of information processing according to the third modified example. In the flowchart of Fig. 10, steps S401 to S403 and step S405 are the same as the corresponding steps in Fig. 4, and therefore their description will be omitted. Also, steps S602 and S603 are the same as the corresponding steps in Fig. 6, and steps S902 and S903 are the same as the corresponding steps in Fig. 9, and therefore their description will also be omitted.

[0071] In the third modified example, the process of the information processing device 101 proceeds to step S1001 after steps S401 to S403 described above. In step S1001, the image quality improving unit 204 performs a threshold determination on the evaluation value of the high-quality image, using the evaluation value of the input image acquired by the evaluation unit 205 in step S403 as a threshold. If the result of the threshold determination indicates that the evaluation value of the high-quality image is less than the evaluation value of the input image (i.e., the threshold), the information processing device 101 proceeds to step S902 and subsequent steps as described above, and then returns to step S1001 after step S603. As described above, in the information processing device 101 of the third modified example, after steps S602 and S603, the information processing device 101 returns to step S1001, thereby repeating the processing from step S1001 onwards. In other words, while the number of determination processes is less than the specified number of times, the information processing device 101 repeats the image quality improvement process by returning to steps S1001 to S603 and then to step S1001 until the evaluation value of the high-quality image becomes higher than the evaluation value of the input image. The subsequent image selection by the selection unit 206 is the same as that in the flowchart of FIG. 9 described above.

[0072] As described above, in the information processing device 101 of the third modified example, by using the evaluation value obtained from the input image as a threshold, it becomes possible to select a high-quality image with a character visibility accuracy equal to or higher than the evaluation value of the input image.

[0073] <Fourth embodiment> In the first to third embodiments described above, an example was described in which an image with high visual recognition accuracy of characters is selected and output as a specific target. In the fourth embodiment described below, an example is described in which a person's face is used as a specific target, and an image with high visual recognition accuracy of the face of the specific target is selected and output. In the fourth embodiment, the configurations of the monitoring system 100 and the information processing device 101 are generally similar to those shown in Figures 2 and 3 described above, and therefore illustrations and descriptions thereof will be omitted.

[0074] Fig. 11(a) is a diagram showing an overview of a surveillance system 100 as an example to which an information processing device 101 according to a fourth embodiment is applied, and shows how a surveillance target is monitored using images captured of a target imaging area 1101. In the case of the fourth embodiment shown in Fig. 11(a), the surveillance system 100 is also configured to include an information processing device 101 and an imaging device 102. Fig. 11(a) shows an example in which a person 1102 exists as a surveillance target within the target imaging area 1101, and therefore the person 1102 appears in the image captured by the imaging device 102.

[0075] 11(b) is a diagram showing an image (referred to as an input image 1111) captured by the imaging device 102 of an area to be photographed 1101 and input to the information processing device 101, and an enlarged image 1112 of a region of a person 1102 in the input image 1111. In the case of the fourth embodiment, the information processing device 101 recognizes the face of the person 1102, including at least the eyes, nose, and mouth, as a specific object 1103.

[0076] In the fourth embodiment, for example, the input image 1111 in Fig. 11(b) is assumed to be an image of the photographing area 1101 shown in Fig. 11(a) photographed at night by the photographing device 102. Therefore, the input image 1111 contains a lot of noise caused by photographing a dark environment with high sensitivity.

[0077] In the fourth embodiment, the image quality improving unit 204 of the information processing device 101 also performs image quality improvement processing on the input image 110 acquired by the image acquisition unit 203. Then, the image quality improving unit 204 sends the high-quality image generated by the image quality improvement processing and the input image 110 from the image acquisition unit 203 to the evaluation unit 205. Also in the fourth embodiment, the image quality improving unit 204 performs image quality improvement processing to reduce noise in the image using an NN model trained for the purpose of noise reduction. In the fourth embodiment, the NN model trained for the purpose of noise reduction is an NN model trained for the purpose of improving the visibility of a specific target, a face, by noise reduction.

[0078] FIG. 11(c) shows a high-quality image 1113 obtained after the image quality improvement unit 204 has performed image quality improvement processing on the input image 110, and an enlarged image 1114 of the region of the person 1102 in the high-quality image 1113. The high-quality image 1113 shown in FIG. 11(c) is an image that has been subjected to image quality improvement processing to reduce noise, and therefore has reduced noise compared to the input image 1111 in FIG. 11(b). However, comparing the enlarged image 1112 in FIG. 11(b) with the enlarged image 1114 in FIG. 11(c), it is clear that the high-quality image 1113 has reduced the sharpness of the eyes, nose, mouth, and other parts of the face of the specific target 1103 due to the image quality improvement processing, reducing the visibility of the face. Therefore, the high-quality image 1113 may not be appropriate as an image for surveillance or analysis by a monitor.

[0079] FIG. 12 is a flowchart showing the flow of information processing performed by each functional unit of the information processing device 101 according to the fourth embodiment. First, in the process of step S1201, the image acquisition unit 203 acquires an input image 1111 from the image capturing device 102. As described in Fig. 11, the input image 1111 is an image captured by the image capturing device 102 capturing an image of the image capturing target area 1101, and includes a person 1102 and a face of a specific target 1103. However, the input image 1111 is an image captured at night with high sensitivity, and therefore contains a lot of noise.

[0080] Next, in the process of step S1202, the image quality improving unit 204 performs image quality improvement processing to reduce noise on the input image 1111, thereby generating a high-image-quality image 1113. In the fourth embodiment, the image quality improving unit 204 generates the high-image-quality image 1113 from the input image 1111 using an NN model that has been trained with the aim of improving the visibility of a face, which is a specific target, by reducing noise.

[0081] Next, in step S1203, the evaluation unit 205 performs a process of evaluating the face visibility of the specific target 1103 in the input image 1111 and the specific target 1103 in the high-quality image 1113, respectively, to obtain an evaluation value. In the evaluation process from the perspective of face visibility as in this embodiment, the evaluation unit 205 recognizes, for example, each feature of the face in the image, such as the eyes, nose, and mouth, and obtains the likelihood of each recognized feature as an evaluation value. The evaluation unit 205 may further recognize facial expressions based on the shapes of the eyes, nose, and mouth, and adjust the evaluation value based on the recognition results. For example, the evaluation unit 205 may increase the evaluation value if it can recognize the facial expression, and may decrease the evaluation value if it cannot recognize the facial expression. Additionally, the evaluation unit 205 may adjust the evaluation value depending on whether or not a mask, glasses, sunglasses, etc. are worn. For example, if it recognizes the wearing of a mask, glasses, or sunglasses, the evaluation value may be decreased, or conversely, increased. Also, for example, if a person normally wears glasses, the evaluation value may be increased if the person is recognized as wearing glasses.

[0082] Furthermore, for example, the evaluation unit 205 may have a face recognizer that performs face recognition processing using a face recognition model that has been trained to recognize facial features such as the eyes, nose, and mouth from multiple faces. In this example, the evaluation unit 205 generates a face image as a face recognition result by the face recognizer, calculates the likelihood of the face image as a face recognition result, and acquires the likelihood as an evaluation value related to the visibility of the face. Furthermore, when performing face recognition processing, the evaluation unit 205 may acquire the likelihood of the face recognition result of a specific person as an evaluation value.

[0083] Next, in the process of step S1204, the selection unit 206 compares the evaluation values ​​obtained by the evaluation unit 205 from both the identified targets in the input image 1111 and the high-quality image 1113, and selects the image with the higher evaluation value. For example, if the identified target 1103 in the input image 1111 has a higher evaluation value, the selection unit 206 selects the input image 1111. On the other hand, if the identified target 1103 in the high-quality image 1113 has a higher evaluation value, the selection unit 206 selects the high-quality image 1113. Next, in step S1205, the output unit 207 outputs the image selected by the selection unit 206 together with its associated information. In the fourth embodiment, too, the image and associated information output from the output unit 207 are recorded in a recording device (not shown) or output to a network.

[0084] As described above, the information processing device 101 of the fourth embodiment can select and output the image having higher visibility of the face, which is a specific target that the monitor focuses on in the monitored object, from the input image and the high-quality image.

[0085] <Fifth embodiment> In the first to fourth embodiments described above, examples have been described in which image quality improvement processing is performed using a pre-trained NN model, or an NN model selected based on the evaluation results of an input image or a high-quality image. In the fifth embodiment described below, an example will be described in which an object detector is used to detect monitored objects, including specific targets, from a high-quality image, and an NN model to be used for image quality improvement processing is selected based on the object detection results. In the fifth embodiment, the application example of the monitoring system and information processing device is the same as that shown in FIG. 1, and therefore illustration and description thereof will be omitted.

[0086] In the fifth embodiment, a ship similar to those in the first to third embodiments will be used as an example of a monitored object, and a character string attached to the ship will be used as an example of a specific object. FIG. 13 is a diagram showing an example of the configuration of a monitoring system 100 and an information processing device 1300 according to the fifth embodiment. The information processing device 1300 of the fifth embodiment is configured to include an object detection unit 1301 in addition to an image acquisition unit 203, an image quality improvement unit 204, an evaluation unit 205, a selection unit 206, and an output unit 207. The image acquisition unit 203, the image quality improvement unit 204, the evaluation unit 205, the selection unit 206, and the output unit 207 are generally similar to those described in the first to fourth embodiments, and therefore detailed description thereof will be omitted. In other words, in the example of the fifth embodiment, the same processing as that described in the first to fourth embodiments may be performed. However, in the case of the fifth embodiment, the processing of the image quality improvement unit 204 and the evaluation unit 205 is partially different from that of the above-mentioned embodiments.

[0087] In the fifth embodiment, the object detection unit 1301 sends the high-resolution image output from the image quality improvement unit 204 and the input image to the evaluation unit 205, and also receives the evaluation result from the evaluation unit 205. The object detection unit 1301 determines whether to execute object detection processing based on the evaluation result received from the evaluation unit 205. If it is determined to execute object detection processing, the object detection unit 1301 executes the object detection processing on the high-resolution image generated by the image quality improvement unit 204. In this embodiment, the object detection unit 1301 performs object detection processing on the high-resolution image using, for example, an object detection model trained to identify the position and type of an object (in this case, a ship) in the image. Then, the object detection unit 1301 notifies the image quality improvement unit 204 of the position of the detected object in the image and the type (category) of the object as the object detection result of the object detection processing. Alternatively, for example, if the object to be detected is determined in advance, the object detection unit 1301 may perform object detection processing by pattern matching using a reference image corresponding to the object to be detected. Furthermore, in this embodiment, an example is given in which the object detection process is performed on a high-quality image, but the object detection process may also be performed on an input image.

[0088] Based on the type of object included in the object detection result, the image quality improvement unit 204 selects an NN model trained to improve the visibility of specific targets within that type of object, and then uses the selected NN model to perform image quality improvement processing on the input image again. That is, in the fifth embodiment, the image quality improvement unit 204 selects an NN model corresponding to the type of object detected in the object detection processing from among multiple NN models trained to improve the visibility of specific targets within each of multiple object categories. As a result, the high-quality image generated by the image quality improvement processing is an image in which the visibility of specific targets within the detected object is further enhanced. The image quality improvement unit 204 then outputs the high-quality image generated by the image quality improvement processing and the input image, and the high-quality image and the input image are sent to the evaluation unit 205 via the object detection unit 1301.

[0089] The evaluation unit 205 again performs evaluation processing on the high-quality image from the image quality improving unit 204 to obtain an evaluation value. Then, the evaluation unit 205 outputs the input image, the high-quality image, and their evaluation values ​​to the selection unit 206. As described above, the selection unit 206 selects either the input image or the high-quality image based on the comparison result of the evaluation values, and outputs the selected image and related information to the output unit 207 .

[0090] Fig. 14 is a flowchart showing the flow of information processing according to the fifth embodiment. In the flowchart of Fig. 14, steps S401 to S403, S405, and S406 are the same as the corresponding steps in Fig. 4, and therefore their description will be omitted. Also, step S603 is the same as the corresponding step in Fig. 6, and step S1001 is the same as the corresponding step in Fig. 10, and therefore their description will also be omitted.

[0091] In the fifth embodiment, the processing of the information processing device 1300 proceeds to step S1400 after step S403. In step S1400, the evaluation unit 205 compares the evaluation value of the input image acquired in step S403 with the evaluation value of the high-quality image. If the comparison shows that the evaluation value of the high-quality image is equal to or greater than the evaluation value of the input image, the processing of the information processing device 1300 proceeds to step S404. On the other hand, if the evaluation value of the high-quality image is less than the evaluation value of the input image, the processing of the information processing device 1300 proceeds to step S1401. Note that in step S1400, the evaluation unit 205 may compare the evaluation value of the high-quality image with a preset threshold value.

[0092] In step S1401, the object detection unit 1301 executes object detection processing on the high-quality image generated by the image quality improvement unit 204 in step S402. That is, if the evaluation value of the high-quality image is less than the evaluation value of the input image in step S1400, the object detection unit 1301 executes processing to detect an object in the high-quality image. Then, the object detection unit 1301 notifies the image quality improvement unit 204 of the position of the detected object in the image and the type (category) of the object as the object detection result of the object detection processing.

[0093] Next, in the process of step S1402, the image quality improving unit 204 selects an NN model based on the object detection result by the object detection unit 1301, and performs image quality improvement processing again on the input image using the selected NN model. After step S1402, the process of the information processing device 1300 proceeds to step S603.

[0094] In step S603, the evaluation unit 205 acquires an evaluation value for the high-quality image regenerated by the image quality improvement unit 204 in step S1402. The evaluation unit 205 then acquires an evaluation value from the high-quality image generated in step S1402, and outputs it to the selection unit 206 together with the evaluation value acquired from the input image in step S403. After step S603, the processing of the information processing device 1300 proceeds to step S404. The processing from step S404 onwards is the same as that described above, and therefore will not be described again.

[0095] As described above, the information processing device 1300 of the fifth embodiment determines whether to perform object detection processing based on the evaluation results of the input image and the high-quality image, and if it determines to perform object detection processing, performs the object detection processing on the high-quality image. Then, based on the object detection result, the information processing device 1300 selects an NN model trained to improve the visibility of specific targets of the detected object, and performs image quality improvement processing on the input image again. That is, in the fifth embodiment, by using an NN model that improves the visibility accuracy of specific targets attached to the detected object, it is possible to perform image quality improvement processing with high visibility of the specific targets.

[0096] In the fifth embodiment, the case where the monitored object is a ship has been exemplified, but the monitored object may also be a person (face) as in the above-described fourth embodiment. When the monitored object is a person's face, the information processing device of the fifth embodiment detects the face using an object detector, selects an NN model trained to improve the visibility of the face, and executes image quality improvement processing.

[0097] Sixth Embodiment In the first to fifth embodiments, an example has been described in which the output destination of an image selected by an information processing device is a recording device. In the following sixth embodiment, an example in which the output destination is a display device will be described. In the sixth embodiment, an image output by the information processing device is displayed on a display device, thereby providing a display UI (user interface) that enables a monitor to check an image of a monitored object while displaying an evaluation result or performing image quality improvement processing. Note that in the sixth embodiment, the configurations of the monitoring system 100 and the information processing device and the information processing flowchart can be applied to any of the first to fifth embodiments described above, and therefore illustrations and descriptions thereof will be omitted.

[0098] Fig. 15 is a diagram showing an example of a display UI on the screen of a display device based on an image output from an information processing device in the sixth embodiment. The display UI shown in Fig. 15 is generated by the output unit 207 and displayed on the screen of the display device by the display unit 306 in Fig. 3. Fig. 15(a) is a diagram showing an example of the screen configuration of the display UI. As shown in Fig. 15(a), the display UI is configured to have at least a selected image display screen 1501, an evaluation result notification screen 1502, and an image quality improvement execution instruction screen 1503. Fig. 15(b) is a diagram showing specific examples of displays on each screen of the display UI.

[0099] The selected image display screen 1501 displays an image 1504 selected and output by the information processing device. The selected image display screen 1501 may also display the image of the frame with the highest evaluation value among the images selected by the information processing device for each frame or among the images selected for every N frames. When checking visibility, consecutive display of multiple frame images is more suitable for monitoring and analysis than a single image. Therefore, the selected image display screen 1501 may also display images of frames before and after the selected image as related information of the selected image. The selected image display screen 1501 may also have a sequence bar 1505 and play / rewind / repeat play buttons 1506. The sequence bar 1505 and play / rewind / repeat play buttons 1506 are UIs that can be operated by the observer, allowing the observer to stop the image, display a specific frame, or perform continuous image playback when checking a desired frame. In addition, although not shown, the selected image display screen 1501 may be provided with a zoom-in / zoom-out button for instructing the zoom-in / zoom-out of the image.

[0100] The evaluation result notification screen 1502 has a notification screen 1507 that outputs the evaluation result or evaluation value performed on the image 1504 in the information processing device. The notification screen 1507 may output, for example, a coordinate position so that the target area in the image can be identified, or may superimpose a rectangle around the target area on the image 1504. Furthermore, when evaluation is performed on multiple frames in the information processing device, the notification screen 1507 may notify the evaluation result for each frame, or may notify the evaluation result that is the maximum value among the frames.

[0101] The image quality improvement execution instruction screen 1503 has UIs that allow the observer to issue instructions to execute any image quality improvement process or to return to the state before the image quality improvement process was executed. The UIs on the image quality improvement execution instruction screen 1503 include, for example, an NR model selection parameter UI 1508, a noise reduction adjustment parameter UI 1509, a high image quality process execution button 1510, and a cancel button 1511. The NR model selection parameter UI 1508 is a UI that allows the user to select an NN model (NR model) in a pull-down format, for example. The noise reduction adjustment parameter UI 1509 is a UI that allows the user to adjust the noise reduction intensity in a pull-down format, for example. The high image quality process execution button 1510 is a button for issuing an instruction to execute generation of a high image quality image for the image 1504 based on the set image quality improvement processing parameters. The cancel button 1511 is a button for issuing an instruction to return the image that has been subjected to image quality improvement processing on the display UI to the image 1504 that was selected by the information processing device. In this way, the high image quality execution instruction screen 1503 is provided with a UI for issuing an instruction to execute any high image quality processing or to return to the state before execution, so that a high image quality processing other than that performed on the image 1504 can be performed and the image desired by the observer can be obtained.

[0102] According to the sixth embodiment, it becomes easier for the monitor to monitor and analyze images selected by the information processing device. Furthermore, according to the sixth embodiment, the monitor can adjust parameters related to the image quality improvement processing while immediately checking the image quality improvement processing results on the screen, thereby enabling the monitor to obtain images that are easier for the monitor to monitor / analyze. In other words, according to the sixth embodiment, a display UI system is provided that allows the monitor to check the evaluation results and perform image quality improvement processing while checking images output from the information processing device, thereby enabling support for the monitor's monitoring and analysis.

[0103] <Other application examples> In the first to sixth embodiments described above, ships and people are given as examples of monitored objects, but the monitored objects are not limited to these and may be various objects such as vehicles such as cars, trucks, motorcycles, and bicycles, airplanes, helicopters, drones, and even animals. Also, while character strings and faces are given as specific targets, the specific targets are not limited to these and may be various objects such as patterns, shapes, designs, and colors on objects, and the information processing of the above-described embodiments may be applied to two or more of these specific targets.

[0104] Furthermore, in the above-described embodiments, an image capturing device is installed in a specific location, and images of objects within the target area are input to an information processing device. However, this is not a limitation. The image capturing device may be a camera mounted on various mobile objects, such as an in-vehicle camera, a wearable camera, an action camera, or a satellite-mounted camera. Of course, the image capturing device is not limited to a camera mounted on a mobile object, and may also include cases where a user on a mobile object, such as a vehicle, captures images using a digital camera or video camera. That is, in this case, the information processing device may perform image quality improvement processing on images captured by an image capturing device mounted on a mobile object as the mobile object moves, and evaluation processing on specific objects captured in the images, and select and output images based on the evaluation results. Furthermore, information processing similar to that of the first to sixth embodiments described above can be applied to this example as well.

[0105] In the above-described embodiments, the image quality improving units 204 and 1301 perform noise reduction processing as an example of image quality improvement processing, but the present invention is not limited to this, and may also perform, for example, demisting processing or super-resolution processing. Furthermore, the image quality improving units 204 and 1301 may perform image quality improvement processing that combines two or more of noise reduction processing, demisting processing, and super-resolution processing.

[0106] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of specific embodiments for implementing the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0107] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) an image quality improvement unit that applies predetermined image quality improvement processing to a first image obtained by photographing to generate a second image; evaluation means for acquiring an evaluation value indicating an evaluation result of the visibility of a specific object in each of the first image and the second image; a selection means for selecting either the first image or the second image based on the evaluation value; An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the selection means selects the image having the higher evaluation value from the first image and the second image. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the image quality improvement means applies the predetermined image quality improvement process to the first image using a neural network model trained to improve the visibility of the specific object in the first image. (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the image quality improvement means has a plurality of neural network models trained to improve the visibility of the specific object in the first image, and performs image quality improvement processing on the first image using a neural network model selected from the plurality of neural network models based on the evaluation value of the first image. (Configuration 5) an object detection means for detecting an object including the specific target from the first image or the second image; The information processing device according to any one of configurations 1 to 4, wherein the image quality improvement means applies image quality improvement processing to the first image using a neural network model that has been trained to improve visibility of the specific target of the object detected by the object detection means. (Configuration 6) The object detection means also detects the type of the object, The information processing device according to configuration 5, wherein the image quality improvement means has a plurality of neural network models trained to improve the visibility of specific targets on objects of different types, and performs image quality improvement processing on the first image using a neural network model selected from the plurality of neural network models according to the type of object detected by the object detection means. (Configuration 7) 7. The information processing device according to configuration 5 or 6, wherein the object detection means performs processing to detect the object when the evaluation value of the second image is lower than the evaluation value of the first image. (Configuration 8) The information processing device described in any one of configurations 1 to 7, characterized in that the image quality improvement means applies the predetermined image quality improvement processing to the first image using a neural network model trained to improve the visibility of an area of ​​the specific object in the first image and a neural network model trained to improve the visibility of an area other than the specific object in the first image. (Configuration 9) 3. The information processing device according to configuration 1 or 2, wherein the image quality improving means adjusts the intensity of the predetermined image quality improving process for the first image based on the evaluation value of the first image. (Configuration 10) the image quality improvement means generates two or more second images by performing two or more different image quality improvement processes on the first image, the evaluation means acquires the evaluation values ​​from the first image and the two or more second images, 10. The information processing device according to any one of configurations 1 to 9, wherein the selection means selects the image with the highest evaluation value from the first image and the two or more second images. (Configuration 11) the evaluation means acquires the evaluation value from each of the first image and the second image for each of a plurality of frames; 11. The information processing device according to any one of configurations 1 to 10, wherein the selection means selects either the first image or the second image based on statistics of the acquired plurality of evaluation values. (Configuration 12) The information processing device according to configuration 11, characterized in that the selection means acquires at least one of the average, median, and maximum of the multiple evaluation values ​​of the first image and the second image as the statistical quantity, and selects the image with the larger statistical quantity from the first image and the second image. (Configuration 13) The information processing device described in any one of configurations 1 to 12, characterized in that, when the evaluation values ​​of the first image and the second image are less than a predetermined threshold, the image quality improvement means repeatedly applies different image quality improvement processes to the first image so that the evaluation value of the second image generated by the predetermined image quality improvement process becomes higher than the predetermined threshold. (Configuration 14) 14. The information processing device according to configuration 13, wherein the image quality improving means uses the evaluation value of the first image as the predetermined threshold value. (Configuration 15) The information processing device according to configuration 13 or 14, characterized in that the selection means selects the second image generated by repeating the image quality improvement process when the evaluation value of the second image becomes equal to or greater than the predetermined threshold value. (Configuration 16) The information processing device described in any one of configurations 13 to 15, characterized in that the selection means selects the image with the highest evaluation value from among the first image and the multiple second images generated by repeating the image quality improvement process when the evaluation values ​​of the first image and the second image generated by repeating the image quality improvement process are less than a predetermined threshold and the number of times the image quality improvement process has been repeated reaches an upper limit. (Configuration 17) 17. The information processing apparatus according to any one of configurations 1 to 16, further comprising output means for recording the image selected by the selection means in recording means or displaying the image on display means. (Configuration 18) The information processing device described in configuration 17, characterized in that the output means generates an image of a user interface including a display screen that displays the image selected by the selection means, a notification screen that notifies the user of the evaluation result by the evaluation means, and an execution instruction screen that instructs the image quality improvement means to execute image quality improvement processing, and outputs the generated image to the display means. (Configuration 19) 19. The information processing device according to any one of configurations 1 to 18, wherein the image quality improving means performs at least one image quality improving process selected from the group consisting of noise reduction, mist reduction, and super-resolution. (Configuration 20) 20. The information processing device according to any one of configurations 1 to 19, wherein the specific object is at least one of a character and a face. (Method 1) an image quality improvement step of performing predetermined image quality improvement processing on the first image obtained by photographing to generate a second image; an evaluation step of acquiring an evaluation value indicating an evaluation result of the visibility of a specific object in each of the first image and the second image; a selection step of selecting either the first image or the second image based on the evaluation value; An information processing method comprising: (Program 1) 20. A program that causes a computer to function as the information processing device according to any one of configurations 1 to 19. [Explanation of symbols]

[0108] 100: Monitoring system, 101: Information processing device, 102: Image capturing device, 203: Image acquisition unit, 204: Image quality improvement unit, 205: Evaluation unit, 206: Selection unit, 207: Output unit

Claims

1. an image quality improvement unit that applies a predetermined image quality improvement process to a first image obtained by photographing to generate a second image; an evaluation means for acquiring an evaluation value indicating an evaluation result of the visibility of a specific object in each of the first image and the second image; a selection means for selecting either the first image or the second image based on the evaluation value; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the selection means selects the image having the higher evaluation value from the first image and the second image.

3. The information processing device according to claim 1, characterized in that the image quality improvement means applies the predetermined image quality improvement processing to the first image using a neural network model trained to improve the visibility of the specific object in the first image.

4. The information processing device according to claim 1, characterized in that the image quality improvement means has a plurality of neural network models trained to improve the visibility of the specific object in the first image, and performs image quality improvement processing on the first image using a neural network model selected from the plurality of neural network models based on the evaluation value of the first image.

5. an object detection means for detecting an object including the specific target from the first image or the second image; The information processing device according to claim 1, characterized in that the image quality improvement means applies image quality improvement processing to the first image using a neural network model that has been trained to improve the visibility of the specific target of the object detected by the object detection means.

6. The object detection means also detects the type of the object, The information processing device according to claim 5, characterized in that the image quality improvement means has a plurality of neural network models trained to improve the visibility of specific targets on objects of different types, and performs image quality improvement processing on the first image using a neural network model selected from the plurality of neural network models according to the type of object detected by the object detection means.

7. 6. The information processing apparatus according to claim 5, wherein the object detection means performs processing to detect the object when the evaluation value of the second image is lower than the evaluation value of the first image.

8. The information processing device described in claim 1, characterized in that the image quality improvement means applies the specified image quality improvement processing to the first image using a neural network model trained to improve the visibility of the area of ​​the specific object in the first image and a neural network model trained to improve the visibility of areas other than the specific object in the first image.

9. 2. The information processing apparatus according to claim 1, wherein the image quality improving means adjusts the intensity of the predetermined image quality improving process for the first image based on the evaluation value of the first image.

10. the image quality improvement means generates two or more second images by applying two or more different image quality improvement processes to the first image, the evaluation means acquires the evaluation values ​​from the first image and the two or more second images, 2. The information processing apparatus according to claim 1, wherein the selection means selects the image with the highest evaluation value from among the first image and the two or more second images.

11. the evaluation means acquires the evaluation value from each of the first image and the second image for each of a plurality of frames; 2. The information processing apparatus according to claim 1, wherein the selection means selects either the first image or the second image based on statistics of the acquired plurality of evaluation values.

12. The information processing device according to claim 11, characterized in that the selection means acquires at least one of the average, median, and maximum of the multiple evaluation values ​​of the first image and the second image as the statistical quantity, and selects the image with the larger statistical quantity from the first image and the second image.

13. The information processing device described in claim 1, characterized in that when the evaluation values ​​of the first image and the second image are less than a predetermined threshold, the image quality improvement means repeatedly applies different image quality improvement processes to the first image so that the evaluation value of the second image generated by the predetermined image quality improvement process becomes higher than the predetermined threshold.

14. 14. The information processing apparatus according to claim 13, wherein the image quality improving means uses the evaluation value of the first image as the predetermined threshold value.

15. 15. The information processing device according to claim 13, wherein the selection means selects the second image generated by repeating the image quality improvement process when the evaluation value of the second image becomes equal to or greater than the predetermined threshold value.

16. The information processing device described in claim 13 or 14, characterized in that when the evaluation value of the first image and the second image generated by repeating the image quality improvement process is less than a predetermined threshold and the number of times the image quality improvement process has been repeated reaches an upper limit, the selection means selects the image with the highest evaluation value among the first image and the multiple second images generated by repeating the image quality improvement process.

17. 2. The information processing apparatus according to claim 1, further comprising output means for recording the image selected by said selection means in recording means or displaying the image on display means.

18. The information processing device according to claim 17, characterized in that the output means generates an image of a user interface including a display screen for displaying the image selected by the selection means, a notification screen for notifying the user of the evaluation result by the evaluation means, and an execution instruction screen for instructing the image quality improvement means to execute image quality improvement processing, and outputs the generated image to the display means.

19. 2. The information processing apparatus according to claim 1, wherein the image quality improving means performs at least one of noise reduction processing, mist reduction processing, and super-resolution processing.

20. 2. The information processing apparatus according to claim 1, wherein the specific object is at least one of a character and a face.

21. an image quality improvement step of performing predetermined image quality improvement processing on a first image obtained by photographing to generate a second image; an evaluation step of acquiring an evaluation value indicating an evaluation result of the visibility of a specific object in each of the first image and the second image; a selection step of selecting either the first image or the second image based on the evaluation value; An information processing method comprising:

22. Computer, an image quality improvement unit that applies a predetermined image quality improvement process to a first image obtained by photographing to generate a second image; an evaluation means for acquiring an evaluation value indicating an evaluation result of the visibility of a specific object in each of the first image and the second image; a selection means for selecting either the first image or the second image based on the evaluation value; A program that causes the device to function as an information processing device having the above.

Images