Information processing device, information processing method, and program

Abstract

This enables efficient and effective image enhancement processing. [Solution] The information processing device includes an image acquisition means for acquiring an image that has been captured; an information acquisition means for acquiring information about the position of an object; a region determination means for determining a local region that is a part of the image and includes an object, based on the information acquired by the information acquisition means; a method determination means for determining a method for improving the image quality of the local region based on the determined local region and the remaining processing time for which high-quality image processing can be performed; and an execution means for performing high-quality image processing on the image according to the determined high-quality image processing method.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, machine learning has been applied to various high-quality image processing application programs. High-quality processing using machine learning includes, for example, noise removal processing, rain, snow, fog, and haze removal processing, and super-resolution processing. Such techniques for high-quality processing using machine learning are also applied to, for example, surveillance systems. For example, in order to determine whether a surveillance target is a suspicious object or not, it is a technique for enhancing the quality of an area including the surveillance target. This technique is used because when a monitor visually checks a video to confirm the surveillance target, the higher the image quality, the clearer the image of the surveillance target area becomes and it becomes easier to confirm. On the other hand, since such a surveillance system deals with video, it is necessary to maintain the frame rate. For high-quality processing performed under such conditions, both improvement in the accuracy of high-quality processing and suppression of latency are required.

[0003] Patent Document 1 discloses a technique for achieving high image quality with a low calculation amount by detecting a local area including features from a captured image and applying a machine learning model corresponding to the direction and illumination conditions in which the local area is captured only to the image of the local area. Patent Document 2 discloses a technique for performing high-quality processing on a local area including a target object on image data without missing the target object for which high-quality processing is desired by updating the local area of the high-quality processing target so as to include the target object based on the in-screen position of the target object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

[0005] The technologies described in Patent Documents 1 and 2 do not determine the method for enhancing the image quality of a local area by considering the remaining processing time within the time required to enhance and render a predetermined number of frames of the monitoring system, and there was room for improvement in the latency suppression method for this image quality enhancement process. In view of the above problems, the present invention aims to enable efficient and effective image quality enhancement processing for images. [Means for solving the problem]

[0006] The information processing apparatus according to the present invention is characterized by comprising: an image acquisition means for acquiring an image that has been captured; an information acquisition means for acquiring information relating to the position of an object; a region determination means for determining a local region that is a part of the image and includes the object, based on the information acquired by the information acquisition means; a method determination means for determining a method for improving the image quality of the local region based on the determined local region and the remaining processing time for which high-quality image processing can be performed; and an execution means for performing high-quality image processing on the image according to the determined high-quality image processing method. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to perform image quality enhancement processing on images efficiently and effectively. [Brief explanation of the drawing]

[0008] [Figure 1] This figure illustrates an example of a usage scenario for the information processing device according to Embodiment 1. [Figure 2] This figure shows an example of the functional configuration of a monitoring system. [Figure 3] This figure shows an example of the hardware configuration of an information processing device. [Figure 4] This is a flowchart that explains the overall flow of processing performed by an information processing device. [Figure 5]This is a flowchart showing the process for determining the local area to be enhanced in image quality according to Embodiment 1. [Figure 6] This figure illustrates the local region targeted for image quality enhancement according to Embodiment 1. [Figure 7] This is a flowchart showing the process for determining the image quality enhancement method according to Embodiment 1. [Figure 8] This diagram illustrates the process for determining the remaining processing time according to Embodiment 1. [Figure 9] This figure shows an example of a table used to determine the image quality enhancement method according to Embodiment 1. [Figure 10] This is a flowchart showing the process for determining the local area to be enhanced in image quality according to Embodiment 2. [Figure 11] This figure illustrates the local region targeted for image quality enhancement according to Embodiment 2. [Figure 12] This figure illustrates an example of a usage scenario for the information processing device according to Embodiment 3. [Figure 13] This figure illustrates the local region targeted for image quality enhancement according to Embodiment 3. [Figure 14] This is a flowchart showing the process for determining the local area to be enhanced in image quality according to Embodiment 4. [Figure 15] This figure shows an example of a display screen according to Embodiment 5. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not intended to limit the invention as defined in the claims, and not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] <Embodiment 1> First, an overview of the environment in which the information processing apparatus according to Embodiment 1 is used will be described. In the present embodiment, an example in which the information processing apparatus according to Embodiment 1 is applied in a situation where the area of a monitoring object in a video is enhanced to high definition while obtaining position information of the monitoring object in the world coordinate system (also referred to as the GPS coordinate system) used for GNSS and AIS will be described. Here, GNSS is the Global Navigation Satellite System, AIS is the Automatic Identification System, and GPS is the Global Positioning System.

[0011] The monitoring system in the present embodiment is used, for example, to determine suspicious vessels. In addition, the monitoring system performs image processing using machine learning on the captured video to enhance the monitoring video to high definition. In the monitoring system, high-definition processing with high high-definition accuracy is performed on the area in the video that includes vessels that are not suspicious vessels among the monitoring objects. The reason for targeting only vessels that are not suspicious vessels is that position information in the GPS coordinate system cannot be obtained for suspicious vessels, whereas for vessels that are not suspicious vessels, position information in the GPS coordinate system can be obtained by installing AIS, so the position in the video can be specified. By enhancing the definition of vessels that are not suspicious vessels for which the position has been specified so that they can be reliably confirmed, it becomes easier to determine other ship shadows as suspicious vessels.

[0012] Therefore, the monitoring system in this embodiment performs high-quality processing with high accuracy only on a local area including ships that are not suspicious ships, rather than on the entire image. Although high-quality processing with high accuracy has a high processing cost, by limiting the processing to a local area including ships that are not suspicious ships, the processing cost can be suppressed and high-quality processing with the required high accuracy can be executed. The method of high-quality processing with high accuracy that is only performed on this local area is determined in consideration of the remaining processing time based on the frame rate of the monitoring system, and the details will be described later. As a result, it is possible to achieve both an improvement in high-quality accuracy and a reduction in latency. Therefore, high-quality processing with high accuracy can be realized for the area including ships that are not suspicious ships while maintaining the frame rate, which ultimately leads to the determination of suspicious ships.

[0013] (Example of usage form) FIG. 1 is a schematic diagram for explaining an example of a usage scenario of an information processing apparatus according to Embodiment 1. FIG. 1 shows a state where the monitoring system according to this embodiment is monitoring an object. The monitoring system includes an information processing apparatus 101 and a camera 102 for capturing a video. The information processing apparatus 101 also communicates with a positioning system 106 that transmits information including position information in a GPS coordinate system such as AIS mounted on a ship. In FIG. 1, an imaging image 103 of a frame at a certain point captured by the camera 102 is shown. In this example, the monitoring objects are two ships (104 and 105), and in the imaging image 103, the ships (104 and 105) are captured at positions where the distance is far and it is difficult to recognize them in the image. Among these two ships, since ship 105 is not a suspicious ship, information including the position information of ship 105 is notified to the monitoring system through the positioning system 106. On the other hand, since ship 104 is a suspicious ship, the monitoring system cannot obtain information from the positioning system 106. In order to detect this suspicious ship 104, the monitoring system performs high-quality processing on the area including ship 105 that is not a suspicious ship in the captured image.

[0014] (Functional configuration of the monitoring system) Referring to Figure 2, the functional configuration of the monitoring system according to this embodiment will be described. Figure 2 is a block diagram showing an example of the functional configuration of the monitoring system 201 according to this embodiment. The monitoring system 201 has an information processing device 101 and an imaging unit 202. The imaging unit 202 corresponds to, for example, the camera 102 shown in Figure 1. The information processing device 101 has an image acquisition unit 203, an information acquisition unit 204, a region determination unit 205, a high-image-quality enhancement method determination unit 206, and a high-image-quality enhancement execution unit 207.

[0015] The image acquisition unit 203 acquires the input image. In this embodiment, the image acquisition unit 203 acquires the image captured by the imaging unit 202. The information acquisition unit 204 acquires information regarding the location of the monitored object (location information). In addition to the location information of the monitored object, the information acquisition unit 204 may also acquire information about the monitored object other than its location. In this embodiment, for example, the location information of a vessel that is not a suspicious vessel (for example, vessel 105 in the example shown in Figure 1) is acquired by the information acquisition unit 204.

[0016] The region determination unit 205 determines a local region containing the object to be monitored based on at least one of the captured image acquired by the image acquisition unit 203 and the information acquired by the information acquisition unit 204. This local region is a part of the captured image. In this embodiment, the region determination unit 205 determines the local region within the captured image based on the position information of the object to be monitored and the captured image. The image quality enhancement method determination unit 206 determines a method for enhancing the image quality of the local region based on the local region determined by the region determination unit 205, in order to satisfy the remaining processing time. Details regarding the determination of the image quality enhancement method will be described later.

[0017] The image enhancement execution unit 207 performs image enhancement processing on the captured image acquired by the image acquisition unit 203 according to the image enhancement method determined by the image enhancement method determination unit 206. In the image enhancement processing in this embodiment, at least one or more partial images of a local region within the captured image to be enhanced are used as input to the image enhancement model, and image enhancement is applied to that local region. In this embodiment, at least one or more partial images are partial images of multiple frames of a local region within the captured image to be enhanced.

[0018] In the example shown in Figure 2, one imaging unit 202 is shown, but the number of imaging units 202 in the monitoring system 201 is not limited to this, and it may have multiple imaging units 202. Also, the information processing device 101 may have other functional units depending on the processing to be performed.

[0019] (Hardware configuration of information processing equipment) Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 101 according to this embodiment. The information processing device 101 according to this embodiment includes a CPU 301, ROM 302, RAM 303, external memory 304, input unit 305, display unit 306, I / O 307, and bus 308. The CPU 301, ROM 302, RAM 303, external memory 304, input unit 305, display unit 306, and I / O 307 are communicated together via the bus 308.

[0020] The CPU (Central Processing Unit) 301 controls various devices connected to the bus 308 and executes programs processed by the information processing unit 101. The ROM (Read Only Memory) 302 stores the BIOS (Basic Input Output System) program and the boot program. The RAM (Random Access Memory) 303 is used as the main memory of the CPU 301. The external memory 304 stores programs and various data processed by the information processing unit 101.

[0021] The input unit 305 is a keyboard or mouse and performs processing related to information input. The display unit 306 outputs the calculation results of the information processing device 101 to the display device according to instructions from the CPU 301. The display device can be of any type, such as a liquid crystal display, projector, or LED (Light Emitting Diode) indicator. The I / O (Input / Output) 307 is an interface for communication with other devices. For example, the camera 102 and the positioning system 106 are connected to the I / O 307. The bus 308 is a bus that connects the various parts of the information processing device 101 so that they can communicate with each other.

[0022] The operation of the information processing device 101 according to this embodiment will be described below with reference to Figures 4 to 9. Figure 4 is a flowchart illustrating the overall processing flow performed by the information processing device 101 according to this embodiment. The processing shown in the flowchart in Figure 4 starts, for example, when monitoring by the monitoring system 201 begins, and steps S402 to S406 are repeatedly executed at predetermined intervals. For example, steps S402 to S406 are executed for each frame of the monitoring video. In the following explanation, the case where the captured image is the captured image 103 shown in Figure 1 will be shown as appropriate.

[0023] In step S401, the information processing device 101 initializes the information it handles. The information processing device 101 loads, for example, the internal and external parameters of the camera 102 (imaging unit 202), the frame rate of the surveillance video, the base method for the image enhancement processing described later, and a table used when determining the image enhancement model described later, from the external memory 304. The information processing device 101 also loads at least one image enhancement model from the external memory 304. Each image enhancement model has a different image enhancement performance (accuracy), and the higher the performance of the model, the longer the processing time required for image enhancement. In addition, the number of input frames can be adjusted for each image enhancement model; the higher the number of input frames, the better the image enhancement performance, but the longer the processing time.

[0024] In step S402, the image acquisition unit 203 acquires the captured image taken by the camera 102 (imaging unit 202) that images the object to be monitored. Through the processing in step S402, the image acquisition unit 203 acquires an image of the object to be monitored (such as ships 104 and 105, as shown in Figure 1).

[0025] In step S403, the information acquisition unit 204 acquires location information of the object to be monitored. In this embodiment, the information acquisition unit 204 acquires the GPS coordinate system location information of the vessel 105, which is not a suspicious vessel, from the positioning system 106 (AIS). In addition to the location information of the object to be monitored, the information acquisition unit 204 may also acquire other information about the object to be monitored if necessary.

[0026] In step S404, the region determination unit 205 determines the local region to be enhanced in image quality, which includes the object being monitored, based on at least one of the captured image acquired by the image acquisition unit 203 and the information acquired by the information acquisition unit 204. In this embodiment, the region determination unit 205 determines the local region to be enhanced in image quality within the captured image based on the GPS coordinate system position information of the object being monitored acquired in step S403 and the captured image acquired in step S402. Details of the process for determining the local region to be enhanced in image quality will be described later with reference to Figure 5.

[0027] In step S405, the image quality enhancement method determination unit 206 determines a method for enhancing the image quality of a local region to satisfy the remaining processing time, based on the local region to be enhanced, as determined by the region determination unit 205. Details of the process for determining the image quality enhancement method will be described later with reference to Figure 7.

[0028] In step S406, the image enhancement execution unit 207 performs image enhancement processing on the captured image acquired by the image acquisition unit 203 according to the image enhancement method determined by the image enhancement method determination unit 206. For local regions within the captured image that are to be enhanced, as determined in step S404, the image enhancement execution unit 207 performs image enhancement processing using the image enhancement method determined in step S405.

[0029] The process for determining the local area to be enhanced in this embodiment will be described below with reference to Figures 5 and 6. Figure 5 is a flowchart showing the flow of the process for determining the local area to be enhanced in step S404 of Figure 4.

[0030] In step S501, the area determination unit 205 acquires the origin of the GPS coordinates of the object to be monitored. In this embodiment, the area determination unit 205 acquires the origin information of the coordinates of a vessel 105 that is not a suspicious vessel and is the object to be monitored. As mentioned above, the coordinates of this vessel 105 are acquired from the positioning system 106, so the origin of the coordinates of the vessel 105 is the origin of the coordinate system measured by the positioning system 106. Therefore, the origin information of the coordinates of the vessel 105, which is the object to be monitored, can be acquired in advance from the positioning system 106.

[0031] In step S502, the region determination unit 205 acquires the origin of the camera 102 (imaging unit 202) in the world coordinate system. In this embodiment, for example, the origin of the camera 102 in the world coordinate system is calculated using the intrinsic and extrinsic parameters of the camera 102. Since this calculation method is a general technique for camera calibration in 3D geometric measurement, the specific calculation method is omitted. Furthermore, the origin of the camera 102 in the world coordinate system calculated in this embodiment is the position of the camera 102.

[0032] In step S503, the region determination unit 205 aligns the origin of the coordinate system of the camera 102 with that of the object being monitored. Specifically, the region determination unit 205 calculates a transformation formula to convert the origin of the GPS coordinate system of the object being monitored to the coordinate value of the camera 102 in the world coordinate system. The region determination unit 205 calculates the transformation formula based on the origin of the GPS coordinates of the object being monitored, acquired in step S501, and the origin of the camera 102 in the world coordinate system, acquired in step S502. The transformation is performed using the physical positional relationship between the camera 102 and the origin of the GPS coordinate system, but since this is a general 3D geometric measurement technique, a detailed explanation of the calculation is omitted.

[0033] In step S504, the region determination unit 205 converts the GPS coordinate system position information of the object to be monitored into pixel coordinates of the camera 102. Specifically, the region determination unit 205 first applies the conversion formula obtained in step S503 to the position information (coordinates) of the ship 105, which is the object to be monitored, to convert the GPS coordinate system coordinates of the ship 105 into the world coordinate system of the camera 102. Then, the region determination unit 205 converts the obtained world coordinate system coordinates of the ship 105 of the camera 102 into the pixel coordinate system of the camera 102, again using the internal and external parameters of the camera 102. In this way, the region determination unit 205 obtains the pixel coordinates of the camera 102 that indicate the position of the object to be monitored.

[0034] In step S505, the region determination unit 205 determines that the object to be monitored is located in a position that is difficult to recognize in the captured image. In this embodiment, the region determination unit 205 determines, for example, that an object to be monitored is located in a position that is difficult to recognize if the physical distance between the imaging position of the camera 102 and the object to be monitored is greater than or equal to a predetermined threshold. The physical distance between the imaging position of the camera 102 and the object to be monitored can be obtained, for example, based on the position information of the vessel 105 acquired from the positioning system 106. For example, in the example of the captured image 103 shown in Figure 1, the region determination unit 205 determines that the vessel 105 is located in a position that is difficult to recognize.

[0035] In step S506, the region determination unit 205 determines a local region within the captured image to be enhanced in image quality, based on the object to be monitored which is located in a difficult-to-recognize position as determined in step S505 and the pixel coordinates of the object to be monitored in camera 102 as obtained in step S504. For example, the region determination unit 205 determines a local region which is a part of the captured image that includes the pixel coordinates of the object to be monitored which is located in a difficult-to-recognize position, is greater than or equal to the size of the object to be monitored in the captured image, and whose size is a constant multiple of the input image size of the image enhancement model.

[0036] Figure 6 illustrates the local area to be enhanced in image quality, as determined by the processing shown in Figures 4 and 5. Here, the captured image 601 before enhancement is divided into multiple rectangular areas by an arbitrary number of units in both the vertical and horizontal directions. The rectangular area 602 containing the ship 105 is set as the local area to be enhanced in image quality. The size of the rectangular area 602 determined as the local area to be enhanced in image quality is a constant multiple of the input image size of the enhancement model.

[0037] The process for determining the image quality enhancement method in this embodiment will be described below with reference to Figures 7 to 9. Figure 7 is a flowchart showing the flow of the process for determining the image quality enhancement method, which is performed in step S405 of Figure 4.

[0038] In step S701, the image quality enhancement method determination unit 206 determines the base method for the image quality enhancement process to be performed on the local region to be enhanced. In this embodiment, the base method for the image quality enhancement process is determined to be either "a method of applying a model with high image quality accuracy only to the local region" or "a method of increasing the number of input frames to the image quality enhancement model only to the local region". Which of these methods will be used as the base method for the image quality enhancement process is predetermined, and information that one of these methods is used is obtained during the initialization process in step S401 in Figure 4. In this embodiment, it will be explained assuming that "a method of applying a model with high image quality accuracy only to the local region" is obtained as the base method.

[0039] In step S702, the image quality enhancement method determination unit 206 determines the remaining processing time that can be used to perform image quality enhancement on the local area to be enhanced. The processing in step S702 will be explained with reference to Figure 8. The image quality enhancement method determination unit 206 determines the total processing time 801. This total processing time 801 is determined, for example, based on the frame rate of the surveillance video acquired in the initialization process in step S401 of Figure 4. Here, as an example, the time of one frame in the frame rate of the surveillance video is taken as the total processing time 801. The image quality enhancement method determination unit 206 also determines processing time A802 and processing time B803. Processing time A802 is the time required for image quality enhancement processing performed on areas other than the local area to be enhanced, and processing time B803 is the time required to generate the image in addition to image quality enhancement processing such as development. Then, the image quality enhancement method determination unit 206 determines the remaining time obtained by subtracting processing time A 802 and processing time B 803 from the total processing time 801 determined in this way as the remaining processing time 804.

[0040] Returning to Figure 7, in step S703, the image quality enhancement method determination unit 206 determines the detailed image quality enhancement method so that the time required for the image quality enhancement processing to be performed on the local area to be enhanced falls within the remaining processing time determined in step S702. If the base method determined in step S701 is a method of applying a model, the image quality enhancement method determination unit 206 selects the method of "selecting and applying the model with the largest computational amount from among the image quality enhancement models that can perform image quality enhancement processing on the local area within the remaining processing time." The method of determining the model here will be described later with reference to Figure 9. Also, if the base method determined in step S701 is a method of increasing the number of input frames, the image quality enhancement method determination unit 206 selects the method of "determining the number of input frames for the image quality enhancement model as the quotient of the remaining processing time and the processing time per frame." Note that the remaining processing time is the dividend and the processing time per frame is the divisor. In this embodiment, since the base method is a method of applying a model, the image quality enhancement method determination unit 206 determines that the detailed method of image quality enhancement is "a method of selecting and applying the model with the largest computational load among the image quality enhancement models that can perform image quality enhancement processing on a local area within the remaining processing time."

[0041] Figure 9 shows an example of a table used when determining the image quality enhancement model in step S703 of Figure 7. The table consists of the items "Model Name," "Computational Amount," and "Processing Time," and lists the model name, computational amount, and processing time for each image quality enhancement model. The computational amount is in units of [MFlops], and a larger computational amount indicates higher image quality enhancement accuracy. The processing time represents the processing time per local region, and is in units of [ms].

[0042] Here, for example, let's assume that the remaining processing time determined in step S702 of Figure 7 is 4 ms. Also, let's assume that there are two local areas to be enhanced in image quality. In this case, since the remaining processing time that allows for image quality enhancement processing to be performed on two local areas is 4 ms, the image quality enhancement processing time allocated to one local area is 2 ms. According to the table shown in Figure 9, the image quality enhancement model with the maximum computational load and a processing time of 2 ms or less per local area is determined to be "Model B". In this way, the method for enhancing the image quality of the local areas is finally determined as "a method of selecting and applying Model B, which has the maximum computational load among the image quality enhancement models that allow for image quality enhancement processing of the local areas within the remaining processing time".

[0043] According to this embodiment, location information in the GPS coordinate system can be obtained, and a portion of the captured image containing the monitored object, which is located in a difficult-to-recognize position, is determined as a local area for image quality enhancement. High-resolution image enhancement processing with high accuracy is then performed on that local area. This makes it possible to efficiently and effectively perform image quality enhancement processing for monitored objects that are not suspicious, whose location in the GPS coordinate system can be obtained, and which are located in a difficult-to-recognize position.

[0044] In the embodiment described above, the region determination unit 205 determines a surveillance target that is difficult to recognize in the captured image based on the condition that the physical distance between the camera 102 and the surveillance target is greater than or equal to a predetermined threshold. However, it is not limited to this condition. The determination may be based on other conditions as long as the condition for selecting a surveillance target that is difficult to recognize in the captured image is one that is difficult to recognize. For example, the region determination unit 205 may determine a surveillance target whose size in the captured image is less than or equal to a threshold as a surveillance target that is difficult to recognize. Alternatively, for example, the region determination unit 205 may determine a surveillance target that has an obstruction within a threshold distance from it as a surveillance target that is difficult to recognize. This makes it possible to select surveillance targets that require high image quality regardless of the condition of being difficult to recognize.

[0045] Furthermore, the remaining processing time acquired by the image quality enhancement method determination unit 206 is determined by using the time of one frame at the frame rate of the surveillance video as the total processing time. However, this does not have to be the time of one frame, as long as there is time to perform the image quality enhancement processing. For example, the image quality enhancement method determination unit 206 may determine the remaining processing time by setting the total processing time 801 to the time of multiple frames. Also, the image quality enhancement method determined by the image quality enhancement method determination unit 206 is not limited to the method described above. If the image quality enhancement processing for the local area to be enhanced can be performed within the remaining processing time, another method may be added. For example, the frame rate may be lowered and the image quality enhancement processing described in this embodiment may be performed on the local area to be enhanced. In this case, the lower limit of the frame rate to be adjusted may be predetermined or determined by user settings. This makes it possible to perform image quality enhancement processing for the local area within the remaining processing time, regardless of the method for determining the remaining processing time.

[0046] Furthermore, if the frame rate cannot be adjusted and enhancing the image quality of a local area would exceed the remaining processing time, the image processing may be performed on the captured image using other methods. For example, image quality enhancement processing may be appropriately skipped for areas other than the local area targeted for enhancement. Here, the areas other than the local area targeted for enhancement that are skipped may be, for example, areas that are far from the position of the monitored object from which the GPS coordinate system position information has been received. Other areas other than the local area targeted for enhancement that are skipped may be areas that are in a different scene from the local area containing the monitored object. For example, if the monitored object is a ship on the water, areas such as mountains, fields, or forests that appear in the captured image may be areas where image quality enhancement processing is skipped. This makes it possible to perform image quality enhancement processing on the local area containing the monitored object even when the frame rate cannot be adjusted.

[0047] Furthermore, although the above-described embodiment does not consider position measurement errors of GNSS, AIS, or GPS (hereinafter referred to as GPS errors), the information processing device 101 may be provided with a function unit for considering these GPS errors. For example, the information processing device 101 may be provided with an error occurrence status determination unit. The error occurrence status determination unit determines, for example, whether the monitoring target is indoors, or whether there is an obstruction nearby, or whether there is bad weather, which are factors that cause GPS errors. The determination of whether the monitoring target is indoors or whether there is an obstruction nearby can be performed, for example, by recognizing the scene or object in the captured image. Also, the determination of bad weather can be performed, for example, by determining whether the contrast value in the captured image is below a threshold. Accordingly, the area determination unit 205 may determine the local area to be improved in image quality while taking GPS errors into consideration. For example, if the error occurrence status determination unit determines that a situation in which GPS errors occur is present, the local area size may be adjusted to include the variation due to GPS errors, according to the physical distance between the imaging position and the monitoring target. The closer the physical distance between the imaging position and the object being monitored, the greater the error in pixel position within the captured image. Therefore, the local area size may be adjusted to be larger. This makes it possible to determine the local area containing the object being monitored without being affected by GPS errors.

[0048] In the embodiment described above, the base method for image enhancement processing was predetermined, but other methods may be used as long as the base method for image enhancement processing is determined at the time the image enhancement method is determined. For example, the user may select and determine the base method for image enhancement processing before starting monitoring. This makes it possible to determine the image enhancement method for the local area to be enhanced, regardless of the form of the method for determining the base method for image enhancement processing.

[0049] <Embodiment 2> Embodiment 1 describes an example in which high-quality image enhancement processing is applied to local areas containing surveillance objects within an captured image, identified based on location information in a GPS coordinate system, in order to satisfy the constraint of remaining processing time. Embodiment 2 describes an example in which high-quality image enhancement processing is applied in order to satisfy the constraint of remaining processing time when there are many surveillance objects for which location information in a GPS coordinate system has been received relative to the remaining processing time. Specifically, the local areas to be enhanced are switched at regular intervals in the time series, or only local areas containing high-priority surveillance objects are enhanced, or local areas are set to include multiple surveillance objects together. In this way, it is possible to perform high-quality image enhancement processing while satisfying the constraint of remaining processing time, even when there are many surveillance objects in the captured image.

[0050] (Examples of usage patterns) In Embodiment 2, the objects of surveillance are three vessels, one more than in Embodiment 1. Of these, one vessel (vessel 104 in the example shown in Figure 11, described later) is a suspicious vessel, while the remaining two vessels (vessels 105 and 1104 in the example shown in Figure 11, described later) are not suspicious vessels. Therefore, information including positional information in the GPS coordinate system can be obtained for either of the two vessels that are not suspicious vessels through the positioning system 106. Other usage scenarios are the same as in the example of Embodiment 1, so the explanation will be omitted.

[0051] The functional configuration of the monitoring system according to Embodiment 2 is the same as that of the monitoring system shown in Figure 2 described in Embodiment 1, so its explanation is omitted. Furthermore, the hardware configuration of the information processing device 101 according to Embodiment 2 is the same as that of the information processing device shown in Figure 3 described in Embodiment 1, so its explanation is omitted.

[0052] The operation of the information processing device 101 according to Embodiment 2 will now be described. The overall flow of processing performed by the information processing device 101 according to Embodiment 2 is the same as the processing in the flowchart shown in Figure 4 described in Embodiment 1, so the explanation will be omitted.

[0053] The process for determining the local area to be enhanced in Embodiment 2 will be described below with reference to Figures 10 and 11. Figure 10 is a flowchart showing the flow of the process for determining the local area to be enhanced in step S404 of Figure 4. The processes in steps S1001 to S1005 of Figure 10 are the same as the processes in steps S501 to S505 of Figure 5 in Embodiment 1, so their explanation will be omitted.

[0054] In step S1006, the region determination unit 205 extracts local regions within the captured image that are to be enhanced in image quality, based on the location of the object to be monitored, which is difficult to recognize, as determined in step S1005, and the pixel coordinates of the object to be monitored, as determined in step S1004, as measured by the camera 102. The region determination unit 205 then selects and determines from the extracted local regions to be enhanced in image quality, based on the remaining processing time.

[0055] First, the region determination unit 205, similar to Embodiment 1, determines a local region that includes the pixel coordinates of the object being monitored, which is located in a position that is difficult to recognize, is greater than or equal to the size of the object being monitored in the captured image, and whose size is a constant multiple of the input image size of the image enhancement model. Next, the region determination unit 205 determines the number of local regions that can be processed within the remaining processing time. Here, as an example, it is determined that only two local regions can be enhanced within the remaining processing time. Next, the region determination unit 205 divides the local regions according to the number of local regions that can be processed within the determined remaining processing time. For example, the region determination unit 205 divides local regions that are located close together into the same group. Next, the region determination unit 205 determines the period for switching the local regions to be enhanced. Here, as an example, it is determined that 5 seconds is the switching period. Finally, the region determination unit 205 assigns the divided local regions to each switching period and selects and determines the local region corresponding to that time.

[0056] Figure 11 illustrates the local areas to be enhanced in image quality, as determined by the processes shown in Figures 4 and 10. Figure 11(a) shows the total local areas determined by the process for determining the local areas to be enhanced in image quality. Here, the captured image 1101 before enhancement is divided into multiple rectangular areas by an arbitrary number of units in both the vertical and horizontal directions. Rectangular areas 1102 and 1103, which contain vessels 105 and 1104 that are not suspicious vessels, respectively, are set as the local areas to be enhanced in image quality.

[0057] Figure 11(b) is an example of an explanatory diagram showing the allocation of local regions for each switching period. In this diagram, local region 1102 is allocated as the local region to be processed for high-quality image processing during the first half of time A (5 seconds in this embodiment), and local region 1103 is allocated as the local region to be processed for high-quality image processing during the second half of time B (5 seconds in this embodiment). Also, as shown in Figure 11(b), if the current time is within time A, the region determination unit 205 outputs local region 1102 as the final local region to be processed for high-quality image processing.

[0058] The process for determining the image quality enhancement method in Embodiment 2 is the same as the process shown in Figure 7 described in Embodiment 1, so its explanation will be omitted.

[0059] According to Embodiment 2, even if there are many non-suspicious objects that can be located in a GPS coordinate system and are difficult to recognize, within the remaining processing time, high-resolution processing to enhance the image quality of the monitored objects can be performed efficiently and effectively.

[0060] In the explanation above, the method for selecting local areas to be enhanced in image quality is to switch the local areas to be enhanced in image quality at predetermined time intervals. However, other methods may be used as long as a local area that can be processed within the remaining processing time can be selected. For example, the area determination unit 205 may select a local area containing a high-priority surveillance target from among local areas containing surveillance targets that have been determined to be in a difficult-to-recognize position. Alternatively, the area determination unit 205 may select a local area to enhance the image quality of surveillance targets that are within a threshold distance in the captured image. A local area set to high priority may be a local area where the physical distance between the imaging position and the surveillance target is greater than or equal to the threshold. A local area set to high priority may also be a local area where the degree of degradation of the area requiring image quality enhancement is greater than or equal to the threshold, or a local area of ​​high importance determined based on information about the surveillance target. Information about the surveillance target may include, for example, information that a vessel is carrying dangerous goods. In this case, the information that the vessel being monitored is carrying dangerous goods is acquired by the information acquisition unit 204. Furthermore, the local areas that are given high priority may be local areas where the physical depth distance between the object being monitored and the imaging position is within a range pre-set by the user, or local areas where the position of the object being monitored within the captured image is within a range pre-set by the user. In addition, the local areas that are given high priority may be those where the direction of movement of the object being monitored is in a direction pre-set by the user. This makes it possible to appropriately determine the local areas to be upgraded to improve image quality, even when there are many objects being monitored relative to the remaining processing time, regardless of the method of selecting the local areas.

[0061] In the example mentioned above, the local regions were divided into two groups of two, with two being the number of local regions that could be processed within the remaining processing time. However, this can be changed to any other number of local regions as long as the image enhancement processing can be performed within the remaining processing time. This makes it possible to select local regions that can be processed within the remaining processing time, regardless of the number of local regions that are divided. Also, in the example mentioned above, the switching period for switching local regions to be enhanced was set to every 5 seconds, but this can be changed to any other time. This makes it possible to switch local regions to be enhanced regardless of a specific switching period.

[0062] <Embodiment 3> Embodiments 1 and 2 described an example of applying high-resolution image enhancement processing with high image quality accuracy, using the case where the monitored object is a ship as an example. Embodiment 3 describes an example of applying high-resolution image enhancement processing with high image quality accuracy to a monitored object other than a ship. The monitored object in Embodiment 3 is, for example, a person whose GPS location can be obtained using a smartphone, or a vehicle such as a car or airplane whose location can be obtained using a car navigation system, or an autonomous mobile robot that is GPS-linked. In Embodiment 3, high-resolution image enhancement processing with high image quality accuracy can be performed on the image while satisfying the remaining processing time, regardless of the form of the monitored object.

[0063] (Examples of usage patterns) Figure 12 is a schematic diagram illustrating an example of a usage scenario for the information processing device according to Embodiment 3. Figure 12 shows the monitoring system according to this embodiment monitoring an object. In Figure 12, the same reference numerals are used for components that are the same as those shown in Figure 1, and their descriptions are omitted. In Figure 12, the captured image 1201 taken by camera 102 shows two people (1202 and 1203), who are the objects being monitored in this embodiment. Of these, person 1203 is holding a smartphone, and their GPS location can be obtained. On the other hand, person 1202 is a suspicious person, and their GPS location cannot be obtained.

[0064] The functional configuration of the monitoring system according to Embodiment 3 is the same as that of the monitoring system shown in Figure 2 described in Embodiment 1, so its explanation is omitted. Furthermore, the hardware configuration of the information processing device 101 according to Embodiment 3 is the same as that of the information processing device shown in Figure 3 described in Embodiment 1, so its explanation is omitted.

[0065] The overall processing flow executed by the information processing device 101 according to Embodiment 3 is the same as the processing in the flowchart shown in Figure 4 described in Embodiment 1. Furthermore, the process for determining the local area to be enhanced in high image quality in Embodiment 3 is the same as the process shown in Figure 5 described in Embodiment 1. Also, the process for determining the high-image-quality enhancement method in Embodiment 3 is the same as the process shown in Figure 7 described in Embodiment 1. Therefore, these explanations are omitted.

[0066] Figure 13 is a diagram illustrating the local area to be enhanced in image quality, as determined by the processing shown in Figures 4 and 5 in Embodiment 3. Here, the captured image 1301 before enhancement is divided into multiple rectangular areas by an arbitrary number of units in both the vertical and horizontal directions, and the rectangular area 1302 containing the person 1203 is set as the local area to be enhanced in image quality.

[0067] In the example mentioned above, the object being monitored was a person, but other objects may be monitored as long as their location information in the GPS coordinate system can be obtained through the positioning system. For example, vehicles such as cars or airplanes whose location can be obtained by a car navigation system, or autonomous mobile robots that are GPS-linked may be monitored. This makes it possible to perform high-resolution processing of local areas within the remaining processing time, regardless of the object being monitored.

[0068] According to Embodiment 3, even for surveillance targets such as people, vehicles, and autonomous mobile robots whose positions can be obtained in the GPS coordinate system and which are in positions that are difficult to recognize, high-resolution processing can be efficiently and effectively performed to enhance the image quality of surveillance targets that are not suspicious.

[0069] <Embodiment 4> In Embodiments 1 to 3 described above, the local area corresponding to the monitored object in the captured image is determined based on the GPS coordinate position information of the monitored object obtained from the positioning system. Embodiment 4 describes an example in which the local area corresponding to the monitored object is determined from the captured image without using the GPS coordinate position information of the monitored object, and high-resolution image enhancement processing with high image quality accuracy is applied to satisfy the constraint of remaining processing time. This makes it possible to perform high-resolution image enhancement processing with high image quality accuracy on the image while satisfying the remaining processing time, even when not linked with the positioning system.

[0070] (Examples of usage patterns) The monitoring system according to Embodiment 4 is not linked to the positioning system 106. Other usage scenarios are the same as those shown in the embodiment in Figure 1, so their explanation is omitted.

[0071] The functional configuration of the monitoring system according to Embodiment 4 is the same as that of the monitoring system shown in Figure 2 described in Embodiment 1. However, in Embodiment 4, the region determination unit 205 determines a local region including the object to be monitored based on the captured image acquired by the image acquisition unit 203. In this embodiment, the region determination unit 205 acquires the position of the object by image recognition on the captured image and determines the local region within the captured image that is to be enhanced in image quality. The other functional units, namely the image acquisition unit 203, the information acquisition unit 204, the image quality enhancement method determination unit 206, and the image quality enhancement execution unit 207, are the same as in Embodiment 1, so their description is omitted. Furthermore, the hardware configuration of the information processing device 101 according to Embodiment 4 is the same as that of the information processing device shown in Figure 3 described in Embodiment 1, so their description is omitted.

[0072] The operation of the information processing device 101 according to Embodiment 4 will now be described. The overall flow of processing performed by the information processing device 101 according to Embodiment 4 is the same as the processing shown in the flowchart in Figure 4 described in Embodiment 1, so the explanation will be omitted.

[0073] The process for determining the local area to be enhanced in Embodiment 4 will be described below with reference to Figure 14. Figure 14 is a flowchart showing the flow of the process for determining the local area to be enhanced in step S404 of Figure 4.

[0074] In step S1401, the region determination unit 205 determines the coordinates of the object to be monitored, which is located in a difficult-to-recognize position, by detecting the object in the captured image. For example, the region determination unit 205 outputs the pixel coordinates of the object to be monitored, which is located in a difficult-to-recognize position, by performing object detection in the captured image using pattern matching image processing. Note that object detection is not limited to pattern matching; other object detection methods may also be applied.

[0075] In step S1402, the region determination unit 205 determines a local region within the captured image to be enhanced in image quality, based on the position information (pixel coordinates) of the object being monitored, which is located in a difficult-to-recognize position, as determined in step S1401. For example, the region determination unit 205 determines a local region from a portion of the captured image such that it includes the pixel coordinates of the object being monitored, is greater than or equal to the size of the object being monitored within the captured image, and its size is a constant multiple of the input image size of the image enhancement model. In this way, the local region to be enhanced in image quality is determined, for example, as shown in the example in Figure 6.

[0076] The process for determining the image quality enhancement method in Embodiment 4 is the same as the process shown in Figure 7 described in Embodiment 1, so its explanation will be omitted.

[0077] According to Embodiment 4, the position of the object to be monitored is detected from the captured image, and a portion of the captured image containing the object to be monitored, which is located in a position that is difficult to recognize, is determined as a local area for image quality enhancement. High-quality image enhancement processing with high image quality enhancement accuracy is then performed on that local area. This makes it possible to efficiently and effectively perform image quality enhancement processing to enhance the image quality of the object to be monitored, even if it is located in a position that is difficult to recognize.

[0078] In the explanation above, it was stated that the coordinates of the monitored object are determined by object detection within the captured image. However, any detection process other than normal detection is acceptable as long as the monitored object can be detected within the captured image; for example, the result of detection by anomaly detection is also acceptable. This makes it possible to perform image quality enhancement processing efficiently and effectively, regardless of the form of object detection.

[0079] <Embodiment 5> The following describes how to notify users of a monitoring system of the implementation details of control that applies high-resolution image processing with high accuracy while suppressing latency, only to local areas including the monitored object, according to Embodiments 1 to 4.

[0080] In the monitoring system according to Embodiment 5, the information processing device 101 has a notification unit in addition to the configuration shown in Figure 2. Note that the configuration other than the notification unit is the same as in the embodiments described above, so its explanation is omitted. Furthermore, the hardware configuration of the information processing device 101 according to Embodiment 5 is the same as the configuration of the information processing device shown in Figure 3 described in Embodiment 1, so its explanation is omitted.

[0081] The notification unit of the information processing device 101 notifies the user of at least one processing result from the image acquisition unit 203, information acquisition unit 204, region determination unit 205, image quality enhancement method determination unit 206, image quality enhancement execution unit 207, and error occurrence status determination unit of the information processing device 101.

[0082] Figure 15 is a diagram showing an example of a display screen showing the results when high-quality image enhancement processing with high accuracy is applied only to a local area of ​​the target for image enhancement, in order to satisfy the remaining processing time, as realized by the notification unit according to Embodiment 5. Figure 15 shows an example in the usage mode described in Embodiment 2, where high-quality image enhancement processing with high accuracy is applied only to a local area of ​​the captured image that includes a monitored object identified based on location information in the GPS coordinate system, in order to satisfy the remaining processing time.

[0083] The display screen 1501 includes a unique information display unit 1502, a high-resolution method display unit 1503, a monitored target information display unit 1504, a high-resolution image display unit 1505, a local image display unit 1506, and a local area selection display unit 1507.

[0084] The unique information display unit 1502 displays, for example, the monitored object. The image quality enhancement method display unit 1503 displays the image quality enhancement method determined by the image quality enhancement method determination unit 206. In this embodiment, it displays which of the following image quality enhancement methods was determined: "a method that applies a model with high image quality accuracy only to a local area" or "a method that increases the number of input frames to the image quality enhancement model only to a local area".

[0085] The monitoring target information display unit 1504 displays information about the monitoring targets acquired from the positioning system. In this embodiment, it displays the number of vessels for which information has been acquired from AIS. The high-resolution image display unit 1505 displays a high-resolution image obtained by applying the high-resolution method described in each embodiment to the image captured by the camera 102. In this embodiment, the information about each monitoring target acquired from the positioning system, as well as the high-resolution area, are superimposed and displayed near the location of the monitoring target.

[0086] The local image display unit 1506 displays an image of the local area containing the monitored object, which has been processed to enhance image quality. The local area selection display unit 1507 displays the location of the local area to be enhanced in the captured image, as explained with reference to Figure 11. In this embodiment, information on the local area that has undergone high-quality image enhancement processing at the current time is also displayed.

[0087] According to this embodiment, it is possible to notify users of the monitoring system of the content in which high-resolution processing with high image quality accuracy has been applied to only a local area including the object being monitored, while suppressing latency.

[0088] The content notified by the unique information display unit 1502, the image enhancement method display unit 1503, the monitored object information display unit 1504, the enhanced image display unit 1505, the local image display unit 1506, and the local area selection display unit 1507 was explained using the usage mode described in Embodiment 2 as an example. However, depending on the usage mode, it is possible to switch whether or not to notify information (results) for each display unit. This makes it possible to notify the user of the monitoring system of the content in which high-resolution processing with high image enhancement accuracy has been applied to a local area including the monitored object, while suppressing latency, regardless of the usage mode of the information processing device 101.

[0089] Furthermore, in the example mentioned above, the control content is notified in a GUI format, but as long as the control content can be notified to the users of the monitoring system, other notification formats may be used, except for the notification of high-resolution images. For example, the content could be notified in text format. This makes it possible to notify content with high-resolution processing applied to a local area including the monitored object, while suppressing latency, regardless of whether the notification unit supports GUI display or GUI display hardware.

[0090] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0091] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0092] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) Image acquisition means for acquiring captured images, Information acquisition means for acquiring information about the position of an object, A region determination means that determines a local region within the image that includes the object, based on the information acquired by the information acquisition means, A method determination means for determining a method for improving the image quality of the local region based on the determined local region and the remaining processing time for performing image quality improvement processing, An information processing apparatus characterized by having an execution means for performing image enhancement processing on the image according to the determined image enhancement method. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the method determination means determines the image enhancement method such that the time required for image enhancement processing performed on the local region falls within the remaining processing time. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the region determination means determines the region including the object from which information has been acquired by the information acquisition means as the local region. (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the region determination means determines the region including the object whose distance from the image acquisition position is greater than or equal to a threshold as the local region. (Composition 5) The information processing apparatus according to any one of configurations 1 to 4, characterized in that the region determination means determines the local region to be a region that includes at least one of the objects whose size in the image is less than or equal to a threshold, and the object whose distance from an occluding object is less than or equal to a threshold. (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that the region determination means selects and determines the local region to be enhanced in image quality from among the regions including the object from which information has been acquired by the information acquisition means. (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, characterized in that the region determination means determines the local region based on the positional information of the object in the world coordinate system. (Composition 8) The execution means performs the image enhancement process on the local region using the image enhancement model, The information processing apparatus according to any one of configurations 1 to 7, characterized in that the method determination means determines the image quality enhancement model with the largest computational load among the image quality enhancement models capable of performing the image quality enhancement processing on the local region within the remaining processing time as the image quality enhancement model to be used by the execution means. (Composition 9) The execution means performs the image enhancement process on the local region using the image enhancement model, The information processing apparatus according to any one of configurations 1 to 8, characterized in that the method determination means determines the number of input frames to the high-resolution model based on the remaining processing time and the processing time per frame in the high-resolution model. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, characterized in that the region determination means determines the local region for which high-image-quality processing is performed according to priority. (Composition 11) The information processing apparatus according to any one of configurations 1 to 10, characterized in that the region determination means determines the local region such that it includes the object whose distance within the image is within a threshold. (Composition 12) An information processing apparatus according to any one of configurations 1 to 11, characterized in that the local region in which high-image-quality processing is performed is switched at predetermined intervals. (Composition 13) An information processing apparatus according to any one of configurations 1 to 12, characterized by having an image acquisition means, an information acquisition means, a region determination means, a method determination means, and a notification means for notifying at least one processing result of the execution means. (Method 1) An information processing method performed by an information processing device, The image acquisition process involves obtaining the captured image, An information acquisition process to obtain information about the position of the object, A region determination step, based on the information acquired in the information acquisition step, determines a local region within the image that includes the object, A method determination step in which a method for improving the image quality of the local area is determined based on the determined local area and the remaining processing time that allows for image quality improvement processing, An information processing method characterized by comprising an execution step of performing an image enhancement process on the image according to the determined image enhancement method. (Program 1) Image acquisition step to acquire the captured image, An information acquisition step to obtain information about the position of the object, A region determination step in which a local region is determined, which is a part of the image and includes the object, based on the information obtained in the information acquisition step, A method determination step in which a method for improving the image quality of the local area is determined based on the determined local area and the remaining processing time that allows for image quality improvement processing, A program for causing a computer to perform an execution step of performing an image enhancement process on the image according to the determined image enhancement method. [Explanation of Symbols]

[0093] 101: Information processing device 201: Monitoring system 202: Imaging unit 203: Image acquisition unit 204: Information acquisition unit 205: Region determination unit 206: High-quality image enhancement method determination unit 207: High-quality image enhancement execution unit

Claims

1. Image acquisition means for acquiring captured images, Information acquisition means for acquiring information about the position of an object, A region determination means that determines a local region within the image that includes the object, based on the information acquired by the information acquisition means, A method determination means for determining a method for improving the image quality of the local region based on the determined local region and the remaining processing time for performing image quality improvement processing, An information processing apparatus characterized by having an execution means for performing image enhancement processing on the image according to the determined image enhancement method.

2. The information processing apparatus according to claim 1, characterized in that the method determination means determines the image enhancement method such that the time required for the image enhancement processing performed on the local region falls within the remaining processing time.

3. The information processing apparatus according to claim 1, characterized in that the region determination means determines the region including the object from which information has been acquired by the information acquisition means as the local region.

4. The information processing apparatus according to claim 1, characterized in that the region determination means determines the region including the object whose distance from the image acquisition position is greater than or equal to a threshold as the local region.

5. The information processing apparatus according to claim 1, characterized in that the region determination means determines the region including at least one of the objects whose size in the image is less than or equal to a threshold, and the object whose distance from the occluding object is less than or equal to a threshold, as the local region.

6. The information processing apparatus according to claim 1, characterized in that the region determination means selects and determines the local region to be enhanced in image quality from among the regions including the object from which information has been acquired by the information acquisition means.

7. The information processing apparatus according to claim 1, characterized in that the region determination means determines the local region based on the positional information of the object in the world coordinate system.

8. The execution means performs the image enhancement process on the local region using the image enhancement model, The information processing apparatus according to claim 1, characterized in that the method determination means determines, among the high-image-quality enhancement models capable of performing the high-image-quality enhancement processing on the local region within the remaining processing time, the high-image-quality enhancement model with the largest computational load as the high-image-quality enhancement model to be used by the execution means.

9. The execution means performs the image enhancement process on the local region using the image enhancement model, The information processing apparatus according to claim 1, characterized in that the method determination means determines the number of input frames to the high-resolution model based on the remaining processing time and the processing time per frame in the high-resolution model.

10. The information processing apparatus according to claim 1, characterized in that the region determination means determines the local region that undergoes high-image-quality processing according to priority.

11. The information processing apparatus according to claim 1, characterized in that the region determination means determines the local region such that it includes the object whose distance within the image is within a threshold.

12. The information processing apparatus according to claim 1, characterized in that the local region in which high-image-quality processing is performed is switched at predetermined intervals.

13. The information processing apparatus according to claim 1, further comprising an image acquisition means, an information acquisition means, a region determination means, a method determination means, and a notification means for notifying at least one processing result of the execution means.

14. An information processing method performed by an information processing device, The image acquisition process involves obtaining the captured image, An information acquisition process to obtain information about the position of the object, A region determination step, based on the information acquired in the information acquisition step, determines a local region within the image that includes the object, A method determination step in which a method for improving the image quality of the local area is determined based on the determined local area and the remaining processing time that allows for image quality improvement processing, An information processing method characterized by comprising an execution step of performing an image enhancement process on the image according to the determined image enhancement method.

15. Image acquisition step to acquire the captured image, An information acquisition step to obtain information about the position of the object, A region determination step in which a local region is determined, which is a part of the image and includes the object, based on the information obtained in the information acquisition step, A method determination step in which a method for improving the image quality of the local area is determined based on the determined local area and the remaining processing time that allows for image quality improvement processing, A program for causing a computer to perform an execution step of performing an image enhancement process on the image according to the determined image enhancement method.

Images