Information processing apparatus, information processing method, information processing system, and program

Abstract

To enable improvement in the quality of an image when the image is compressed and expanded and subsequently subjected to image processing.SOLUTION: An information processing apparatus has: image acquisition means that acquires an image; image processing means that performs first image processing on the image; meta data acquisition means that acquires meta data related to the image to be transmitted to a reception-side information processing apparatus; compression means that compresses the image after the first image processing; and communication means that transmits the meta data and the compressed image to the reception-side information processing apparatus. The image processing means determines the details of the first image processing on the basis of the details of second image processing performed according to a result of analysis of the meta data in the reception-side information processing apparatus.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing technique for handling image information. [Background technology]

[0002] There is known a system in which an image captured by an imaging device such as a security camera is transmitted via a network to a receiving device such as a server in a monitoring room, and the server processes the image and stores the image. For example, image noise reduction is one example of image processing performed by the server. Patent Document 1 discloses a method for determining image analysis processing to be performed by a small robot equipped with a security camera and the server, respectively. Patent Document 2 discloses a method for switching image processing functions at the source of image data depending on the destination of the image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 038100 [Patent Document 2] JP 2007-88924 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned system, the imaging device compresses an image so that the image data size fits within the network bandwidth before transmitting it. Then, the server that receives the image via the network expands the compressed image, and then processes and stores the image. Here, for example, if the image data size is large, the imaging device compresses the image at a high compression rate. However, when an image is compressed at a high compression rate, much of the information contained in the original captured image is lost. For this reason, the image that is received and expanded after being compressed becomes an image that deviates greatly from the original captured image. In this case, even if the server side performs image processing on the image that is received and expanded after being compressed, there is a possibility that the image quality cannot be improved.

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to enable improvement in image quality when a compressed image is decompressed and then image processed. [Means for solving the problem]

[0006] The information processing device of the present invention comprises an image acquisition means for acquiring an image, an image processing means for performing first image processing on the image, a metadata acquisition means for acquiring metadata related to an image to be transmitted to a receiving information processing device, a compression means for compressing the image after the first image processing, and a communication means for transmitting the metadata and the compressed image to the receiving information processing device, and is characterized in that the image processing means determines the content of the first image processing based on the content of a second image processing performed in the receiving information processing device in accordance with the analysis result of the metadata. Effect of the Invention

[0007] According to the present invention, it is possible to improve the quality of an image when a compressed image is decompressed and then image processed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration to which an information processing device is applied. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3] FIG. 1 is a diagram illustrating an example of a network structure of a trained model according to a first embodiment. [Figure 4] 4 is a flowchart of information processing according to the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a network structure of a trained model according to the second embodiment. [Figure 6] 10 is a flowchart of information processing according to the second embodiment. [Figure 7] FIG. 13 illustrates an example of a setting screen. [Figure 8] FIG. 11 is a diagram showing another example of the setting screen. [Figure 9] 13 is a flowchart of information processing according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiment may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, the same or similar configurations and processing steps are given the same reference symbols, and duplicated explanations are omitted.

[0010] <First embodiment> Fig. 1 is a diagram showing an example of a configuration of an information processing system to which an information processing device according to this embodiment is applied. The information processing system shown in Fig. 1 is a system including an edge device 100 which is an example of a first information processing device on a transmitting side that captures and transmits an image, and a server 150 which is an example of a second information processing device on a receiving side that receives the image transmitted from the edge device 100. It is assumed that the edge device 100 and the server 150 are connected via a network such as the Internet. In this embodiment, the edge device 100 is assumed to be, for example, a surveillance camera, and the server 150 is assumed to be a server in a surveillance room that receives the image transmitted from the surveillance camera (edge ​​device 100), but is not limited thereto.

[0011] In addition, in this embodiment, an example of information processing that improves the image quality of an image in an information processing device will be described. Here, factors that cause an image to have low image quality include, for example, noise, blur, aberration, loss, a low-resolution image that originally has a low resolution, and a decrease in contrast due to weather conditions such as fog, haze, snow, and rain at the time of shooting. In addition, for example, an image that was originally of high image quality may become of low image quality due to loss of information due to compression and decompression. For this reason, the information processing device of this embodiment performs image processing that improves the image quality of an image of low image quality due to these factors. Examples of image processing that improves image quality include noise reduction, blur removal, aberration correction, loss compensation, edge emphasis, restoration of information lost in compression and decompression, super-resolution processing for low-resolution images, and contrast correction that corrects contrast reduction due to weather at the time of shooting. The image processing of this embodiment includes not only processing that improves the image quality of an image that has become of low image quality at the time of shooting, but also processing that improves the image quality of an image that was originally of high image quality but has become of low image quality, and hereinafter these are collectively referred to as degradation restoration processing. In addition, in this embodiment, as an example of degradation restoration processing, information processing that improves the image quality of an image using a CNN (Covolutional Neural Network) is given. In the first embodiment described below, a noise reduction process for an image containing noise will be taken as an example of the degradation restoration process.

[0012] <Edge device hardware configuration> The edge device 100 includes a CPU 101, a RAM 102, a ROM 103, a large-capacity storage device 104, a general-purpose interface (I / F) 105, and a network I / F 106, and these components are interconnected by a system bus 107. The edge device 100 also includes an imaging device 108, an input device 109, an external storage device 110, and a display device 111, which are connected via the general-purpose I / F 105, but of course, is not limited to this example.

[0013] The imaging device 108 captures an image and obtains captured image data (here, for example, RAW image data corresponding to the Bayer array). The captured image data is sent to the CPU 101 via the general-purpose I / F 105. Note that hereinafter, unless otherwise specified, image data handled in the edge device 100 and the server 150 of this embodiment will be simply referred to as an image.

[0014] The CPU 101 uses the RAM 102 as a work memory, executes a program stored in the ROM 103, and generally controls each component of the edge device 100 via the system bus 107. The CPU 101 also executes an information processing program stored in the ROM 103 to perform various information processing such as image analysis processing, development processing, image processing, compression processing, metadata acquisition and assignment processing, generation and storage of setting information, and communication, which will be described later. Details of the information processing by the CPU 101 of the edge device 100 will be described later. The CPU 101 also sends the image that has undergone the information processing to the network I / F 106. The CPU 101 also performs processing to generate an image for display from a captured image, and outputs the display image to the display device 111 (for example, various image display devices such as a liquid crystal display) via the general-purpose I / F 105. As a result, an image captured by the imaging device 108, a setting screen to be described later, and the like are displayed on the display device 111. The display device 111 may be a display device integrated with a touch panel.

[0015] The mass storage device 104 includes, for example, an HDD or SSD, and stores various data and image data handled by the edge device 100. The CPU 101 writes data to the mass storage device 104 via a system bus 107, and reads data stored in the mass storage device 104. Note that the programs executed by the CPU 101 may be stored in the mass storage device 104, or may be downloaded from an external device, for example.

[0016] The network I / F 106 is an interface for connecting to the Internet, etc. The CPU 101 executes a communication program included in the information processing program to access the server 150 from the network I / F 106 via the Internet, and transmits the image and the like after the above-described information processing to the server 150.

[0017] The general-purpose I / F 105 is a serial bus interface such as USB, IEEE1394, HDMI (registered trademark), etc. The CPU 101 obtains data from an external storage device 110 (such as various storage media such as a memory card, a CF card, an SD card, a USB memory, etc.) and writes data to the external storage device 110 via the general-purpose I / F 105. The CPU 101 also accepts user instructions from an input device 109 such as a mouse or a keyboard via the general-purpose I / F 105.

[0018] <Server hardware configuration> The server 150 includes a CPU 151 , a ROM 152 , a RAM 153 , a mass storage device 154 , and a network I / F 155 , and these components are connected to each other via a system bus 156 .

[0019] The CPU 151 controls the overall operation of the server 150 by reading out the programs stored in the ROM 152 and executing various processes. The CPU 151 uses the RAM 153 as a temporary storage area such as a main memory or a work area. The CPU 151 also executes a communication program included in the information processing program stored in the ROM 152 to communicate with the edge device 100 from the network I / F 155 via the Internet and receive images and the like from the edge device 100. The CPU 151 also executes the information processing program to perform various information processes such as decompression processing, metadata analysis processing, image processing, and generation and storage of setting information, which will be described later. Details of the information processing by the CPU 151 of the server 150 will be described later.

[0020] Here, the image transmitted from the edge device 100 to the server 150 is an image that has been subjected to compression processing, as described later. Therefore, the CPU 151 performs decompression processing corresponding to the compression processing performed on the image received from the edge device 100, and the image after the decompression processing is set as a processing target image for image processing. Although details will be described later, in the case of the first embodiment, the CPU 151 performs degradation restoration processing as image processing on the processing target image using a trained machine learning model (hereinafter referred to as a trained model), and outputs the image after the degradation restoration processing. In addition, the CPU 151 generates an image for display from the image after the degradation restoration processing, and outputs the display image to the display device 159 (for example, various image display devices such as a liquid crystal display) via the general-purpose I / F 157. As a result, a high-quality image after the degradation restoration processing is displayed on the display device 159. Note that the display device 159 may be a display device integrated with a touch panel.

[0021] The large-capacity storage device 154 is a large-capacity secondary storage device including an HDD, an SSD, etc., that stores image data and various programs. The programs executed by the CPU 151 may be stored in the large-capacity storage device 154, or may be downloaded from an external device. The network I / F 155 is an interface for connecting to the Internet, and communicates with the edge device 100 via the Internet and receives images transmitted from the edge device 100. The general-purpose I / F 157 is, for example, a serial bus interface such as USB, IEEE1394, HDMI (registered trademark), etc. The general-purpose I / F 157 is also connected to an input device 158 such as a mouse or keyboard. Therefore, the server 150 can receive user instructions from the input device 158 such as a mouse or keyboard via the general-purpose I / F 157, and the CPU 151 can perform processing based on the user instructions.

[0022] The edge device 100 and the server 150 have other components in addition to those described above, but the description thereof will be omitted here. The configuration of the information processing system according to this embodiment is an example, and is not limited to the example of FIG. 1. For example, the functions of the server 150 may be subdivided, and image reception and degradation restoration processing may be performed by separate devices. Alternatively, the edge device 100 may not include the imaging device 108, and an imaging device provided outside the edge device 100 may be connected, and the image captured by the external imaging device may be subjected to the above-mentioned information processing and the like and transmitted to the server 150.

[0023] <Function block configuration> Fig. 2 is a diagram showing an example of the functional configuration of the entire information processing system according to this embodiment. Each functional unit shown in Fig. 2 is realized by executing an information processing program according to this embodiment in the CPU 101 of the edge device 100 and the CPU 151 of the server 150. Note that all or part of the functional units shown in Fig. 2 may be implemented in hardware.

[0024] First, the functional configuration of the edge device 100 will be described. The imaging section 200 is an image acquisition section that acquires an image captured by the imaging device 108. It is assumed that the captured image acquired by the imaging section 200 is a RAW image. The image analysis unit 201 analyzes the captured image by a predetermined analysis process. In this embodiment, since a noise reduction process is taken as an example of the degradation restoration process, the image analysis result by the image analysis unit 201 is, for example, a numerical value indicating the amount of noise contained in the captured image, or information (noise map) in which the amount of noise of each pixel is arranged two-dimensionally. There are various known analysis processes as the image analysis process for generating a numerical value of the amount of noise or a noise map from an image, and any of these analysis processes may be used. Note that, in this embodiment, an example of analyzing the amount of noise is taken as the predetermined analysis process, but the present invention is not limited to this example, and for example, the amount of blurring, aberration, loss, resolution, contrast reduction, etc. of the image may be analyzed. The developing unit 202 develops the RAW image of the captured image to create a developed image. The developed image is assumed to be an RGB image. The development process is also known, so a detailed description thereof will be omitted.

[0025] The setting unit 208 stores the setting information input by the user via the input device 109 in Fig. 2 in the setting storage unit 209, and updates the setting information already stored in the setting storage unit 209 with the setting information input by the user. The setting unit 208 also updates the setting information stored in the setting storage unit 209 based on information received from the server 150 via the communication unit 207. The information received from the server 150 is information indicating the content of image processing performed by the server 150. In this embodiment, the image processing (deterioration restoration processing) performed by the server 150 is called second image processing, and information indicating the content of the second image processing is called second image processing information.

[0026] The image processing unit 203 performs image processing on the image developed by the development unit 202. The image processing unit 203 of the edge device 100 can perform a combination of one or more of a plurality of types of image processing, such as low-pass filter processing (LPF processing), color space conversion processing, reduction, color reduction, and degradation restoration processing. The image processing unit 203 of the edge device 100 determines and performs one or more of the plurality of types of image processing, based on second image processing information indicating the content of the second image processing performed by the server 150. Hereinafter, the image processing performed by the image processing unit 203 of the edge device 100 will be referred to as a first image processing.

[0027] In the first embodiment, an example will be given in which the server 150 executes noise reduction processing as the second image processing, and the image processing unit 203 of the edge device 100 executes low-pass filter processing and YUV422 conversion processing of color space conversion processing as the first image processing. That is, the image processing unit 203 can execute multiple types of degradation restoration processing as described above, but in the first embodiment, an example will be given in which the image data size is reduced by performing low-pass filter processing and color space conversion processing, and the degradation restoration processing is performed by the server 150.

[0028] The model storage unit 210 is a functional unit that stores a trained model used when the image processing unit 203 of the edge device 100 performs degradation restoration processing as the first image processing. However, in the case of the first embodiment, the image processing unit 203 of the edge device 100 performs only image processing of low-pass filter processing and color space conversion processing, and does not perform degradation restoration processing, so the trained model in the model storage unit 210 is not used. An example in which the image processing unit 203 of the edge device 100 performs the first image processing using the trained model in the model storage unit 210 will be described in the second embodiment described later.

[0029] The metadata adding unit 205 acquires metadata related to an image to be transmitted to the information processing device on the receiving side, and adds the metadata to the image after the image processing by the image processing unit 203. That is, the metadata adding unit 205 adds the acquired metadata to the processing target image that is transmitted from the edge device 100 to the server 150 and is to be subjected to the second image processing (degradation restoration processing) on ​​the server 150 side. Any of various known methods may be used as a method for acquiring and adding metadata. Also, in the present embodiment, an example has been given in which metadata is added to the processing target image and transmitted to the server 150, but the metadata may be separately transmitted to the server 150.

[0030] Here, the metadata includes one or more pieces of information including shooting information, image analysis information, development information, image processing information, processing target area information, and compression information. The shooting information is information about the shooting performed by the imaging device 108, and includes, for example, gain information at the time of shooting. The image analysis information is information about the image analysis by the image analysis unit 201, and includes, for example, a numerical value indicating the amount of noise contained in the image and a noise map. The development information is information about the development by the development unit 202, and includes, for example, gamma information. The image processing information is first image processing information indicating the contents of the first image processing by the image processing unit 203, and in the case of the first embodiment, includes information indicating that low-pass filter processing and color space conversion processing have been performed. The processing target area information is information indicating the area that has been subjected to image processing, and includes, for example, information indicating that the entire area or a part of the image has become the processing target area. The compression information is information about the compression processing by the compression unit 206, and includes, for example, a compression method and a compression rate. In the case of this embodiment, it is assumed that the metadata includes all of these pieces of information.

[0031] The compression unit 206 performs compression processing on each image of the multiple frames that have been subjected to the first image processing by the image processing unit 203, and outputs the image after the compression processing (hereinafter referred to as a compressed image). Note that the compression method and compression rate in the compression processing by the compression unit 206 can be determined or calculated by a known method based on the network bandwidth usage rate, the image data size, etc. The communication unit 207 transmits the above-mentioned metadata and the compressed image from the edge device 100 to the server 150. In addition, the communication unit 207 receives, from the server 150, second image processing information indicating the content of the second image processing. The edge device 100 can also output the image resulting from the first image processing by the image processing unit 203 as an output image 204 to the display device 111 or the like for display.

[0032] Next, the functional configuration of the server 150 will be described. The communication unit 250 receives the compressed image and metadata transmitted from the edge device 100 . The decompression unit 251 decompresses the compressed image. The image after the decompression process by the decompression unit 251 becomes an image to be processed in the second image processing performed by the image processing unit 253 at the subsequent stage. The metadata analysis unit 252 analyzes the metadata attached to the image to be processed. The analysis result of the metadata is used when determining the content of the second image processing performed in the image processing unit 253 at the subsequent stage.

[0033] The image processing unit 253 performs a second image processing on the processing target image based on the analysis result of the metadata by the metadata analysis unit 252. The second image processing performed by the image processing unit 253 of the server 150 is a degradation restoration processing, and in this embodiment, an example of noise reduction processing is given. In this embodiment, the image processing unit 253 of the server 150 executes noise reduction processing using a trained model stored in the model storage unit 257. Then, an image of the degradation restoration processing result (noise reduction processing result) by the image processing unit 253 is output as an output image 254.

[0034] The setting unit 255 stores the setting information input by the user via the input device 158 in the setting storage unit 256, and updates the setting information already stored in the setting storage unit 256 with the setting information input by the user. The setting unit 255 also updates the setting information stored in the setting storage unit 256 based on the first image processing information included in the metadata received from the edge device 100.

[0035] <Network structure of machine learning model> 3 is a diagram showing an example of a network structure of a trained model used in degradation restoration processing (noise reduction processing in the first embodiment) executed by the image processing unit 253 of the server 150. In the example of FIG. 3, the input to the trained model network is input data 301 obtained by performing known preprocessing on an input image, which is an image to be processed, and the noise map 302 described above. Note that in FIG. 3, t represents time, and t relatively later than the image at t=0 is expressed as a positive value, and t relatively earlier is expressed as a negative value.

[0036] 3 illustrates an example in which input data 301 is arranged in chronological order for each of a plurality of frames, the input data 301 and a noise map are input to a trained model network, and output data 306 (t=0) for an input image at time t=0 is output. The noise map 302 is included in the image analysis result by the image analysis unit 201 of the edge device 100 described above, and is transmitted to the server 150 as metadata, so that the server 150 can obtain the noise map by analyzing the metadata.

[0037] The network structure of the trained model illustrated in FIG. 3 has a two-stage configuration. In the first stage processing, three frames of input data 301 and one frame of noise map 302 are input to execute a first inference 303, and output an intermediate output 304. The intermediate output 304 is, for example, image format data. Subsequently, in the second stage processing, intermediate outputs 304, which are the output of three times of the first inference 303, are input to execute a second inference 305, and output data 306 is output. Here, two of the intermediate outputs 304 input to the second inference 305 reuse the results of the previous and previous inferences. That is, the intermediate outputs t=-1 to 1 obtained as the output of the first inference 303 in the previous inference and the intermediate outputs t=-2 to 0 obtained as the output of the first inference 303 in the previous inference are saved, and these intermediate outputs 304 are reused. Note that, although the example of FIG. 3 uses a plurality of input data 301 and one output data 306, this embodiment is not limited to this example. For example, the number of input data and the number of output data may each be 1 or more, or the output data 306 may be an output corresponding to a different time other than t=0 of the input data. The input data 301 does not have to be continuous data in a time series. The noise map 302 and the intermediate output 304 do not have to be data in an image format, and may have a different width, height, and number of channels from the input data 301.

[0038] <Processing flow of the entire system> Next, various processes performed in the information processing system of this embodiment will be described with reference to the flowchart of Fig. 4. Fig. 4(A) is a flowchart showing the flow of information processing performed in each functional unit of the edge device 100 shown in Fig. 2, and Fig. 4(B) is a flowchart showing the flow of information processing performed in each functional unit of the server 150 shown in Fig. 2. In each of the subsequent flowcharts, the symbol "S" represents a processing step (process).

[0039] First, the information processing performed by the edge device 100 will be described with reference to the flowchart of FIG. In S401, the imaging unit 200 acquires a captured image from the imaging device . Next, in S402, the image analysis unit 201 analyzes the captured image acquired by the imaging unit 200. As described above, the image analysis unit 201 acquires, through image analysis processing, a numerical value indicating the amount of noise contained in the image and a noise map (noise map 302 in FIG. 3) in which the noise amount of each pixel is two-dimensionally arranged. Next, in S403, the development unit 202 develops the RAW image of the captured image to generate a developed image (RGB image).

[0040] Next, in S404, the image processing unit 203 reads the setting information stored in the setting storage unit 209, and determines the content of the first image processing to be performed in the subsequent step S405 based on the setting information. As described above, the setting information stored in the setting storage unit 209 is updated based on the second image processing information indicating the content of the second image processing to be performed by the server 150. As described above, the image processing unit 203 can execute a combination of one or more image processes among a plurality of types of image processing such as low-pass filter processing, color space conversion processing, reduction, color reduction, and degradation restoration processing. Therefore, the image processing unit 203 determines one or more image processes among the plurality of types of image processing based on the content of the second image processing to be performed by the server 150. In the case of the first embodiment, the image processing unit 203 executes the image processing of low-pass filter processing and color space conversion processing (YUV422 conversion processing) as described above.

[0041] Next, in S405, the image processing unit 203 executes the image processing determined in S404 on the image developed by the development unit 202, that is, in the case of the first embodiment, low-pass filter processing and color space conversion processing. Next, in S406, the metadata adding unit 205 adds the above-mentioned metadata to the image (which will be the image to be processed by the server 150) after image processing by the image processing unit 203. As described above, the metadata to be added includes one or more pieces of information including shooting information, image analysis results (noise map 302, etc.), development information, first image processing information indicating the contents of the first image processing, and compression information. In this embodiment, all of this information is included in the metadata.

[0042] Next, in S407 , the compression unit 206 compresses the image after the first image processing has been performed by the image processing unit 203 . Thereafter, in S408, the communication unit 207 transmits the compressed image and the metadata to the server 150.

[0043] Next, the information processing performed by the server 150 will be described with reference to the flowchart of FIG. In S451 , the communication unit 250 receives the compressed image transmitted from the edge device 100 . Next, in S452, the decompression unit 251 decompresses the compressed image received by the communication unit 250, and acquires the image to be processed by the server 150.

[0044] Next, in S453, the metadata analysis unit 252 analyzes the metadata attached to the image to be processed. In this embodiment, the metadata includes shooting information in the edge device 100, image analysis results (such as the noise map 302), development information, first image processing information, and compression information.

[0045] Next, in S454, the image processing unit 253 determines the content of the second image processing to be performed in the subsequent S456 based on the setting information stored in the setting storage unit 256 and the metadata analyzed by the metadata analysis unit 252. The image processing unit 253 selects, based on the shooting gain value and gamma of the development information included in the metadata, and the noise map of the image analysis result, from the model storage unit 257, a trained model for noise reduction processing corresponding to those gain value, gamma, and noise map.

[0046] Then, in S455, the image processing unit 253 performs image processing on the processing target image after the decompression processing in the decompression unit 251, in this embodiment, noise reduction processing using a trained model selected from the model storage unit 257. That is, the image processing unit 253 performs highly accurate noise reduction processing by inputting the processing target image to a trained model selected based on the gain value at the time of shooting of the shooting information and the gamma information of the development information in the edge device 100, and the noise map. Thereafter, in S456, the image processing unit 253 outputs the image after the image processing as the output image 254.

[0047] As described above, the edge device 100 determines and executes the content of the first image processing to be performed by the image processing unit 203 based on the content of the second image processing performed by the server 150. Then, the edge device 100 transmits to the server 150 an image to which metadata including first image processing information indicating the first image processing and shooting information, image analysis information, development information, and compression information and the like is added. The server 150 determines and executes the content of the second image processing to be performed by the server 150 based on the metadata, thereby achieving a highly accurate degradation restoration process. As a result, according to this embodiment, the server 150 that receives the image compressed and transmitted by the edge device 100 can improve the image quality of the image by image processing after decompression.

[0048] In the above-described embodiment, an example was given in which the gain at the time of shooting and the like were sent to the server 150 as the shooting information of the edge device 100, but other shooting information, for example, information such as the distance to the subject, the focal length, the size of the image sensor, and the exposure, may be sent to the server 150. In this case, the server 150 can further improve the accuracy of the degradation restoration process by using information such as the distance to the subject, the focal length, the size of the image sensor, and the exposure in the degradation restoration process.

[0049] In the above-described embodiment, the noise reduction process is given as an example of the degradation restoration process performed by the server 150, but the process is not limited to this example. The image processing performed by the server 150 may be a degradation restoration process for any one of a plurality of types of degradation elements such as blur, aberration, compression, low resolution, loss, and contrast reduction caused by the weather at the time of shooting, or a combination of a plurality of these. In the present embodiment, an example of degradation restoration process based on a trained model has been described, but the image processing based on the trained model is not limited to degradation restoration process. The image processing based on the trained model may be, for example, image processing for detecting the position and posture of a specific object such as a person or a car from an image, or image recognition processing such as reading a license plate.

[0050] Second Embodiment Next, an example will be described as an information processing system according to a second embodiment, in which the edge device 100 and the server 150 share the responsibility for executing the degradation restoration process. In the second embodiment, an example will be given in which the image processing content to be performed by the edge device 100 and the server 150 is determined based on the amount of noise in the captured image, the image data size, the bandwidth usage rate of the network for transmitting the image, and the amount of computational resources used by the edge device 100. Note that the configuration of the information processing system is similar to that of the first embodiment described above, and a description of the content common to the first embodiment will be omitted, and the following description will focus on the differences from the first embodiment.

[0051] In the second embodiment, a noise reduction process will be described as an example of degradation restoration processing that is performed separately by the edge device 100 and the server 150. In the second embodiment, an example will be given in which the edge device 100 transmits an intermediate output of the noise reduction process using a trained model to the server 150, and the server 150 further performs noise reduction processing using the trained model on the received intermediate output.

[0052] In the case of the second embodiment, the image processing unit 203 of the edge device 100 determines the content of the first image processing based on the setting information stored in the setting storage unit 209 and the noise information (amount of noise) of the image analyzed by the image analysis unit 201. Specifically, the image processing unit 203 determines to perform noise reduction processing based on the setting information stored in the setting storage unit 209, and further determines a stronger noise reduction intensity as the amount of noise in the image analyzed by the image analysis unit 201 increases. As a result, the stronger the amount of noise in the image, the stronger the noise reduction processing becomes, and it becomes possible to reduce the difference in pixel values ​​between frames and reduce the image data size. In addition, in the case of the second embodiment, the image processing unit 203 of the edge device 100 performs noise reduction processing using a trained model corresponding to the noise reduction processing. In the second embodiment, it is assumed that a plurality of trained models corresponding to the noise reduction processing are prepared in advance, such as a trained model that emphasizes noise reduction in the spatial direction and a trained model with a low processing load.

[0053] In addition, the image processing unit 203 of the second embodiment may determine the contents of the first image processing based on one or more of the image data size, the bandwidth usage rate of the network, the amount of computational resources used by the edge device 100, and the like, in addition to the amount of noise in the image. Specifically, the image processing unit 203 of the second embodiment sets a stronger noise reduction intensity, for example, as the image data size increases. This makes it possible to reduce the difference in pixel values ​​between frames and reduce the image data size. In addition, the image processing unit 203 sets a stronger noise reduction intensity, for example, as the bandwidth usage rate of the network for transmitting the image increases. This makes it possible to reduce the difference in pixel values ​​between frames and reduce the image data size. In addition, the image processing unit 203 selects a trained model for trained noise reduction with a lower processing load, for example, as the amount of computational resources used by the edge device 100 increases, so as to reduce the load on the edge device 100. In this way, the image processing unit 203 of the second embodiment determines the contents of the first image processing based on the setting information of the setting storage unit 209 and one or more of the image noise amount, the image data size, the bandwidth usage rate of the network, the amount of computational resources used by the edge device 100, and the like.

[0054] In the second embodiment, the edge device 100 includes the amount of noise in the image and the noise reduction intensity during the noise reduction process in the metadata. Then, the edge device 100 of the second embodiment adds metadata to the intermediate output of the noise reduction process based on the trained model in the image processing unit 203 and transmits the intermediate output to the server 150.

[0055] Also in the second embodiment, the image processing unit 253 of the server 150 determines the content of the second image processing based on the setting information of the setting storage unit 256 and the analysis result of the metadata. In the present embodiment, the image processing unit 253 of the server 150 sets a stronger noise reduction intensity to perform noise reduction processing as the amount of noise in the image included in the metadata increases. Note that the image processing unit 253 may determine the noise reduction intensity for the second image processing based on the relationship between the amount of noise in the image included in the metadata sent from the edge device 100 and the noise reduction intensity during the noise reduction processing in the edge device 100.

[0056] In addition, the image processing unit 253 of the server 150 of this embodiment selects a trained model and a noise reduction intensity corresponding to the content of the noise reduction processing performed in the edge device 100 based on the analysis result of the metadata. For example, when the image processing unit 203 of the edge device 100 performs image processing using a trained model that emphasizes noise reduction in the spatial direction, the image processing unit 253 selects a trained model that emphasizes consistency in the noise reduction results in the time direction between multiple images. This can improve the consistency in the time direction of the output image after the noise reduction processing. Also, for example, when the edge device 100 performs image processing using a trained model with a low processing load, the image processing unit 253 selects a trained model with a high processing load. This can improve the accuracy of the noise reduction processing as a whole information processing system, that is, improve the image quality by the noise reduction processing. In this embodiment, the edge device 100 transmits the intermediate output and the noise reduction intensity of the noise reduction processing based on the trained model to the server 150 as described above. Therefore, in the server 150, the noise reduction processing using the trained model is further performed on the intermediate output of the noise reduction processing in the edge device 100.

[0057] <Network structure of machine learning model in the second embodiment> 5 is a diagram showing an example of a network structure of a trained model used in the image processing unit 203 of the edge device 100 and the image processing unit 253 of the server in the second embodiment. Note that a description of common contents similar to those in the network structure shown in FIG. 3 in the first embodiment will be omitted, and the following description will focus on the differences.

[0058] FIG. 5A is a diagram showing an example of a network structure of a trained model in the image processing unit 203 of the edge device 100. Inputs to the network are input data 301 obtained by performing known preprocessing on the input image and a noise reduction intensity 501. The noise reduction intensity 501 is information expressed by a numerical value, and the image processing unit 203 determines the numerical value of the noise reduction intensity 501 based on noise information (amount of noise) of the captured image analyzed by the image analysis unit 201. As described above, the image processing unit 203 sets a stronger noise reduction intensity as the amount of noise in the image analyzed by the image analysis unit 201 increases, and this enables a stronger noise reduction process as the amount of noise in the captured image increases. The image processing unit 203 of the edge device 100 executes a first inference 502 using the determined numerical value of the noise reduction intensity as an input, and outputs an intermediate output 304. In the case of the second embodiment, information indicating the contents of the first inference 502 is included in the metadata, and the intermediate output 304 is compressed and transmitted to the server 150. Therefore, in the case of the second embodiment, the decompression unit 251 of the server 150 performs decompression processing on the compressed intermediate output 304 .

[0059] 5B is a diagram showing an example of a network structure of a trained model in the image processing unit 253 of the server 150. Inputs to the network are the intermediate output 304, which is the output of three rounds of the first inference 502 by the image processing unit 203 of the edge device 100, and the noise reduction strength 503. The image processing unit 253 determines the noise reduction strength 503 based on the amount of noise in the image analyzed by the edge device 100 and information indicating the content of the first inference 502, and also determines the trained model to be used in the second inference 504. Then, the image processing unit 253 executes the second inference 504 using the intermediate output 304 and the noise reduction strength 503 expanded by the expansion unit 251 as inputs, and outputs the output data 306.

[0060] <Processing flow of the entire system in the second embodiment> Next, various processes performed in the information processing system of the second embodiment will be described with reference to the flowchart of Fig. 6. Fig. 6(A) is a flowchart showing the flow of information processing performed in each functional unit of the edge device 100 according to the second embodiment, and Fig. 6(B) is a flowchart showing the flow of information processing performed in each functional unit of the server 150. Note that a description of the contents common to the processing steps listed in the first embodiment will be omitted, and the following description will focus on the differences.

[0061] First, information processing performed by the edge device 100 according to the second embodiment will be described with reference to the flowchart in Fig. 6(A). In the second embodiment, the edge device 100 proceeds to processing in S602 after acquiring a captured image in S401. In S602, the image analysis unit 201 analyzes the captured image from the imaging unit 200. In the image analysis process of the second embodiment, the amount of noise contained in the captured image is analyzed, and the numerical value of the amount of noise is output. In the edge device 100 of the second embodiment, after the development process in S403, the process of S604 is performed.

[0062] In S604, the image processing unit 203 determines the content of the first image processing to be performed in the subsequent step S605. Here, the image processing unit 203 of the second embodiment reads information that the edge device 100 and the server 150 share the noise reduction processing of the degradation restoration processing from the setting information stored in the setting storage unit 209. Then, the image processing unit 203 of the second embodiment sets the noise reduction intensity 501 that is stronger as the noise amount of the image analyzed by the image analysis unit 201 increases. As described above, the image processing unit 203 may determine the noise reduction intensity and select the trained model based on one or more of the image noise amount, the image data size, the network bandwidth usage rate, the edge device 100 computational resource usage, and the like, in addition to the image noise amount. That is, the image processing unit 203 determines the first image processing content to be performed in S605 based on one or more of the setting information of the setting storage unit 209, the image noise amount, the image data size, the network bandwidth usage rate, and the edge device 100 computational resource usage.

[0063] Next, in S605, the image processing unit 203 processes the image developed by the developing unit 202 according to the content of the first image processing determined in S604. The subsequent processes from S406 to S408 are the same as those described above, and therefore the description will be omitted.

[0064] Next, the information processing performed by the server 150 of the second embodiment will be described with reference to the flowchart of Fig. 6(B). In the case of the second embodiment, the server 150 proceeds to the processing of S652 after the reception processing in S451, the decompression processing in S452, and the metadata analysis processing in S453.

[0065] In S654, the image processing unit 253 determines the content of the second image processing to be performed in the subsequent step S655 based on the setting information of the setting storage unit 256 and the metadata analyzed in S453. In this embodiment, the image processing unit 253 performs noise reduction processing by setting a noise reduction intensity 503 that increases as the amount of noise in the image included in the metadata increases. Also, as described above, the image processing unit 253 selects a trained model and noise reduction intensity 503 corresponding to the noise reduction processing performed in the edge device 100 based on the analysis information of the metadata.

[0066] Next, in S655, the image processing unit 253 processes the image expanded by the expansion unit 251 according to the image processing content determined in S654. The subsequent processing in S456 is the same as that described above, and therefore the explanation will be omitted.

[0067] As described above, in the second embodiment, the edge device 100 and the server 150 share the responsibility of executing the degradation restoration process. In the case of the second embodiment, the edge device 100 performs noise reduction processing on the image before compression to reduce the image data size, thereby making it possible to lower the compression rate during the compression processing performed by the compression unit 206. As a result, more image information before compression in the edge device 100 remains in the image expanded in the server 150, and it is possible to suppress degradation of image quality during image processing in the server 150. That is, even in the second embodiment, it is possible to improve the image quality of an image by image processing after expanding a compressed image.

[0068] <Third embodiment> Next, as a third embodiment, an example will be described in which the edge device 100 and the server 150 share the degradation restoration process and execute it, and the edge device 100 can display a preview of the image processing result according to the change of the setting information by the user. In the third embodiment, noise reduction processing and super-resolution processing will be taken as examples of degradation restoration processing executed separately by the edge device 100 and the server 150. In the third embodiment, the image processing unit 203 of the edge device 100 can execute, for example, noise reduction processing and super-resolution processing as the degradation restoration processing, and the edge device 100 displays the result of the degradation restoration processing in the image processing unit 203 as a preview image. In addition, the image processing unit 253 of the server 150 of the third embodiment also executes noise reduction processing and super-resolution processing as the degradation restoration processing. In the third embodiment, since the edge device 100 and the server 150 execute the degradation restoration processing separately, the edge device 100 determines the content of the degradation restoration processing to be executed by the image processing unit 203 of the edge device 100 based on the content of the degradation restoration processing handled by the server 150. In addition, in the configuration and processing of the information processing system of the third embodiment, explanations of the contents in common with the configurations and processing described in each of the above-mentioned embodiments will be omitted, and the following explanation will focus on the differences from the above-mentioned embodiments.

[0069] Fig. 7 is a diagram showing an example of a setting screen displayed on the display device 111 of the edge device 100. Fig. 7 illustrates an example of a case where there is no change in the setting information. In the setting screen 700 illustrated in FIG. 7, a processing target area setting 705, a noise reduction strength setting 706, and a super-resolution setting 707 are displayed as examples of setting items whose setting information can be changed by the user of the edge device 100. The setting information of each of these setting items can be arbitrarily operated by the user via the input device 109, that is, can be arbitrarily changed by the user. The changed setting information is unconfirmed setting information until the setting confirmation button 704 is pressed. When the setting information of the setting screen 700 is changed by the user and the setting confirmation button 704 is not pressed, the setting unit 208 writes information that there is unconfirmed setting information whose setting has been changed, into the setting storage unit 209. When the setting information of the setting screen 700 is changed by the user and the setting confirmation button 704 is pressed and confirmed, the setting unit 208 updates the setting information stored in the setting storage unit 209 with the setting information whose setting has been changed and confirmed. In the case of the third embodiment, the setting information stored in the setting storage unit 209 is included in metadata similar to that described above and transmitted to the server 150.

[0070] An image 701 of the setting screen 700 in FIG. 7 is an example of a preview image of the image processing result in the edge device 100, and is the output image 204 in FIG. 2. While the setting screen 700 is displayed, the image processing unit 203 of the edge device 100 performs degradation restoration processing according to the setting information of each setting item of the setting screen 700, and displays the degradation restoration processing result as the preview image 701. That is, when the setting information of each setting item of the setting screen 700 is changed, even if the changed setting information is unconfirmed, the image processing unit 203 performs degradation restoration processing according to the unconfirmed setting information, and displays the degradation restoration processing result as the preview image 701. This allows the user of the edge device 100 to check the result of the degradation restoration processing based on the setting information, including the case where the setting information is unconfirmed.

[0071] Furthermore, when there is no change in the setting information and when the changed setting information is confirmed, the edge device 100 displays the frame surrounding the setting information in each setting item in the setting screen 700 as, for example, a thin line. On the other hand, when the setting information is changed and the change is unconfirmed, the edge device 100 highlights the frame surrounding the setting information in the corresponding setting item in the setting screen 700 as, for example, a thick line. As described above, the changed setting information is unconfirmed setting information until the setting confirmation button 704 is pressed, and the preview image 701 displayed on the setting screen 700 of the edge device 100 is an image showing the image processing result based on the unconfirmed setting information. In the case of the setting screen 700 illustrated in FIG. 7, the frame lines surrounding the setting information of the processing target area setting 705, the noise reduction strength setting 706, and the super-resolution setting 707 are all displayed as thin lines, so that the setting information is not changed or is confirmed after the change. 7, the processing target area setting 705 indicates that the entire image is the processing target area, the noise reduction intensity setting 706 indicates that the noise reduction intensity is 1.1 times, and the super-resolution setting 707 indicates that the super-resolution is 2 times. Therefore, the preview image 701 on the setting screen 700 is an image in which the noise reduction process with the noise reduction intensity of 1.1 times and the super-resolution process with the super-resolution of 2 times have been performed on the entire image.

[0072] However, an image in which super-resolution processing has been performed on the entire image has a large image data size, and there is a possibility that the image will not fit within the network bandwidth when it is transmitted from the edge device 100 to the server 150. For this reason, the image transmitted from the edge device 100 to the server 150 is an image in which super-resolution processing has not been performed on the entire image. For this reason, in the second embodiment, the server 150 performs super-resolution processing on the processing target area of ​​the entire image as the second image processing. That is, in the second embodiment, the edge device 100 determines the first image processing for the image to be transmitted to the server 150 to be noise reduction processing based on the contents of the second image processing in the server 150 (i.e., that super-resolution processing is performed). As a result, in the second embodiment, the image transmitted from the edge device 100 to the server 150 is an image in which noise reduction processing has been performed on the processing target area of ​​the entire image.

[0073] Fig. 8 is a diagram showing an example of a setting screen displayed on the display device 111 of the edge device 100, similarly to Fig. 7. Fig. 8 describes a case where the setting information has been changed and the change in the setting information is not yet confirmed. A processing target area setting 805, a noise reduction intensity setting 806, and a super-resolution setting 807 in a setting screen 800 in Fig. 8 are examples of setting items for which the user can change the setting information, similarly to the example in Fig. 7. Therefore, when the setting information on the setting screen 800 is changed by the user, the setting unit 208 writes information to the setting storage unit 209 that there is changed setting information that has not yet been confirmed.

[0074] 8 is an example of a preview image of the image processing result in the edge device 100, similar to the preview image 701 in FIG. 7, and is the output image 204 in FIG. 2. As with the setting screen 700 described above, while the setting screen 800 is being displayed, the image processing unit 203 of the edge device 100 performs degradation restoration processing according to the setting information of each setting item of the setting screen 800, and displays the result of the degradation restoration processing as the preview image 801. This allows the user of the edge device 100 to check the result of the degradation restoration processing based on the setting information, including the unconfirmed case.

[0075] Here, in the case of the setting screen 800 illustrated in Fig. 8, the frame lines surrounding the setting information in each setting item of the processing target area setting 805 and the noise reduction intensity setting 806 are thin lines, so that the setting information has been changed or has been confirmed after being changed. Note that in the case of the setting screen 800 illustrated in Fig. 8, the noise reduction intensity setting 806 is set to 1.1 times the noise reduction intensity as in the example of Fig. 7, but the processing target area setting 805 is set to setting information for a part of the image as the processing target area rather than the entire image. On the other hand, the super-resolution setting 807 is set to 4 times super-resolution, and the frame lines surrounding the setting information are thick lines, so that the setting information of this super-resolution setting 807 is changed and unconfirmed information.

[0076] In the case of the setting screen 800 illustrated in Fig. 8, the image processing unit 203 of the edge device 100 performs noise reduction processing with a noise reduction intensity of 1.1 times and super-resolution processing with a resolution of 4 times on a processing target area 802, which is a partial area in the image. Therefore, the degradation restoration processing result 803 in the preview image 801 in Fig. 8 is an image representing the result of performing noise reduction processing with a noise reduction intensity of 1.1 times and resolution processing with a resolution of 4 times on the processing target area 802. Note that in the preview image 801 in Fig. 8, the dotted line frame surrounding the processing target area 802 and the degradation restoration processing result 803 is illustrated for the purpose of explanation, and the dotted line is not displayed in the actual preview image, but the dotted line frame may be displayed for the sake of visibility to the user.

[0077] In the case of the setting screen 800 illustrated in Fig. 8, the edge device 100 transmits an image to the server 150 as an image that has not been subjected to unconfirmed image processing. In the case of the setting screen 800 illustrated in Fig. 8, the setting information of the super-resolution setting 807 is unconfirmed, while the setting information of the processing target area setting 805 and the noise reduction intensity setting 806 has not been changed or has been changed and then confirmed. Therefore, the image transmitted from the edge device 100 to the server 150 is an image in which the noise reduction processing has been performed on the processing target area 802 in Fig. 8, but the super-resolution processing has not been performed.

[0078] Then, the edge device 100 transmits the compressed image after performing noise reduction processing on the processing target area 802 to the server 150 together with metadata. For this reason, in the degradation restoration processing on the image received from the edge device 100 and expanded in the server 150, the noise reduction processing is not performed on the processing target area 802, but the super-resolution processing is performed. Also, since the setting information of the 4x super-resolution processing on the processing target area 802 in the edge device 100 is unconfirmed, the server 150 uses the setting information of the super-resolution processing before the change, for example, 2x super-resolution processing, when performing the super-resolution processing on the processing target area 802.

[0079] In the edge device 100, when the setting screen 700 illustrated in Fig. 7 or the setting screen 800 illustrated in Fig. 8 is displayed, the setting unit 208 writes information that the setting screen is being displayed in the setting storage unit 209. On the other hand, when ending the display of the setting screen 700 or the setting screen 800, the setting unit 208 writes information that the setting screen is not being displayed in the setting storage unit 209. In addition, although Fig. 7 and Fig. 8 show an example of the setting screen in the image confirmation mode using the display device 111 and the input device 109, the display and the user input may be performed by other methods. For example, the display and the user input may be performed via the network I / F 106.

[0080] <Processing flow of the entire system in the third embodiment> Next, various processes performed in the information processing system of the third embodiment will be described with reference to the flowchart of Fig. 9. Fig. 9(A) is a flowchart showing the flow of information processing performed in each functional unit of the edge device 100 according to the third embodiment, and Fig. 9(B) is a flowchart showing the flow of information processing performed in each functional unit of the server 150. Note that a description of the contents common to the processing steps listed in the first and second embodiments will be omitted, and the following description will focus on the differences.

[0081] First, information processing performed by the edge device 100 according to the third embodiment will be described with reference to the flowchart in Fig. 9(A). In the third embodiment, the edge device 100 performs captured image acquisition processing in S401, image analysis processing in S402, and development processing in S403, and then proceeds to processing in S904.

[0082] In S904, the image processing unit 203 determines the content of image processing to be performed in the subsequent step S905 based on the setting information stored in the setting storage unit 209 and the noise information (amount of noise) of the image analyzed by the image analysis unit 201. Here, when the setting screen is being displayed, the image processing unit 203 determines the content of image processing necessary for checking the image processing result to be performed in the subsequent step S905 in the case where there is no unconfirmed setting information based on the information stored in the setting storage unit 209. In the case of this embodiment, the image processing unit 203 determines the content of image processing to include noise reduction processing. This is because the processing common to the image to be transmitted to the server 150 and the image when the user checks the settings on the edge device 100 is performed in the subsequent step S905, and the image processing required only when checking the settings on the edge device 100 is performed in the subsequent step S909.

[0083] In addition, in this embodiment, the image processing unit 203 of the edge device 100 does not perform degradation restoration processing that causes the data size of the image to be transmitted to the server 150 to exceed the data size that fits within the network bandwidth. In the example of this embodiment, the image processing unit 203 of the edge device 100 does not perform degradation restoration processing such as super-resolution processing that doubles the vertical and horizontal sizes of the entire image as described in FIG. 7. That is, this is to prevent the image data size to be transmitted to the server 150 from increasing significantly and becoming unable to fit within the network bandwidth. Note that whether the image data size increases significantly may be determined by whether the degradation restoration processing is image processing that increases the amount of data beyond a threshold value that is predetermined based on the network bandwidth.

[0084] When the process proceeds to the next step S905, the image processing unit 203 performs the image processing determined in S904 on the image developed by the development unit 202. After the image processing in S905, the edge device 100 performs the processing in S406 and the processing in S906 in parallel. The processing in S409 and the subsequent steps is the same as that described above, and therefore will not be described.

[0085] The processing from S906 onwards corresponds to the change of the setting information on the setting screen as described above in the edge device 100. In S906, the image processing unit 203 determines whether or not the setting screen is being displayed, and if it is being displayed, the processing proceeds to S907, and if not, the processing ends. In S907, the image processing unit 203 determines whether there is any changed setting information that has not been confirmed. If there is any unconfirmed setting information, the process proceeds to S908, and if not, the process proceeds to S910.

[0086] When the process proceeds to S908, the image processing unit 203 determines the image processing content to be performed in the following step S909 based on the setting information stored in the setting storage unit 209, the amount of noise in the image analyzed by the image analysis unit 201, and the image processing content performed in S905. This is because, when displaying the image after image processing based on the setting information determined to be unconfirmed in S907 on a screen for the user to confirm, image processing other than the common image processing (processing already performed in S905) for the image to be transmitted to the server 150 is performed in S909.

[0087] In the next step S909, the image processing unit 203 processes the image after the image processing performed in step S905 according to the image processing content determined in step S908. Note that the image processing performed in step S909 is referred to as a third image processing. Next, in S910, the image processing unit 203 causes the display device 111 to display a preview image such as that shown in Fig. 7 or 8 described above. That is, if image processing is performed in S905 and there is unconfirmed setting information, in S910, the image processing unit 203 causes the display device 111 to display a preview image according to the result of the image processing in S909. After the processing of S910, the edge device 100 ends the processing.

[0088] Next, information processing performed by the server 150 according to the third embodiment will be described with reference to the flowchart in Fig. 9(B). In the case of the third embodiment, the server 150 performs a reception process in S451, a decompression process in S452, and a metadata analysis process in S453, and then proceeds to processing in S954.

[0089] In S954, the image processing unit 253 of the server 150 determines the image processing content to be performed in S955 based on the setting information stored in the setting storage unit 256 and the metadata analyzed by the metadata analysis unit 252.

[0090] Here, for example, it is assumed that the edge device 100 has performed image processing according to the setting screen 700 of FIG. 7. In the case of the example of the setting screen 700 of FIG. 7 described above, the image transmitted from the edge device 100 to the server 150 is an image on which noise reduction processing has been performed on the entire image, but super-resolution processing has not been performed. On the other hand, the setting storage unit 256 of the server 150 stores setting information for noise reduction processing and super-resolution processing as setting information for degradation restoration processing. Therefore, the image processing unit 253 determines from the stored setting information that noise reduction processing and super-resolution processing are set, and from the analyzed metadata that the edge device 100 has performed noise reduction processing on the entire image, but has not performed super-resolution processing. In this case, the image processing unit 253 determines to perform super-resolution processing on the entire image as the content of the image processing to be performed in S955.

[0091] Also, for example, assume that the edge device 100 has performed image processing according to the setting screen 800 in FIG. 8. In the case of the example of the setting screen 800 in FIG. 8 described above, the edge device 100 performs noise reduction processing on a part of the processing target area, but the setting information for the super-resolution processing is unconfirmed. On the other hand, the image processing unit 253 determines that the noise reduction processing and the super-resolution processing are set based on the saved setting information, and that the edge device 100 has performed noise reduction processing on a part of the processing target area based on the analyzed metadata, and that the setting information for the super-resolution processing is unconfirmed. In this case, the image processing unit 253 determines that the edge device 100 will perform super-resolution processing on a part of the processing target area as the content of the image processing to be performed in S955, and determines the resolution in the super-resolution processing to be the magnification before the change (for example, 2x).

[0092] Then, in S955, the image processing unit 253 executes degradation restoration processing on the image that has been decompressed by the decompression unit 251 in accordance with the image processing content determined in S954. Thereafter, in the server 150, the image after the image processing (degradation restoration processing) by the image processing unit 253 is output as an output image 254 to, for example, the display device 159.

[0093] In the third embodiment, when a user changes the setting information in the edge device 100, the user can check the image resulting from the degradation restoration process using the changed setting information. Also, in the third embodiment, even if the setting information is changed in the edge device 100, if the change in the setting information is not confirmed, the server 150 performs degradation restoration process with contents corresponding to the setting information before the change. Also, in the third embodiment, the server 150 also performs image processing that is not required for image confirmation on the setting screen of the edge device 100.

[0094] <Other embodiments> In the above-described embodiment, a surveillance camera is given as an example of the edge device 100, and a server in a surveillance room that receives images from the surveillance camera (edge ​​device 100) is given as an example of the server 150, but the system of this embodiment is not limited to these. The edge device 100 may be, for example, a digital camera, a camera of a smartphone or a tablet terminal, a camera mounted on a moving object such as a vehicle, or a camera mounted on a head-mounted display. The system of this embodiment can also be applied to a system in which image data captured by such a camera is transmitted to a personal computer or a server via wireless or wired communication.

[0095] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0096] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) image acquisition means for acquiring an image; image processing means for performing a first image processing on the image; a metadata acquisition means for acquiring metadata related to an image to be transmitted to a receiving information processing device; a compression means for compressing the image after the first image processing; a communication means for transmitting the metadata and the compressed image to the receiving information processing device; having The information processing device characterized in that the image processing means determines the content of the first image processing based on the content of a second image processing performed in the receiving information processing device according to the analysis result of the metadata. (Configuration 2) the second image processing performed by the receiving information processing device is a degradation restoration process of an image, 2. The information processing device according to configuration 1, wherein the image processing means determines the content of the first image processing based on the degradation restoration processing performed in the information processing device on the receiving side. (Configuration 3) The information processing device according to configuration 2, wherein the degradation restoration processing includes one or more of noise reduction, blur removal, aberration correction, defect completion, edge enhancement, restoration of information lost due to compression, super-resolution, and contrast correction. (Configuration 4) 4. The information processing apparatus according to any one of configurations 1 to 3, wherein the first image processing is processing for reducing a data size of an image. (Configuration 5) 5. The information processing device according to configuration 4, wherein the first image processing includes one or more processes among low-pass filtering, color space conversion, reduction, color reduction, and degradation restoration. (Configuration 6) 4. The information processing apparatus according to any one of configurations 1 to 3, wherein the first image processing is processing for reducing a difference in pixel values ​​between frames of the image. (Configuration 7) The information processing device according to any one of configurations 1 to 3, wherein the first image processing and the second image processing are processing using a trained machine learning model. (Configuration 8) The information processing device according to configuration 7, wherein the machine learning model used in the first image processing is a machine learning model having a lower processing load than the machine learning model used in the second image processing. (Configuration 9) The first image processing is a process using a trained machine learning model that performs noise reduction in a spatial direction, 3. The information processing device according to configuration 1 or 2, wherein the second image processing is processing using a trained machine learning model that performs noise reduction in the time direction between multiple images. (Configuration 10) image analysis means for performing a predetermined analysis process on the image; 3. The information processing device according to configuration 1 or 2, wherein the image processing means performs the first image processing based on an analysis result by the image analysis means. (Configuration 11) The image analysis means analyzes a noise amount of the image as the predetermined analysis process, The information processing device according to configuration 10, characterized in that the image processing means determines the content of the first image processing based on the amount of noise in the image and the content of the second image processing performed in the receiving information processing device. (Configuration 12) The information processing device according to any one of configurations 1 to 11, wherein the metadata includes one or more pieces of information: information regarding shooting when the image was captured, information regarding development of the captured image, analysis information indicating the results of a predetermined analysis processing on the image, information indicating the contents of the first image processing, information indicating the area to be subject to the first image processing, a compression method used in the compression by the compression means, and a compression rate used in the compression. (Configuration 13) An information processing device according to any one of configurations 1 to 12, characterized in that the contents of the first image processing and the second image processing are determined based on one or more pieces of information among the data size of the image to be transmitted, the bandwidth usage rate of the network through which the image is transmitted, and the amount of computing resources used when the first image processing is performed. (Configuration 14) 14. The information processing apparatus according to any one of configurations 1 to 13, further comprising a setting unit that sets the content of the first image processing in response to an input from a user. (Configuration 15) a display means for displaying a setting screen for a user to set the content of the first image processing, 15. The information processing apparatus according to configuration 14, wherein the image processing means determines the content of the first image processing depending on whether the setting screen is being displayed or not. (Configuration 16) The information processing device described in configuration 15, characterized in that when there is setting information with unconfirmed content on the setting screen, the image processing means displays an image subjected to the first image processing in accordance with the unconfirmed setting information on the display means. (Configuration 17) the first image processing is a combination of one or more image processings, The information processing device according to any one of configurations 1 to 16, characterized in that the first image processing performed by the image processing means on the image to be transmitted to the receiving information processing device does not include image processing in which the data size of the image to be transmitted exceeds a threshold value. (Configuration 18) a first information processing device including an image acquisition means for acquiring an image, an image processing means for performing a first image processing on the image, a metadata acquisition means for acquiring metadata related to the image to be transmitted to a receiving information processing device, a compression means for compressing the image after the first image processing, and a communication means for transmitting the metadata and the compressed image to a second information processing device; the second information processing device including a communication means for receiving the metadata and the compressed image transmitted from the first information processing device, a decompression means for decompressing the received compressed image, a metadata analysis means for analyzing the received metadata, and a second image processing means for performing a second image processing on the decompressed image, the first image processing means of the first information processing device determines the content of the first image processing based on the content of the second image processing performed in the second information processing device according to the analysis result of the metadata; The information processing system according to the present invention, wherein the second image processing means of the second information processing apparatus determines the content of the second image processing based on an analysis result of the metadata. (Method 1) an image acquisition step of acquiring an image; an image processing step of performing a first image processing on the image; a metadata acquisition step of acquiring metadata related to an image to be transmitted to a receiving information processing device; a compression step of compressing the image after the first image processing; a communication step of transmitting the metadata and the compressed image to a receiving information processing device; having An information processing method characterized in that, in the image processing step, the content of the first image processing is determined based on the content of a second image processing performed in the receiving information processing device in accordance with the analysis result of the metadata. (Method 2) a first information processing step including an image acquisition step of acquiring an image, an image processing step of performing a first image processing on the image, a metadata acquisition step of acquiring metadata related to the image to be transmitted to a receiving information processing device, a compression step of compressing the image after the first image processing, and a communication step of transmitting the metadata and the compressed image to a second information processing device; the second information processing step including a communication step of receiving the metadata and the compressed image transmitted from the first information processing device, a decompression step of decompressing the received compressed image, a metadata analysis step of analyzing the received metadata, and a second image processing step of performing a second image processing on the decompressed image, In the first image processing step of the first information processing step, content of the first image processing is determined based on content of the second image processing performed in the second information processing device according to an analysis result of the metadata, An information processing method, comprising: determining, in the second image processing step of the second information processing step, content of the second image processing based on a result of analysis of the metadata. (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0097] 100: Edge device, 200: Imaging unit, 201: Image analysis unit, 202: Development unit, 203: Image processing unit, 205: Metadata attachment unit, 206: Compression unit, 207: Communication unit, 208: Setting unit, 209: Setting storage unit, 210: Model storage unit, 150: Server, 250: Communication unit, 251: Decompression unit, 252: Metadata analysis unit, 253: Image processing unit, 255: Setting unit, 256: Setting storage unit, 257: Model storage unit

Claims

1. image acquisition means for acquiring an image; image processing means for performing a first image processing on the image; a metadata acquisition means for acquiring metadata related to an image to be transmitted to a receiving information processing device; a compression means for compressing the image after the first image processing; a communication means for transmitting the metadata and the compressed image to the receiving information processing device; having The information processing device characterized in that the image processing means determines the content of the first image processing based on the content of a second image processing performed in the receiving information processing device according to the analysis result of the metadata.

2. the second image processing performed by the receiving information processing device is a degradation restoration process of an image, 2. The information processing apparatus according to claim 1, wherein said image processing means determines the content of said first image processing based on said degradation restoration processing performed in said information processing apparatus on the receiving side.

3. 3. The information processing device according to claim 2, wherein the degradation restoration processing includes one or more of noise reduction, blur removal, aberration correction, defect completion, edge enhancement, restoration of information lost due to compression, super-resolution, and contrast correction.

4. 4. The information processing apparatus according to claim 1, wherein the first image processing is processing for reducing a data size of an image.

5. 5. The information processing apparatus according to claim 4, wherein the first image processing includes one or more processes selected from the group consisting of low-pass filtering, color space conversion, reduction, color reduction, and degradation restoration.

6. 4. The information processing apparatus according to claim 1, wherein the first image processing is processing for reducing a difference in pixel values ​​between frames of the image.

7. The information processing apparatus according to claim 1 , wherein the first image processing and the second image processing are processing using a trained machine learning model.

8. The information processing device according to claim 7 , wherein the machine learning model used in the first image processing has a lower processing load than the machine learning model used in the second image processing.

9. The first image processing is a process using a trained machine learning model that performs noise reduction in a spatial direction, 3 . The information processing apparatus according to claim 1 , wherein the second image processing is processing using a trained machine learning model that performs noise reduction in a time direction between a plurality of images. 4 .

10. image analysis means for performing a predetermined analysis process on the image; 3. The information processing apparatus according to claim 1, wherein the image processing means performs the first image processing based on a result of analysis by the image analysis means.

11. The image analysis means analyzes a noise amount of the image as the predetermined analysis process, 11. The information processing apparatus according to claim 10, wherein the image processing means determines the content of the first image processing based on the amount of noise in the image and the content of the second image processing performed in the receiving information processing apparatus.

12. 2. The information processing device according to claim 1, wherein the metadata includes one or more of the following information: information regarding shooting when the image was captured, information regarding development of the captured image, analysis information indicating the results of a predetermined analysis processing on the image, information indicating the content of the first image processing, information indicating the area to be subject to the first image processing, a compression method used in the compression by the compression means, and a compression rate used in the compression.

13. The information processing device according to claim 1, characterized in that the content of the first image processing and the content of the second image processing are determined based on one or more of the following information: the data size of the image to be transmitted, the bandwidth usage of the network through which the image is transmitted, and the amount of computing resources used when the first image processing is performed.

14. 2. The information processing apparatus according to claim 1, further comprising a setting unit for setting the content of the first image processing in response to an input from a user.

15. a display means for displaying a setting screen for a user to set the content of the first image processing, 15. The information processing apparatus according to claim 14, wherein said image processing means determines the content of said first image processing depending on whether said setting screen is being displayed or not.

16. The information processing device according to claim 15, characterized in that, when there is unconfirmed setting information on the setting screen, the image processing means displays, on the display means, an image that has been subjected to the first image processing in accordance with the unconfirmed setting information.

17. the first image processing is a combination of one or more image processings, 2 . The information processing device according to claim 1 , wherein the first image processing performed by the image processing means on the image to be transmitted to the receiving information processing device does not include image processing in which the data size of the transmitted image exceeds a threshold value.

18. a first information processing device including an image acquisition means for acquiring an image, an image processing means for performing a first image processing on the image, a metadata acquisition means for acquiring metadata related to the image to be transmitted to a receiving information processing device, a compression means for compressing the image after the first image processing, and a communication means for transmitting the metadata and the compressed image to a second information processing device; the second information processing device including a communication means for receiving the metadata and the compressed image transmitted from the first information processing device, a decompression means for decompressing the received compressed image, a metadata analysis means for analyzing the received metadata, and a second image processing means for performing a second image processing on the decompressed image, the first image processing means of the first information processing device determines the content of the first image processing based on the content of the second image processing performed in the second information processing device according to the analysis result of the metadata; The information processing system according to the present invention, wherein the second image processing means of the second information processing apparatus determines the content of the second image processing based on an analysis result of the metadata.

19. an image acquisition step of acquiring an image; an image processing step of performing a first image processing on the image; a metadata acquisition step of acquiring metadata related to an image to be transmitted to a receiving information processing device; a compression step of compressing the image after the first image processing; a communication step of transmitting the metadata and the compressed image to a receiving information processing device; having An information processing method characterized in that, in the image processing step, the content of the first image processing is determined based on the content of a second image processing performed in the receiving information processing device in accordance with the analysis result of the metadata.

20. a first information processing step including an image acquisition step of acquiring an image, an image processing step of performing a first image processing on the image, a metadata acquisition step of acquiring metadata related to the image to be transmitted to a receiving information processing device, a compression step of compressing the image after the first image processing, and a communication step of transmitting the metadata and the compressed image to a second information processing device; the second information processing step including a communication step of receiving the metadata and the compressed image transmitted from the first information processing device, a decompression step of decompressing the received compressed image, a metadata analysis step of analyzing the received metadata, and a second image processing step of performing a second image processing on the decompressed image, In the first image processing step of the first information processing step, content of the first image processing is determined based on content of the second image processing performed in the second information processing device according to an analysis result of the metadata, An information processing method, comprising: determining, in the second image processing step of the second information processing step, content of the second image processing based on a result of analysis of the metadata.

21. Computer, image acquisition means for acquiring an image; image processing means for performing a first image processing on the image; a metadata acquisition means for acquiring metadata related to an image to be transmitted to a receiving information processing device; a compression means for compressing the image after the first image processing; a communication means for transmitting the metadata and the compressed image to the receiving information processing device; having A program that causes the image processing means to function as an information processing device that determines the content of the first image processing based on the content of the second image processing performed in the receiving information processing device in accordance with the analysis results of the metadata.

Images